We present a nonhydrostatic finite-volume global atmospheric model formulation for numerical weather prediction with the Integrated Forecasting System (IFS) at ECMWF and compare it to the established operational spectral-transform formulation. The novel Finite-Volume Module of the IFS (henceforth IFS-FVM) integrates the fully compressible equations using semi-implicit time stepping and non-oscillatory forward-in-time (NFT) Eulerian advection, whereas the spectral-transform IFS solves the hydrostatic primitive equations (optionally the fully compressible equations) using a semi-implicit semi-Lagrangian scheme. The IFS-FVM complements the spectral-transform counterpart by means of the finite-volume discretization with a local low-volume communication footprint, fully conservative and monotone advective transport, all-scale deep-atmosphere fully compressible equations in a generalized height-based vertical coordinate, and flexible horizontal meshes. Nevertheless, both the finite-volume and spectral-transform formulations can share the same quasi-uniform horizontal grid with co-located arrangement of variables, geospherical longitude–latitude coordinates, and physics parameterizations, thereby facilitating their comparison, coexistence, and combination in the IFS.

We highlight the advanced semi-implicit NFT finite-volume integration of the fully compressible equations of IFS-FVM considering comprehensive moist-precipitating dynamics with coupling to the IFS cloud parameterization by means of a generic interface. These developments – including a new horizontal–vertical split NFT MPDATA advective transport scheme, variable time stepping, effective preconditioning of the elliptic Helmholtz solver in the semi-implicit scheme, and a computationally efficient implementation of the median-dual finite-volume approach – provide a basis for the efficacy of IFS-FVM and its application in global numerical weather prediction. Here, numerical experiments focus on relevant dry and moist-precipitating baroclinic instability at various resolutions. We show that the presented semi-implicit NFT finite-volume integration scheme on co-located meshes of IFS-FVM can provide highly competitive solution quality and computational performance to the proven semi-implicit semi-Lagrangian integration scheme of the spectral-transform IFS.

Notwithstanding the achievements made over the last decades

The spectral-transform (ST) – also known as pseudo-spectral – method was
introduced in NWP following the work by

The SL schemes are subject to a topological realizability
condition based on the Lipschitz number which is related to the flow
deformation

At ECMWF, the first forecast model using the ST method became operational in
1983, and the technique is still successfully applied today with the
efficient SISL integration of the hydrostatic primitive equations in the
Integrated Forecasting System (IFS)

The uncertainties concerning the SISL integration based on the ST method with
regard to emerging and future HPC architectures is one of the main reasons
for ECMWF and its European partners to look into alternative nonhydrostatic,
all-scale global model formulations and discretization schemes to be
incorporated in the IFS. With this objective in mind, the Finite-Volume
Module of the IFS (henceforth IFS-FVM) is under development at ECMWF

By default, IFS-FVM employs 3-D semi-implicit integrators for the
nonhydrostatic fully compressible equations

Including orography in the implicit part involves multiplications which are standardly performed in grid-point space in the context of the ST method. In principle, one could carry out the necessary multiplications in spectral space but this is usually avoided because of computational complexity.

, leaving the associated effects solely to the explicit non-linear residual. Furthermore, in the constant-coefficient semi-implicit scheme different (i.e. split) boundary conditions are applied in the linear operator and the non-linear residual. Although still a research issue, the constant-coefficient semi-implicit scheme may incur reduced stability under more complex orography for future high-resolution forecasts in the nonhydrostatic regime.At ECMWF, the reign of the ST method with the SISL integrators still
continues, but future challenges, especially with respect to HPC,
nonhydrostatic modelling, and complex orography, can be foreseen. The IFS-FVM
represents an alternative dynamical core formulation that can complement
IFS-ST with regard to these issues. However, to make IFS-FVM a useful option
for global medium-range weather forecasting at ECMWF, it needs to be shown
that the model formulation can provide (at least) comparable solution quality
to the established IFS. In particular, a fundamental scientific question is
whether a second-order FV method on the co-located meshes employed in IFS-FVM
can sustain the accuracy of the ST method of IFS-ST. Another important
question concerns the computational efficiency of IFS-FVM. At ECMWF and
generally in NWP, tight constraints exist with regard to the runtime of the
forecast models on the employed supercomputers. Therefore, we will evaluate
the basic efficiency in terms of the time to solution of the current IFS-FVM
formulation relative to the operational hydrostatic IFS-ST and its
nonhydrostatic extension. In the present paper, these issues are investigated
using relevant atmospheric flow benchmarks such as those defined in the
context of the Dynamical Core Model Intercomparison Project

The paper is organized as follows. Section

The IFS comprises a comprehensive model infrastructure to perform data assimilation and to run deterministic and probabilistic global weather forecasts with various ranges and resolutions, supplemented with preprocessing and post-processing capabilities. The dynamical core lies at the heart of the NWP model infrastructure.

Summary of the main formulation features of IFS-FVM and IFS-ST. For
IFS-ST, information about the hydrostatic formulation and its nonhydrostatic
extension is provided (see main text for description). Abbreviations are as follows:
finite element (FE), finite difference (FD), spectral transform (ST),
finite volume (FV), two time level (2-TL), semi-implicit (SI),
iterative-centred implicit (ICI). A summary of variables is provided in
Table

IFS-FVM solves the deep-atmosphere

The shallow-atmosphere equations,
the default in IFS-ST, are available by means of a simple switch

Building on the formulation of moist-precipitating dynamics described in

A quantity which appears in various right-hand-side (RHS) terms of the
momentum Eq. (

Another important aspect of the governing Eqs. (

All governing equations are formulated with
respect to generalized curvilinear coordinates embedded in a geospherical
framework. At the most elementary level, the generalized curvilinear
coordinate formulation can be used to implement fixed terrain-following
levels with appropriate boundary conditions, but the model formulation
optionally permits quite general moving meshes in the vertical and the
horizontal directions. Symbols associated with the geometric aspects of the
model are the transformed curvilinear coordinates

Note that additional RHS terms not explicitly provided in the governing
Eqs. (

To facilitate a compact description of the integration scheme, each of the
governing Eqs. (

As an example,

Specification of prognostic model variables and corresponding
parameters in the template
scheme (

Building on the earlier works by

There are two alternative implementations of the NFT advective transport
scheme

Given the preceding discussion, the semi-implicit solution procedure proceeds
from an atmospheric state at

Given the tendencies from physics parameterization

From Eqs. (

As far as the adiabatic dynamics is concerned, the

Overall, the 3-D implicit scheme with respect to

The discretization framework of IFS-FVM combines a structured-grid FD–FV
method in the vertical with an unstructured

Note that although the presented IFS-FVM formulation assumes an unstructured mesh with indirect addressing in the horizontal, the model may exploit structured or semi-structured grids on future HPC architectures.

FV approach in the horizontalThe schematic in Fig.

Schematic of the median-dual mesh in 2-D. The edge connecting nodes

For a differentiable vector field

Note that in IFS-FVM,

For IFS-FVM, the mesh generation and mesh data structures, as well as the
nearest-neighbour distributed-memory communication using MPI, are handled by
ECMWF's Atlas library, comprehensively described in

In the present work, the programming of the discrete differential
operators (

We note that the GCR solver for the Helmholtz problem (

The spectral-transform IFS (denoted as IFS-ST in this paper) that is
operational at ECMWF is based on the hydrostatic primitive equations (HPEs).
The HPEs are formulated in a hybrid sigma–pressure terrain-following vertical
coordinate following

The HPEs analytically filter internal acoustic modes but support the external Lamb mode.

and buoyant modes, and the 3-D SL advectionThe FE implementation of the discrete vertical integral operator is based on the Galerkin method using cubic B splines as basis functions.

The IFS-ST uses a discrete spherical harmonics representation of the spectral
space

The semi-Lagrangian advection scheme in IFS-ST is based on

The IFS also includes various research options which are not yet applied in
the operational configuration, most notably the nonhydrostatic (NH)
formulation based on the fully compressible equations

The IFS physics parameterization package at ECMWF is applied in the same configuration throughout the medium-range, sub-seasonal, and seasonal forecasting systems. The physics package includes parameterization of radiation, moist convection, clouds and stratiform precipitation, surface processes, sub-grid-scale turbulence, and orographic and non-orographic gravity wave drag.

The locations of the octahedral reduced Gaussian grid nodes are
shown in

In IFS-ST, the physics–dynamics coupling employs the SLAVEPP
(semi-Lagrangian averaging of physical parameterization) scheme

SLAVEPP is applied to tendencies from radiation, moist
convection, and the cloud scheme, whereas tendencies from turbulence and
gravity wave drag parameterizations are incorporated with first order at

The physics–dynamics coupling in IFS-FVM differs from IFS-ST. As explained in
Sect.

The IFS-FVM code has its own interface to the IFS physics parameterizations.
Among others, it involves conversion between IFS-FVM's variables and those
employed in IFS-ST (see Table

Examples are
conversions between mixing ratios

In the current
high-resolution deterministic IFS forecasts on the O1280 grid, the radiation
scheme is called every hour, compared to the semi-implicit model time step

With the current formulations, the time step

As with the classical reduced Gaussian grid of

We study the solution quality and also the computational efficiency of IFS-FVM in comparison to the established IFS-ST. The hydrostatic IFS-ST represents a proven formulation for global medium-range NWP at ECMWF, and the aim is to reproduce its results with the novel IFS-FVM.

Baroclinic instability represents a common and relevant test problem to
evaluate the performance of global NWP models in the large-scale hydrostatic
regime. The underlying processes are fundamental to the life cycle
(i.e. formation to decay) of high- and low-pressure systems in the
mid-latitude “storm tracks” of the Earth's atmosphere. Here, we adopt the
experimental set-up for a baroclinic wave life cycle used in the 2016 edition
of DCMIP following

To study the accuracy of the novel nonhydrostatic IFS-FVM based on the
finite-volume discretization, we verify its solution quality against IFS-ST
based on the spectral-transform approach. This comparison is performed in
Sect.

We use two different sizes of the octahedral reduced Gaussian grid for
comparing the solution quality of IFS-FVM and IFS-ST. Considered are very
coarse (O160,TCo159) and coarse (O320,TCo319) grids by current NWP standards,
corresponding to about 64 and 32 km nominal horizontal grid spacings,
respectively; see Sect.

For instance, the 1 m height difference corresponds to about 0.1 hPa near the surface, which is negligible with regard to the subsequent analysis.

. For the comparison of computational efficiency in Sect.Dry baroclinic instability at day 10:
panels

Dry baroclinic instability at day 10:
panels

Dry baroclinic instability at day 15: pressure on the lowest full
level (hPa) obtained with IFS-FVM (default backward Euler scheme with

All IFS-FVM and IFS-ST results presented in this section were obtained without any explicit diffusion or regularization.

For the dry and moist configurations, the baroclinic instability evolution
starts from two zonal jet flows in the mid-latitudes of each global
hemisphere that are in thermal wind balance with the meridional temperature
gradient. The definition of the balanced initial state is given by analytical
functions provided in

Dry baroclinic instability at day 15 when the triggering of the baroclinic instability was applied in the Northern Hemisphere only: pressure on the lowest full level (hPa) obtained with IFS-FVM and IFS-ST using the (O320,TCo319) grid.

Figures

Dry baroclinic instability at day 15: kinetic energy spectra
obtained with IFS-FVM and IFS-ST using the (O320,TCo319) grid. The blue
vertical line indicates the spatial scale corresponding to 4 times the
nominal grid spacing of the O320 octahedral grid, which also represents the
cubic truncation scale with TCo319 applied in IFS-ST. The spectra are shown
on model levels near the surface and at

Kinetic energy spectra evaluated at day 15 in Fig.

The unsplit NFT
MPDATA advection

Moist-precipitating baroclinic instability at day 10: surface
precipitation rate (mm day

Moist-precipitating baroclinic instability at day 15: surface
precipitation rate (mm day

Next we present results for the moist-precipitating baroclinic instability
with coupling to the IFS cloud parameterization. Figure

The precipitation rate represents the liquid and rain (excluding ice and snow) sedimentation flux at the surface.

for the (O160,TCo159) and (O320,TCo319) grids at day 10. For any of these grids, both model formulations show five rainbands with essentially identical phase, as emphasized by the overlay with the 0.5 mm dayMoist-precipitating baroclinic instability at day 15: pressure on
the lowest full level (hPa) obtained with IFS-FVM and IFS-ST coupled to the
same IFS cloud microphysics parameterization. The IFS-FVM results
in

Moist-precipitating baroclinic instability: time series of minimum
pressure on the lowest full level

Moist-precipitating baroclinic instability at day 15: kinetic energy
spectra obtained with IFS-FVM and IFS-ST using the (O320,TCo319) grid. The
blue vertical line indicates the spatial scale corresponding to 4 times
the nominal grid spacing of the O320 octahedral grid, which also represents
the cubic truncation scale with TCo319 applied in IFS-ST. The spectra are
shown on model levels near the surface and at

The computational efficiency of NWP models is crucial. For current HPC
architectures and model resolutions, the operational IFS-ST at ECMWF
represents one of the most efficient dynamical core formulations for global
NWP. The IFS-FVM is envisaged for future applications in the nonhydrostatic
regime running on future HPC architectures, but its computational performance
on the current HPC facility at ECMWF sheds some light on its potential; see
also

Figure

Each node on this supercomputer consists of two Intel Xeon EP E5-2695 v4 “Broadwell” processors, each with 18 cores, which for the 350 compute nodes employed results in a total of 12 600 cores. Here, a hybrid MPI–OpenMP parallelization with six threads was used by all three dynamical cores.

. Importantly, while IFS-ST used the constant time step of 450 s, IFS-FVM employed variable time stepping according to the maximum permitted advective Courant number; therefore, in order to obtain realistic numbers for IFS-FVM, the timings were evaluated between day 10 and 15 in the fully non-linear stage of the baroclinic instability evolution.Figure

Elapsed time to run 1 day of the dry baroclinic instability benchmark similar to the current HRES configuration at ECMWF, i.e. the three different models – the nonhydrostatic IFS-FVM designated as FV(NH), the hydrostatic IFS-ST designated as ST(H), and the nonhydrostatic IFS-ST designated as ST(NH) – are set up for the O1280/TCo1279 horizontal grid (corresponding to about 9 km grid spacing) with 137 stretched vertical levels and employ 350 nodes of ECMWF's Cray XC40.

Supporting substantially higher resolution in global NWP may ultimately demand local numerical discretizations to solve the governing nonhydrostatic equations in NWP models in a computationally efficient manner. The IFS-FVM successfully implements such a discretization and thus complements the operational hydrostatic IFS-ST and its nonhydrostatic extension at ECMWF. At the same time, the IFS-FVM introduces several useful new features into the IFS, such as conservative and monotone advective transport, deep-atmosphere all-scale governing equations, and fully flexible unstructured FV meshes with optional variable resolution or meshes defined about the nodes of the operational octahedral grid.

The paper highlighted the semi-implicit NFT finite-volume integration of the fully compressible equations of the novel IFS-FVM considering comprehensive moist-precipitating dynamics with coupling to the IFS cloud parameterization by means of a generic interface applicable for coupling to the full IFS physics parameterization package. Developments such as the new horizontal–vertical directionally split NFT advective transport scheme based on MPDATA, variable time stepping, effective preconditioning of the Krylov-subspace solver for the elliptic Helmholtz problem arising in the semi-implicit scheme, and an efficient node-based implementation of the median-dual FV approach provide a basis for the overall efficacy of IFS-FVM and application in global NWP at ECMWF.

The IFS-ST is applied successfully for operational forecasting at ECMWF and is therefore considered an appropriate reference model. It was shown that the presented semi-implicit NFT finite-volume integration scheme on co-located meshes can achieve comparable solutions to the proven spectral-transform IFS-ST. Here, the study focused on medium- and extended-range simulation of the dry and moist-precipitating baroclinic instability benchmark at various resolutions. While the baroclinic instability benchmark aims at global atmospheric dynamics in the hydrostatic regime, referenced supplementary studies with IFS-FVM emphasize non-orographic and orographic flows in the nonhydrostatic regime. In addition to solution quality, we have demonstrated highly competitive computational efficiency of the presented semi-implicit NFT finite-volume integration of IFS-FVM in comparison to the semi-implicit semi-Lagrangian integration of IFS-ST.

Common aspects of the finite-volume and spectral-transform model formulations
are the octahedral reduced Gaussian grid, the co-location of variables, the
geospherical framework, and the physics parameterizations. Sharing these
properties facilitates the comparison of the different discretizations and
physics–dynamics coupling. Moreover, it provides numerous benefits for the
general IFS model infrastructure, data assimilation, and ensemble system.
Ongoing work advances IFS-FVM to full-physics global medium-range NWP at
convection-resolving resolutions

Model codes developed at ECMWF are the intellectual property of
ECMWF and its member states, and therefore the IFS code is not publicly
available. Access to a reduced version of the IFS code may be obtained from
ECMWF under an OpenIFS licence (see

The model output data can be downloaded from

We consider the advection operator

The proposed scheme implements mass-compatible second-order Strang splitting
as explained in the following. The overall semi-implicit integration of the
fully compressible Eqs. (

Compared to the unsplit scheme, the particular horizontal–vertical splitting also does not incur any additional parallel communication in the context of the horizontal domain decomposition of IFS-FVM.

. Overall, the horizontal–vertical splitting ofThe bespoke preconditioner solves for the solution error

IFS-FVM's ability to accommodate complex mesh geometries results from two aspects
of its formulation: the horizontal unstructured-mesh FV discretization and
generalized curvilinear coordinate mappings embedded in a geospherical framework

In the geospherical curvilinear coordinate framework of

Consistent with

Furthermore, in the momentum Eq. (

List of variables.

List of physical constants.

CK did most of the developments and numerical experiments presented in the paper. CK and PKS are the main developers of IFS-FVM. WD is the main developer of the Atlas library employed by IFS-FVM. RK contributed to developments of the advection scheme and time stepping. SM set IFS-ST up for the test cases and performed experiments for the comparison to IFS-FVM. ZPP contributed to the efficient coding of IFS-FVM. JS contributed to the preconditioning of the elliptic solver. NPW provided support with regard to the general IFS framework.

The authors declare that they have no conflict of interest.

We would like to thank two reviewers for their useful comments. Helpful discussions with the physics parameterization team at ECMWF are gratefully acknowledged. This work was supported in part by funding received from the European Research Council under the European Union's Seventh Framework Programme FP7/2012/ERC (grant agreement no. 320375), the Horizon 2020 Research and Innovation Programme ESCAPE (grant agreement no. 671627), and ESIWACE (grant agreement no. 675191). Rupert Klein thanks ECMWF for support under their fellowship programme and acknowledges partial support of his contributions by the Deutsche Forschungsgemeinschaft, grant CRC 1114 “Scaling Cascades in Complex Systems”, project A02. Zbigniew Piotrowski acknowledges partial support from the “Numerical weather prediction for sustainable Europe” project, carried out within the FIRST TEAM programme of the Foundation for Polish Science co-financed by the European Union under the European Regional Development Fund. Edited by: Paul Ullrich Reviewed by: two anonymous referees