This paper describes MicroHH 1.0, a new and open-source
(

In this paper, we present a description of MicroHH 1.0, a new computational
fluid dynamics code for the simulation of turbulent flows in doubly periodic
domains, with a focus on those in the atmosphere. MicroHH is designed for the
direct numerical simulation (DNS) technique but also supports the large-eddy
simulation (LES) technique. Its applications range from neutral channel flows
to cloudy atmospheric boundary layers in large domains. MicroHH is written in
C

Even though we started from scratch, many of the ideas are the results of our
experiences with other codes. Here, DALES

This paper is built up as follows: in Sect.

The dynamical core of MicroHH solves the conservation equations of mass,
momentum, and energy under the anelastic approximation

The conservation of mass is formulated using Einstein summation as

Under the Boussinesq approximation (

The thermodynamic relation between the fluctuations of virtual potential
temperature, pressure, and density under the anelastic approximation is (see

The corresponding momentum equation is written in the flux form in order to
assure momentum conservation. The hydrostatic balance

Under the Boussinesq approximation, the two equations simplify to

The equation to acquire the pressure is diagnostic because density
fluctuations are neglected in the mass conservation equation under the
anelastic approximation (Eq.

The conservation equation of an arbitrary scalar

MicroHH provides multiple options for the energy conservation equation. The
conservation equation for potential temperature for dry dynamics

A third, more simplified mode, is available for dry dynamics under the
Boussinesq approximation. Here, the equation of state (Eq.

With a slight modification to the definition of

MicroHH is discretized on a staggered Arakawa C-grid, where the scalars are
located in the center of a grid cell and the three velocity components are at the
faces. The code can work with stretched grids in the vertical dimension. The
grid is initialized from a vertical profile that contains the heights of the
cell centers. The locations of the faces are determined consistently with the
spatial order of the interpolations that are described in
Sect.

There is the option to apply a uniform translation velocity to the grid and thus to let the grid move with the flow. This so-called Galilean transformation is allowed as the Navier–Stokes equations are invariant under translation. It has the potential to allow for larger time steps and to increase the accuracy of simulations.

In order to solve the governing equations, MicroHH generates at
initialization three-dimensional fields of the prognostic variables. These
are the three velocity components (Eqs.

The generation of turbulence requires perturbations to the initial fields.
MicroHH has two options to superimpose perturbations on any of the prognostic
variables. These perturbations can be random noise of which the amplitude and
location can be controlled, as well as two-dimensional rotating vortices with
an axis aligned with the

The prognostic equations are solved using low-storage Runge–Kutta time
integration schemes. Such schemes require two fields per variable: one that
contains the actual value, which we denote with

For the third-order scheme, the vectors

For the fourth-order scheme, the vectors

The reduced truncation error of the fourth-order scheme makes the scheme
preferable over the third-order scheme under many conditions (see
Sect.

The spatial operators are based on finite differences. The code supports
second-order and fourth-order accurate discretizations following

We define two second-order interpolation operators, one with a small stencil and one with a wide stencil, as

The gradient operators, denoted with letter

The fourth-order operators, written down in the same notation, are defined as

The centered and biased fourth-order gradient operators are

The lateral boundaries in MicroHH are periodic. The bottom and top boundary
conditions can be formulated in their most general form as the Robin boundary
condition:

MicroHH makes use of ghost cells in order to avoid the need of biased schemes
for single interpolation or gradient operators near the wall. The values at
the ghost cells are derived making use of the boundary conditions following

We use the previously introduced notation to describe the more complex
operators and expand them for illustration. The advection term is discretized
in the flux form, where

MicroHH has a fully kinetic energy-conserving fourth-order advection scheme

Velocity interpolations, such as those in Eq. (

To increase the overall accuracy of the second-order advection operator,
there is an option available to only increase the interpolation part to
fourth order:

We apply a discretization for diffusion that can be written as the divergence
of a gradient, using the building blocks defined earlier in this section. As
this operator is identical in all directions, we present it in one direction
only:

For the fourth-order accurate operator, a seven-point stencil is used:

Whereas diffusion can be computed with fourth-order accuracy using a
five-point stencil, we use a seven-point stencil, as it extends naturally to
non-uniform grids as explained in

Schematic of the diffusion discretization near the wall. The green node is the evaluation point at the center of the first cell above the wall, the red nodes are the stencil of the divergence operator, and yellow nodes show the stencils of the four gradient operators over which the divergence is evaluated. White nodes indicate the extent of the stencil.

Equations (

The velocity values at the next time step can be computed as soon as the
pressure is known, using

Solving Eq. (

MicroHH supports the potential (

The base state density

The effects of a rotating reference frame on an

With the governing equations described in Sect.

The same approach is followed for all scalars, including the thermodynamic
variables discussed in Sect.

The LES implementation of MicroHH uses a surface model that is constrained to
rough surfaces and high Reynolds numbers, which is a typical configuration for
atmospheric flows. This model computes the surface fluxes of the horizontal
momentum components and the scalars (including thermodynamic variables) using
Monin–Obukhov similarity theory (MOST) (

Following MOST, the friction velocity

The same procedure for scalars is followed, with

The functions

With the equations above, the surface fluxes, surface values, and near-surface
gradients can be computed but only if the Obukhov length

The first option is fixed momentum fluxes and a fixed surface buoyancy flux. Both the
friction velocity

The second option is a fixed horizontal velocity

The third option is a fixed surface velocity

MicroHH provides two options to introduce a large-scale pressure force into
the model. The first is to enforce a constant mass flux through the domain in
the streamwise direction. In this approach, the desired large-scale velocity

The second option is to enforce a large-scale pressure force through the
geostrophic wind components

Large-scale sources and sinks, representing, for instance, large-scale
advection or radiative cooling, can be prescribed for each variable
separately. The user has to provide vertical profiles of large-scale sources
and sinks

A second method of introducing large-scale thermodynamic effects is through
the inclusion of a large-scale vertical velocity. In this case, each scalar
gets an additional source term

MicroHH has the option to damp gravity waves in the top of the simulation
domain in a so-called buffer layer. The source term

MicroHH is written in C

MicroHH is enabled to run on fast and energy-efficient graphical processing
units (GPUs). This promising technique has been pioneered in atmospheric flows
by

The code uses the Message Passing Interface (MPI) in order to run on a large number of cores. The three-dimensional simulation domain is split into vertically oriented columns standing on a two-dimensional grid. The code assigns one MPI task to each grid column using the MPI_Cart_create function and uses this grid to detect the IDs of neighboring processes. In order to avoid complex packing routines, we make use of MPI datatypes wherever possible. The MPI calls are hidden in an interface to avoid the need to manually write MPI calls.

The input/output (IO) is entirely based on MPI-IO, the parallel IO framework that comes with MPI, to ensure that three-dimensional fields and two-dimensional cross sections are stored as single files. We have chosen MPI-IO in order to limit the number of files resulting from simulations on a large number of processes and to allow for restarts on a different number of processes. In order to keep complexity of the IO as low as possible, we make use of the MPI_Sub_array function in combination with MPI_File_write_all in order to write the fields.

MicroHH depends on several external software tools or libraries. It uses the
CMake (

In order to run a simulation with MicroHH, a sequence of steps needs to be
taken. Each simulation has an

To prepare a simulation with the name

After the previously described initialization phase, the execution of

The last mode in which the code can run is the post-processing mode. By running

A large set of output statistics can be computed during runtime at user-specified time intervals. The statistics module provides vertical profiles of means, second-, third- and fourth-order moments of all prognostic variables, as well as advective and diffusive fluxes. Furthermore, there are multiple diagnostic variables, such as the pressure, the pressure variance, and its vertical flux. The thermodynamics generate their own statistics based on the chosen option. The moist thermodynamics provides a large set of cloud statistics. There is a separate module for budget statistics, which provides the budgets of all components of the Reynolds stress tensor, and those of the variance and vertical flux of the thermodynamic variables.

Convergence of the spatial discretization error in the
two-dimensional Taylor–Green vortex. Subscript

One of the key features of the MicroHH statistics routine is that an
arbitrary mask can be passed into the routine over which the statistics are
calculated. This allows, for instance, for a very simple way of computing
conditional statistics in updrafts or clouds, which is demonstrated later in
Sect.

It is possible to save two-dimensional cross sections and three-dimensional
fields of any of the prognostic and diagnostic variables of the model, as
well as of derived variables. This output can be made at user-specified time
intervals. Cross sections can be made in any chosen

Time evolution of the kinetic energy change

In this section, we present a series of cases intended to validate MicroHH
under a wide range of settings. Each of these test cases is available in the

The two-dimensional Taylor–Green vortex (

Normalized numerical Prandtl model solutions for velocity

We use the analytical form at

Figure

The second test of the dynamical core consists of combined evaluation of
kinetic energy (

Velocity means

The loss of kinetic energy as a function of time is shown in
Fig.

To test the buoyancy routine and the option to put the domain on a slope, a
Prandtl-type anabatic slope flow

Budgets of variances and turbulence kinetic energy (TKE, defined as

The fluid, which was initially at rest, goes through a series of decaying
oscillations after the buoyancy flux is applied at the surface. Eventually,
it reaches the steady state corresponding to the Prandtl model solution.
Numerical integration was performed sufficiently long for the oscillation
amplitude to become a small fraction of the amplitude of the first
oscillation. Comparison of horizontal wind

For fully turbulent flows, the numerical solutions cannot be compared against
any analytical test cases. Therefore, we validate our results against a
channel flow at a Reynolds

Figure

The findings in the previous paragraph are further corroborated by the
spectra shown in Fig.

Spectra of all velocity components and pressure compared against

The final evaluation of the dynamical core without parameterizations enabled
is based on the direct numerical simulation of a turbulent katabatic flow.
Here, a buoyancy-driven slope flow is simulated following the setup of

Turbulent motion starts quickly after the buoyancy flux is applied at the surface. It is characterized by random, large-amplitude fluctuations of velocity and buoyancy fields in the near-slope core region and shows quasi-periodic oscillatory behavior at larger distances from the slope. Mean profiles of the along-slope velocity component and buoyancy, as well as profiles of second-order turbulence statistics, such as kinematic turbulent fluxes of momentum and buoyancy, and velocity component and buoyancy fluctuation variances, were evaluated by averaging the simulated flow fields spatially over the along-slope planes and temporally over five oscillation periods beyond the transition stage.

For comparison, the same katabatic flow case was reproduced using the
numerical code (hereafter referred to as FS09) that was employed to simulate
turbulent slope flows in

Numerical results obtained with both numerical codes testify that stable
environmental stratification in combination with negative surface buoyancy
forcing in the katabatic flow leads to an effective suppression of vertical
turbulent exchange in the flow region adjacent to the slope. This suppression
results in a shallow near-surface sublayer with strong buoyancy gradients
(Fig.

As a first test case of MicroHH in LES mode, we present that of

The results show the formation of a well-mixed layer with an overlying
capping inversion (see Fig.

The BOMEX shallow cumulus case

Profile of the mean along-slope velocity

This case produces non-precipitating shallow cumulus. It has a large-scale cooling applied that represents radiation, as well as a large-scale drying to allow the atmosphere to relax to a steady state. In addition, a large-scale vertical velocity is applied over a certain height range to reproduce the appropriate synoptic conditions.

The simulation is run for 6 h. Statistics are recorded during the final
hour, including conditional statistics for the cloud-covered fields (

To evaluate the LES mode for stable atmospheric conditions, the GABLS1 LES
intercomparison case

Vertical profiles of horizontally averaged potential
temperature

BOMEX LES intercomparison (S03). Shown are the domain mean, and
conditionally sampled cloud (

Figure

GABLS1 LES intercomparison

Speedup from a strong-scaling experiment

The parallel performance of MicroHH has been evaluated in strong-
(

The strong-scaling experiment shows that increasing the number of processors
leads to faster simulations. The speedup is initially close to linear, but
each consecutive increase in the number of cores makes the model less
efficient. Based on these results, we conclude that for the chosen test case
and for the used supercomputers, a work load of approximately

The weak scaling shows almost 90 % efficiency going from 512 to 8192 cores; beyond that, the scaling falls off to 80 %. This can be explained by physical properties of the machine; beyond 8192 cores, a simulation no longer fits on one midplane (a physical unit consisting of 8192 cores), leading to slower communication.

The GPU implementation of MicroHH allows for fast simulations on a single
device. The current state-of-the-art GPUs feature 12 GB of memory; thus,
simulations of maximally

To test the performance of such simulations, the performance of MicroHH on an
NVIDIA Quadro K6000 (using CUDA 6.5) has been compared against the Max Planck
Institute for Meteorology's cluster Thunder (two Intel Xeon E5-2670 CPUs per
node, 16 cores per node). Three benchmark cases have been chosen: the BOMEX
moist convection case on grids of

The results shown in Table

To date, several studies have been published that make use of MicroHH or data
generated with MicroHH. The work of

There are several ongoing projects to extend the model. Currently, a
parameterization for microphysics has been developed, and an interactive land
surface model is under development. In addition, the immersed boundary method
following

Speedup of GPU simulation compared to respective CPU simulation performed on

This paper has presented a full description of MicroHH, a new computational fluid dynamics code for simulations of turbulent flows in the atmospheric boundary layer. The governing equations and their implementation have been presented, and a broad validation under a wide range of settings has been shown. MicroHH delivers the expected error convergence of the spatial and temporal schemes, and has proven to be mass, momentum, and energy conserving. The code delivers good performance in weak- and strong-scaling experiments. Its current limitations are the absence of horizontal boundary conditions other than periodic, and the limited set of available physical parameterizations. Both limitations will be addressed in future versions of the code.

MicroHH has its own website at

Table

Overview of used constants.

The authors declare that they have no conflict of interest.

Version 1.0 of MicroHH was completed thanks to extensive discussions and exchange of code with many of the developers of DALES (Huug Ouwersloot, Stephan de Roode, Arnold Moene, and Jordi Vilà-Guerau de Arellano), PALM (Björn Maronga and Siegfried Raasch), UCLA-LES (Linda Schlemmer and Bjorn Stevens), and ICON (Leonidas Linardakis). We thank Alan Shapiro for the discussions on the bottom boundary condition for pressure for buoyancy-driven flows and for his comments on the manuscript. We thank Elie Bou-Zeid and an anonymous reviewer for their constructive comments on the manuscript. The authors gratefully acknowledge the Gauss Centre for Supercomputing (GCS) for providing computing time for a GCS Large-Scale Project on the GCS share of the supercomputer JUQUEEN at the Jülich Supercomputing Centre (JSC).The article processing charges for this open-access publication were covered by the Max Planck Society. Edited by: Paul Ullrich Reviewed by: Elie Bou-Zeid and one anonymous referee