Accurate calculations of shortwave reflectances in clear-sky aerosol-laden atmospheres are necessary for various applications in atmospheric sciences. However, computational cost becomes increasingly important for some applications such as data assimilation of top-of-atmosphere reflectances in models of atmospheric composition. This study aims to provide a benchmark that can help in assessing these two requirements in combination. We describe a protocol and input data for 44 080 cases involving various solar and viewing geometries, four different surfaces (one oceanic bidirectional reflectance function and three albedo values for a Lambertian surface), eight aerosol optical depths, five wavelengths, and four aerosol types. We first consider two models relying on the discrete ordinate method: VLIDORT (in vector and scalar configurations) and DISORT (scalar configuration only). We use VLIDORT in its vector configuration as a reference model and quantify the loss of accuracy due to (i) neglecting the effect of polarization in DISORT and VLIDORT (scalar) models and (ii) decreasing the number of streams in DISORT. We further test two other models: the 6SV2 model, relying on the successive orders of scattering method, and Forward-Lobe Two-Stream Radiance Model (FLOTSAM), a new model under development by two of the authors. Typical mean fractional errors of 2.8 % and 2.4 % for 6SV2 and FLOTSAM are found, respectively. Computational cost depends on the input parameters but also on the code implementation and application as some models solve the radiative transfer equations for a range of geometries while others do not. All necessary input and output data are provided as a Supplement as a potential resource for interested developers and users of radiative transfer models.

Accurate radiative transfer calculations in the Earth's atmosphere are
necessary for some applications such as remote sensing of the atmospheric and
surface properties, numerical weather prediction, and climate modelling. In
this study, we are interested more specifically in the calculation of
radiances in the shortwave spectrum in clear-sky aerosol-laden atmospheres of
the Earth for reasons explained below. Assuming perfect knowledge of the
optical properties of the atmosphere and surface and a plane-parallel
atmosphere, solving the radiative transfer equation is a very well posed
physical problem that essentially reduces to a (not-so-easy) algorithmic and
numerical problem. Accurate methods have existed for a long time, and testing
the accuracy of newly developed methods has been standard practice in the
radiative transfer community for some time (

Methods for computing shortwave radiances by solving accurately the radiative
transfer equation are often computationally expensive. In data assimilation
or satellite retrieval applications for which radiances have to be computed
many times and on many profiles, the computational cost has always been an
issue for pragmatic reasons (a numerical model that is too slow to be run on
the fastest computer is virtually useless). However, developing fast or very
fast radiative transfer models capable of computing radiances (and not only
vertical fluxes) has received little attention so far. Exceptions include
early attempts to use neural networks for fast and accurate radiative
transfer in the longwave spectrum

Forecasts and reanalyses of atmospheric aerosols have been produced
operationally by combining numerical models and aerosol observations in data
assimilation systems

An alternative pathway to the assimilation of AOD retrieved from passive
satellite measurements is the direct assimilation of these measurements

To our knowledge, only a limited number of studies have been able to
assimilate shortwave radiances in atmospheric composition models.

For operational data assimilation purposes, the computational cost of the
radiance observation operator is crucial

How computational speed is achieved depends on the desired application. Some
methods, such as the discrete ordinate method or the Monte Carlo method

Keeping this introduction in mind, the objective of this study is to provide a benchmarking tool for evaluating shortwave radiances in the clear-sky aerosol-laden atmosphere under a wide range of aerosol conditions. The benchmarking is designed to assess both the accuracy and computational cost of radiative transfer models. Although data assimilation of aerosol radiances is the key motivation for this work, we report computational cost both for single and multiple geometries to make the scope more general. We provide all necessary input and output data for this benchmarking as a Supplement in NetCDF and ACSII format so that it becomes an available resource for model developers and potential model users. The protocol is described in the next section. It is followed by a test of the protocol in different RTMs and some conclusions.

The detailed benchmark protocol is provided as a Supplement. Only the main
features are summarized here. We consider five wavelengths spanning the
shortwave spectrum: 470, 550, 660, 865, and 1024

The molecular and aerosol profiles are described below. It should be noted that we assume the atmosphere to be plane-parallel; i.e. there is no correction applied for sphericity of the Earth's atmosphere.

We use a midlatitude summer atmospheric profile

We define the vertical profile of the aerosol extinction coefficient
(

The aerosol optical depth of a layer

Four aerosol types have been defined. Aerosol size distributions are assumed
to be log-normal with the parameters shown in Table

Aerosol physical and optical properties for the different aerosol
types considered in the benchmark.

Aerosol optical properties were computed by the authors using their own Mie
routines, and results were cross-checked for consistency. The Mie routine used
to generate the input data was rewritten in Fortran based on the
implementation of

For each aerosol type, we provide the aerosol optical depths at the 440, 670,
865, and 1024

The phase matrix is computed and provided at 50 000 points evenly spaced in
scattering angle from 0 to 180

The accurate reconstruction of the original phase function from the Legendre
decomposition (

Figure

First (i.e.

Two surface reflectance models are selected: a Lambertian and an oceanic
bidirectional reflectance distribution function (BRDF) model. For the
Lambertian model, three surface albedos were chosen, namely 0, 0.05, and 0.1.
The spectral dependence of the surface reflectance is ignored; i.e. the
albedo is assumed to be the same for the five wavelengths considered. The second
surface reflectance model is the oceanic glitter model from

Wind speed is set to 10

The convention for the azimuthal angle used in the

We have selected eight cases for aerosol loadings in the atmosphere, with AOD at
550

We have further selected four solar and four viewing zenith angles –

Summary of the proposed geometrical and environmental conditions used in the simulations.

Statistical measures of DISORT and scalar VLIDORT reflectances
against vector VLIDORT (first row). Errors of the last rows are defined as
the absolute value of the relative error, using the VLIDORT (vector)
reflectance as the reference. VLIDORT is the configuration with 32 streams
and the TMS correction activated.

We aim to provide a standardized dataset to evaluate the possibility of assimilating satellite radiances in operational forecasts of atmospheric composition. Thus, two variables are important to estimate for each case: the simulated reflectance (or radiance) that would be observed by the satellite and a “sound” estimate of the radiative transfer code runtime.

We will describe the accuracy of the models in terms of the reflectance,

The runtime may be difficult to estimate and compare between different models. We aim to measure the runtime of the computations excluding input/output operations, as typical applications would embed the RTM code in a larger operational system. The runtime of the RTM should be measured in a single-thread configuration. Ideally, the simulations should be performed on the same core of the computer and with the same system load conditions (except for the RTM) as other threads on the operating system (in particular with a high input/output load) could hamper the correct timing of the model. Multiple repetitions of the computations can be performed to get better estimates.

Given a set of atmospheric and aerosol conditions, some models can compute,
almost without extra cost, the radiances for several viewing geometries,
while other models require multiple simulations to achieve the same. For this
protocol, this would introduce a difference of a factor of circa 80 (the
number of viewing geometries) in the reported runtimes. Depending on the
application, this feature of some models may or may not be relevant. For this
reason, we present here the runtimes of the tested models in two
configurations: one taking advantage of the simultaneous output for multiple
viewing geometries and another one without this feature. In the second
configuration (called “ind” in Sect.

The VLIDORT model is an independent implementation of the discrete ordinate
method based in part on earlier work by

The DISORT model

Given a solar zenith angle, the DISORT model can provide outputs for multiple
viewing geometries. In this study, the number of geometries for the
mult timing configuration is the number of

Although we know the DISORT model to be too slow for online data assimilation
of aerosol radiances, we find it useful to document here the trade-off
between accuracy and computing time as a function of the number of streams
used in the calculations. Figure

There is a big gain in accuracy when increasing the number of streams from 4
to 8 and little further gain in accuracy beyond 16 streams. A deterioration
in accuracy is observed beyond 96 streams in the case of the BRDF surface,
which may be due to the DISORT model being coded in single-precision float.
While the dependence of model accuracy on the number of streams is well known

Accuracy (in black) and computing times (in red) for the DISORT
model as a function of the number of streams used. The Lambertian and oceanic
BRDF surface cases are shown with solid and dashed lines, respectively. The
accuracy is shown in terms of mean fractional error against VLIDORT (vector).
The computing times are an average for 20 geometries and were estimated on a
processor AMD Opteron 6378, 2.4 GHz. Please note the logarithmic scales for
the number of streams used in the DISORT calculations (

The 6SV2 model is a radiative transfer model based on the successive order of
scattering (SOS) method

Molecular scattering and absorption vertical profiles are embedded in the
code. Both are internally computed according to the atmospheric profile as
part of “core” SOS routines. It is therefore not appropriate to prescribe
our own molecular scattering and absorption profiles, so we use the 6SV2
defaults instead. The difference in the total molecular scattering optical
depth is less than 0.26 % with respect to the files provided in this
protocol. We have coupled the BRDF calculations to the 6SV2 model, but it
should be noted that the atmosphere–surface coupling of the oceanic BRDF
model in 6SV2 does not use the full surface reflection matrix but only the
first column of it (i.e. the column that relates reflected

The 6SV2 model can only compute one viewing geometry in each model call (i.e. the ind configuration is exactly the same as the mult configuration).

We also test in this study a preliminary version of the Forward-Lobe
Two-Stream Radiance Model (FLOTSAM), which is being developed at the European
Centre for Medium-Range Weather Forecasts (ECMWF). Unlike a lookup-table-based
solar RTM, FLOTSAM permits an arbitrary layering of different types of
scatterers, including clouds, aerosols, and molecules. The model has been
designed to be fast enough to be used in iterative assimilation and retrieval
schemes. It exploits the fact that particle phase functions typically contain
a “narrow forward lobe” associated with scattering angles of less than a
few degrees and a “wide forward lobe” associated with scattering angles of
the order of 15

FLOTSAM is designed to be used in iterative schemes in which multiple
calculations are performed with different profiles of optical properties but
for the same viewing geometry. This is implemented by allowing the user to
perform the set-up calculations once for a particular solar zenith angle,
viewing zenith angle, and viewing azimuth angle and then to reuse the
information in subsequent radiance calculations for different optical
profiles. We assess the computational cost for both the ind and the
mult configurations, with the latter consisting of

The protocol has been tested for the five model versions described above:
VLIDORT (vector), VLIDORT (scalar), DISORT, 6SV2, and FLOTSAM. As explained
in Sect.

With as many as 44 080 cases considered in this study, we have to find
original ways to visualize the results as explained below. Along with the
average reflectance computed for each model, the following statistical
measures have also been computed:

Runtimes of the models are presented in an absolute measure (seconds), and no reference is needed. Finally, we will comment on possible trade-offs between accuracy and computing time.

Figure

Histograms of the computed reflectances for the VLIDORT model. Panel

Same as Fig.

A first comparison of the relative error is shown in
Fig.

Same as Fig.

Same as Fig.

Figure

Same as Table

Synthetic error statistics are shown in Table

In this section, we present statistics of computing times for three of the models used in this study. All the computing times presented below were estimated on an Intel i5-4690, 3.5 GHz workstation running Linux (kernel 2.6.32), GNU C++ and Fortran compiler version 4.4.7, on a single processor and with exclusive use of the workstation. We have compiled the models using their default compilation flags, i.e. “-O” for 6SV2, “-O3” and “-march=native” for FLOTSAM, and “-O3” for DISORT.

We start this section with the 6SV2 model which can only be used in
ind configuration. Figure

Average runtimes in seconds. The standard deviation is shown in parentheses. The mult runtimes account for 640 profiles for FLOTSAM, 20 profiles for DISORT, and 1 profile per model call for 6SV2. The one-off set-up costs of FLOTSAM are invoked only once per instance of the executable, no matter how many radiances are subsequently calculated, so are the same for the ind and mult experiments.

As discussed above, the DISORT and FLOTSAM models offer efficiency savings
when multiple profiles are computed together. In the case of DISORT, multiple
viewing geometries can be computed for one solar zenith angle. In the case of
FLOTSAM, a smaller saving is that the geometry set-up calculations may be
computed once for a given solar/viewing geometry and then reused for
multiple atmospheric profiles with different optical properties. The runtime
of these models is, in general, not sensitive to the viewing geometry,
wavelength, aerosol type, or AOD. However, there are some differences between
the two types of surface reflectance. Table

For the 6SV2 model, values of Table

Table

In this study, we have presented a comprehensive benchmarking methodology and protocol, with an unprecedented number of cases combining different viewing geometries, aerosol optical depths, aerosol types, wavelengths, and surface types. The VLIDORT model in its vector configuration (i.e. considering polarization) is used as the reference model. Preliminary results in terms of accuracy and computing time are presented for three models: the well-known models DISORT and 6SV2, on the one hand, and FLOTSAM, a new fast model under development by two of the authors of this study, on the other. All models perform better when using Lambertian surface reflectance than when using an oceanic BRDF. All the models perform better under low AOD, and the scalar models have lower accuracies at shorter wavelengths. For the Lambertian surface cases, the mean fractional errors of the models are 0.8 %, 1.5 %, and 1.9 % for DISORT, 6SV2, and FLOTSAM, respectively. The BRDF cases show a larger mean fractional error with 1.5 %, 6.6 %, and 4 % for DISORT, 6SV2, and FLOTSAM, respectively.

The DISORT and 6SV2 models show comparable computing time, between tenths of a second and seconds, but DISORT can provide the solution at multiple viewing geometries for each atmospheric condition, while 6SV2 cannot. FLOTSAM is very fast, with computing times of much less than 1 ms per profile. However, it should be noted that this model is still under development and its accuracy could improve further. Moreover, all models present a tail of cases with larger errors and/or computing times. It could be interesting to characterize these better to investigate if screening out those cases can be an option in data assimilation applications. Also, a fast tangent linear and adjoint computation is required for the integration of an RTM in a variational data assimilation system.

The protocol and input and output data are available as a potential resource for interested developers and users of RTMs willing to benchmark their models.

The detailed protocol is available in PDF format. All
inputs necessary to the benchmark are available as ASCII text files. The
surface BRDF is available as a Fortran routine from the Michael Mishchenko website

The authors acknowledge support from the Copernicus Atmosphere Monitoring Service implemented by ECMWF on behalf of the European Commission. The authors thank Didier Tanré for his interesting comments on the draft of the manuscript. The authors would also like to thank the 6SV2 development team for their help in resolving small issues with the model and Vijay Natraj and Robert Spurr and the two anonymous reviewers for their comments in the discussion phase. Computing was partly performed on the ESPRI data and computing centre of the Institut Pierre-Simon Laplace.Edited by: Andrea Stenke Reviewed by: two anonymous referees