Identifying fast and robust numerical solvers is a critical issue that needs to
be addressed in order to improve projections of polar ice sheets evolving in a
changing climate. This work evaluates the impact of using advanced numerical
solvers for transient ice-flow simulations conducted with the JPL–UCI Ice Sheet
System Model (ISSM). We identify optimal numerical solvers by testing a broad
suite of readily available solvers, ranging from direct sparse solvers to
preconditioned iterative methods, on the commonly used Ice Sheet Model
Intercomparison Project for Higher-Order ice sheet Models benchmark tests.
Three types of analyses are considered: mass transport, horizontal stress
balance, and incompressibility. The results of the fastest solvers for
each analysis type are ranked based on their scalability across mesh size and
basal boundary conditions. We find that the fastest iterative solvers are

Fast and efficient numerical simulations of ice flow are critical to
understanding the role and impact of polar ice sheets (Greenland Ice Sheet,
GIS, and Antarctica Ice Sheet, AIS) on sea-level rise in a changing climate. As
reported in the Intergovernmental Panel on Climate Change AR5 Synthesis Report

The traditional approach to address this combined challenge is to solve a
simplified set of equations for stress balance, relying on approximations to
the stress tensor, which drastically reduce the number of degrees of freedom
(DOFs). These approximations have been extensively documented in the literature

The higher-order formulation

Finally, for the interior of the ice sheet, ISMs rely on the Shallow Ice
Approximation

This list of model approximations is not exhaustive and does not include hybrid
approaches such as the L1L2 formulation that mixes both SSA and SIA
approximations. For readers who are interested in this topic, a comprehensive
classification can be found in

The Ice Sheet System Model (ISSM) framework relies on a massively parallelized
thermo-mechanical finite-element ice-sheet model that was developed to simulate
the evolution of Greenland and Antarctica in a changing climate

Using a direct parallel solver provides a robust and stable numerical scheme.
However, this approach tends to be slow and memory-intensive for large
problems, where the number of DOFs approaches 100 000 or more. As noted by

This work specifically focuses on this widely used formulation, as it currently
represents the most computationally demanding model (short of full Stokes)
capable of capturing vertical as well as horizontal shear stresses necessary to
model an entire basin

The HO model represents the next, most complete formulation and represents a
significant computational bottleneck compared to its 2-D and 1-D counterparts,
which are significantly less demanding because of the drastic reduction in the
number of DOFs required for vertically collapsed 2-D meshes (SSA) or local 1-D
analytical formulations (SIA). In contrast to the studies focused on specific,
customized solvers for approximate flow models (i.e.,

The manuscript is structured as follows. In Sect. 2, we describe the ISMIP-HOM experiments that we consider and the approach adopted for testing different numerical methods. In Sect. 3, we summarize efficient baseline solvers for transient simulations using the ISSM framework. In Sect. 4, we discuss the timing results from testing a wide range of solvers which, in addition to enabling large-scale simulations, yield significant speed-ups in solution time. We then conclude on the scope of this study and summarize our findings.

In order to identify optimal numerical solvers for a broad class of transient ice-flow simulations, we test a suite of PETSc solvers on synthetic ice-flow experiments with varying basal sliding conditions. We consider the effectiveness of competing solvers (in terms of speed) using the ISMIP-HOM tests, since these experiments represent a suite of accepted benchmark tests that are commonly used in the community to validate higher-order (3-D) approximations of the stress balance equations. We first use Experiment F of the ISMIP-HOM tests to evaluate competing numerical solvers since it entails a transient ice-flow simulation with two test cases involving distinct basal sliding regimes. This transient simulation allows us to independently test solvers on each analysis component (mass transport, horizontal stress balance, and incompressibility) underlying a transient simulation in ISSM and evaluate the performance of competing solvers for models using different basal sliding conditions. Experiment F is representative of the type of physics solved for in many scenarios of ice sheets retreating and advancing onto downward or upward-sloping bedrocks. It is therefore wide-ranging in terms of applicability and happens to be a commonly accepted benchmark experiment that is used by many ISMs. However, since Experiment F specifies a constant viscosity for ice, we also consider ISMIP-HOM Experiment A as it includes a nonlinear rheology for ice. While this is only a static test, Experiment A allows us to evaluate the performance of solvers applied to the horizontal stress balance equations for simulations using a more physically realistic model of ice rheology, albeit only for a nonsliding case. For testing the impact of different basal sliding conditions on solver performance, we refer to the results from Experiment F, which includes both sliding and no-slip basal conditions.

Experiment F consists of simulating the flow of a 3-D slab of ice (10 km square,
1 km thick) over an inclined bed (3

ISSM results for the ISMIP-HOM benchmark Experiment F transient simulation after
1500 years. Surface velocity (m a

Experiment A simulates the flow of a 3-D slab of ice (80 km square, 1 km thick)
over an inclined bed (0.5

Our approach for identifying efficient numerical methods for each analysis
component of the transient simulation in ISSM is to independently test
combinations of preconditioning matrices with iterative methods on the system
of equations resulting from the finite-element discretization of the stress-balance and mass-transport equations. Since we rely on the HO formulation, the
stress balance only solves the horizontal stress balance and requires an
additional step to solve for the vertical velocities. Here, we use the
incompressibility equation and an

For each system of equations, we test a wide range of solvers including the
default solver (MUMPS) and preconditioned iterative methods provided by PETSc.
When referring to the solvers available through the PETSc interface, we rely on
the abbreviations used in the PETSc libraries by labeling a preconditioning
matrix as a PC and an iterative method as a KSP (Krylov subspace method). Here
a preconditioning matrix improves the spectral properties of the problem (i.e.,
the condition number) without altering the solution provided by the iterative
method. Since the Jacobian of the system of equations resulting from the finite-element discretization of the horizontal stress balance is symmetric positive
definite, a wide range of iterative solvers and preconditioners are applicable
and potentially efficient. For a complete review of potential solvers we point
to

Horizontal velocity analysis: timing results for the fastest solvers (top 10 %) tested on ISMIP-HOM Experiment F. The top (1 %) timing results are distinguished using color-filled symbols. Both basal boundary conditions for Experiment F are shown: frozen bed (upper half) and sliding bed (lower half). Each solver is represented by the combination of a preconditioner (horizontal rows) and a Krylov subspace method (vertical columns) using PETSc abbreviations. Simulations are performed using four mesh sizes (denoted by the symbols in the legend) and four CPU cases (denoted by the colors in the legend). Only the fastest CPU case (i.e., color) is displayed. Red shaded boxes highlight solver combinations that rank among the fastest methods for all model sizes and both bed conditions (i.e., four symbols in the top and bottom frame).

The slab of ice in Experiment F is modeled using four levels of mesh
refinement. The smallest, coarsest-resolution model consists of 2000
elements resulting from a

To study the impact of using a nonlinear viscosity model for ice on solver speed and convergence, we follow the same methodology applied to Experiment F (i.e., same solvers, discretization strategy, and CPU cases) and evaluate the performance of solvers applied to the stress-balance equations for Experiment A. However, we omit testing of the largest model size (i.e., 1 024 000 elements) due to the intense computational resources necessary for this model size and the limited information gained by this prognostic test relative to the more comprehensive transient model. Simulations of Experiment A were performed more recently on the Pleiades cluster using upgraded Broadwell nodes (two 14-core Intel Xeon E5-2680v4 processors per node with 128 GB per node) with ISSM version 4.11 and PETSc version 3.7. Updates to the ISSM code from version 4.2.5 to version 4.11 have added new capabilities that are not used in this study. The solution methods and algorithms between these versions are the same, and the results from this study apply to all intermediate versions that users may be working with.

Incompressibility analysis: timing results for the fastest solvers (top 5 %) tested on ISMIP-HOM Experiment F.
The top (1 %) timing results are distinguished using color-filled symbols.
Red shaded boxes highlight solver combinations that rank among the
fastest methods for all model sizes and both bed conditions (i.e., four symbols in the top and bottom frame).
See Fig.

Mass-transport analysis: timing results for the fastest solvers (top 5 %) tested on ISMIP-HOM Experiment F.
The top (1 %) timing results are distinguished using color-filled symbols.
Red shaded boxes highlight solver combinations that rank among the
fastest methods for all model sizes and both bed conditions (i.e., four symbols in the top and bottom frame).
See Fig.

Weak scalability for simulating ISMIP-HOM Experiment F using the default ISSM
solver (MUMPS), compared with a combination of robust solvers (selected from the
highlighted solvers in Figs.

Strong scalability of the default ISSM solver (MUMPS) compared with a
combination of robust solvers (selected from the highlighted solvers in Figs.

Horizontal velocity analysis: timing results for the fastest solvers (top 15 %) tested on ISMIP-HOM Experiment A.
The top (1 %) timing results are distinguished using color-filled symbols.
See Fig.

Scalability of the default ISSM solver (MUMPS) compared with a preconditioned
iterative method (PC

Since our primary interest is identifying fast, stable solvers for transient
ice-flow simulations, we first present the results from Experiment F. Using
the profiling features in ISSM, we evaluate the timing results for each
simulation (measured in seconds), which consists of the CPU time associated
with assembling the stiffness matrix and load vector, solving the system of
equations, and updating the input from the solution. Here, solving the system
of equations represents the majority of the reported CPU time. Since this
study focuses on the relative performance of the tested solvers, the timing
results for different methods are directly comparable, as the additional steps
are consistent across the tested methods and do not bias the results. Only the
fastest results for each model size for solving the horizontal velocity
analysis (fastest 10 %), the incompressibility analysis (fastest 5 %), and the
mass-transport analysis (fastest 5 %) are shown in Figs.

For Experiment F, we highlight the most robust solvers in Figs.

Figures

In considering the magnitude of the slopes representing the weak and strong
scalability, we recall that our timing results include routines outside of the
solver procedure (i.e., assembling the stiffness matrix, load vector, and
updating the input from the solution) that are not necessarily scalable.
However, the relative scalability (i.e., differences in slope) between the
preconditioned iterative methods and the direct solver illustrates the
differences in performance between these approaches. Optimal weak scalability
for a solver implies a horizontal slope in Fig.

To show the impact of nonlinear viscosity on the efficiency of the presented
solvers, we plot the timing results for solving the stress balance equations in
Experiment A (Fig.

Solving the horizontal velocity analysis dominates the CPU time needed to solve
a transient simulation since this analysis involves more DOFs, and has a much
higher condition number, than the mass-transport and incompressibility analyses.
Our results, however, show that this bottleneck can be significantly reduced
for moderate-sized models (i.e., 16 000 to 128 000 elements) by using any of the
highlighted solvers, which leads to significant speed-ups relative to the
default solver (i.e.,

Most of the limitations associated with using the default solver on large
models arise from the LU Factorization phase in the MUMPS solver, which is not
yet parallelized. This could be remedied by switching on the out-of-core
computation capability for this decomposition, but this has not been
successfully tested yet and would potentially shift the problem of memory
limitations to disk space and read/write speeds (the size of the matrices being
significant). Furthermore, Fig.

In evaluating the effect of using a nonlinear viscosity model for ice on solver
performance, we see that many of the methods which efficiently solve the
horizontal velocity analysis for Experiment A (Fig.

In practice, users may experience issues with numerical convergence when
applying some of the iterative methods presented in Figs.

While the relative rankings of the tested solvers presented in this work are specific to the ISMIP-HOM experiments, applying these methods to simulations using realistic model parameterizations (e.g., data-driven boundary conditions, anisotropic meshes, and complex geometries) also results in significant speed-ups compared to the default solver, though these computations are not shown here. We acknowledge that in relation to using synthetic test cases, real-world model parameterizations may affect the convergence and relative performance of the iterative solvers tested in this work. However, since any of the highlighted solvers are significantly more efficient than using a direct solver, our results provide a useful starting point for modelers looking for efficient methods to use for specific ice-flow simulations.

We recommend that future refinement of these results include customization of
the PETSc components, which can lead to significant performance gains over the
default values, and include more realistic geometries with varying degrees of
anisotropy. Finally, it should be noted that the presented optimal solvers do
not require a supercomputer and may be used with fewer CPUs than the number
indicated by the symbol color in Figs.

The results presented herein offer guidance for selecting fast and robust
numerical solvers for transient ice-flow simulations across a broad range of
model sizes and basal boundary conditions. Here, the highlighted solvers offer
significant speed-ups (

The results from this work are reproducible using ISSM (versions 4.2.5–4.11)
with the corresponding PETSc solvers used for each analysis type. The
current version of ISSM is available for download at

This work was performed at the Jet Propulsion Laboratory, California Institute of Technology, and at the Department of Earth System Science, University of California Irvine, under two contracts: one a grant from the National Science Foundation (NSF) (award ANT-1155885) and the other a contract with the National Aeronautics and Space Administration (NASA), funded by the Cryospheric Sciences Program and the Modeling Analysis and Prediction Program. Resources supporting the numerical simulations were provided by the NASA High-End Computing (HEC) Program through the NASA Advanced Supercomputing (NAS) Division at Ames Research Center. We would like to acknowledge the insights and help from Jed Brown, as well as helpful feedback from the anonymous reviewers. This is UTIG contribution 3033.Edited by: J. Fyke Reviewed by: two anonymous referees