Articles | Volume 10, issue 1
https://doi.org/10.5194/gmd-10-413-2017
https://doi.org/10.5194/gmd-10-413-2017
Development and technical paper
 | 
27 Jan 2017
Development and technical paper |  | 27 Jan 2017

The compression–error trade-off for large gridded data sets

Jeremy D. Silver and Charles S. Zender

Related authors

Interannual variability in the Australian carbon cycle over 2015–2019, based on assimilation of Orbiting Carbon Observatory-2 (OCO-2) satellite data
Yohanna Villalobos, Peter J. Rayner, Jeremy D. Silver, Steven Thomas, Vanessa Haverd, Jürgen Knauer, Zoë M. Loh, Nicholas M. Deutscher, David W. T. Griffith, and David F. Pollard
Atmos. Chem. Phys., 22, 8897–8934, https://doi.org/10.5194/acp-22-8897-2022,https://doi.org/10.5194/acp-22-8897-2022, 2022
Short summary
Was Australia a sink or source of CO2 in 2015? Data assimilation using OCO-2 satellite measurements
Yohanna Villalobos, Peter J. Rayner, Jeremy D. Silver, Steven Thomas, Vanessa Haverd, Jürgen Knauer, Zoë M. Loh, Nicholas M. Deutscher, David W. T. Griffith, and David F. Pollard
Atmos. Chem. Phys., 21, 17453–17494, https://doi.org/10.5194/acp-21-17453-2021,https://doi.org/10.5194/acp-21-17453-2021, 2021
Short summary
Comparison of formaldehyde tropospheric columns in Australia and New Zealand using MAX-DOAS, FTIR and TROPOMI
Robert G. Ryan, Jeremy D. Silver, Richard Querel, Dan Smale, Steve Rhodes, Matt Tully, Nicholas Jones, and Robyn Schofield
Atmos. Meas. Tech., 13, 6501–6519, https://doi.org/10.5194/amt-13-6501-2020,https://doi.org/10.5194/amt-13-6501-2020, 2020
Short summary
The potential of Orbiting Carbon Observatory-2 data to reduce the uncertainties in CO2 surface fluxes over Australia using a variational assimilation scheme
Yohanna Villalobos, Peter Rayner, Steven Thomas, and Jeremy Silver
Atmos. Chem. Phys., 20, 8473–8500, https://doi.org/10.5194/acp-20-8473-2020,https://doi.org/10.5194/acp-20-8473-2020, 2020
Short summary
Estimating global gross primary productivity using chlorophyll fluorescence and a data assimilation system with the BETHY-SCOPE model
Alexander J. Norton, Peter J. Rayner, Ernest N. Koffi, Marko Scholze, Jeremy D. Silver, and Ying-Ping Wang
Biogeosciences, 16, 3069–3093, https://doi.org/10.5194/bg-16-3069-2019,https://doi.org/10.5194/bg-16-3069-2019, 2019
Short summary

Related subject area

Numerical methods
A dynamical core based on a discontinuous Galerkin method for higher-order finite-element sea ice modeling
Thomas Richter, Véronique Dansereau, Christian Lessig, and Piotr Minakowski
Geosci. Model Dev., 16, 3907–3926, https://doi.org/10.5194/gmd-16-3907-2023,https://doi.org/10.5194/gmd-16-3907-2023, 2023
Short summary
GStatSim V1.0: a Python package for geostatistical interpolation and conditional simulation
Emma J. MacKie, Michael Field, Lijing Wang, Zhen Yin, Nathan Schoedl, Matthew Hibbs, and Allan Zhang
Geosci. Model Dev., 16, 3765–3783, https://doi.org/10.5194/gmd-16-3765-2023,https://doi.org/10.5194/gmd-16-3765-2023, 2023
Short summary
Leveraging Google's Tensor Processing Units for tsunami-risk mitigation planning in the Pacific Northwest and beyond
Ian Madden, Simone Marras, and Jenny Suckale
Geosci. Model Dev., 16, 3479–3500, https://doi.org/10.5194/gmd-16-3479-2023,https://doi.org/10.5194/gmd-16-3479-2023, 2023
Short summary
An improved subgrid channel model with upwind-form artificial diffusion for river hydrodynamics and floodplain inundation simulation
Youtong Rong, Paul Bates, and Jeffrey Neal
Geosci. Model Dev., 16, 3291–3311, https://doi.org/10.5194/gmd-16-3291-2023,https://doi.org/10.5194/gmd-16-3291-2023, 2023
Short summary
A model instability issue in the National Centers for Environmental Prediction Global Forecast System version 16 and potential solutions
Xiaqiong Zhou and Hann-Ming Henry Juang
Geosci. Model Dev., 16, 3263–3274, https://doi.org/10.5194/gmd-16-3263-2023,https://doi.org/10.5194/gmd-16-3263-2023, 2023
Short summary

Cited articles

Baker, A. H., Xu, H., Dennis, J. M., Levy, M. N., Nychka, D., Mickelson, S. A., Edwards, J., Vertenstein, M., and Wegener, A.: A methodology for evaluating the impact of data compression on climate simulation data, in: Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing, Vancouver, BC, Canada, 23–27 June 2014, ACM, 203–214, 2014.
Berger, T.: Rate-Distortion Theory, in: Wiley Encyclopedia of Telecommunications, John Wiley & Sons, Inc., https://doi.org/10.1002/0471219282.eot142, 2003.
Caron, J.: Converting GRIB to netCDF-4: Compression studies, presentation to the workshop "Closing the GRIB/netCDF gap", European Centre for Medium Range Weather Forecasts (ECMWF), 24–25 September 2014, Reading, UK, available at: http://www.ecmwf.int/sites/default/files/elibrary/2014/13711-converting-grib-netcdf-4.pdf, last access: 17 June 2016, 2014.
Dee, D., Uppala, S., Simmons, A., Berrisford, P., Poli, P., Kobayashi, S., Andrae, U., Balmaseda, M., Balsamo, G., Bauer, P., and Bechtold, P.: The ERA-Interim reanalysis: Configuration and performance of the data assimilation system, Q. J. Roy. Meteor. Soc., 137, 553–597, 2011.
Deutsch, L. P.: DEFLATE compressed data format specification version 1.3, Tech. Rep. IETF RFC1951, Internet Engineering Task Force, 2008.
Download
Short summary
Many modern scientific research projects generate large amounts of data. Storage space is valuable and may be limited; hence compression is vital. We tested different compression methods for large gridded data sets, assessing the space savings and the amount of precision lost. We found a general trade-off between precision and compression, with compression well-predicted by the entropy of the data set. A method introduced here proved to be a competitive archive format for gridded numerical data.