Journal cover Journal topic
Geoscientific Model Development An interactive open-access journal of the European Geosciences Union
Geosci. Model Dev., 10, 413-423, 2017
http://www.geosci-model-dev.net/10/413/2017/
doi:10.5194/gmd-10-413-2017
© Author(s) 2017. This work is distributed
under the Creative Commons Attribution 3.0 License.
Development and technical paper
27 Jan 2017
The compression–error trade-off for large gridded data sets
Jeremy D. Silver and Charles S. Zender

Data sets

Sample CAM SE model output
C. S. Zender
doi:10.4225/49/576ca64db2a14
Sample output of the mineral Dust Entrainment And Deposition (DEAD) model
C. S. Zender
doi:10.4225/49/576c95f254b67
Sample MERRA analysis
C. S. Zender
doi:10.4225/49/576c934f73be7
Sample output from the Weather, Research and Forecasting model
J. D. Silver
doi:10.4225/49/576c900f7d289
Sample MOZART model output
J. D. Silver
doi:10.4225/49/576c8ba706ac9
Publications Copernicus
Download
Short summary
Many modern scientific research projects generate large amounts of data. Storage space is valuable and may be limited; hence compression is vital. We tested different compression methods for large gridded data sets, assessing the space savings and the amount of precision lost. We found a general trade-off between precision and compression, with compression well-predicted by the entropy of the data set. A method introduced here proved to be a competitive archive format for gridded numerical data.
Many modern scientific research projects generate large amounts of data. Storage space is...
Share