A methodology for evaluating the impact of data compression on climate simulation data

AMS Citation:
Baker, A., and Coauthors, 2014: A methodology for evaluating the impact of data compression on climate simulation data. Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing, 203-214, doi:10.1145/2600212.2600217.
Date:2014-07-01
Resource Type:article
Title:A methodology for evaluating the impact of data compression on climate simulation data
Abstract: High-resolution climate simulations require tremendous computing resources and can generate massive datasets. At present, preserving the data from these simulations consumes vast storage resources at institutions such as the National Center for Atmospheric Research (NCAR). The historical data generation trends are economically unsustainable, and storage resources are already beginning to limit science objectives. To mitigate this problem, we investigate the use of data compression techniques on climate simulation data from the Community Earth System Model. Ultimately, to convince climate scientists to compress their simulation data, we must be able to demonstrate that the reconstructed data reveals the same mean climate as the original data, and this paper is a first step toward that goal. To that end, we develop an approach for verifying the climate data and use it to evaluate several compression algorithms. We find that the diversity of the climate data requires the individual treatment of variables, and, in doing so, the reconstructed data can fall within the natural variability of the system, while achieving compression rates of up to 5:1.
Peer Review:Refereed
Copyright Information:Copyright 2014 Association for Computing Machinery.
OpenSky citable URL: ark:/85065/d7wd429q
Publisher's Version: 10.1145/2600212.2600217
Author(s):
  • Allison Baker - NCAR/UCAR
  • Haiying Xu - NCAR/UCAR
  • John Dennis - NCAR/UCAR
  • Michael Levy - NCAR/UCAR
  • Doug Nychka - NCAR/UCAR
  • Sheri Mickelson - NCAR/UCAR
  • James Edwards - NCAR/UCAR
  • Mariana Vertenstein - NCAR/UCAR
  • Al Wegener
  • Random Profile

    NCAR ASSOC DIR

    Recent & Upcoming Visitors