Light-weight parallel Python tools for earth system modeling workflows

AMS Citation:
Paul, K., S. Mickelson, J. M. Dennis, H. Xu, and D. Brown, 2015: Light-weight parallel Python tools for earth system modeling workflows. 2015 IEEE International Conference on Big Data, Institute of Electrical and Electronics Engineers (IEEE), Santa Clara, CA, US.
Date:2015-10-29
Resource Type:conference material
Title:Light-weight parallel Python tools for earth system modeling workflows
Abstract: In the last 30 years, earth system modeling has become increasingly data-intensive. The Community Earth System Model (CESM) response to the next Intergovernmental Panel on Climate Change (IPCC) assessment report (AR6) may require close to 1 Billion CPU hours of computation and generate up to 12 PB of raw data for post-processing. Existing post-processing tools are serial-only and impossibly slow with this much data. To improve the post-processing performance, our team has adopted a strategy of targeted replacement of the "bottleneck software" with light-weight parallel Python alternatives. This allows maximum impact with the least disruption to the CESM community and the shortest delivery time. We developed two light-weight parallel Python tools: one to convert model output from time-slice to time-series format, and one to perform fast time-averaging of time-series data. We present the motivation, approach, and results of these two tools, and our plans for future research and development.
Peer Review:Non-refereed
Copyright Information:Copyright Author(s). This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
OpenSky citable URL: ark:/85065/d7s1847f
Author(s):
  • Kevin Paul - NCAR/UCAR
  • Sheri Mickelson - NCAR/UCAR
  • John M. Dennis - NCAR/UCAR
  • Haiying Xu - NCAR/UCAR
  • David Brown - NCAR/UCAR
  • Random Profile

    PROJ SCIENTIST III

    Recent & Upcoming Visitors