Light-weight parallel python tools for climate model workflows [poster]

AMS Citation:
Mickelson, S., K. Paul, J. M. Dennis, and W. G. Strand, 2014: Light-weight parallel python tools for climate model workflows [poster]. AGU Fall Meeting 2014, American Geophysical Union, San Francisco, CA, US.
Date:2014-12-16
Resource Type:conference material
Title:Light-weight parallel python tools for climate model workflows [poster]
Abstract: It is expected that the data required for the next Intergovernmental Panel on Climate Change (IPCC) Assessment Report (AR6) will increase by more than a factor of 10 to an expected 25 terabytes per model. Experiences from the last Coupled Model Intercomparison Project (CMIP5), which assembled the data used for the last IPCC Assessment Report (AR5), concluded that the processing, archiving, and post-run diagnostic operations required on such large model output took almost as long to complete as the model runs themselves! As a result, we have been investigating and developing light-weight Python-based tools to parallelize the time-intensive post-run steps in the climate model workflow. In particular, we have developed a parallel Python tool for converting time-slice model output to time-series format, and we have more recently developed a parallel Python tool to perform fast time-averaging of time-series data, an operation needed for many diagnostic computations. These tools are designed to be light-weight, easy to install, with very few dependencies, and that can be easily inserted into the climate model workflow with negligible disruption. In this work, we present the motivation, approach, and results of the two light-weight parallel Python tools that we have developed, as well as our plans for future research and development.
Peer Review:Non-refereed
Copyright Information:Copyright Author(s). This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
OpenSky citable URL: ark:/85065/d7wq02t1
Author(s):
  • Sheri Mickelson - NCAR/UCAR
  • Kevin Paul - NCAR/UCAR
  • John Dennis - NCAR/UCAR
  • Warren Strand - NCAR/UCAR
  • Random Profile

    SR SCI EMERITUS

    Recent & Upcoming Visitors