Coordinated Model Evaluation Capabilities
Coordinated Model Evaluation Capabilities (CMEC) is an effort to bring together a diverse set of analysis packages that have been developed to facilitate the systematic evaluation of Earth System Models (ESMs). CMEC includes capabilities that are supported by the U.S. Department of Energy, Office of Biological and Environmental Research (BER), and Regional and Global Model Analysis (RGMA), along with capabilities contributed by community-based experts and international agencies.
Model
Summaries
(PMP)
Extremes
(TECA)
Biogeochemistry
(ILAMB)
Biogeochemistry
(IOMB)
CMEC is motivated by the need for comprehensive and holistic evaluation of Earth system models and datasets, so as to identify persistent biases and improve data credibility. With widespread and rapid growth in the number of available metrics and diagnostic tools, a lack of standards within the evaluation community have meant that running even a single evaluation tool can require extensive user intervention. Given the significant commonality in how these evaluation tools operate, interoperability is a natural goal achievable through robust and light-weight standards.
The three goals of the CMEC project include: (1) to develop robust and light-weight standards for operation of evaluation packages and their output; (2) to develop accompanying tools for installation of evaluation packages, coordinated execution of evaluation packages, and obtaining data products necessary for operation of these tools; and (3) to build connections across groups, centers, and individual investigators performing model evaluation.
CMEC aims to operate on a variety of Earth system data products including but not limited to: Earth system model data such as contributions to the Coupled Model Intercomparison Project (CMIP), reanalysis data products, and other data products produced by global modeling centers or individual research groups.
To advance this effort, participation is sought from new and existing ESM evaluation packages. Inclusion is not limited to packages for CMIP analysis, and both metrics and diagnostics packages are welcome.
PMP provides diagnostic summaries of physical atmospheric model variables on seasonal, annual, and inter-annual time scales. It compares these variables with global satellite remote sensing and reanalysis data sets, and scores model performance based on RMSE or other metrics. PMP is useful for producing quick, high-level diagnostic summaries of physical atmosphere model performance.
Quick links: Repository, Installation, and Results
TECA is a high-performance, general purpose tool for detecting discrete weather events, such as tropical cyclones, in climate model output. Its core is a map-reduce framework, implemented in C++, that utilizes MPI and OpenMP parallelism. It features Python bindings for the core architecture, which allows rapid prototyping new detectors while taking advantage of the high-performance parallelism of the C++ core.
Quick links: Repository, Installation, and Documentation
ILAMB provides a variety of in-depth diagnostics of more than 24 terrestrial biogeochemical and hydrological model variables on annual and inter-annual time scales. It compares these variables with over 60 site-based, regional, and global observational data sets, and scores model performance based on a combination bias, RMSE, and seasonal cycle metrics. Relationships between many biogeochemical variables and physical driver variables are calculated from model results and compared with observational estimates. ILAMB is useful for detailed exploration of land biogeochemical and hydrological model responses and provides an interactive interface designed to enable the user to more rapidly understand the underlying drivers of those responses.
Quick links: Repository and Installation
IOMB provides a variety of in-depth diagnostics of marine biogeochmical model variables on annual and inter-annual time scales. It compares a growing number of variables with site-based, transect, regional, and global observational data sets, and scores model performance based on a combination of bias, RMSE, and seasonal cycle metrics. IOMB is useful for detailed exploration of ocean biogeochemical model responses and provides an interactive interface designed to enable the user to more rapidly understand the underlying drivers of those responses.
Quick links: Repository and Installation