Model Evaluation Packages

This table describes the metrics and diagnostics available in CMEC packages

Package Contents Source Documentation
Analyzing Scales of Precipitation (ASoP) Coherence: Temporal intermittency metrics, spatial intermittency metrics, plots of correlations of precipitation in space and time over regional blocks.
Spectral: Histogram overlap metric, file (for each dataset) containing 100-bin distribution of counts over the time period for each gridpoint, plots of 1d histograms of precipitation, maps of actual and fractional contribution to precipitation rate by intensity range
Github repository User Guide
Cyclone Metrics Package (CyMeP) Statistics: mean, bias, Pearson correlation, Spearman correlation, hit rate, false alarm ratio
Statistics measure the following storm characteristics: tropical cyclone days, storm genesis, storm intensity (minimum sea level pressure, maximum 10-meter wind speed, accumulated cyclone energy, pressure accumulated cyclone energy), latitude of lifetime-maximum intensity
Figures: Scorecards, cyclone trajectories, spatial density maps, Taylor diagrams, interannual cycle line plots, seasonal cycle line plots
Github repository CyMeP README
Drought Metrics Metrics: Mean precipitation, mean SPI6, mean SPI36, seasonality of precipitation, long term monthly mean, fractional area coverage of drought, proportion of dry months, count of dry months, drought intensity, probability of drought initiation, probability of drought termination, total score
Figures: Taylor diagram, heatmap of total scores
Github repository Drought Metrics README
International Land Model Benchmarking (ILAMB) Metrics: A set of scores for spatial and temporal statistics of a model compared with observations for different quantities. Scores include overall score, bias score, RMSE score, and others.
Metrics categories include Ecosystem and Carbon Cycle, Hydrology Cycle, Radiation and Energy Cycle, Forcings, and Relationships (Gross Primary Productivity, Leaf area index, and Evapotranspiration).
Figures: contour maps, time series plots, Taylor diagrams
Github repository Documentation website
International Ocean Model Benchmarking (IOMB) Metrics: A set of scores that account for spatial and temporal statistics of a model compared with observations for different quantities. Scores include overall score, bias score, RMSE score, and others.
Metrics categories include Ecosystems, Nutrients, Carbon, Physical Drivers, and Relationships.
Figures: contour maps, time series plots, Taylor diagrams
Github repository (shared with ILAMB) Documentation website (shared with ILAMB)
NOAA Model Diagnostics Task Force (MDTF) Process Oriented Diagnostics (PODs):
ENSO_MSE
ENSO_RWS
Extratropical Variance
Madden-Julian Oscillation Propogation and Amplitude
Madden-Julian Oscillation Spectra and Phasing
Madden-Julian Oscillation Teleconnections
Coupling between Soil Moisture and Evapotranspiration
Wavenumber-Frequency Spectra
Convective Transition Diagnostics
Mixed Layer Depth
Ocean Sufrace Latent Heat Flux Diagnostic
Diurnal Cycle of Precipitation
Surface Temperature Extremes and Distribution Shape
Tropical Pacific Sea Level Diagnostic
Github repository Documentation website
Project Website
PCMDI Metrics Package (PMP) Mean Climate: Statistics that characterize atmospheric variables and compare models with observations, including bias, mean, correlation, root mean square error, and standard deviation.
Monsoon (Wang): Correlation, rmsn, threat score for defined regions (North America, South America, North Africa, South Africa, Asia, Australia, and "all" monsoon domains).
Monsoon (Sperber): Decay index, duration, onset index, slope for defined regions (All-India, Northern Australia, Sahel, Gulf of Guinea, North America, South America).
Diurnal Cycle: Daily mean precipitation, standard deviation of mean diurnal cycle, standard deviation of hourly means, standard deviation of daily means, Fourier harmonics (amplitude and phase)
Modes of Variability: Statistics for the Northern Annular Mode, Northern Atlantic Oscillation, Sourthern Annular Mode, Pacific North American Pattern, Pacific Decadal Oscillation.
Madden-Julian Oscillation metrics: East power, West power, East-west power ratio
El Niño–Southern Oscillation (ENSO) performance: Cold tongue bias, trade wind bias, dry equator bias, double intertropical convergence zone bias, seasonalities of biases, ENSO amplitude, ENSO life cycle, ENSO asymmetry, ENSO diversity
El Niño–Southern Oscillation teleconnections: ENSO amplitude, ENSO pattern, ENSO seasonality, DJF precipitation teleconnection, DJF surface temperature teleconnection, JJA precipitation teleconnection, JJA surface temperature teleconnection
El Niño–Southern Oscillation processes: Cold tongue bias, trade winds bias, ENSO pattern, ENSO amplitude, ENSO seasonality, ENSO asymmetry, ocean-atmosphere Bjerknes feedback (atmosphere branch, ocean-atmosphere branch, and ocean branch), balance of Bjerknes feedback and heat fluxes on sea surface temperature anomalies
Precipitation variability: Forced annual, forced semi-annual, unforced interannual, unforced seasonal-annual, unforced sub-seasonal, unforced synoptic
Github repository Documentation website