EvalML

Run evaluation pipelines for data-driven weather models built with Anemoi.

Getting started

1. Installation
2. Credentials setup
3. Workspace setup

Features:

• Experiments: compare model performance via standard and diagnostic verification
• Showcasing: produce visual material for specific events
• Sandboxing: generate an isolated inference development environments for any model

Quick example

To run an experiment, prepare a demo config file like the one below and adapt it to your setup:

# yaml-language-server: $schema=../workflow/tools/config.schema.json
description: |
  Demo experiment: compare two forecaster checkpoints against the same baseline and truth data.

# Optional: used in the output directory name. If omitted, the config file name is used.
config_label: co2-forecasters-demo

# Choose one date style:
# 1. A regular range with a run frequency (shown here)
# 2. An explicit list of ISO timestamps for case studies or showcases
dates:
  start: 2020-01-01T00:00
  end: 2020-01-10T00:00
  frequency: 60h
  # Optional: skip specific initialisation dates
  # blacklist:
  #   - 2020-01-05T00:00

runs:
  # Each item is either `forecaster`, `temporal_downscaler` or `baseline`
  - forecaster:
      # `checkpoint` may point to a supported MLflow run URL, a Hugging Face `.ckpt` URL, or a local checkpoint path.
      checkpoint: https://servicedepl.meteoswiss.ch/mlstore#/experiments/228/runs/2f962c89ff644ca7940072fa9cd088ec
      # Labels are what appear in plots, tables, and reports.
      label: Stage D - N320 global grid with CERRA finetuning
      # Lead times follow start/end/step in hours.
      steps: 0/120/6
      # `config` points to the inference config template for the run. If omitted, evalml uses the bundled default for the run type.
      config: resources/inference/configs/sgm-forecaster-global.yaml
      # Optional extra dependencies needed by this checkpoint at inference time.
      extra_requirements:
        - git+https://github.com/ecmwf/anemoi-inference.git@0.8.3
  - forecaster:
      checkpoint: https://mlflow.ecmwf.int/#/experiments/103/runs/d0846032fc7248a58b089cbe8fa4c511
      label: M-1 forecaster
      steps: 0/120/6
      config: resources/inference/configs/sgm-forecaster-global_trimedge.yaml
  # A temporal_downscaler entry can optionally embed its driving forecaster config inline.
  # - temporal_downscaler:
  #     checkpoint: /path/to/temporal_downscaler.ckpt
  #     label: My temporal downscaler
  #     steps: 0/120/1
  #     forecaster:
  #       checkpoint: https://...
  #       steps: 0/120/6
  - baseline:
      label: COSMO-E
      root: /store_new/mch/msopr/ml/COSMO-E
      steps: 0/120/6

truth:
  label: COSMO KENDA
  root: /scratch/mch/fzanetta/data/anemoi/datasets/mch-co2-an-archive-0p02-2015-2020-6h-v3-pl13.zarr

experiment:
  stratification:
    regions:
      - icon  # first item will be used as default verification region
      - jura
      - mittelland
      - voralpen
      - alpennordhang
      - innerealpentaeler
      - alpensuedseite
    root: /store_new/mch/msopr/ml/regions/Prognoseregionen_LV95_20220517
  # Optional: categorical thresholds used for binary skill scores, one entry per parameter.
  thresholds:
    TOT_PREC:
      gt: [0.0, 1, 5]
    T_2M:
      lt: [273.15]
      gt: [288.15, 298.15]
  dashboard:
    # Stratification dimensions to include in the experiment dashboard (any of season, region, init_hour).
    stratification:
      - season
  # Optional: named scorecards comparing each forecaster against a chosen baseline.
  scorecards:
    enabled: true
    sections:
      short_range:
        # Baseline label — must match the `label` field of a baseline entry in `runs`.
        baseline: COSMO-E
        # Lead-time range as start/stop/step (hours).
        lead_times: "0/120/6"
        # Stratification dimension to use as scorecard columns (e.g. region, season).
        stratification: region
        # Variables and metrics as scorecard rows. Format: VAR:METRIC1,METRIC2,...
        # Supported metrics: RMSE, R2, ETS, POD, FAR (categorical metrics require thresholds).
        variables:
          - "T_2M:RMSE,R2"
          - "TOT_PREC:RMSE,ETS"

locations:
  # All workflow outputs are written under this root.
  output_root: output/

profile:
  # Passed through to Snakemake. Tune this block to match your cluster or local executor.
  executor: slurm
  global_resources:
    # Limits total concurrent GPU use across submitted jobs.
    gpus: 16
  default_resources:
    slurm_partition: "postproc"
    cpus_per_task: 1
    mem_mb_per_cpu: 1800
    runtime: "1h"
  jobs: 50
  batch_rules:
    # Group many small plotting jobs into fewer submissions.
    plot_forecast_frame: 32

The runs list accepts forecaster, temporal_downscaler, and baseline entries. For dates, you can either provide a start / end / frequency block as above or an explicit list of ISO timestamps for case-study style runs. Stratification, thresholds, dashboard, and scorecard settings are all grouped under the experiment key.

Precipitation (TOT_PREC) from forecaster and temporal downscaler runs is expected to be accumulated since the start of the forecast, in kg m-2 (mm), like ICON.

You can then run it with:

evalml experiment path/to/experiment/config.yaml --report

Truth sources

The truth.root value selects how the ground truth is loaded:

• Anemoi dataset — a path ending in .zarr, read as an anemoi analysis dataset (e.g. KENDA); other zarr layouts are not supported.

• DWH / jretrievedwh — a jretrievedwh: marker string fetching surface observations (e.g. SMN) live from the MeteoSwiss data warehouse. Variables are mapped to ICON names in SI units (temperatures in K, pressure in Pa, precipitation as the hourly sum); wind U_10M/V_10M are derived from speed + direction, RELHUM_2M is read directly and TW_2M is derived from temperature, dew point and station pressure.

  Marker syntax (station selection is required; pick one of group/locations/bbox):

  truth:
    label: SwissMetNet
    root: jretrievedwh:SwissMetNet                  # station group (default if none is given)
    # root: jretrievedwh:locations=ARO,KLO,LUG      # explicit nat_abbr list
    # root: jretrievedwh:bbox=45.8,47.8,5.9,10.5    # minlat,maxlat,minlon,maxlon

  Optional keys are appended with ;, e.g. jretrievedwh:bbox=45.7,48.0,5.8,10.8;use_limitation=40. use_limitation is passed to jretrieve's --use-limitation and selects which stations are returned (default: 40).

  Prerequisites: jretrievedwh.py must be on $PATH (falls back to /oprusers/osm/opr.inn/bin/jretrievedwh.py) and DWH credentials must be available — see Credentials setup. No data is pre-downloaded — the obs are queried at verification time.

• INCA-CH — a path into the INCA archive (any path with an INCA directory), read directly from the NetCDF files. Step 0 of each INCA run is an analysis, and only step 0 is used: the truth at valid time t comes from the run started at t. The data are on the 1 km INCA grid (710×640, Swiss LV03), so forecasts are mapped onto it by nearest neighbour, as for a gridded zarr truth.

  truth:
    label: INCA analysis
    root: /store_new/mch/msclim/INCA

  Variables (evalml name ← INCA file prefix):

  Group	evalml name ← INCA
  Temperature (K)	T_2M ← TT, TD_2M ← TD, TW_2M ← TW (wet bulb), T_G ← TG (surface)
  Humidity (%)	RELHUM_2M, derived from TT/TD
  Wind (10-min products)	SP_10M ← FF_10min, DD_10M ← DD_10min, VMAX_10M ← WG_10min; U_10M, V_10M are derived from FF_10min/DD_10min
  Precipitation (kg m-2)	TOT_PREC ← RR, TOT_SNOW ← RS (CombiPrecip based). Use period names such as TOT_PREC1, TOT_PREC6
  Precipitation type	PTYPE ← PT (CombiPrecip based), recoded to the ICON convention (WMO code table 4.201: 0 none, 1 rain, 3 freezing rain, 5 snow, 7 rain and snow, 10 hail)
  Cloud, sun, lightning	CLCT ← CT (%), SUNSHINE_REL ← SU (%), LIGHTNING_PROB ← LP
  Freezing level (m)	HZEROCL ← Z0, SNOWLMT ← ZS
  Convection (J/kg)	CAPE_MU ← CP, CIN_MU ← CN (from {YYYY}/analysis/{MM}/)

  Notes:

  • Use a label different from the INCA nowcast baseline (labelled INCA in the examples), otherwise both appear as "INCA" in the reports.
  • Precipitation rates (mm/h) are turned into hourly amounts: the rate stamped t is taken as the mean over (t − 10 min, t], and the hourly amount is the mean of the available rates in the hour. An hour without any run is NaN, as are the later totals of that forecast.
  • Missing files are filled with NaN and reported in the log.
  • Parameters INCA does not provide (e.g. PS, PMSL) are skipped.
  • Availability differs per variable (as of 2026-10): the main archive starts in May 2021 and runs to January 2026, the 10-min wind products start in May 2025. Most months miss a few runs.
  • INCA has no mixed-layer parcel: CAPE_ML/CIN_ML are not verified against it.
  • Meteograms need station truth; with a gridded truth they show a placeholder.
  • Scorecard rows for variables a run or baseline lacks are left empty.

Fields of the derive-diagnostic post-processor

The derive-diagnostic post-processor of anemoi-plugins-meteoswiss adds SP_10M, DD_10M, RELHUM_2M, TW_2M, HZEROCL, SNOWLMT, CAPE_ML, CIN_ML, CAPE_MU, CIN_MU, PTYPE and TOT_SNOW to the output of an ML run (see resources/inference/configs/sgm-temporal-downscaler-global_trimedge_multi.yaml). They are written as ICON-CH1 GRIB (templates_index_icon.yaml) and follow the ICON conventions: CIN is positive (missing without a level of free convection), PTYPE uses WMO code table 4.201, TOT_SNOW is accumulated from the start of the forecast in kg m-2, like TOT_PREC.

• ICON baselines lack some of them: DD_10M, RELHUM_2M, TW_2M (earthkit-meteo, as the post-processor) and TOT_SNOW (from SNOW_GSP) are derived from their components; a field present in the GRIB files is always used as is.
• Truth: INCA provides all but CAPE_ML/CIN_ML; a KENDA zarr has CAPE/CIN and the components of the derived fields, but no HZEROCL, SNOWLMT, PTYPE or TOT_SNOW. Parameters a truth source lacks are skipped with a warning.
• DD_10M is scored on the shortest angular error (e.g. 350° vs 10° is 20°).
• PTYPE only gets categorical scores, from eq thresholds on its codes (e.g. PTYPE: {eq: [0, 1, 3, 5, 7]}); its continuous scores are NaN, and it has no score maps.
• All participants are scored on the same points: where a forecast is missing at a point with valid truth, the scores of that region and time step are NaN. Fields that are undefined in places (CIN without a level of free convection, HZEROCL/SNOWLMT where ICON leaves them undefined) therefore get NaN scores, unless they are listed in score_on_common_points (see below).

Scoring on common points

Some fields are undefined in parts of the domain: for instance, ICON leaves HZEROCL/SNOWLMT undefined where the level is below the ground and CIN where there is no level of free convection. A single undefined point makes the scores of that region and time step NaN. List such fields to score them only where every participant has a value:

experiment:
  score_on_common_points:
    - HZEROCL
    - SNOWLMT
    - CIN_MU

For each reference time and lead time, the workflow builds a mask per participant (baselines and candidate runs) of the points where its forecast has a value, and keeps only the points valid in all masks. Every participant and the truth are then scored on these points only.

• The points change with each reference time and lead time; where every participant has a value, the scores are the same as without the setting.
• The scores of the listed fields depend on the set of participants: adding or removing a run or baseline changes them, and reruns the verification.
• Fields not listed are not affected. The listed fields must be in experiment.params.

Holdout stations

With station truth (jretrievedwh:), you can name stations to verify as a separate group, for example stations withheld from observation nudging, to get an independent check:

experiment:
  station_holdout: [CHM, FRE, RUE, BAS]   # station nat_abbr

Every model and baseline is then verified separately on three station groups: all, holdout (the listed stations) and holdin (all others). The dashboard gets a station-group selector; scorecards and the standard plots use all.

• Holdout stations missing from the truth are ignored; if none are present, or the truth is gridded (an anemoi dataset or INCA), only all is computed and a warning is logged.
• The list is written to a CSV file that the workflow tracks, so changing it reruns the affected verification and inference.
• An inference config can read the same list from holdout_stations.csv in its working directory (e.g. holdout_station_file: holdout_stations.csv for the nudge_toward_observation filter).

Installation

This project uses uv. Download and install it with

curl -LsSf https://astral.sh/uv/install.sh | sh

then, install the project and its dependencies with uv sync and activate the virtual environment with source .venv/bin/activate.

Credentials setup

Some experiments are stored on the ECMWF-hosted MLflow server: https://mlflow.ecmwf.int. To access these runs in the evaluation workflow, you need to authenticate using a valid token. Run the following commands once to log in and obtain a token:

uv pip install anemoi-training --no-deps
anemoi-training mlflow login --url https://mlflow.ecmwf.int

You will be prompted to paste a seed token obtained from https://mlflow.ecmwf.int/seed. After this step, your token is stored locally and used for subsequent runs. Tokens are valid for 30 days. Every training or evaluation run within this period automatically extends the token by another 30 days. It’s good practice to run the login command before executing the workflow to ensure your token is still valid.

DWH / jretrieve credentials

To use jretrievedwh: as a truth source, provide a client ID and secret for the MeteoSwiss service account. Set them as environment variables:

export JRETRIEVE_CLIENT_ID=your-client-id
export JRETRIEVE_CLIENT_SECRET=your-client-secret

or place them in a .env file at the project root (next to .jretrievedwh-conf.prod.py):

JRETRIEVE_CLIENT_ID=your-client-id
JRETRIEVE_CLIENT_SECRET=your-client-secret

The .env file is listed in .gitignore and is never committed.

Workspace setup

By default, data produced by the workflow will be stored under output/ in your working directory. We suggest that you set up a symlink to a directory on your scratch:

mkdir -p $SCRATCH/evalenv/output
ln -s $SCRATCH/evalenv/output output

This way data will be written to your scratch, but you will still be able to browse it with your IDE.

If you are using VSCode, we advise that you install the YAML extension, which will enable config validation, autocompletion, hovering support, and more.

Workflow development guidelines

This section defines the conventions to follow when adding or modifying rules, scripts, and outputs inside workflow/.

Rule names

Rules use the pattern {module}_{operation}[_{sub_operation}] in snake_case. The module prefix groups rules by their place in the pipeline:

Module	Rules prefix	Purpose
{module}.smk	data_	Input data preparation
{module}.smk	inference_	Model checkpoint retrieval, environment setup, and execution
{module}.smk	verification_	Metrics calculation, aggregation, and metric plots
{module}.smk	plot_	Forecast visualisation (frames, animations, meteograms)
{module}.smk	report_	Dashboard and HTML report generation

Aggregate target rules follow the pattern {module}_all (e.g. inference_all, verification_metrics_all). Top-level entry points are named after the workflow mode: experiment_all, showcase_all, sandbox_all.

Script names

Scripts live in workflow/scripts/ and mirror the rule that calls them:

{module}_{operation}.py         # standard Python scripts
{module}_{operation}.mo.py      # interactive Marimo notebooks

Examples: inference_prepare.py, verification_metrics.py, verification_aggregation.py, plot_forecast_frame.mo.py.

In some cases, a single script may be shared by more than one rule (e.g. inference_prepare.py is used by both inference_prepare_forecaster and inference_prepare_temporal_downscaler).

Log file paths

Every rule declares a log file under:

{OUT_ROOT}/logs/{rule_name}/{wildcards}.log

When a rule produces a sentinel file (.ok) to mark successful completion, it is placed alongside the log:

{OUT_ROOT}/logs/{rule_name}/{wildcards}.ok

Multiple wildcards are joined with a hyphen: {run_id}-{init_time}.log.

Output directory layout

All outputs are rooted at OUT_ROOT (from locations.output_root in the config):

{OUT_ROOT}/
├── data/
│   ├── runs/{env_id}/                        # per-environment artefacts (shared across config changes)
│   │   ├── inference-last.ckpt
│   │   ├── requirements.txt
│   │   ├── venv.squashfs
│   │   └── {config_hash}/                    # per-run-config artefacts
│   │       ├── summary.md
│   │       ├── verif_aggregated.nc
│   │       └── {init_time}/                  # per-initialisation-time artefacts
│   │           ├── config.yaml
│   │           ├── grib/
│   │           └── verif.nc
│   ├── baselines/{baseline_id}/
│   │   └── {init_time}/verif.nc
│   └── observations/
├── logs/                                      # one sub-directory per rule
└── results/{experiment_name}/               # final products
    ├── dashboard/
    ├── plots/
    └── scorecards/

Wildcard conventions

Wildcard	Format	Example
{env_id}	{type}-{model_id}-{env_hash}	forecaster-1a2b-c3d4
{run_id}	{env_id}/{config_hash}	forecaster-1a2b-c3d4/e5f6
{baseline_id}	baseline-{hash}	baseline-3f9a
{init_time}	%Y%m%d%H%M	202001011200
{experiment}	{YYYYMMDD}_{label}_{hash}	20260331_demo_a1b2
{param}	variable name	T_2M, TOT_PREC
{region}	geographic region slug	switzerland, globe
{leadtime}	zero-padded hours	000, 006, 024