Improve CI - #226
Open
clairemerker wants to merge 40 commits into
Open
Conversation
Contributor
|
cscs-ci run |
Contributor
|
cscs-ci run |
Co-authored-by: Mary McGlohon <Mary.McGlohon@meteoswiss.ch>
Adds an integration test that runs the full evalml experiment pipeline and asserts that key verification metrics match reference values from a known-good run. --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch> Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln003.cscs.ch>
This PR implements minor changes to the existing meteogram integration test. ### Summary of changes * rename test and config to meteogram_small * change date to 2024-08-01 (official start of ICON archive) * additional parameters to cover all use cases (derived, aggregated and instantaneous)
Contributor
|
cscs-ci run |
Contributor
|
cscs-ci run |
2 similar comments
Contributor
|
cscs-ci run |
Contributor
|
cscs-ci run |
Under the service account, uv needs to be installed for every CI run since there is no global installation. --------- Co-authored-by: Mary McGlohon <Mary.McGlohon@meteoswiss.ch> Co-authored-by: clairemerker <34312518+clairemerker@users.noreply.github.com>
Two different pipelines: - longtest: intergration tests - heavytest: heavy integration tests (full end to end) The idea is to trigger heavytest weekly or nighly using the CI cron, and longtest in PRs using the comment call.
Fix truth in config test move configs to test folder --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>
Contributor
|
cscs-ci run heavytest |
… testing-hackathon
Contributor
|
cscs-ci run heavytest |
Contributor
|
cscs-ci run heavytest |
Adds test coverage diagnostics - to the unit tests - to the integration tests with subprocess coverage
Contributor
Author
|
cscs-ci run longtest |
expose logs from evalml from pipeline to balfrin --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>
Contributor
|
cscs-ci run longtest |
Contributor
|
cscs-ci run heavytest |
AI assisted implementation! Adds an opt-in mechanism to replay pre-computed inference output instead of running inference, so the integration tests can run without needing inference. Truth and baselines still come from the DWH and /store_new: only inference is frozen. Adds a new top-level config key fixture_root (absent by default → production unchanged): - unset: inference_execute runs anemoi-inference exactly as previously - set: inference_execute symlinks frozen GRIB from <fixture_root>/data/runs/<run_id>/<init_time>/grib into the run workdir and marks the step done
Adds testing config for scoremaps. --------- Co-authored-by: clairemerker <34312518+clairemerker@users.noreply.github.com>
Contributor
Author
|
cscs-ci run longtest |
Contributor
|
cscs-ci run heavytest |
Contributor
|
cscs-ci run longtest |
- New tests/integration/test_dashboard_small.py + dashboard_small.yaml, exercising the dashboard pipeline against a frozen inference fixture - Move fixture_root for dashboard/meteogram small tests to a shared team location under /store_new - small cleanup to tools/summarize_logs.py to make benign error-checking less bulky
clairemerker
commented
Jul 30, 2026
| @@ -0,0 +1,133 @@ | |||
| #!/usr/bin/env python3 | |||
Contributor
Author
There was a problem hiding this comment.
Should we move this to the ci folder?
There was a problem hiding this comment.
It depends, if you only want to use it for ci probably yes. If it might be useful for users and developers, here might be good.
The question is if we want to reuse it for all snakemake projects, then it be cool if was in its own repo or kind of a utility repo that then all snakemake repos can use.
| - | | ||
| banner "Install uv" | ||
| export UV_CACHE_DIR="$SCRATCH/.cache/uv" # wheel cache, persistent across pipelines | ||
| # Set eckit ECKIT_GEO_CACHE_PATH for grid downloads |
Contributor
Author
There was a problem hiding this comment.
This is not really related to uv, rename the script or put somewhere else?
| @@ -1,4 +1,4 @@ | |||
| lead_time: 120h | |||
| lead_time: 12h | |||
Contributor
Author
There was a problem hiding this comment.
Revert, has no effect on testing anyways I think.
Suggested change
| lead_time: 12h | |
| lead_time: 120h |
|
|
||
| # # yaml-language-server: $schema=../../../workflow/tools/config.schema.json | ||
| # description: | | ||
| # Evaluate skill of ICON-CH1 single. |
Contributor
Author
There was a problem hiding this comment.
Suggested change
| # Evaluate skill of ICON-CH1 single. | |
| # Evaluate skill of Varda-single hierarchical 2 level mesh. |
|
|
||
| # # yaml-language-server: $schema=../../../workflow/tools/config.schema.json | ||
| # description: | | ||
| # Evaluate skill of ICON-CH1 single. |
Contributor
Author
There was a problem hiding this comment.
Suggested change
| # Evaluate skill of ICON-CH1 single. | |
| # Evaluate skill of Varda-single forecaster. |
| # yaml-language-server: $schema=../workflow/tools/config.schema.json | ||
| # yaml-language-server: $schema=../../../workflow/tools/config.schema.json | ||
| description: | | ||
| Evaluate skill of ICON-CH1 single. |
Contributor
Author
There was a problem hiding this comment.
Suggested change
| Evaluate skill of ICON-CH1 single. | |
| Evaluate skill of Varda-single forecaster. |
huppd
reviewed
Jul 30, 2026
| export PATH="$PWD/.local/bin:$PATH" | ||
| # uv: bare metal assumes a user-managed env so evalml ships no installer | ||
| export UV_INSTALL_DIR="$PWD/.local/bin" | ||
| curl -LsSf https://astral.sh/uv/install.sh | env INSTALLER_NO_MODIFY_PATH=1 sh |
There was a problem hiding this comment.
I would suggest to pin the uv version:
Suggested change
| curl -LsSf https://astral.sh/uv/install.sh | env INSTALLER_NO_MODIFY_PATH=1 sh | |
| UV_VERSION='0.11.32' | |
| curl -LsSf "https://astral.sh/uv/${UV_VERSION}/install.sh" | env INSTALLER_NO_MODIFY_PATH=1 sh |
huppd
reviewed
Jul 30, 2026
huppd
left a comment
huppd
left a comment
There was a problem hiding this comment.
I haven't looked into it in greate detail. But I like the split of the ci yamls, and the nice READMEs.
There was a problem hiding this comment.
I have a slight feeling that this and the .cfg could be also in the tests folder. Maybe something to think about. I am not sure which is the better place.
| @@ -0,0 +1,133 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
It depends, if you only want to use it for ci probably yes. If it might be useful for users and developers, here might be good.
The question is if we want to reuse it for all snakemake projects, then it be cool if was in its own repo or kind of a utility repo that then all snakemake repos can use.
| DEST = ROOT / "tests" / "log" | ||
| DEST.mkdir(exist_ok=True) | ||
|
|
||
| ERROR_KEYWORDS = {"error", "exception", "traceback", "failed", "oom", "killed"} |
There was a problem hiding this comment.
A friend told me we might want to add
Suggested change
| ERROR_KEYWORDS = {"error", "exception", "traceback", "failed", "oom", "killed"} | |
| ERROR_KEYWORDS = {"error", "exception", "traceback", "failed", "oom", "killed", "cancel"} |
When heavytests fail, produce some stats. See below for example: varda-single-1.0.yaml: 2898/3326 pass, 428/3326 fail (13%) | tol=1e-02 | diff range [0.00e+00 – 3.07e+00], median 4.69e-07 | failed vars: PMSL(72), SP_10M(140), TD_2M(48), TOT_PREC1(8), TOT_PREC6(14), T_2M(146) Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>
Contributor
|
cscs-ci run heavytest |
- test running evalml --mec and asserting verSYNOP_*.nc files are produced (inference frozen) - resources/mec/namelist.jinja2: switch prefix_in to ekf - workflow/rules/verif_obs.smk: guard FFV2 rules under if config[ffv2] - workflow/Snakefile: guard ffv2_all under mec+ffv2 both non-None --------- Co-authored-by: clairemerker <34312518+clairemerker@users.noreply.github.com>
# Enable forecasters-ich1 in the heavytest integration suite Adds `forecasters-ich1.yaml` to the heavytest `CONFIGS` list together with its generated expected-metrics file, extends the metric check to handle configs that produce more than one forecaster run, replaces the commented-out generator with a `--regenerate-expected` flag, and fixes a CI-only failure in the plotting stage that made this config unrunnable. ## Changes | File | Diff | Notes | | --- | --- | --- | | `tests/integration/expected/forecasters-ich1.yaml` | +7466 | Generated expected file — 72 entries × ~100 metrics across every region/season/init_hour combination | | `tests/integration/test_configs.py` | +206 / −87 | Multi-source expected format, `_collect_failures` helper, expected-file generation | | `tests/integration/configs/forecasters-ich1.yaml` | +122 / −21 | The config being enabled | | `tests/conftest.py` | +108 | `--regenerate-expected` option, per-config selection and validation | | `tests/integration/README.md` | +57 | How to regenerate an expected file | | `workflow/scripts/prefetch_natural_earth.py` | +72 | New — Natural Earth cache warm-up | | `workflow/rules/plot.smk` | +17 | Wires the warm-up ahead of the plot fan-out | 93% of the diff is the generated expected file; the reviewable surface is ~690 lines. ## Multi-source metric checking A config with several forecasters writes several `verif_aggregated_*.nc` files, and the previous check only looked at `nc_files[0]` and its first non-truth source — so additional runs were silently unverified. Expected entries are now keyed by source hash-prefix (e.g. `forecaster-b30a-4d02`), and every non-truth source found across all output files is compared. `assert checked_sources == set(expected)` guards against a vacuous pass: if a run's outputs are missing a source that the expected file expects, the test fails rather than quietly checking less. Mismatches are collected rather than raised on first hit, so a failure reports full pass/fail counts, diff range and median, and per-variable failure counts instead of a single value. The `isinstance(expected, list)` branch is deliberate back-compat for `varda-single-1.0.yaml`, whose expected file is still in the older flat-list format. It can be deleted once that file is regenerated, which needs operational DWH access for varda's truth. ## Regenerating an expected file The generator was a commented-out block run as `python tests/integration/test_configs.py`. Being dead code it had drifted: it looped over `CONFIGS` but globbed all of `output/data/runs/**` inside the loop, so every config would have been written the same content, and it called `math.isfinite()` without importing `math`. It is now live code behind a flag that names one config exactly: ``` pytest tests/integration/test_configs.py -m heavytest --regenerate-expected=varda-single-1.0.yaml ``` Repeat the option for several configs, or pass `=all`. Comparison and regeneration read values through the same `_metric_value()` helper, so an expected file cannot be generated with a different selection than the one asserted against — the duplication that let the old block rot. Three sharp edges, all aborting before the experiment starts: a bare `--regenerate-expected` is rejected, since regenerating every config costs hours of GPU each and must be spelled `=all`; a partial name is rejected with the valid names listed, because `-k varda-single` would also match a future `varda-single-2.0.yaml`; and selecting no metric test is rejected, since forgetting `-m heavytest` would otherwise report a green "N passed" having regenerated nothing. Only the `.nc` files the run rewrote are used, tracked by snapshotting mtimes before the experiment — `output/data/runs/` is shared across configs, so a plain glob would write one config's runs into another's expected file. ## Prerequisite: Natural Earth cache warm-up Enabling this config is what first ran 1620 scoremap plot jobs in parallel, which exposed a pre-existing race. The plots draw coastlines via earthkit-plots → cartopy, which downloads Natural Earth shapefiles into a cache under `$HOME` on first use. `ci/cscs.yml` sets `HOME="$PWD/home"`, recreated fresh every pipeline, so that cache is always cold in CI: the plot jobs all start at once, race to download the same files, and one reads a half-written shapefile and dies with `struct.error: unpack requires a buffer of N bytes`, aborting the workflow at ~5% completion. The new `plot_prefetch_natural_earth` localrule downloads and fully reads the three 50m datasets the plots use, once and serially, and both scoremap rules take its marker as an input. Local runs reuse a persistent `$HOME` and so were never affected, which is why this only ever surfaced in CI. ## Test results `forecasters-ich1.yaml` **passes** — the workflow completes all 1725 steps and every metric is within tolerance. `varda-single-1.0.yaml` **fails** on 2 of 3326 metrics - expected and irrelevant for this PR. `--regenerate-expected` has not been run end to end: its generation logic reproduces the committed `forecasters-ich1.yaml` bit-for-bit from that run's `.nc` files and every guard above is verified, but no expected file in this PR was produced by the flag itself.
Louis-Frey
added a commit
that referenced
this pull request
Aug 10, 2026
Adds a baseline-only integration config (ICON-CH2-CTRL against the KENDA-CH1 zarr, two inits, TOT_PREC6 at +6/+12 h) and a longtest driving `evalml experiment` over it, exercising the full SAL chain end to end without a GPU: schema, SalConfig validators, experiment_all target expansion, verification_sal_baseline, verification_sal.py and verification.sal. The test needs no blessed reference files. It leans on internal redundancy instead: A is recomputed from the row's own mean columns (exact), and truth_mean is cross-checked across the two output files that reach the same accumulation window via different (reftime, leadtime) pairs -- with rtol 1e-9, since agreement is close but not bit-exact given the zarr accumulates chunk-wise -- plus component bounds and an all-NaN guard. S and L are deliberately not pinned: they pass through a discrete object segmentation, so a tight tolerance would be flaky and a loose one would catch little. The config writes to a dedicated output_root so the exactly-one-CSV assertion cannot trip over an unrelated experiment sharing output/. The two unit tests cover the script's pre-flight guards (lead time below the accumulation period, non-zarr truth); both fire before any I/O, so they run on GitHub Actions. No CI changes: the CSCS longtest pipeline runs `pytest tests/integration -m longtest` over the whole directory, so this activates when that job lands with #226.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Capabilities of evalml that need testing
Proposed solution
We disentangle testing of inference in evalml and testing of evalml core functionality (e.g. experiment reports, meteograms, animations). To do so, we generate reference inference data that we can flexilbly reuse for testing. Heavy end-to-end tests still run the whole workflow including inference.
0. unit tests
Unit tests are run using Github Actions on every push to PRs to main.
Trigger locally with
pytest1. shorter integration tests (to be run with every PR) - longtest
Those tests can be triggered on a PR with the comment
cscs-ci run longtestor locally usingci/run-integration-coverage.sh longtest tests/integration. They need to be run before merging a PR.Here we use baseline data or frozen inference data only, as this is readily available and lengthy inference can be avoided. We aim to set up a minimal configs that test real-world cases.
test_meteogram_small.pywith Varda, ICON-CH1-CTRL and ICON-CH2-CTRL, 2024-08-01T00:00, SwissMetNet truth, T_2M/SP_10M/TOT_PREC6, GVE/SAEtest_animation_small.pywith ICON-CH2-CTRL (restrict lead times to only 2)test_dashboard_small.pywith ICON-CH1-CTRL and INCAtest_scoremaps_small.pywith ICON-CH2-CTRL, 2025-04-01T00:00 - 2025-04-01T06:00, KENDA-CH1 truth, T_2M/TOT_PREC6, bias, Switzerlandtest_mec_small.py: Varda forecaster, 2025-08-01T12:00 - 2025-08-02T00:00, mec (without ffv2)2. long integration tests (to be run weekly) - heavytest
Those tests can be triggered on a PR with the comment
cscs-ci run heavytestor locally usingci/run-integration-coverage.sh heavytest tests/integration. They are run nighly on this branch currently using the CI cron.test_configs.pyto test inference and model verificationforecasters-ich1.yaml: forecaster evaluationexperiment, 2025-03-01T00:00 - 2025-03-02T00:00, baseline ICON-CH2-EPS, KENDA-CH1 truth, multiple params, regions, thresholds, dashboard, currently no scoremapsaifs-single.yaml:forecasters-ich1-oper-fixed.yaml:forecasters-ich1-oper.yaml:forecasters-ich1_mec_ffv2.yaml:varda-single-1.0.yaml: forecaster and temporal downscalerexperiment, 2025-01-01T00:00 / 2025-04-01T00:00 / 2025-07-01T00:00 / 2025-10-01T00:00, baselines INCA / ICON-CH2-CTRL / ICON-CH1-CTRL, SwissMetNet truth, scorecards, multiple params, regions, thresholdsTodo and missing tests