Skip to content

Feat/add tool compare inference - #248

Open
lclanzi wants to merge 4 commits into
testing-hackathonfrom
feat/add-tool-compare-inference
Open

Feat/add tool compare inference#248
lclanzi wants to merge 4 commits into
testing-hackathonfrom
feat/add-tool-compare-inference

Conversation

@lclanzi

@lclanzi lclanzi commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Adds a Marimo interactive notebook to compare raw GRIB outputs from two evalml inference runs field-by-field.

The tool reads GRIB messages using eccodes, compares corresponding fields between two runs, and presents a pandas result table showing:

  • Maximum absolute difference
  • Mean absolute difference
  • Pass/fail status based on configurable tolerance thresholds

Usage workflow

  1. Run evalml and generate an inference output.
  2. Rename the output folder to preserve the first run.
  3. Run evalml again and generate a new inference output.
  4. Launch the comparison notebook:
uv run marimo run tools/compare_grib.mo.py
  1. Provide path of two evalml runs. Examples:
    PathA = /scratch/mch/llanzila/test/evalml/output_ground_truth/data/runs/temporal_downscaler-sruc-a157-on-forecaster-sruc-94fd/21c5/202501010000/grib
    PathB = /scratch/mch/llanzila/test/evalml/output/data/runs/temporal_downscaler-sruc-a157-on-forecaster-sruc-94fd/21c5/202501010000/grib

An example of the output is shown below

Screenshot 2026-08-04 110023

@jonasbhend jonasbhend left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool addition and very nice summaries of the differences.

Would it make sense to isolate the logic for equivalence testing of grib files into a module for use in testing?

@lclanzi

lclanzi commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

I think the challenging part is that this type of testing always requires two evalml outputs, one serving as the reference (ground truth/baseline) and one generated with the current version. I'm not sure how maintainable that would be in the long term, and it is also not straight forward to implement imo.

That's why I added it as a standalone tool rather than integrating it into the test suite, but I'm happy to discuss other approaches

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants