Skip to content

Wire the unwired test suites into CI, and fix runANN #163

Description

@HugoFara

TESTING.md states openly that only tests/runff is validated in CI. Current coverage:

Suite In CI? Notes
tests/runff main.yml, macos.yml The only physics regression check
tests/unit ✅ (added in #156) Model-level, 22 cases
tests/python/test_wheel.py wheels.yml Import and smoke test only
tests/runANN Broken — see below
tests/python/idealizedwind.py Also the best undocumented example
tests/python/farsite_flat.py
tests/python/percolation.py
tests/mnh_ideal, tests/mnh_real_nested Need a Meso-NH build
tests/run.bash Aggregate runner, never invoked by CI

tests/runANN fails immediately, and has since it was committed

run.bash line 1 does:

../../bin/ANN_test Rothermel.ffann modelrun.csv print | grep result > result.txt
if ! diff result.txt result.txt.ref; then

result.txt.ref is not in the repository. git ls-files tests/runANN/ returns five files and that is not one of them. Running it:

diff: result.txt.ref: No such file or directory
ANN results differs from reference.

It looks like result.txt was committed where result.txt.ref was meant — but a plain rename is not the fix. Running the test overwrites result.txt, and the regenerated output differs from the committed one in 13 of 1000+ lines, all last-digit float noise:

- result: 12412.1 observed 12412.1
+ result: 12412.2 observed 12412.1

So the check needs a numerical tolerance the way compare_nc.py already has, not an exact diff. As written it would be permanently red on any machine but the one that produced the file.

This matters more than it looks: runANN is the only coverage for ANNPropagationModel and BMapLoggerForANNTraining, which the unit suite in #156 cannot reach — both read a .ffann network in their constructor and abort when it is missing. Those two models are also the inverse-modelling path, one of the more distinctive things in the codebase.

On test_threading.py

It does not exist on dev or master — it arrives with the free-threading branch. So the free-threading work currently has no automated guard, and wiring it up should be part of landing that branch rather than a separate task.

Suggested approach

.github/workflows/invariants.yml (added in #158) is a good template — it builds pyforefire in a venv and runs a Python suite with no LFS fixtures.

In order of value: fix runANN and wire it; then the tests/python/ examples, which double as documentation and so are worth keeping executable; then decide whether mnh_* is testable at all without a Meso-NH build.


Drafted by Claude Opus 5 from a codebase audit. Reviewed by a maintainer before filing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions