TESTING.md states openly that only tests/runff is validated in CI. Current coverage:
| Suite |
In CI? |
Notes |
tests/runff |
✅ main.yml, macos.yml |
The only physics regression check |
tests/unit |
✅ (added in #156) |
Model-level, 22 cases |
tests/python/test_wheel.py |
✅ wheels.yml |
Import and smoke test only |
tests/runANN |
❌ |
Broken — see below |
tests/python/idealizedwind.py |
❌ |
Also the best undocumented example |
tests/python/farsite_flat.py |
❌ |
|
tests/python/percolation.py |
❌ |
|
tests/mnh_ideal, tests/mnh_real_nested |
❌ |
Need a Meso-NH build |
tests/run.bash |
❌ |
Aggregate runner, never invoked by CI |
tests/runANN fails immediately, and has since it was committed
run.bash line 1 does:
../../bin/ANN_test Rothermel.ffann modelrun.csv print | grep result > result.txt
if ! diff result.txt result.txt.ref; then
result.txt.ref is not in the repository. git ls-files tests/runANN/ returns five files and that is not one of them. Running it:
diff: result.txt.ref: No such file or directory
ANN results differs from reference.
It looks like result.txt was committed where result.txt.ref was meant — but a plain rename is not the fix. Running the test overwrites result.txt, and the regenerated output differs from the committed one in 13 of 1000+ lines, all last-digit float noise:
- result: 12412.1 observed 12412.1
+ result: 12412.2 observed 12412.1
So the check needs a numerical tolerance the way compare_nc.py already has, not an exact diff. As written it would be permanently red on any machine but the one that produced the file.
This matters more than it looks: runANN is the only coverage for ANNPropagationModel and BMapLoggerForANNTraining, which the unit suite in #156 cannot reach — both read a .ffann network in their constructor and abort when it is missing. Those two models are also the inverse-modelling path, one of the more distinctive things in the codebase.
On test_threading.py
It does not exist on dev or master — it arrives with the free-threading branch. So the free-threading work currently has no automated guard, and wiring it up should be part of landing that branch rather than a separate task.
Suggested approach
.github/workflows/invariants.yml (added in #158) is a good template — it builds pyforefire in a venv and runs a Python suite with no LFS fixtures.
In order of value: fix runANN and wire it; then the tests/python/ examples, which double as documentation and so are worth keeping executable; then decide whether mnh_* is testable at all without a Meso-NH build.
Drafted by Claude Opus 5 from a codebase audit. Reviewed by a maintainer before filing.
TESTING.mdstates openly that onlytests/runffis validated in CI. Current coverage:tests/runffmain.yml,macos.ymltests/unittests/python/test_wheel.pywheels.ymltests/runANNtests/python/idealizedwind.pytests/python/farsite_flat.pytests/python/percolation.pytests/mnh_ideal,tests/mnh_real_nestedtests/run.bashtests/runANNfails immediately, and has since it was committedrun.bashline 1 does:result.txt.refis not in the repository.git ls-files tests/runANN/returns five files and that is not one of them. Running it:It looks like
result.txtwas committed whereresult.txt.refwas meant — but a plain rename is not the fix. Running the test overwritesresult.txt, and the regenerated output differs from the committed one in 13 of 1000+ lines, all last-digit float noise:So the check needs a numerical tolerance the way
compare_nc.pyalready has, not an exactdiff. As written it would be permanently red on any machine but the one that produced the file.This matters more than it looks:
runANNis the only coverage forANNPropagationModelandBMapLoggerForANNTraining, which the unit suite in #156 cannot reach — both read a.ffannnetwork in their constructor and abort when it is missing. Those two models are also the inverse-modelling path, one of the more distinctive things in the codebase.On
test_threading.pyIt does not exist on
devormaster— it arrives with the free-threading branch. So the free-threading work currently has no automated guard, and wiring it up should be part of landing that branch rather than a separate task.Suggested approach
.github/workflows/invariants.yml(added in #158) is a good template — it buildspyforefirein a venv and runs a Python suite with no LFS fixtures.In order of value: fix
runANNand wire it; then thetests/python/examples, which double as documentation and so are worth keeping executable; then decide whethermnh_*is testable at all without a Meso-NH build.Drafted by Claude Opus 5 from a codebase audit. Reviewed by a maintainer before filing.