Move every checker message into a catalog, and clean up the review output

[arav-agarwal2]

The review output for a failing submission looked off: one warning repeated ten times against the wrong file, the rule ID printed twice in every annotation, three notations for spec sections, no guidance on how to fix anything, and wording that drifted because every message was an inline f-string next to its check. This PR moves all wording into a data file and cleans up the output on top of it.

The catalog

src/submission_checker/data/messages.yaml holds, per rule, a title, the spec section, and each message as a template with an optional fix:

accuracy-coverage:
  title: Accuracy at the mandatory points
  spec: §5.3
  messages:
    missing-bands:
      text: 'No accuracy results at a point in: {bands}'
      fix: 'Run accuracy at one point in each of: {bands}.'

• In code, a call names a rule and a key and passes the values: err("accuracy-coverage", "missing-bands", path, bands=...). The ok/warn/err helpers render through submission_checker/messages.py. CheckResult gains key, title and fix; the JSON report only gains fields.
• Severity stays in code, beside the rule. It's §9.1's failure action and decides the verdict, so no wording change can make a failing submission pass.
• Ships with the package; changes only with a release. There's deliberately no --messages flag or environment-variable override.
• Templates are str.format restricted to plain names. {name!r} and {x:.4f} work; {a.b} and {a[0]} are rejected when the catalog loads.
• _shared holds the messages several rules emit: file not found, parse errors, Pydantic field errors (now runtime_settings.min_duration_ms, not a tuple).
• fragment(rule, key, ...) renders sub-problems for rules that collect several before reporting (power descriptor, nodes used, steady state), so those words live in the catalog too.

Migration

All 196 result sites moved over: 167 by an AST converter, the rest by hand, plus 35 sub-problem strings. Existing wording is kept, so wording changes can be reviewed on their own in the YAML. Of 1214 existing tests, one needed updating, because band names now read "High Concurrency".

Keeping code and catalog together

tests/submission_checker/test_messages.py reads the checker's source and asserts:

• every key used exists;
• every call passes exactly the placeholders its template (and fix) needs;
• no catalog message is unused;
• every template renders;
• every rule has a title and a § section.

tests/conftest.py runs the suite with the catalog in strict mode, so a message that can't render fails the test that produced it. Outside tests it degrades to a generic message rather than stopping a check.

Review output

Before → after, on test_submissions/sub_g:

	Before	After
Annotation	title=submission-checker: model-name-valid::[model-name-valid] … (#3.2)	title=Benchmark model name (§3.2)::… plus Fix: …
Accuracy gate	10 identical warnings, all blaming r64/results.json	1 per curve, on the model directory
Table / summary columns	Rule, § Ref, Severity, Message, Path	Check (title + rule ID), Severity, Message + Fix, Path
Repeated findings	one row each	one row, (×N), +N more paths
Section references	#5.4, #3–6, inline §5.3	§5.4, §§3–6

Fix hints are written for the 29 failures submitters hit most: directory layout, missing files, model name, region and accuracy coverage, tps_utilization, shared paths, system-description consistency, the Offline point. The rest can be added in the YAML.

Checks

Full suite: 1466 passed (non-integration). ruff and mypy are clean. A wheel built from this branch includes data/messages.yaml, and the installed CLI renders titles for every result.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅