Skip to content

[Backend 11] test(training): a conformance kit for step runners, with its mutations - #1214

Draft
maocheng23 wants to merge 5 commits into
maocheng/backend-10-boundariesfrom
maocheng/backend-11-conformance
Draft

maocheng23 wants to merge 5 commits into
maocheng/backend-10-boundariesfrom
maocheng/backend-11-conformance

Conversation

@maocheng23

@maocheng23 maocheng23 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Next step of the training-backend design (#1177), stacked on the import-scan PR. The stack so far was held to a recorder that compares two revisions of the same runner. A second runner needs more: a statement of what it has to reproduce that does not depend on which recipes happen to exist, a check of what a recording cannot show (the order of commits and checkpoints), and proof that these checks can fail. This is the design's Conformance section and delivery step 5.

Change

tests/test_runtime/conformance.py (a kit, not a test module) and tests/test_runtime/test_conformance.py.

Part What it is
declaration_cases() one recorded case per thing a program declares to a runner: reduction: window (DFlash), reduction: microbatch (DSpark), statistic: replay (DSpine), evaluation diagnostics (EAGLE3, evaluated). Two ranks, a window of two. A protocol in the registry that declares something without a case fails DeclarationTest, so a new reduction or statistic needs its case before a runner can claim it
MUTATIONS three broken references, trained through a new around_training= hook of the recorder that every rank enters around its run: gradients scaled by the world size, clipping skipped, the learning-rate schedule shifted by one step
ProtocolChecks a mixin: the real run loop over a runner, a stream that is read once and a ledger of what was read, handed to the runner, applied, committed, snapshotted and checkpointed. The window spans the epoch boundary (three batches an epoch, two epochs, a window of two: three updates); the runner is handed the stream's batches, each once and in order; an update consumes one whole window; a commit names every sample its step's update consumed and follows it; nothing is committed for an incomplete window or after the run raised; a checkpoint's state is collected, and the checkpoint written, with every update committed and no acknowledgement in flight; a complete run returns. protocol_violations(ledger) is the rule as a function
RunnerChecks a mixin: one job per declaration in one process, compared with the reference runner bit for bit: every update's metrics, gradient norm, learning rate and weights, and the exported draft, the optimizer state and the generator at the end. KitProgram weighs micro-batches differently; its loss reads the step and the noise it draws; it reports one ratio only when detailed metrics are asked for; under microbatch it returns terms and a numerator without a graph; under replay it forms its forward's argument itself, outside the forward, from pooled sums that depend on the prepass's mode and generator. One job runs in bfloat16, where the optimizer's fp32 masters hold more than the weights. Plus the single-use handle, a batch without a declared input, evaluation (the declaration decides the diagnostics, weights and generator are left alone), and resume in bfloat16 through two further updates

A runner is checked in one process by two test classes that set make_runner; the reference runner's are in this PR. The recorded cases train what specforge train assembles, which is the reference: no configuration key selects another runner, so a candidate is recorded by substituting it around every rank's run (record(cases, around_training=...)).

Evidence

  • Every mutation changes every declaration case. More specifically: the world-size scaling shows in the first update's reported gradient norm and the shifted schedule in the first applied learning rate.
  • The kit's own checks fail when they should (KitCanFailTest): a runner that skips the window's rescale fails the window job and passes the microbatch one (whose rescale is by one); a runner that ignores the terms cannot train the microbatch job; a replay on another generator state, and a prepass run in evaluation mode, fail the replay job; a runner that passes step zero to every forward, or never asks for detailed metrics, fails the window job; one that rounds its fp32 masters to the weights fails the bfloat16 job and passes the fp32 one; a resume that loses the masters, or only the schedule's position, fails the resume job; a run loop that does not wait for commits before a checkpoint fails the protocol; nine hand-written ledgers that break the commit contract are each reported with the reason.
  • The kit runs in 17 seconds on this machine, four recordings of four two-rank cases included. On a host that cannot place two ranks the recordings skip.
  • Full CPU suite: 2,032 passed; the failure set (15 failed, 1 error, all environmental on this machine) is identical to main's.

Not in the kit

  • Sequence parallelism: the design's first version includes an EAGLE3 case with data-parallel and sequence-parallel sizes above one. It needs flash_attn and GPUs and is not recorded here.
  • Commit order across ranks: the ledger is one process. tests/test_runtime/test_async_durable_ack.py has the two-rank acknowledgement tests; they are not parameterized by runner.
  • A prepass in the protocol's ledger: the protocol checks use a program without a replay statistic.
  • Evaluation across ranks with unequal denominators or an empty shard: the one evaluated case is EAGLE3 with two records on each rank.
  • An accelerator: the runner jobs build both runners on the host, because the reference's generator there is the host's.
  • Frozen modules and partition blocks: only a sharding runner acts on them, and the kit's program declares none.
  • A candidate runner of several ranks: the mixins hold it in one process; recording it needs a substitution for its topology, which only the one-rank runner of the next PR has.
  • The run record written last: arrives with the checkpoint-layout change (delivery step 8).
  • Stored trajectories: recordings are compared on one machine and torch build, so none is checked in.

After Codex review

One read-only Codex round (maocheng/backend-10-boundaries...HEAD) asked of each check whether a wrong runner could pass it. Several could, and the kit was tightened in a follow-up commit:

  • P1 the microbatch job passed a runner that ignored the terms (on one rank the program's own numerator was the same loss); the ledger approved a commit for a batch the update never consumed; resume passed with lost fp32 masters or a restarted schedule; the mutation recordings failed in setUpClass on a host with one accelerator.
  • P1, wording the first description read as if a new runner's recordings were covered by setting make_runner. They are not, and the text above now says what is.
  • P2 evaluation checked mode and state only; an asynchronous acknowledgement could win the race and hide a missing flush; the validator ignored how a run ended; jobs compared loss, norm and weights only; the replay job could not tell sums from means; the clipping assertion required a first-step difference clipping does not guarantee; the program allocated on the host.

Each of these has a test now that fails when the property is broken (see Evidence).

Second Codex round

A second read-only round asked again, of the tightened kit, whether a wrong runner or run loop could pass. Closed in a further commit, each with a probe in KitCanFailTest or a hand-written ledger that fails when the property is broken:

  • P1 the ledger did not compare what the runner was handed with what the stream read, and counted a window in samples; a complete run could end in an exception and pass; a runner could bypass replay_arguments, or run its prepass in evaluation mode, and match; no job compared a bfloat16 run with the reference, so masters rounded to the weights passed.
  • P2 the flush check now looks at the moment a checkpoint's state is collected and at acknowledgements in flight; the resume probe loses only the schedule's position; jobs compare the gradient norm and the exported draft, read the step and the detail flag; the mutation recordings tell a missing draft from a host without two ranks and count a difference only beyond an accelerator's own repeatability; the refused batch is checked by its exception type.

Not covered, and listed above under "Not in the kit": a prepass in the ledger, cross-rank evaluation cases, and an accelerator.

🤖 Generated with Claude Code

maocheng23 and others added 5 commits November 18, 2026 14:36
tests/test_runtime/conformance.py is what a step runner is held to, after
the training-backend design:

- declaration_cases(): one recorded case per declaration a program makes
  to a runner (each reduction, a replay statistic, evaluation
  diagnostics), not per recipe. A registry entry that declares something
  without a case fails.
- MUTATIONS: gradients scaled by the world size, clipping skipped, the
  learning-rate schedule shifted by one step. The recorder trains each
  through a hook around every rank's run; every one changes every case.
- ProtocolChecks: the run loop over a runner with a consume-once stream
  and a ledger: the window spans the epoch boundary, samples are
  committed after their update and in order, an incomplete window is not
  committed, commits are flushed before a checkpoint.
- RunnerChecks: one job per declaration in one process against the
  reference runner's losses, gradient norms and weights, bit for bit,
  plus the single-use handle, evaluation and resume.

The checks are shown to fail: a runner without the window's rescale, a
replay on another generator state, and ledgers that break the commit
contract.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review findings on the kit:

- The microbatch job passed a runner that ignored the terms: on one rank
  the program's own numerator was the same loss. The kit program's
  numerator carries no graph there, so such a runner cannot train it.
- The ledger records what the runner was handed; a commit has to name
  exactly what the update of its step consumed, nothing is committed
  after the run raised, and a run that ends cleanly committed every
  update. An asynchronous acknowledgement takes a moment, so a checkpoint
  that does not wait for it is seen.
- Resume runs in bfloat16 and through two further updates: lost fp32
  masters and a schedule that starts over both fail it.
- Jobs compare every metric, the learning rate, the weights and the
  optimizer state bitwise through the recorder's comparison, check that
  the evaluation declaration reaches the program and that a batch without
  a declared input is refused; the replay statistic is used in a way
  that tells sums from means.
- The mutation recordings skip on a host that cannot place two ranks; the
  assertion on clipping's first update, which clipping does not
  guarantee, is dropped. The program allocates on its weights' device.
- DESIGN.md says what the recorded cases train (the reference) and what
  the kit does not cover.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Second review round on the conformance kit:

- The ledger compares what the runner was handed with what the stream
  read, batch by batch with all of a batch's samples, so a sample that
  was read and never trained on cannot be committed; a window is counted
  in micro-batches and a commit names every sample of it. A complete run
  has to return, and a checkpoint's state has to be collected with every
  update committed and no acknowledgement in flight.
- The kit program forms its replay argument itself, outside the forward,
  from sums that depend on the prepass's mode and generator; its loss
  reads the step and it reports one ratio only when detailed metrics are
  asked for. A runner that bypasses replay_arguments, runs a prepass in
  evaluation mode, or drops the schedule or the flag now differs from the
  reference.
- One job runs in bfloat16 against the reference, so a runner that rounds
  its fp32 masters to the weights fails; jobs also compare the gradient
  norm and the exported draft. The two arms of every job seed the
  generator after the runner is built.
- The probes of resume lose the masters or only the schedule's position.
- The mutation recordings tell a case the recorder has no draft for from
  a host without two ranks, and count a difference only beyond what two
  accelerator runs of one revision differ by.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant