Repository navigation
[Backend 11] test(training): a conformance kit for step runners, with its mutations - #1214
Draft
maocheng23 wants to merge 5 commits into
Draft
maocheng23 wants to merge 5 commits into
maocheng23 wants to merge 5 commits into
Conversation
tests/test_runtime/conformance.py is what a step runner is held to, after the training-backend design: - declaration_cases(): one recorded case per declaration a program makes to a runner (each reduction, a replay statistic, evaluation diagnostics), not per recipe. A registry entry that declares something without a case fails. - MUTATIONS: gradients scaled by the world size, clipping skipped, the learning-rate schedule shifted by one step. The recorder trains each through a hook around every rank's run; every one changes every case. - ProtocolChecks: the run loop over a runner with a consume-once stream and a ledger: the window spans the epoch boundary, samples are committed after their update and in order, an incomplete window is not committed, commits are flushed before a checkpoint. - RunnerChecks: one job per declaration in one process against the reference runner's losses, gradient norms and weights, bit for bit, plus the single-use handle, evaluation and resume. The checks are shown to fail: a runner without the window's rescale, a replay on another generator state, and ledgers that break the commit contract. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review findings on the kit: - The microbatch job passed a runner that ignored the terms: on one rank the program's own numerator was the same loss. The kit program's numerator carries no graph there, so such a runner cannot train it. - The ledger records what the runner was handed; a commit has to name exactly what the update of its step consumed, nothing is committed after the run raised, and a run that ends cleanly committed every update. An asynchronous acknowledgement takes a moment, so a checkpoint that does not wait for it is seen. - Resume runs in bfloat16 and through two further updates: lost fp32 masters and a schedule that starts over both fail it. - Jobs compare every metric, the learning rate, the weights and the optimizer state bitwise through the recorder's comparison, check that the evaluation declaration reaches the program and that a batch without a declared input is refused; the replay statistic is used in a way that tells sums from means. - The mutation recordings skip on a host that cannot place two ranks; the assertion on clipping's first update, which clipping does not guarantee, is dropped. The program allocates on its weights' device. - DESIGN.md says what the recorded cases train (the reference) and what the kit does not cover. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Second review round on the conformance kit: - The ledger compares what the runner was handed with what the stream read, batch by batch with all of a batch's samples, so a sample that was read and never trained on cannot be committed; a window is counted in micro-batches and a commit names every sample of it. A complete run has to return, and a checkpoint's state has to be collected with every update committed and no acknowledgement in flight. - The kit program forms its replay argument itself, outside the forward, from sums that depend on the prepass's mode and generator; its loss reads the step and it reports one ratio only when detailed metrics are asked for. A runner that bypasses replay_arguments, runs a prepass in evaluation mode, or drops the schedule or the flag now differs from the reference. - One job runs in bfloat16 against the reference, so a runner that rounds its fp32 masters to the weights fails; jobs also compare the gradient norm and the exported draft. The two arms of every job seed the generator after the runner is built. - The probes of resume lose the masters or only the schedule's position. - The mutation recordings tell a case the recorder has no draft for from a host without two ranks, and count a difference only beyond what two accelerator runs of one revision differ by. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Next step of the training-backend design (#1177), stacked on the import-scan PR. The stack so far was held to a recorder that compares two revisions of the same runner. A second runner needs more: a statement of what it has to reproduce that does not depend on which recipes happen to exist, a check of what a recording cannot show (the order of commits and checkpoints), and proof that these checks can fail. This is the design's Conformance section and delivery step 5.
Change
tests/test_runtime/conformance.py(a kit, not a test module) andtests/test_runtime/test_conformance.py.declaration_cases()reduction: window(DFlash),reduction: microbatch(DSpark),statistic: replay(DSpine),evaluation diagnostics(EAGLE3, evaluated). Two ranks, a window of two. A protocol in the registry that declares something without a case failsDeclarationTest, so a new reduction or statistic needs its case before a runner can claim itMUTATIONSaround_training=hook of the recorder that every rank enters around its run: gradients scaled by the world size, clipping skipped, the learning-rate schedule shifted by one stepProtocolChecksprotocol_violations(ledger)is the rule as a functionRunnerChecksKitProgramweighs micro-batches differently; its loss reads the step and the noise it draws; it reports one ratio only when detailed metrics are asked for; undermicrobatchit returns terms and a numerator without a graph; underreplayit forms its forward's argument itself, outside the forward, from pooled sums that depend on the prepass's mode and generator. One job runs in bfloat16, where the optimizer's fp32 masters hold more than the weights. Plus the single-use handle, a batch without a declared input, evaluation (the declaration decides the diagnostics, weights and generator are left alone), and resume in bfloat16 through two further updatesA runner is checked in one process by two test classes that set
make_runner; the reference runner's are in this PR. The recorded cases train whatspecforge trainassembles, which is the reference: no configuration key selects another runner, so a candidate is recorded by substituting it around every rank's run (record(cases, around_training=...)).Evidence
KitCanFailTest): a runner that skips the window's rescale fails thewindowjob and passes themicrobatchone (whose rescale is by one); a runner that ignores the terms cannot train themicrobatchjob; a replay on another generator state, and a prepass run in evaluation mode, fail thereplayjob; a runner that passes step zero to every forward, or never asks for detailed metrics, fails thewindowjob; one that rounds its fp32 masters to the weights fails the bfloat16 job and passes the fp32 one; a resume that loses the masters, or only the schedule's position, fails the resume job; a run loop that does not wait for commits before a checkpoint fails the protocol; nine hand-written ledgers that break the commit contract are each reported with the reason.main's.Not in the kit
flash_attnand GPUs and is not recorded here.tests/test_runtime/test_async_durable_ack.pyhas the two-rank acknowledgement tests; they are not parameterized by runner.After Codex review
One read-only Codex round (
maocheng/backend-10-boundaries...HEAD) asked of each check whether a wrong runner could pass it. Several could, and the kit was tightened in a follow-up commit:setUpClasson a host with one accelerator.make_runner. They are not, and the text above now says what is.Each of these has a test now that fails when the property is broken (see Evidence).
Second Codex round
A second read-only round asked again, of the tightened kit, whether a wrong runner or run loop could pass. Closed in a further commit, each with a probe in
KitCanFailTestor a hand-written ledger that fails when the property is broken:replay_arguments, or run its prepass in evaluation mode, and match; no job compared a bfloat16 run with the reference, so masters rounded to the weights passed.Not covered, and listed above under "Not in the kit": a prepass in the ledger, cross-rank evaluation cases, and an accelerator.
🤖 Generated with Claude Code