Skip to content

[Backend 5] Take checkpoint state from the step runner - #1208

Draft
maocheng23 wants to merge 3 commits into
maocheng/backend-04-evaluatefrom
maocheng/backend-05-checkpoint-state
Draft

maocheng23 wants to merge 3 commits into
maocheng/backend-04-evaluatefrom
maocheng/backend-05-checkpoint-state

Conversation

@maocheng23

@maocheng23 maocheng23 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Next step of the training-backend design (#1177), stacked on the evaluation PR. To save a checkpoint the run loop reached into the backend for its state dict, into the strategy for the draft-weight filter and a pre-checkpoint hook, and probed the backend with getattr for whether its optimizer state is replicated. To resume, the trainer loaded the backend's state itself. A second runner could not own any of that.

Change

Part Before After
StepRunner (training/contracts.py) — gains checkpoint_state(), load_resume(state) and agree(error, value=); CheckpointState is the value that crosses
DataParallelRunner.checkpoint_state — one backend call, split by who saves what: the draft's weights under canonical keys (complete on rank 0), this rank's own state, the state every rank shares (the DDP optimizer), the loss kernels this rank has realised
Kernel agreement ExecutionSelection.agree(), a collective inside the objectives package, called from CanonicalTrainStrategy.before_checkpoint() runner.agree(None, value=kernels) from the run loop; ExecutionSelection.agree and the strategy hook are deleted
TrainerController held the strategy and the backend holds neither; takes strategy_name as a value
Trainer resume backend.load_state_dict(resume["backend"]) runner.load_resume(resume["backend"])
TrainingBackend optimizer_state_is_replicated probed with getattr declared on the base class

One runner verb returns the whole state because the backend gathers it in one call today. The design's separate draft_state and save_resume, and the checkpoint manager's own collectives (_all_ok, the barriers, the best-checkpoint broadcast, the resume gather), belong to the change of the checkpoint sequence and its files (delivery step 8, which needs the maintainers' decision 5). They are untouched here.

Behaviour-change ledger

# What Before After Kind
1 Files a checkpoint writes, and their contents — identical; recorded trajectories read every checkpoint (see Evidence) unchanged
2 Order of the first two collectives of a save gather of the kernel names, then the model gather (FSDP) the model gather, then the gather of the kernel names; both before anything is written reordered, same values
3 Ranks resolved different loss kernels ValueError("Loss implementations differ across ranks: ...") RuntimeError("ranks disagree: [('loss kernels', {...}), ...]") same point of failure, other type and text
4 Group of the kernel agreement the default group the backend's data-parallel group same group in every run the builders assemble
5 TrainerController(...) strategy=, backend= strategy_name= internal API
6 A rank fails in its own part of collecting checkpoint state (copying the optimizer, the RNG state, selecting the draft's weights) or of restoring resume state it raised and its peers waited in the next collective the backend and the runner raise RankFailure for that work; the run loop and the trainer report it through runner.agree, its peers raise, and it raises what it failed with. A failure inside the model gather propagates as before the design's rule for these two calls
7 Resume — one more gather (agree) after load_resume added collective, on resume only
8 A DDP run without an optimizer (test doubles) shared file holds replicated_optimizer_state: None the key is absent; the reader accepts both file content
9 An injected strategy whose factory draws from the generator, on resume the generator was restored before the factory ran after it (the canonical constructor draws nothing, and the next PR removes the factory) order, for injected strategies only
10 An injected strategy that kept the no-op before_checkpoint, or whose name differs from algorithm_name no kernel gather; the checkpoint recorded the strategy's name one gather per save; the checkpoint records algorithm_name for injected strategies only

Evidence

  • Recorded trajectories, which hold every checkpoint of every run (draft weights, every rank's optimizer and RNG state, the replicated optimizer state, the data position): 90 cases, 0 differences against main.
  • Full CPU suite: 1,965 passed; the failure set (15 failed, 1 error, all environmental on this machine) is identical to main's.
  • New tests: the runner's agreement (equal outcomes, an error on another rank, an error of its own, differing values, one process); checkpoint_state split and load_resume restoring optimizer and generator; a checkpoint is not written when the ranks disagree. The two-rank gloo test of kernel disagreement now goes through runner.agree.

Not verified

  • CUDA: not run. Resume across ranks with sharded optimizer state is covered by CUDA-gated tests only (test_checkpoint_resume.py), which I edited and did not run; on CPU the end-to-end resume test covers one rank.
  • The recorder has no resumed run yet.

After Codex review

One read-only Codex round (maocheng/backend-04-evaluate...HEAD). Fixed in a follow-up commit on this branch:

  • P1 a rank that failed while collecting its checkpoint state never reached the agreement, and a rank whose resume state did not restore left its peers to enter fit alone: both now go through runner.agree (ledger rows 6 and 7). The failing rank still raises its own exception.
  • P2 ranks other than the first handed an empty dict to the draft filter; now only a rank that holds the gathered model state runs it.
  • P2 / P3 agree: one outcome is not compared with itself (a NaN or a tensor value raised at one rank); an empty error string is an error.
  • P2 / P3 tests: the one-process agreement test no longer depends on whether an earlier test left a process group; the resume test checks optimizer state, not the generator alone.
  • P3 DESIGN.md said where DDP writes its optimizer state incorrectly.

Not changed, on purpose:

  • The agreement runs on the backend's data-parallel group and the checkpoint manager's own collectives on the world. They are the same group in every run the builders assemble (ledger row 4); the manager's collectives move with the checkpoint sequence (delivery step 8).
  • The kernels written to disk come from the resume contract (checkpoint_extra), the kernels compared come from the runner. In an assembled run both are one live dictionary. The recorder does not compare that key, so the claim "files unchanged" rests on the code for it; removing the second source belongs to the run-record change.

Second Codex round

A second read-only round on the fixes above. Fixed in a further commit:

  • P1 the first fix caught any failure of checkpoint_state(). A rank that fails inside the FSDP model gather is already apart from its peers, which are in a tensor collective, and entering the agreement there waits where main ended the job. Only rank-local work is agreed now (ledger row 6): RankFailure in contracts.py, raised by the backend and the runner around exactly that work.
  • P2 the draft's weights are selected by the rank that writes the shared file, as on main, not by whichever rank holds a non-empty model state (under sharding a non-writer can hold its ignored modules); the kernel record is copied inside the protected work; the failing rank raises its own exception whatever the agreement raises; CheckpointState.replicated is documented as the optimizer state it is today.
  • Ledger rows 9 and 10 record two effects on injected strategies that the first description missed.

Not changed: the two declined points above (the reviewer agreed the reasons hold for every assembled run). A rank that is terminated by a signal during its local work does not reach the agreement; that is the launcher's to tear down. No test restores two sharded ranks: FSDP does not run on this machine's CPU, and the recorder has no resumed run.

🤖 Generated with Claude Code

maocheng23 and others added 3 commits November 18, 2026 13:43
The run loop built a checkpoint by reaching into the backend for its state
dict, into the strategy for the draft-weight filter and a pre-checkpoint hook,
and probed the backend for whether its optimizer state is replicated.

- StepRunner gains checkpoint_state (the draft's weights under canonical keys,
  this rank's state, the state every rank shares, the realised loss kernels),
  load_resume and agree. DataParallelRunner builds them from the backend and
  the strategy.
- The ranks' loss kernels are compared through runner.agree before a
  checkpoint is written. ExecutionSelection.agree, a collective inside the
  objectives package, is gone with the strategy's before_checkpoint hook.
- TrainerController holds neither the strategy nor the backend; it takes the
  strategy's name as a value.

The files a checkpoint writes are unchanged, and recorded trajectories are
bit-identical before and after.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review findings on the checkpoint-state change:

- A rank that fails collecting its checkpoint state (after the model
  gather) or restoring its resume state now reports it through
  runner.agree, so its peers raise instead of waiting in the next
  collective. The rank that failed still raises what it failed with.
- Only a rank that holds the gathered model state runs the draft filter;
  the others used to be handed an empty dict.
- agree: any string is a failure, an empty one included; one outcome is
  not compared with itself (a NaN or a tensor value raised at one rank).
- Tests: the one-process agreement no longer depends on whether an
  earlier test left a process group; the resume test checks optimizer
  state as well as the generator.
- DESIGN.md said where DDP writes its optimizer state incorrectly.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Second review round on the checkpoint-state change:

- The previous fix caught any failure of checkpoint_state() and entered
  the agreement. A rank that fails inside the FSDP model gather is
  already apart from its peers, which are inside a tensor collective;
  entering a gather there waits instead of ending the job. The backend
  and the runner now raise RankFailure for their local work only (the
  optimizer copy, the RNG state, selecting the draft's weights, the
  kernel record; restoring optimizer and RNG), and the run loop and the
  trainer report exactly that. Anything else propagates as it did.
- The draft's weights are selected by the rank that writes the shared
  file, as before this stack, not by whichever rank holds a non-empty
  model state: under sharding a non-writer can hold its ignored modules.
- The kernel record is copied inside the runner's protected work; the
  rank that failed raises its own exception whatever the agreement
  raises.
- CheckpointState.replicated is documented as what it is today, the
  optimizer state.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant