Repository navigation
[Backend 5] Take checkpoint state from the step runner - #1208
Draft
maocheng23 wants to merge 3 commits into
Draft
maocheng23 wants to merge 3 commits into
maocheng23 wants to merge 3 commits into
Conversation
The run loop built a checkpoint by reaching into the backend for its state dict, into the strategy for the draft-weight filter and a pre-checkpoint hook, and probed the backend for whether its optimizer state is replicated. - StepRunner gains checkpoint_state (the draft's weights under canonical keys, this rank's state, the state every rank shares, the realised loss kernels), load_resume and agree. DataParallelRunner builds them from the backend and the strategy. - The ranks' loss kernels are compared through runner.agree before a checkpoint is written. ExecutionSelection.agree, a collective inside the objectives package, is gone with the strategy's before_checkpoint hook. - TrainerController holds neither the strategy nor the backend; it takes the strategy's name as a value. The files a checkpoint writes are unchanged, and recorded trajectories are bit-identical before and after. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review findings on the checkpoint-state change: - A rank that fails collecting its checkpoint state (after the model gather) or restoring its resume state now reports it through runner.agree, so its peers raise instead of waiting in the next collective. The rank that failed still raises what it failed with. - Only a rank that holds the gathered model state runs the draft filter; the others used to be handed an empty dict. - agree: any string is a failure, an empty one included; one outcome is not compared with itself (a NaN or a tensor value raised at one rank). - Tests: the one-process agreement no longer depends on whether an earlier test left a process group; the resume test checks optimizer state as well as the generator. - DESIGN.md said where DDP writes its optimizer state incorrectly. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Second review round on the checkpoint-state change: - The previous fix caught any failure of checkpoint_state() and entered the agreement. A rank that fails inside the FSDP model gather is already apart from its peers, which are inside a tensor collective; entering a gather there waits instead of ending the job. The backend and the runner now raise RankFailure for their local work only (the optimizer copy, the RNG state, selecting the draft's weights, the kernel record; restoring optimizer and RNG), and the run loop and the trainer report exactly that. Anything else propagates as it did. - The draft's weights are selected by the rank that writes the shared file, as before this stack, not by whichever rank holds a non-empty model state: under sharding a non-writer can hold its ignored modules. - The kernel record is copied inside the runner's protected work; the rank that failed raises its own exception whatever the agreement raises. - CheckpointState.replicated is documented as what it is today, the optimizer state. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Next step of the training-backend design (#1177), stacked on the evaluation PR. To save a checkpoint the run loop reached into the backend for its state dict, into the strategy for the draft-weight filter and a pre-checkpoint hook, and probed the backend with
getattrfor whether its optimizer state is replicated. To resume, the trainer loaded the backend's state itself. A second runner could not own any of that.Change
StepRunner(training/contracts.py)checkpoint_state(),load_resume(state)andagree(error, value=);CheckpointStateis the value that crossesDataParallelRunner.checkpoint_stateExecutionSelection.agree(), a collective inside the objectives package, called fromCanonicalTrainStrategy.before_checkpoint()runner.agree(None, value=kernels)from the run loop;ExecutionSelection.agreeand the strategy hook are deletedTrainerControllerstrategy_nameas a valueTrainerresumebackend.load_state_dict(resume["backend"])runner.load_resume(resume["backend"])TrainingBackendoptimizer_state_is_replicatedprobed withgetattrOne runner verb returns the whole state because the backend gathers it in one call today. The design's separate
draft_stateandsave_resume, and the checkpoint manager's own collectives (_all_ok, the barriers, the best-checkpoint broadcast, the resume gather), belong to the change of the checkpoint sequence and its files (delivery step 8, which needs the maintainers' decision 5). They are untouched here.Behaviour-change ledger
ValueError("Loss implementations differ across ranks: ...")RuntimeError("ranks disagree: [('loss kernels', {...}), ...]")TrainerController(...)strategy=,backend=strategy_name=RankFailurefor that work; the run loop and the trainer report it throughrunner.agree, its peers raise, and it raises what it failed with. A failure inside the model gather propagates as beforeagree) afterload_resumereplicated_optimizer_state: Nonebefore_checkpoint, or whosenamediffers fromalgorithm_namealgorithm_nameEvidence
main.main's.checkpoint_statesplit andload_resumerestoring optimizer and generator; a checkpoint is not written when the ranks disagree. The two-rank gloo test of kernel disagreement now goes throughrunner.agree.Not verified
test_checkpoint_resume.py), which I edited and did not run; on CPU the end-to-end resume test covers one rank.After Codex review
One read-only Codex round (
maocheng/backend-04-evaluate...HEAD). Fixed in a follow-up commit on this branch:fitalone: both now go throughrunner.agree(ledger rows 6 and 7). The failing rank still raises its own exception.agree: one outcome is not compared with itself (a NaN or a tensor value raised at one rank); an empty error string is an error.DESIGN.mdsaid where DDP writes its optimizer state incorrectly.Not changed, on purpose:
checkpoint_extra), the kernels compared come from the runner. In an assembled run both are one live dictionary. The recorder does not compare that key, so the claim "files unchanged" rests on the code for it; removing the second source belongs to the run-record change.Second Codex round
A second read-only round on the fixes above. Fixed in a further commit:
checkpoint_state(). A rank that fails inside the FSDP model gather is already apart from its peers, which are in a tensor collective, and entering the agreement there waits wheremainended the job. Only rank-local work is agreed now (ledger row 6):RankFailureincontracts.py, raised by the backend and the runner around exactly that work.main, not by whichever rank holds a non-empty model state (under sharding a non-writer can hold its ignored modules); the kernel record is copied inside the protected work; the failing rank raises its own exception whatever the agreement raises;CheckpointState.replicatedis documented as the optimizer state it is today.Not changed: the two declined points above (the reviewer agreed the reasons hold for every assembled run). A rank that is terminated by a signal during its local work does not reach the agreement; that is the launcher's to tear down. No test restores two sharded ranks: FSDP does not run on this machine's CPU, and the recorder has no resumed run.
🤖 Generated with Claude Code