Repository navigation
[Backend 10] test(training): read the training plane's import rules from the source - #1213
Draft
maocheng23 wants to merge 6 commits into
Draft
maocheng23 wants to merge 6 commits into
maocheng23 wants to merge 6 commits into
Conversation
An AST scan over specforge/ checks five rules of the training-backend design: a collective across samples appears only in a runner; the frame meets the other parts through the contracts only; the run loop holds no runner, backend, objective or model; a runner decides from declarations, never from a registry; a program does not know how it is trained. Imports inside functions, relative imports and attribute use such as torch.distributed.get_rank count; imports under TYPE_CHECKING do not. Each rule lists today's exceptions with the reason (the model-parallel axes, and the run loop's remaining topology reads). An exception that no longer matches anything fails, so the list can only shrink. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review findings on the scan: - Exceptions were per file: a new collective in an excepted file passed. Each exception now lists exactly the names the module reaches; one more or one fewer fails. - Names are resolved through import aliases, so `import torch as t; t.distributed.all_reduce` counts and a local variable named like a module does not. Only whole attribute chains count. - One step through the package: a re-exported name is followed to its origin and a helper function brings what its body reaches, so `specforge.utils.rank_0_priority` counts as its barriers and a runner imported from `trainer.py` as the runner. - Only the branch of a TYPE_CHECKING test that never executes is skipped; literal importlib.import_module / __import__ calls count. - The first rule also covers the checkpoint manager, the profiler and trainer.py; the run-loop rule also forbids the optimizer and the backends package; the runner rule covers optimizer.py and forbids the draft registry instead of all of specforge.config; value types of config.draft are a shared contract. - The exceptions are described from what the code does (ten modules: six of a model-parallel axis, four uses by the run loop and its helpers). DESIGN.md states import restrictions and what the scan cannot see. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rigin The scan's index of the package followed a re-exported name to where it comes from, but for `from x import a, b` it gave every name the origin of the first: `trainer.ParallelConfig` resolved to `backend.FSDPTrainingBackend`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Second review round on the import scan: - Names are resolved in the scope they are used in: a function's own import, parameter or assignment hides the module's, and an import in one function says nothing about another. - Every use is counted, an imported name called by itself included, and an exception lists how many times its module reaches each forbidden name: a second broadcast in an excepted file fails. - A re-exported name is followed with the rest of its chain (`utils.dist.barrier`), through any number of modules; a helper's summary is what its own scope uses, imports in a module-level `try` included. - Literal dynamic imports count with the attributes after the call, by keyword and relative to a literal package. - TYPE_CHECKING is typing's name however it was imported, not anything spelled like it. - Of config.draft only the package's own names are a shared contract; its codecs, validators and registry are not. - The run loop may not import layers, losses or torch.optim; a program may not import torch.optim. No covered module may star-import from the package. - The scan's own tests follow names through a package written for the test instead of through real modules' aliases. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Next step of the training-backend design (#1177), stacked on the construction PR. The previous PRs moved every collective across samples into the step runner and made the parts meet through
training/contracts.py. Nothing keeps it that way: oneimport torch.distributedin an objective, or onefrom .runner import ...in the run loop, and the property is gone without a test failing. The design's API rules name an import scan as the enforcement.Change
tests/test_runtime/test_training_boundaries.py: an AST scan overspecforge/, with no import of torch. Five rules:trainer.pytorch.distributed,specforge.distributedcontracts.py,frame.py,program.pyspecforgecontroller.py,specforge/eval/training.runner,training.backend,training.backends,optimizer,algorithms,modeling,layers,losses,torch.optimrunner.py,backend.py,optimizer.pyalgorithms,modeling,config.draft.registrytraining,runtime,launch,eval,optimizer,torch.optim,torch.nn.parallelThe shared contracts are
training.contracts,runtime.contractsand the names of theconfig.draftpackage itself (its value types); the modules inside it (codecs, validators, the registry) are not shared.What counts as reaching a name. An import at module level or inside a function, with relative imports resolved; every use of the imported name, through its alias and with the attributes that follow (
import torch as t; t.distributed.barrier()), a call of a name imported by itself included; a literalimportlib.import_moduleor__import__, with what is taken from the returned module; and through the package: a name another module re-exports is followed to its origin with the rest of the chain (specforge.utils.dist.barrier), and a helper function brings what its own body uses (specforge.utils.rank_0_prioritycounts as its barriers). Names are resolved in the scope they are used in: a function's own import, parameter or assignment hides the module's. Only the branch of atyping.TYPE_CHECKINGtest that never executes is skipped. No covered module may star-import from the package.Exceptions are pinned. A module that breaks a rule today is listed with the reason and with each forbidden name it reaches and how many times; one use more or one fewer fails the test, so a second broadcast in an excepted file is seen. There are ten, all under the first rule. Six are a model-parallel axis, whose groups share a sequence or a tensor and never samples:
cached_flash.py,eagle_rollout.py,layers/ring/utils.py(sequence parallelism),layers/linear.py,embedding.py,lm_head.py(tensor parallelism). Four are the run loop's own uses of the process group, which the design moves behindrunner.agreeand a topology value:controller.py(the progress bar's rank, rank 0's broadcast verdict on whether an evaluation runs, the world size written into a checkpoint),checkpoint.py(the gather of write errors, the barriers, the best-checkpoint broadcast, the resume gather),profiling.py(the rank in its output's name),trainer.py(the world size compared on resume).specforge/training/DESIGN.mdgains a Boundaries section with the rules.What the scan does not see, stated in the test and the design note: which process group a collective runs on (a reduction bound to the world instead of the sequence group changes no name); what a method called on a value does (the run loop holds a runner and nothing here stops it from reaching through it); a decision taken from
spec.nameinstead of a declaration; a name computed at run time. An exception list in the repository can also be extended in the same change that needs it; that is for review to see.Evidence
torch.distributedimport inside a function ofobjectives/reductions.py;import torch as t; t.distributed.all_reducethere;from specforge.utils import rank_0_prioritythere;import specforge.utils as u; u.dist.barrier()there;importlib.import_module(name="torch.distributed").barrier()there; atorch.distributed.barrier()and a secondbroadcast_object_listadded tocontroller.py, which is an excepted file;from specforge.utils import dist as pg; pg.barrier()there; the runner imported intocontroller.pydirectly and throughspecforge.training.trainer;gather_id_unionimported into the evaluator from the control-plane package, which re-exports it in a multi-name import; the objective registry imported intobackend.py. On the base of the stack the first rule reportsobjectives/base.py(microbatch_mean), which the microbatch PR removed.TYPE_CHECKINGhowever it was imported, literal dynamic imports. Re-exports and helpers are followed through a small package written for the test, so no real module's alias is frozen by it.main's.After Codex review
One read-only Codex round (
maocheng/backend-09-construction...HEAD) against the first version, which matched import statements and plaintorch.distributed.xchains and excepted whole files. It found that version too easy to pass, and the scan was rewritten in a follow-up commit:rank_0_priority, or a runner re-exported bytrainer.py, was not followed: now one step through the package.TYPE_CHECKINGbranches, literal dynamic imports;config.draftvalue types as a shared contract and its registry as the forbidden lookup; the optimizer andtorch.nn.parallelforbidden to programs;optimizer.pyin the runner's scope; the exceptions described from what the code does.Not addressed, and stated above as limits: process groups at the binding sites, and method calls on values.
Second Codex round
A second read-only round found the rewritten scan still passable, and one plain bug in it. The core was rewritten once more:
from x import a, bwas given the origin of the first, so a helper re-exported in such a statement was not followed.utils.dist.barrier); a helper that used a collective by its imported name contributed nothing; an import in one function changed how a name in another resolved; a second use of an already-pinned operation passed; a dynamic import lost the operation taken from the module. All five are covered above and in Evidence.TYPE_CHECKINGis recognised by its binding; imports inside a module-leveltrycount for the module's helpers; ofconfig.draftonly the package's own names are shared; the run loop may not importlayers,lossesortorch.optim; the scan's tests no longer requirespecforge.utils.distto exist; the design note says that rank 0's two broadcast verdicts need more thanrunner.agreeto move.Still limits, stated above: process groups at the binding sites, method calls on values, and names computed at run time.
🤖 Generated with Claude Code