Skip to content

[Backend 10] test(training): read the training plane's import rules from the source - #1213

Draft
maocheng23 wants to merge 6 commits into
maocheng/backend-09-constructionfrom
maocheng/backend-10-boundaries
Draft

maocheng23 wants to merge 6 commits into
maocheng/backend-09-constructionfrom
maocheng/backend-10-boundaries

Conversation

@maocheng23

@maocheng23 maocheng23 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Next step of the training-backend design (#1177), stacked on the construction PR. The previous PRs moved every collective across samples into the step runner and made the parts meet through training/contracts.py. Nothing keeps it that way: one import torch.distributed in an objective, or one from .runner import ... in the run loop, and the property is gone without a test failing. The design's API rules name an import scan as the enforcement.

Change

tests/test_runtime/test_training_boundaries.py: an AST scan over specforge/, with no import of torch. Five rules:

Rule Modules May not reach
A collective across samples appears only in a runner objectives, drafts, losses, layers; the frame; the run loop and the evaluator; the checkpoint manager, the profiler and trainer.py torch.distributed, specforge.distributed
The frame imports the shared contracts only contracts.py, frame.py, program.py anything else in specforge
The run loop imports no runner, backend, optimizer, objective or model controller.py, specforge/eval/ training.runner, training.backend, training.backends, optimizer, algorithms, modeling, layers, losses, torch.optim
A runner decides from declarations and imports no registry runner.py, backend.py, optimizer.py algorithms, modeling, config.draft.registry
A program imports nothing that trains it objectives, drafts, losses, layers training, runtime, launch, eval, optimizer, torch.optim, torch.nn.parallel

The shared contracts are training.contracts, runtime.contracts and the names of the config.draft package itself (its value types); the modules inside it (codecs, validators, the registry) are not shared.

What counts as reaching a name. An import at module level or inside a function, with relative imports resolved; every use of the imported name, through its alias and with the attributes that follow (import torch as t; t.distributed.barrier()), a call of a name imported by itself included; a literal importlib.import_module or __import__, with what is taken from the returned module; and through the package: a name another module re-exports is followed to its origin with the rest of the chain (specforge.utils.dist.barrier), and a helper function brings what its own body uses (specforge.utils.rank_0_priority counts as its barriers). Names are resolved in the scope they are used in: a function's own import, parameter or assignment hides the module's. Only the branch of a typing.TYPE_CHECKING test that never executes is skipped. No covered module may star-import from the package.

Exceptions are pinned. A module that breaks a rule today is listed with the reason and with each forbidden name it reaches and how many times; one use more or one fewer fails the test, so a second broadcast in an excepted file is seen. There are ten, all under the first rule. Six are a model-parallel axis, whose groups share a sequence or a tensor and never samples: cached_flash.py, eagle_rollout.py, layers/ring/utils.py (sequence parallelism), layers/linear.py, embedding.py, lm_head.py (tensor parallelism). Four are the run loop's own uses of the process group, which the design moves behind runner.agree and a topology value: controller.py (the progress bar's rank, rank 0's broadcast verdict on whether an evaluation runs, the world size written into a checkpoint), checkpoint.py (the gather of write errors, the barriers, the best-checkpoint broadcast, the resume gather), profiling.py (the rank in its output's name), trainer.py (the world size compared on resume).

specforge/training/DESIGN.md gains a Boundaries section with the rules.

What the scan does not see, stated in the test and the design note: which process group a collective runs on (a reduction bound to the world instead of the sequence group changes no name); what a method called on a value does (the run loop holds a runner and nothing here stops it from reaching through it); a decision taken from spec.name instead of a declaration; a name computed at run time. An exception list in the repository can also be extended in the same change that needs it; that is for review to see.

Evidence

  • The rules hold on this branch with the ten exceptions and no others.
  • The scan can fail. Planted and reported: a torch.distributed import inside a function of objectives/reductions.py; import torch as t; t.distributed.all_reduce there; from specforge.utils import rank_0_priority there; import specforge.utils as u; u.dist.barrier() there; importlib.import_module(name="torch.distributed").barrier() there; a torch.distributed.barrier() and a second broadcast_object_list added to controller.py, which is an excepted file; from specforge.utils import dist as pg; pg.barrier() there; the runner imported into controller.py directly and through specforge.training.trainer; gather_id_union imported into the evaluator from the control-plane package, which re-exports it in a multi-name import; the objective registry imported into backend.py. On the base of the stack the first rule reports objectives/base.py (microbatch_mean), which the microbatch PR removed.
  • Unit tests of the scan itself: function-level imports, aliases, every use counted, names resolved per scope (an import in another function, a parameter, an assignment), whole attribute chains, relative imports, both branches of TYPE_CHECKING however it was imported, literal dynamic imports. Re-exports and helpers are followed through a small package written for the test, so no real module's alias is frozen by it.
  • Full CPU suite: 2,006 passed; the failure set (15 failed, 1 error, all environmental on this machine) is identical to main's.

After Codex review

One read-only Codex round (maocheng/backend-09-construction...HEAD) against the first version, which matched import statements and plain torch.distributed.x chains and excepted whole files. It found that version too easy to pass, and the scan was rewritten in a follow-up commit:

  • P1 a new collective in an excepted file passed: exceptions are now pinned to exact names.
  • P1 the checkpoint manager's, the profiler's and the trainer's collectives were out of scope: now in scope, as pinned exceptions.
  • P1 a helper such as rank_0_priority, or a runner re-exported by trainer.py, was not followed: now one step through the package.
  • P2 aliases, local names that shadow nothing, TYPE_CHECKING branches, literal dynamic imports; config.draft value types as a shared contract and its registry as the forbidden lookup; the optimizer and torch.nn.parallel forbidden to programs; optimizer.py in the runner's scope; the exceptions described from what the code does.

Not addressed, and stated above as limits: process groups at the binding sites, and method calls on values.

Second Codex round

A second read-only round found the rewritten scan still passable, and one plain bug in it. The core was rewritten once more:

  • P1, a bug each name of a multi-name from x import a, b was given the origin of the first, so a helper re-exported in such a statement was not followed.
  • P1 a re-exported name lost the rest of its chain (utils.dist.barrier); a helper that used a collective by its imported name contributed nothing; an import in one function changed how a name in another resolved; a second use of an already-pinned operation passed; a dynamic import lost the operation taken from the module. All five are covered above and in Evidence.
  • P2 TYPE_CHECKING is recognised by its binding; imports inside a module-level try count for the module's helpers; of config.draft only the package's own names are shared; the run loop may not import layers, losses or torch.optim; the scan's tests no longer require specforge.utils.dist to exist; the design note says that rank 0's two broadcast verdicts need more than runner.agree to move.

Still limits, stated above: process groups at the binding sites, method calls on values, and names computed at run time.

🤖 Generated with Claude Code

maocheng23 and others added 6 commits November 18, 2026 14:29
An AST scan over specforge/ checks five rules of the training-backend
design: a collective across samples appears only in a runner; the frame
meets the other parts through the contracts only; the run loop holds no
runner, backend, objective or model; a runner decides from declarations,
never from a registry; a program does not know how it is trained.

Imports inside functions, relative imports and attribute use such as
torch.distributed.get_rank count; imports under TYPE_CHECKING do not.
Each rule lists today's exceptions with the reason (the model-parallel
axes, and the run loop's remaining topology reads). An exception that no
longer matches anything fails, so the list can only shrink.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review findings on the scan:

- Exceptions were per file: a new collective in an excepted file passed.
  Each exception now lists exactly the names the module reaches; one
  more or one fewer fails.
- Names are resolved through import aliases, so `import torch as t;
  t.distributed.all_reduce` counts and a local variable named like a
  module does not. Only whole attribute chains count.
- One step through the package: a re-exported name is followed to its
  origin and a helper function brings what its body reaches, so
  `specforge.utils.rank_0_priority` counts as its barriers and a runner
  imported from `trainer.py` as the runner.
- Only the branch of a TYPE_CHECKING test that never executes is skipped;
  literal importlib.import_module / __import__ calls count.
- The first rule also covers the checkpoint manager, the profiler and
  trainer.py; the run-loop rule also forbids the optimizer and the
  backends package; the runner rule covers optimizer.py and forbids the
  draft registry instead of all of specforge.config; value types of
  config.draft are a shared contract.
- The exceptions are described from what the code does (ten modules: six
  of a model-parallel axis, four uses by the run loop and its helpers).
  DESIGN.md states import restrictions and what the scan cannot see.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rigin

The scan's index of the package followed a re-exported name to where it
comes from, but for `from x import a, b` it gave every name the origin of
the first: `trainer.ParallelConfig` resolved to `backend.FSDPTrainingBackend`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Second review round on the import scan:

- Names are resolved in the scope they are used in: a function's own
  import, parameter or assignment hides the module's, and an import in
  one function says nothing about another.
- Every use is counted, an imported name called by itself included, and
  an exception lists how many times its module reaches each forbidden
  name: a second broadcast in an excepted file fails.
- A re-exported name is followed with the rest of its chain
  (`utils.dist.barrier`), through any number of modules; a helper's
  summary is what its own scope uses, imports in a module-level `try`
  included.
- Literal dynamic imports count with the attributes after the call, by
  keyword and relative to a literal package.
- TYPE_CHECKING is typing's name however it was imported, not anything
  spelled like it.
- Of config.draft only the package's own names are a shared contract;
  its codecs, validators and registry are not.
- The run loop may not import layers, losses or torch.optim; a program
  may not import torch.optim. No covered module may star-import from
  the package.
- The scan's own tests follow names through a package written for the
  test instead of through real modules' aliases.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant