feat(optim): add the SFT and REINFORCE baseline trainers (SPEC R8, milestone M4)

[minion1227]

The gap

SPEC §1.3 R8 claims sep-CMA-ES > SFT > RS > REINFORCE on all 4 tasks (Table 4), and §9 pins both recipes exactly. Two consumers of that ordering are already merged:

• analysis/convergence.py → R8_EXPECTED_ORDER names "sft" and "reinforce"
• configs/trinity.yaml ships imitation_sft / rl_reinforce entries plus sft: / reinforce: hyperparameter blocks

…but nothing in src/ could ever produce either run. The verifiers exist with no producer, so R8 was unverifiable at its source, and those config knobs had zero readers. This PR adds the producers.

optim/baselines.py said so itself — its docstring carried a written IOU deferring both "to a follow-up". This is that follow-up.

What's here

optim/sft.py — the SLM is frozen, so the only trainable tensor is the head weight W ∈ R^{n_a × d_h}. With frozen features the routing head is a multinomial logistic regression over precomputed hidden states, and SFT is imitation learning: for each query, build a teacher categorical over pool models from the measured per-(query, model) solve rates, then minimize softmax cross-entropy against it. SPEC §9 recipe verbatim — Adam, lr 1e-6, batch 64, head-only.

optim/reinforce.py — one-step (contextual-bandit) policy gradient, ∇_W J = E[(r − b)·∇_W log π(a|h)], with b an EMA advantage baseline carried in from previous iterations. Because b is a function of past batches only it is independent of the action being scored, so it subtracts nothing in expectation — the gradient stays unbiased while losing most of its variance, which is what makes REINFORCE usable on a reward this noisy. (Using the post-update baseline would leak the batch's own sampled actions into its baseline; there's a regression test pinning the carried-in semantics.)

Both implement BaseTrainer and return the documented trainer summary, so their results are directly comparable with sep-CMA-ES and Random Search.

Why it's low-risk

• Purely additive, default-off. No existing caller changes behaviour; nothing is wired into the submission or training path.
• Reuses the canonical schema path. coordinator.warmstart.load_labels / pack_warmstart_theta are reused rather than reimplemented, so these cannot drift from the on-disk matrix schema. A run produces a θ of exactly spec.n_total, so R8 compares like with like.
• Zero marginal cost. Labels come from the oracle-ceiling matrices already cached on disk — no new API calls.
• Stays torch-free at import. torch is imported lazily inside the fitting functions, so import trinity.optim still imports no torch — the invariant test_shaped_fitness.py::test_no_torch_imported enforces. Test files are named test_torch_* so alphabetical collection keeps them after that guard.

Deliberately not folded into warmstart.py

warmstart.fit_agent_head also fits agent rows by cross-entropy, but it is a pure-numpy full-batch GD helper whose only job is to produce a CMA-ES initial mean x0 (bespoke lr, L2, disagreement reweighting, no optimizer state, no trainer summary). This module is an R8 competitor and must reproduce the SPEC recipe verbatim. Sharing a loss family is expected — same statistical problem — but a warm-start and a baseline are different experiments, and collapsing them would make R8 compare CMA-ES against its own initializer.

Scope, stated honestly

Only the head's agent rows are fit: the oracle matrices carry per-model correctness and nothing about roles, so there is no role supervision to learn from. Role rows stay 0 (uniform) and SVF scales stay 1.0 (identity), matching pack_warmstart_theta.

The old baselines.py docstring claimed these "cannot be validated in a CPU-only environment". That was too strong and is corrected in this PR: both are head-only with small tensors, and their update rules are exercised on CPU (convergence on toy bandit problems, gradient/unbiasedness properties, budget matching). What genuinely still needs a GPU is producing the Table 4 numbers for R8, not the trainers themselves. This PR does not claim to have reproduced Table 4.

Verification

• pytest tests/ -q → 1830 passed, 0 failed (95 of them new, covering both trainers)
• ruff check src/ scripts/repo_governance/ → clean
• import trinity.optim confirmed to leave torch out of sys.modules

Note on CI: main has unresolved merge-conflict markers in src/trinity/analysis/__init__.py (introduced by dc15f97), so the package does not import and this PR is red on the merge commit. This branch touches no file in analysis/, and the 1830-passed run above was taken with those markers resolved locally.

The fix is not contested: #402, #405, #407, #408 and #403 all rewrite that file to the byte-identical result (every one is index 9916618..6865bc2), so merging any one of them repairs main and the rest become no-ops on that file. Note that #407 / #408 / #403 are already green — they touch that file for their own feature and resolved it in passing. Only PRs that don't touch it (this one, #416, #412) are red.

🤖 Generated with Claude Code

[minion1227]

CI red here is main, not this branch

The lint, typecheck, tests failure is 45 Ruff invalid-syntax errors, all from unresolved merge-conflict markers in src/trinity/analysis/__init__.py, introduced on main by dc15f97. This branch touches only src/trinity/optim/ and two new test files — nothing under analysis/ — so the failure is on the merge base, not in these changes.

The conflict is nested: a <<<<<<< HEAD / ======= / >>>>>>> f319b1f pair sits inside an outer conflict closed by >>>>>>> 9e31ba3, tangling three branches (union_diagnostics, oracle_column, definition_of_done).

Correction to my earlier note

I first wrote that every open PR is red for this reason. That is not accurate, and the real picture is more useful:

• fix: resolve leftover merge-conflict markers breaking analysis/__init__.py (main does not import) #402, fix(analysis): resolve unresolved merge-conflict markers in __init__ (package failed to import) #405, feat(analysis): verify SPEC R4 (TRINITY > random routing) offline #407, feat(competition): submission preflight go/no-go with per-benchmark gap-to-king #408, feat(competition): king-progression timeline from competition.history #403 all already contain the fix, and all five produce the byte-identical file — every one is index 9916618..6865bc2, the union resolution keeping all three import blocks.
• feat(analysis): verify SPEC R4 (TRINITY > random routing) offline #407 / feat(competition): submission preflight go/no-go with per-benchmark gap-to-king #408 / feat(competition): king-progression timeline from competition.history #403 are green precisely because they touch analysis/__init__.py for their own feature and resolved it in passing.
• Only PRs that don't touch that file — this one, feat(coordinator): add the SPEC §3.5 alternative heads (R10 ablation parity) #416, feat(analysis): routing-composition diagnostic (empirical model/role usage, collapse detection) #412 — are red.

So this isn't blocked on the two dedicated fix PRs specifically. Merging any one of those five repairs main; the others then reduce to no-ops on that file. Whichever lands first, no resolution needs arbitrating — there is only one resolution in play.

#415 and #416 both go green on a re-run once that happens, with no push from me.