feat(coordinator): add the R9 encoder-side ablations (no_svf / last_token) - #426
Merged
Merged
Conversation
…oken) SPEC R9's verifier is merged (analysis/ablations.py, James-CUDA#376) and consumes {full, no_svf, no_thinker, no_trirole, last_token}, but nothing in src/ could produce those numbers. James-CUDA#425 added the two role ablations; this adds the two that concern the feature pipeline. no_svf is a pure theta transformation: the SVF block is set to the identity (all scales 1.0), which is exactly what SVFAdapter.identity_scales() already calls 'no adaptation'. It reuses the canonical params.pack/unpack, keeps theta at full width so an ablated run stays comparable with the full one, and changes no code path. last_token reads the appended EOS instead of SPEC 3.2's penultimate token. slm.py gains a token_index parameter defaulting to -2, so the shipped config and every existing caller are byte-for-byte unchanged; the default is also declared at class level so instances built via object.__new__ (as the offline tests do) keep reading the canonical position. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
James-CUDA
force-pushed
the
minion_encoder_ablations
branch
from
July 23, 2026 11:36
fb2cd4d to
b136629
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
SPEC §1.3 R9 — "removing SVF / Thinker / tri-role / penultimate-token all hurt" — has a merged verifier and no producer.
analysis/ablations.py(#376) takes{full, no_svf, no_thinker, no_trirole, last_token}and reports each ablation's drop, but those four names appear nowhere insrc/outside docstrings and the example JSON atscripts/ablations_report.py:11.#425 added the two role ablations. This adds the two that concern the feature pipeline, completing the set the verifier asks for.
no_svf— a pure θ transformationRemoving SVF adaptation is running the encoder at its stock singular values, i.e. all scales 1.0. That is exactly the vector
SVFAdapter.identity_scales()already hands out, so the ablation is defined by shipped code rather than by a new literal:params.pack/unpack, so it cannot drift from the θ layout.spec.n_total), so an ablated run stays directly comparable with the full model's and needs no separateParamSpec.is_svf_ablated()is included as a run-manifest assertion, and it's deliberately exact: a run whose scales merely sit near 1.0 is not ano_svfrun, and there's a test pinning that.last_token— one default-preserving parameterslm.pyhardcodedout.hidden_states[-1][0, -2, :]. It now readsself.token_index, which defaults to-2.from_configreads an optionalhidden_state.token_index; absent, it stays at SPEC §3.2's penultimate token. A test assertsfrom_config()on the repo's ownconfigs/trinity.yamlstill yields-2.object.__new__— whichtests/test_slm_head_input_suffix.pydoes — keeps reading the canonical position. That existing test file is untouched.make_ablated_encoderrejects an explicittoken_index=alongside a variant rather than silently overriding it: two intentions for one knob should be an error.Why a parameter rather than a subclass
A
LastTokenEncodersubclass would avoid editingslm.py, butencode()is a single method — overriding it means duplicating ~40 lines of tokenize / EOS-append / forward / L2-normalize. A copy that drifts from the real extraction would silently corrupt the ablation, which is the one thing this PR exists to measure. A default-preserving parameter keeps exactly one extraction path.Scope, stated honestly
This ships the producers, not R9's numbers. Filling in the accuracies needs benchmark runs on the GPU box, as does any call into
CoordinatorEncoder. What is offline-verifiable — and tested — is the θ transformation, the position registry, the tensor-level read semantics, and the wiring that carries a token index from config into the encoder.Why it's low-risk
no_svftouches no code path.last_tokenis a defaulted parameter; every existing caller and the shipped config behave identically.import trinity.coordinator.encoder_ablationspulls no torch, and the validation branches ofmake_ablated_encoderraise before the lazy encoder import — pinned by a subprocess test, since the in-processtest_no_torch_importedguards are order-fragile (test_union_oracle.pyuses the robust subprocess form; both checks here follow it, withPYTHONPATHpointed at the checkout under test).Verification
Full suite, A/B against the same tree without these files:
The 11 are pre-existing
test_report_script_*subprocess failures, untouched by this branch.ruff check src/ scripts/repo_governance/→ clean;mypyon both touched modules → cleantests/test_slm_head_input_suffix.pyandtests/test_torch_coordinator_policy.pypass unchanged🤖 Generated with Claude Code