Require every blocking gate to prove it can fail (check-liveness ratchet) - #996
Merged
Conversation
The 2026-07-25 review found two load-bearing rules that were green while broken: ADR-0016's "Python never ships in skills" with no gate, and an exact-identity digest contract whose test mocked Validate with the code under test's own digest function. PR #995 fixed both instances. This is the class. A gate verified only against a green tree is indistinguishable from `exit 0` -- delete its detector and CI stays green. Measured across the registry: 32 of 54 BLOCKING script-backed gates had no test anywhere under tests/ demonstrating they FAIL on what they claim to detect. 17 had no test at all, including skill.schema and skill.triggers, which gate every SKILL.md in the corpus. The closure lives in Go because the registry does. An earlier bash/Python projection of seed.go missed four gates registered outside that file (always.git-config-hygiene, always.retrieval-manifest-paths, contract.verdict-corpus, workflow.install-drift) -- the authority is gates.Default.All(), and reading it any other way reproduces exactly the projection-drift ADR-0016 is named after. It mirrors the existing TestRatchetLibConsumersRouteLibEdits closure and is self-extending. Mechanism is the repo's proven shrink-only ratchet: the gates lacking a witness are pinned in scripts/.gate-negative-witness-grandfather; a NEW blocking gate must ship with a witness; a pinned gate that gains one must be pruned; a pinned gate that leaves the registry must be pruned; and the growth guard reads HEAD, so a change cannot exempt its own new gate in the same diff. The surviving count prints on every run. Ratcheted down by one immediately, to prove the mechanism moves rather than just counts: tests/scripts/validate-skill-schema.bats is a real witness for skill.schema (32 -> 31). It drives the actual validator against a fixture tree and asserts exit 1 on each missing required field. One fixture-fidelity trap worth recording: the first draft's negatives passed for the WRONG reason. Descriptions in this corpus embed `Triggers: "x"`, and unquoted that colon makes YAML read a nested mapping -- so the gate failed on a parse error, not the schema violation each test claimed to witness. Caught because the positive case failed too. Descriptions are now single-quoted and the missing-name case is confirmed to fail on "'name' is a required property". Verified: the meta-test's own three failure directions each seeded and observed (unpin a witness-less gate -> FAIL; pin a gate that gained a witness -> FAIL demanding the prune; pin a gate absent from the registry -> FAIL); 5/5 bats on the new witness; go build, go vet, full go test ./... green; the bats path-filter wiring tests green; make regen-check clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CJNsMBvvKCYktzaFo3E6Fu
… test go.test-isolation caught the new test: its git-exec env-scrub ratchet rose from 3 to 4 because negative_witness_test.go execs `git show` with cmd.Dir set but no cmd.Env. Relying on TestMain's process-level ScrubGitDiscoveryEnv was not enough — the repo's contract is a per-command scrub, and it should not depend on a TestMain elsewhere in the package staying correct. Under a hook-leaked GIT_DIR even a directory-scoped git call resolves against the LEAKED repository (age-gate-scripts-worktree-gitdir-p62wo, ek8v). Adds a package-local scrubbedGitEnv twin of gitDiscoveryEnv and wires it into the one git call. Ratchet is back at baseline 3 with this file no longer listed. Worth recording for the PR's own subject matter: go.test-isolation is on the check-liveness allowlist as lacking a proven negative — and it just caught a real defect anyway. The pin measures TEST COVERAGE of a gate, not its efficacy. An unproven gate may work fine; the point is that nothing would tell us if it stopped. Verified: check-test-isolation.sh PASS (git-unscrubbed=3/3); full go test ./... green; meta-test green at 31; growth guard still fires on a seeded self-allowlist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CJNsMBvvKCYktzaFo3E6Fu
boshu2
added a commit
that referenced
this pull request
Jul 28, 2026
…to fail
Under set -euo pipefail a '! grep' line is a silent no-op — errexit never
fires for !-inverted pipelines — so the forbidden-vocabulary guards in the
rpi, plan, implement, learn, ms, scaffold, and security validators could
never fail their script. Witnessed RED (seeded tokens, all seven exited 0),
converted each to the if-grep-exit-1 house form, and reworded learn's
boundary sentence whose own denial ('emit a lifecycle receipt') trips the
now-live guard. tests/scripts/skill-validator-liveness.bats pins the class:
a static no-inert-negation sweep plus seeded behavioral liveness per skill.
W1 of age-skill-overhaul-reboot-sjv7v (#996 doctrine applied to skill
validators).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A shrink-only ratchet requiring every blocking script-backed gate to have a test that demonstrates it failing on what it detects — plus the first ratchet-down, a real negative witness for
skill.schema.This is P3 from the
2026-07-25review: fix the inert-check class, not just the two instances.Why
PR #995 fixed two load-bearing rules that were green while broken — ADR-0016's "Python never ships in skills" with no gate, and an exact-identity digest contract whose test mocked Validate with the code under test's own digest function. Both were instances of one thing: a check nobody had ever proven executes.
A gate verified only against a green tree is indistinguishable from
exit 0. Delete its detector and CI stays green.Measured across the registry: 32 of 54 blocking script-backed gates had no test anywhere under
tests/demonstrating they can fail. 17 had no test at all — includingskill.schemaandskill.triggers, which gate everySKILL.mdin the corpus.The closure lives in Go because the registry does. My first measurement parsed
seed.gofrom Python and undercounted by four —always.git-config-hygiene,always.retrieval-manifest-paths,contract.verdict-corpus, andworkflow.install-driftare registered outside that file. The authority isgates.Default.All(), and reading it any other way reproduces exactly the projection-drift ADR-0016 is named after. The test mirrors the existingTestRatchetLibConsumersRouteLibEditsclosure and is self-extending — a new gate is covered the moment it registers.How it works
Same mechanism the repo already uses for preamble adoption and shipped Python:
scripts/.gate-negative-witness-grandfatherand exempt — zero churn.HEAD, so a change cannot exempt its own new gate in the same diff."Negative witness" is deliberately generous about form — bats, shell integration tests, and Python
returncode != 0all count. The bar is somebody proved it can fail, not a particular framework. The teeth come from the growth guard, not from the regex being exhaustive.How I tested
The meta-test's own liveness — all four failure directions seeded and observed, because a check-liveness ratchet that was only ever seen passing would be self-refuting:
skill.schema)HEADThe growth-guard direction was verified after the commit, since it correctly stands down while no
HEADversion of the snapshot exists (the initial-snapshot state).The ratchet-down —
tests/scripts/validate-skill-schema.bats, 5/5 green. Drives the real validator against a fixture tree (the script anchorsREPO_ROOTtodirname $0/..) and asserts exit 1 for each missing required field. Count goes 32 → 31.A fixture-fidelity trap worth recording: the first draft's negatives passed for the wrong reason. Descriptions in this corpus embed
Triggers: "x", and unquoted that colon makes YAML read a nested mapping — so the gate failed on a parse error, not the schema violation each test claimed to witness. Caught only because the positive case failed too. Descriptions are now single-quoted, and the missing-name case is confirmed to fail on'name' is a required property. A negative that fires for the wrong reason is not a witness.Everything else —
go build,go vet, fullgo test ./...green; the bats path-filter wiring tests (test-bats-path-filter-wiring,test-pathfilter-gate-coverage) green, so the new bats file is correctly routed;shellcheck -S warningclean;make regen-checkclean.Not checked: the 31 gates still on the allowlist remain unproven — that is the point of the ratchet, not an oversight. They are now counted and can only decrease.
Checklist
make build && make testpasses (if Go changes)Generated by Claude Code