Skip to content

test: realign antigravity + pi mapping assertions with pruned references#1987

Merged
obra merged 2 commits into
obra:devfrom
gaurav0107:fix/stale-per-harness-mapping-tests
Jul 15, 2026
Merged

test: realign antigravity + pi mapping assertions with pruned references#1987
obra merged 2 commits into
obra:devfrom
gaurav0107:fix/stale-per-harness-mapping-tests

Conversation

@gaurav0107

@gaurav0107 gaurav0107 commented Jul 13, 2026

Copy link
Copy Markdown

Targets the dev branch.

Who is submitting this PR? (required)

Field Value
Your model + version Claude Opus 4.8 (1M context) — model ID claude-opus-4-8[1m]
Harness + version Claude Code 2.1.143
All plugins installed superhuman@superhuman, superpowers@claude-plugins-official
Human partner who reviewed this diff Gaurav Dubey (@gaurav0107)

What problem are you trying to solve?

The plugin-infrastructure test suite fails on dev (and on main, 6.1.1). Two per-harness mapping tests assert content that was intentionally deleted:

$ git checkout dev
$ bash tests/antigravity/test-antigravity-tools.sh
FAIL: mapping does not document view_file as the file/skill-read tool   # exit 1

$ node tests/pi/test-pi-extension.mjs
✖ pi tools reference documents pi-specific mappings
  AssertionError: The input did not match the regular expression /write/   # exit 1 (1 of 6 cases)

Root cause: commit e7ddc25 ("Prune per-harness tool-mapping boilerplate", 2026-06-24) deliberately removed the skill-loading explainers and the generic action→tool tables from antigravity-tools.md and pi-tools.md, trimming each file "to the harness-specific notes that still carry weight (subagent dispatch, task tracking…)". That commit did not touch tests/, so the two content-assertion tests still grep for tokens the prune removed — view_file, IsSkillFile, run_command, grep_search (antigravity) and read/write/edit/bash (pi). With no CI in the repo, the breakage went unnoticed.

What does this PR change?

Updates the two orphaned tests to assert only the harness-specific mappings e7ddc25 kept (subagent dispatch and task tracking) instead of the boilerplate it removed. No reference, skill, or product content is touched — only the stale test assertions (and their now-inaccurate header comment / PASS string).

Is this change appropriate for the core library?

Yes. These are the repo's own plugin-infrastructure tests under tests/. No new dependency, no skill-behavior change, no third-party integration — it makes the existing suite green again on dev.

What alternatives did you consider?

  1. Restore the pruned content to the reference files so the tests pass unchanged — rejected: it directly contradicts e7ddc25's stated intent, and the restore-direction attempt PR Document Antigravity tool mappings for skill loading #1902 ("Document Antigravity tool mappings for skill loading") was already closed. Reintroducing the boilerplate would re-break what the maintainer deliberately trimmed.
  2. Delete the two content-assertion tests entirely — rejected: the reference files still exist and still document harness-specific mappings worth guarding. Trimming the assertions keeps meaningful coverage (subagent dispatch, task artifact, SKILL.md link) while dropping only the pruned tokens.

Does this PR contain multiple unrelated changes?

No. One root cause — e7ddc25 orphaned the per-harness mapping-content tests. Both files fail for the identical reason and are fixed the identical way. One problem.

Existing PRs

Environment tested

Harness Harness version Model Model version/ID
Claude Code 2.1.143 Claude Opus 4.8 (1M context) claude-opus-4-8[1m]

Tests run locally on macOS (bash + node --test).

New harness support (required if this PR adds a new harness)

N/A — this PR does not add a harness.

Evaluation

Not a behavior/skill change, so there is no eval-session delta to report. Verification instead:

  • Before (on dev): tests/antigravity/test-antigravity-tools.sh → exit 1; tests/pi/test-pi-extension.mjs → 5 pass / 1 fail.
  • After: antigravity → PASS; pi → 6 pass / 0 fail.
  • Also ran tests/antigravity/run-tests.sh (all passed) and the repo shell-lint tests (pass). Confirmed each surviving asserted token is actually present in the pruned reference file.

Rigor

  • If this is a skills change: … — N/A, not a skills change (tests only).
  • This change was tested adversarially, not just the happy path — verified failing-first on both dev and main, and confirmed the new assertions match the pruned references.
  • I did not modify carefully-tuned content (Red Flags table, rationalizations, "human partner" language) — no skill/prose content touched.

Human review

  • A human has reviewed the COMPLETE proposed diff before submission — @gaurav0107 reviewed the full two-file diff and approved it.

Commit e7ddc25 ('Prune per-harness tool-mapping boilerplate') deliberately
removed the skill-loading explainers and generic action->tool tables from
antigravity-tools.md and pi-tools.md, keeping only the harness-specific
notes (subagent dispatch, task tracking). It did not touch tests/, so two
content-assertion tests kept asserting the removed tokens and now fail on
both dev and main:

  - tests/antigravity/test-antigravity-tools.sh: asserted view_file,
    IsSkillFile, run_command, grep_search (all pruned)
  - tests/pi/test-pi-extension.mjs: asserted read/write/edit/bash (pruned)

Update both to assert only the surviving harness-specific mappings. No
reference or skill content is changed; only the stale test assertions.
@muunkky

muunkky commented Jul 14, 2026

Copy link
Copy Markdown

The antigravity half holds up. I stripped the mapping table out of antigravity-tools.md and your test correctly fails on invoke_subagent, which is the one token that only ever appears in the table. That's the right shape.

The pi half doesn't hold. All four of the new tokens also occur in the prose, so the assertion survives the thing it exists to detect. On your branch, with the pi mapping table deleted:

$ grep -v '^|' skills/using-superpowers/references/pi-tools.md > t && mv t skills/using-superpowers/references/pi-tools.md
$ node --test tests/pi/test-pi-extension.mjs
# pass 6
# fail 0

subagent matches the intro line, pi-subagents and Task both match the Subagents paragraph, and TODO.md matches the Task lists paragraph. Task is the one I'd look at twice: it matches "do not fabricate Task calls", a sentence whose entire job is to say Pi has no Task tool.

So the test passes on a pi-tools.md with no mapping table at all, which is the regression e7ddc25 caused in the first place.

Scoping the match to the table rows closes it:

const rows = text.split('\n').filter((line) => line.startsWith('|'));
assert.ok(rows.some((row) => /subagent/i.test(row)), 'mapping table documents subagent dispatch');
assert.ok(rows.some((row) => /todo|task/i.test(row)), 'mapping table documents task tracking');

That fails on the stripped file and passes on dev. Being straight about it: that snippet is the diff in my #1983, which does the pi half and nothing else. Take it or write your own — yours covers antigravity too, so it's the better PR either way and I'll close mine.

Disclosure: agent-assisted — Claude Opus 4.8 (claude-opus-4-8[1m]), Claude Code 2.1.208. Plugins: gitban (muunkky.github.io/gitban-site), frontend-design, skill-creator, claude-patent-creator-standalone. Not superpowers itself. Grounded in the runs above, against your branch at 8296a98.

The pi tokens (subagent, pi-subagents, Task, TODO.md) also appear in the
surrounding prose, so matching the whole file passed even with the mapping
table deleted — the exact regression this test exists to catch. Filter to
table rows (lines starting with '|') so the assertion fails when the table
is gone and passes on dev.

Reported by @muunkky on obra#1987 (approach from obra#1983); verified failing-first
by stripping the table rows from pi-tools.md.
@gaurav0107

Copy link
Copy Markdown
Author

Thanks @muunkky — this is exactly right, and I reproduced it before changing anything. Stripping the table rows from pi-tools.md and re-running the test, my token list passed 6/0, so the assertion was surviving the regression it exists to catch. Task matching "do not fabricate Task calls" was the clearest tell.

Pushed the table-scoped version (0873545) — filtering to rows that start with |, then asserting /subagent/i and /todo|task/i against those rows. Verified it now fails on a table-stripped pi-tools.md and passes on dev. The approach and snippet are from your #1983 — credited in the commit message.

I checked the antigravity half against the same test: it already fails when the table is stripped (invoke_subagent only appears in the row), so I left it as-is. Since this PR now covers both halves, feel free to close #1983 — and thanks for doing the adversarial pass.

Disclosure: agent-assisted — Claude Opus 4.8 (claude-opus-4-8[1m]), Claude Code 2.1.143, reviewed and approved by @gaurav0107. Plugins unchanged from the PR description.

muunkky added a commit to muunkky/superpowers that referenced this pull request Jul 14, 2026
Asked whether another team could pick this up. Four things say no, and
three of them are live bugs for us too.

1. The PR template was a fabricated-attestation generator. The disclosure
   table shipped with its answers pre-filled, including 'Human partner who
   reviewed this diff: Cameron Rout (@muunkky)'. Hand this to another team
   and their first PR attests a human who never read their diff — the
   obra#1906 kill, except the account is real. It is also why our own four PRs
   under-disclosed: the template had answers, so nobody read the machine.
   Every cell now says how to read it, and the reviewer cell is empty by
   construction.

2. fork-setup.sh did not exist. SKILL.md told you to keep one and showed
   its contents in a code block, and nobody had ever created it — so on a
   fresh clone there is no .git/info/exclude and no fetch-only upstream,
   and 'git add -A' stages .gitban/ and docs/prds/ into the upstream PR.
   The guardrail the entire code/artifact split rests on was prose.
   Documented is not shipped. It now exists, is idempotent, and proves the
   ignore works with a probe file instead of asserting it.

3. Both audit scripts hardcoded ME=muunkky. Run by anyone else they
   enumerate our threads, find none of theirs, and print 'clean'. SKILL.md
   makes exactly this argument about check-upstream's manifest and then
   hardcodes the account two lines later. Identity now comes from the
   origin/upstream remotes; missing remotes exit non-zero.

4. The gitban-pr overlay had drifted from its appended copy in SKILL.md.
   Resynced.

Verified: fork-setup rescues a clone that was staging .gitban/card.md and
docs/prds/x.md — after it, both are invisible. All three scripts pass
shellcheck; the sweep still finds our four PRs and now sees the obra#1987
thread.
muunkky added a commit to muunkky/superpowers that referenced this pull request Jul 14, 2026
Asked whether we were still confident in the open PRs. Checked instead of
answering, and obra#1982 — the strongest of the three — was resting on
'0 of 3 on dev, 3 of 3 with the change' with no transcript on disk.

Re-ran it. n=8 per arm, trees verified to differ by exactly our 9 lines:
dev releases the finished subagents 3 of 8 (two agents call TaskStop as an
explicit numbered step), ours 8 of 8. Both published figures were wrong in
our favour. obra re-runs eval claims across independent sessions; he would
have found this.

The change still works, but only under the question it actually teaches:
'names the slot as the reason' is 0 of 8 on dev vs 7 of 8 with it. dev's
agents do sometimes stop subagents — for context hygiene, never for
capacity. That is a better argument than the one we were making, and it is
true.

Also logged: I miscounted this eval three times with greps, in both
directions, including calling the adversarial arm a total failure when it
was a total success. Every miscount was caught only by opening the
transcripts. That is the same defect we reported to gaurav0107 on obra#1987 the
same morning. Grep proposes; reading decides.

Body corrected on-thread with the retraction stated plainly.
cwest added a commit to cwest/superpowers that referenced this pull request Jul 15, 2026
e7ddc25 (obra#1847) intentionally trimmed the skill-loading explainer and the
generic action→tool table from antigravity-tools.md, pruning to "the
harness-specific notes that still carry weight." But tests/antigravity still
asserted the removed tokens (view_file, IsSkillFile, SKILL.md, run_command,
grep_search), so the antigravity suite has been red on dev ever since.

Realign the test with the slimmed reference: assert only what the doc still
documents — subagent dispatch (invoke_subagent, self/research types), the
write_to_file/replace_file_content tools, and the task artifact mechanism —
plus the SKILL.md Platform Adaptation link. Drop the assertions for the
intentionally-removed tokens.

Agents don't need the removed details: verified with a clean agy run where the
plugin installs, skills load, and brainstorming auto-triggers without them.

Related: obra#1987 independently proposed realigning these assertions (antigravity
+ pi); this is the narrower antigravity-only change.
@cwest

cwest commented Jul 15, 2026

Copy link
Copy Markdown

Backing this one — and I've closed my overlapping PR (#1992) in favor of it.

I hit the same red antigravity suite on dev and first went the other way:
restoring the pruned rows in antigravity-tools.md. Then I re-read #1847
that content was cut deliberately ("the harness-specific notes that still
carry weight") — so re-adding it fights the intent. Realigning the stale
tests, as you've done, is the right call.

Reproduced locally:

  • tests/antigravity/run-tests.sh → green
  • the pi mapping case flips red→green against the slimmed pi-tools.md
  • a clean agy run with the plugin installed loads skills and auto-triggers
    brainstorming without those mapping lines present — the empirical version
    of "agents mostly don't need 'em"

Honest caveat: I couldn't run pi extension cases 2–5 headlessly (they load the
TS extension and my setup wouldn't), so I verified the mapping assertion, not
those four.

Thanks for taking the pi half down to the table rows too.

@obra
obra merged commit 7a81eb7 into obra:dev Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants