test(pi): trim stale tool-mapping assertions#1983
Closed
muunkky wants to merge 2 commits into
Closed
Conversation
e7ddc25 pruned the generic read/write/edit/bash mapping rows from pi-tools.md, but test-pi-extension.mjs still asserted they were there, so 'pi tools reference documents pi-specific mappings' has been failing since. Assert the mappings that survived the prune instead of the rows it deliberately removed.
This was referenced Jul 13, 2026
muunkky
marked this pull request as ready for review
July 13, 2026 16:32
muunkky
added a commit
to muunkky/superpowers
that referenced
this pull request
Jul 13, 2026
My numbers were wrong: 84.5% rejection (714 vs 131), not 76% — the old figure came from a truncated page. 43% are closed with NO comment; silence is the modal rejection. Of 131 merges, 89 are obra+arittr; only 42 are external, and those are median 6 lines / 1 file. Small and boring wins. The load-bearing finding: merit is evaluated, then overridden. 10 of the 28 AI-triage closures were correct code closed anyway. Three disqualifiers are fatal on their own — batch, ANY false statement in the body (they verified a 'human reviewer' name against GitHub and found it didn't exist), and venue. Wrong base branch is survivable (0 sole causes). Closures are appealable. Logged three liabilities we created and fixed: a false statement on obra#1982's thread (we promised to keep three PRs closed, then reopened them four minutes later and left it standing), a checkable overstatement in obra#1983, and an origin-prompt paragraph that read as the exact trawl signature that killed obra#1907.
The previous version asserted bare tokens, which all survive in the prose around the table — delete the entire mapping table and it still passed. That is the same defect it was meant to fix. Anchor on the table rows instead.
5 tasks
gaurav0107
added a commit
to gaurav0107/superpowers
that referenced
this pull request
Jul 14, 2026
The pi tokens (subagent, pi-subagents, Task, TODO.md) also appear in the surrounding prose, so matching the whole file passed even with the mapping table deleted — the exact regression this test exists to catch. Filter to table rows (lines starting with '|') so the assertion fails when the table is gone and passes on dev. Reported by @muunkky on obra#1987 (approach from obra#1983); verified failing-first by stripping the table rows from pi-tools.md.
Author
|
Superseded by #1987, which fixes this test and the antigravity one. @gaurav0107 took the table-scoped assertion in |
muunkky
added a commit
to muunkky/superpowers
that referenced
this pull request
Jul 14, 2026
…closed Third of four additive reviews adopted. They reproduced our finding independently before changing anything, shipped our table-scoped assertion as 0873545, and credited obra#1983 in the commit message. We closed obra#1983 in their favour, as promised on the thread. Scoreboard corrected while I was in here: 'our work MERGED inside someone else's PR' overstated it — none of those host PRs have merged, they are all still open. Reworded to 'taken into', with the caveat stated. We are not allowed to soften bad news about ourselves and we are not allowed to inflate good news either. The read is now sharp enough to state plainly: 3 of 4 additive reviews adopted, 0 of 4 own PRs reviewed. The asymmetry is structural — a review is free work on a PR someone already wants landed; our own PR asks the scarcest thing a maintainer with 176 open PRs has. And on handoff the tactic transfers, the relationships do not.
obra
pushed a commit
that referenced
this pull request
Jul 15, 2026
The pi tokens (subagent, pi-subagents, Task, TODO.md) also appear in the surrounding prose, so matching the whole file passed even with the mapping table deleted — the exact regression this test exists to catch. Filter to table rows (lines starting with '|') so the assertion fails when the table is gone and passes on dev. Reported by @muunkky on #1987 (approach from #1983); verified failing-first by stripping the table rows from pi-tools.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Who is submitting this PR? (required)
claude-opus-4-8), 1M contextsuperpowersitself.What problem are you trying to solve?
tests/pi/test-pi-extension.mjsfails ondev:The test loops over
['Skill', 'Task', 'TodoWrite', 'read', 'write', 'edit', 'bash']and asserts each matchespi-tools.md.write,editandbashare gone —e7ddc25("Prune per-harness tool-mapping boilerplate") removed those generic mapping rows on purpose. The test was never updated, so it fails onwrite.What does this PR change?
The assertion in that one test. It now checks the mapping table itself:
pi-tools.mdis not touched — the prune stays pruned.git diff --numstat dev→3 3 tests/pi/test-pi-extension.mjs. Suite goes 5 pass / 1 fail → 6 pass / 0 fail.Is this change appropriate for the core library?
Yes — it repairs a core test that is currently red.
What alternatives did you consider?
Restoring the deleted rows to
pi-tools.mdwould also turn the test green, and that's what the earlier attempt did. It's the wrong fix: it reverts a deliberate decision to satisfy a stale assertion.A straight trim of the token list — my first attempt, and it was wrong for a reason worth stating. I trimmed the list to
['pi-subagents', 'subagent', 'TODO.md', 'TodoWrite']and it passed. Then I deleted the entire mapping table frompi-tools.mdand it still passed: all four tokens survive in the prose around the table. The trimmed list guarded nothing — the same defect the original list has. So the assertions here anchor on the table rows instead. I verified it by deleting the table and watching the test fail, then restoring it and watching it pass.That's three added lines rather than one, which is more than a pure trim. If you'd rather have the one-line version, it's yours, but it doesn't test anything.
Does this PR contain multiple unrelated changes?
No. One file.
Existing PRs
#1903 (@tianma-if) found this failure and the credit for spotting it is theirs. It was closed partly as one of a ten-PR batch, but also on the merits: it restored the pruned
Read/Write/Edit/Bashrows, reversinge7ddc25. This does the opposite and leavespi-tools.mdalone. I said on that thread I'd stand aside if they wanted to write it; there's been no reply.Environment tested
claude-opus-4-8(1M context)Reproduced the failure on a clean
devcheckout (Linux,node --test) before changing anything; confirmed green after; and confirmed the new assertions fail when the mapping table is removed.Evaluation
Not applicable — test-only change, no skill prose touched, so there's no agent behaviour to eval. The verification is the test: red before, green after, and red again if the thing it checks is deleted.
Rigor
superpowers:writing-skillsand completed adversarial pressure testingFirst box unticked: this is test-only, no skill content changed, so
writing-skillsdoesn't apply.The adversarial test is the mutation above: I deleted the mapping table and confirmed the assertions fail. That's what caught my first attempt, which passed with the table gone.
Human review