Skip to content

test(pi): trim stale tool-mapping assertions#1983

Closed
muunkky wants to merge 2 commits into
obra:devfrom
muunkky:fix/pi-tool-mapping-test
Closed

test(pi): trim stale tool-mapping assertions#1983
muunkky wants to merge 2 commits into
obra:devfrom
muunkky:fix/pi-tool-mapping-test

Conversation

@muunkky

@muunkky muunkky commented Jul 13, 2026

Copy link
Copy Markdown

Who is submitting this PR? (required)

Field Value
Your model + version Claude Opus 4.8 (claude-opus-4-8), 1M context
Harness + version Claude Code 2.1.207
All plugins installed gitban (muunkky.github.io/gitban-site), frontend-design, skill-creator, claude-patent-creator-standalone. Not superpowers itself.
Human partner who reviewed this diff Cameron Rout (@muunkky)

What problem are you trying to solve?

tests/pi/test-pi-extension.mjs fails on dev:

$ node --test tests/pi/test-pi-extension.mjs
not ok 6 - pi tools reference documents pi-specific mappings
  name: 'AssertionError'
# pass 5
# fail 1

The test loops over ['Skill', 'Task', 'TodoWrite', 'read', 'write', 'edit', 'bash'] and asserts each matches pi-tools.md. write, edit and bash are gone — e7ddc25 ("Prune per-harness tool-mapping boilerplate") removed those generic mapping rows on purpose. The test was never updated, so it fails on write.

What does this PR change?

The assertion in that one test. It now checks the mapping table itself:

-  for (const expected of ['Skill', 'Task', 'TodoWrite', 'read', 'write', 'edit', 'bash']) {
-    assert.match(text, new RegExp(expected));
-  }
+  const rows = text.split('\n').filter((line) => line.startsWith('|'));
+  assert.ok(rows.some((row) => /subagent/i.test(row)), 'mapping table documents subagent dispatch');
+  assert.ok(rows.some((row) => /todo|task/i.test(row)), 'mapping table documents task tracking');

pi-tools.md is not touched — the prune stays pruned. git diff --numstat dev3 3 tests/pi/test-pi-extension.mjs. Suite goes 5 pass / 1 fail → 6 pass / 0 fail.

Is this change appropriate for the core library?

Yes — it repairs a core test that is currently red.

What alternatives did you consider?

Restoring the deleted rows to pi-tools.md would also turn the test green, and that's what the earlier attempt did. It's the wrong fix: it reverts a deliberate decision to satisfy a stale assertion.

A straight trim of the token list — my first attempt, and it was wrong for a reason worth stating. I trimmed the list to ['pi-subagents', 'subagent', 'TODO.md', 'TodoWrite'] and it passed. Then I deleted the entire mapping table from pi-tools.md and it still passed: all four tokens survive in the prose around the table. The trimmed list guarded nothing — the same defect the original list has. So the assertions here anchor on the table rows instead. I verified it by deleting the table and watching the test fail, then restoring it and watching it pass.

That's three added lines rather than one, which is more than a pure trim. If you'd rather have the one-line version, it's yours, but it doesn't test anything.

Does this PR contain multiple unrelated changes?

No. One file.

Existing PRs

#1903 (@tianma-if) found this failure and the credit for spotting it is theirs. It was closed partly as one of a ten-PR batch, but also on the merits: it restored the pruned Read/Write/Edit/Bash rows, reversing e7ddc25. This does the opposite and leaves pi-tools.md alone. I said on that thread I'd stand aside if they wanted to write it; there's been no reply.

Environment tested

Harness Harness version Model Model version/ID
Claude Code 2.1.207 Claude Opus 4.8 claude-opus-4-8 (1M context)

Reproduced the failure on a clean dev checkout (Linux, node --test) before changing anything; confirmed green after; and confirmed the new assertions fail when the mapping table is removed.

Evaluation

Not applicable — test-only change, no skill prose touched, so there's no agent behaviour to eval. The verification is the test: red before, green after, and red again if the thing it checks is deleted.

Rigor

  • If this is a skills change: I used superpowers:writing-skills and completed adversarial pressure testing
  • This change was tested adversarially, not just on the happy path
  • I did not modify carefully-tuned content (Red Flags table, rationalizations, "human partner" language) without extensive evals

First box unticked: this is test-only, no skill content changed, so writing-skills doesn't apply.

The adversarial test is the mutation above: I deleted the mapping table and confirmed the assertions fail. That's what caught my first attempt, which passed with the table gone.

Human review

  • A human has reviewed the COMPLETE proposed diff before submission

e7ddc25 pruned the generic read/write/edit/bash mapping rows from
pi-tools.md, but test-pi-extension.mjs still asserted they were there,
so 'pi tools reference documents pi-specific mappings' has been failing
since. Assert the mappings that survived the prune instead of the rows
it deliberately removed.
@muunkky muunkky closed this Jul 13, 2026
@muunkky muunkky reopened this Jul 13, 2026
@muunkky
muunkky marked this pull request as ready for review July 13, 2026 16:32
muunkky added a commit to muunkky/superpowers that referenced this pull request Jul 13, 2026
My numbers were wrong: 84.5% rejection (714 vs 131), not 76% — the old figure
came from a truncated page. 43% are closed with NO comment; silence is the
modal rejection. Of 131 merges, 89 are obra+arittr; only 42 are external, and
those are median 6 lines / 1 file. Small and boring wins.

The load-bearing finding: merit is evaluated, then overridden. 10 of the 28
AI-triage closures were correct code closed anyway. Three disqualifiers are
fatal on their own — batch, ANY false statement in the body (they verified a
'human reviewer' name against GitHub and found it didn't exist), and venue.
Wrong base branch is survivable (0 sole causes). Closures are appealable.

Logged three liabilities we created and fixed: a false statement on obra#1982's
thread (we promised to keep three PRs closed, then reopened them four minutes
later and left it standing), a checkable overstatement in obra#1983, and an
origin-prompt paragraph that read as the exact trawl signature that killed
obra#1907.
The previous version asserted bare tokens, which all survive in the prose
around the table — delete the entire mapping table and it still passed. That
is the same defect it was meant to fix. Anchor on the table rows instead.
gaurav0107 added a commit to gaurav0107/superpowers that referenced this pull request Jul 14, 2026
The pi tokens (subagent, pi-subagents, Task, TODO.md) also appear in the
surrounding prose, so matching the whole file passed even with the mapping
table deleted — the exact regression this test exists to catch. Filter to
table rows (lines starting with '|') so the assertion fails when the table
is gone and passes on dev.

Reported by @muunkky on obra#1987 (approach from obra#1983); verified failing-first
by stripping the table rows from pi-tools.md.
@muunkky

muunkky commented Jul 14, 2026

Copy link
Copy Markdown
Author

Superseded by #1987, which fixes this test and the antigravity one. @gaurav0107 took the table-scoped assertion in 0873545 — verified it now fails on a pi-tools.md with the mapping table stripped and passes on dev, which is the regression the old token list was missing.

@muunkky muunkky closed this Jul 14, 2026
muunkky added a commit to muunkky/superpowers that referenced this pull request Jul 14, 2026
…closed

Third of four additive reviews adopted. They reproduced our finding
independently before changing anything, shipped our table-scoped assertion
as 0873545, and credited obra#1983 in the commit message. We closed obra#1983 in
their favour, as promised on the thread.

Scoreboard corrected while I was in here: 'our work MERGED inside someone
else's PR' overstated it — none of those host PRs have merged, they are all
still open. Reworded to 'taken into', with the caveat stated. We are not
allowed to soften bad news about ourselves and we are not allowed to
inflate good news either.

The read is now sharp enough to state plainly: 3 of 4 additive reviews
adopted, 0 of 4 own PRs reviewed. The asymmetry is structural — a review is
free work on a PR someone already wants landed; our own PR asks the
scarcest thing a maintainer with 176 open PRs has. And on handoff the
tactic transfers, the relationships do not.
obra pushed a commit that referenced this pull request Jul 15, 2026
The pi tokens (subagent, pi-subagents, Task, TODO.md) also appear in the
surrounding prose, so matching the whole file passed even with the mapping
table deleted — the exact regression this test exists to catch. Filter to
table rows (lines starting with '|') so the assertion fails when the table
is gone and passes on dev.

Reported by @muunkky on #1987 (approach from #1983); verified failing-first
by stripping the table rows from pi-tools.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant