ci: three gates could pass without checking anything, and only one had a way to notice

[wallstop]

What happened

Session 221 hit three gates that report success while checking nothing, in one session, by
accident. None of them was found by running it — all three are silent by construction, because the
tree they scan is clean, and "found nothing" and "looked at nothing" produce identical output.

1. lint-tests.ps1 -Paths a b c linted only a. pwsh -File binds the first token to -Paths
   and drops the rest unless a ValueFromRemainingArguments sibling catches them. Six scripts were
   missing it. .llm/context.md documented that exact multi-file invocation. Measured: -Paths
   with all 1,118 test files reported 0 advisories where the same files report 72.
   Fixed in Order every comparison; fix three silent gates #555, enforced by a new PWS005.
2. A pre-filter in test-shell-portability.sh disabled all six sections it was added to.
   $( ) strips trailing newlines, so the last candidate never matched the membership test — and
   the probe sorted last for every token. Caught only by injecting a violation per section and
   diffing against the unmodified script. Fixed in Order every comparison; fix three silent gates #555, with a self-check in the helper.
3. A --fix in the new comparison-direction linter corrupted an interpolated string. This one
   was caught, but only because the result did not compile. Had it compiled, nothing would have
   said so.

Why this is its own ticket

Each fix is local, and the class is not. The property that matters is: can this gate go red? For
a scanner over a clean corpus, a green run is not evidence that it can. Three of our gates could not,
and we have on the order of sixty.

Prior art in this repository is the same shape and was written down at the time:
test-shell-portability.sh already carried a comment saying each replaced pattern "was verified to
select exactly the same lines as the grep it replaces, over the real corpus PLUS a synthetic line
per pattern -- the corpus contains no violations, so agreement on it alone would only prove the
absence of false positives, not that the checks still fire." That is exactly the discipline that was
then not applied to the next change to the same file.

Ask

1. Inventory the gates whose corpus is clean by construction — every linter in lint:repo and
   validate:tests that scans the tree and reports violations. For each, does a self-test inject a
   violation and assert it is reported, or does it only assert a clean run passes?
2. Add the missing red halves. The pattern already exists in several self-tests
   (test-lint-csharp-naming.ps1, test-lint-comparison-direction.js, the new PWS005 probe):
   write a fixture that violates, assert non-zero exit and the file/line in the message.
3. Consider a meta-check: a linter registered in lint:repo with no corresponding self-test that
   can make it fail is itself reportable. test-run-repo-lint.js already asserts "no linter in
   scripts/ has been left unreachable"; "no linter has been left unfalsifiable" is the same idea
   one level up.

Not blocking anything — all three known instances are fixed. This is about the ones nobody has
tripped over yet.

[wallstop]

A fourth instance, and the most instructive one: I reintroduced the class while fixing it.

The fix for (1) replaced Resolve-Path -LiteralPath $p with
[IO.Path]::GetFullPath((Join-Path (Get-Location).Path $p)) — to make the -Paths branch linear
instead of quadratic. Join-Path '/repo' '/abs/path' yields /repo/abs/path, so an absolute
caller path was mangled into one that does not exist, and then skipped silently. Measured: an
absolute path reported 0 advisories where the same file reports 6. Resolve-Path had handled both
forms; IsPathRooted restores that.

It survived the verification that caught the original bug, because every check I ran passed
relative paths — which is what git ls-files and agent:preflight produce. The test that found
it was asking "what other shape can a caller use?" rather than re-running the one that had just
gone green.

That sharpens the ask in this issue. A red half is necessary and not sufficient: the fixture has to
cover the input shapes, not just the violation. For a -Paths script that is at least
{relative, absolute, multiple, mixed} — all four are one line each and the last two are where both
bugs lived.

[wallstop]

A fifth instance, found by review rather than by me — and it is the sharpest one.

Rewriting scripts/tests/test-random-quality-outcomes.mjs for two stream widths, I replaced
everything from one helper to end-of-file and silently dropped four assertion blocks: the
quality-vs-source attribute sync, the expected-fail rank guard, the missing-report assertion, and
the CLI spawn path.

That last one exists for exactly the reason this issue does. Its own comment, written in an earlier
session:

Everything above imports the helpers directly, which leaves the command-line path — the only path
the workflow actually uses — unexercised. A direct-run guard that silently fails produces no
verdict at all, and a step that prints nothing reads exactly like a clean battery, so the CLI
is spawned here rather than trusted.

I deleted the guard against this issue's own failure mode, inside the session that filed this issue,
while writing it. The remaining test still printed a clean line, so nothing said so. Cursor Bugbot
caught it; the full CI matrix could not have, because the assertions removed are precisely the ones
that make a silent pass visible. Restored in eeaa19f, now per width.

The ratio is the finding

Five instances. Three were mine, all introduced while removing this exact failure mode. Two of
those three were caught by something other than my own verification — a C# compiler and a review
bot. Only one I found myself, and only because I ran a control.

So the ask in this issue should be weighted toward mechanisms over discipline. "Remember to add a red
half" is what failed here, five times, in the hands of someone actively thinking about it. The
meta-check proposed above — a linter registered in lint:repo with no self-test that can make it
fail is itself reportable — is worth more than the inventory, because it is the only item on the
list that does not depend on anyone remembering.

[wallstop]

Ten gates that could not go red, and the fixtures that prove they now can

PR #557.

One could not fail under any input

scripts/extract-code-samples.js:

// Exit with error if no samples found (CI check)
if (csharpSamples.length === 0) {
  console.warn("\n⚠️ Warning: No C# code samples found in documentation!");
  process.exit(0);
}

The comment states the intent and the line under it does the opposite. There was no other failure
path in the file — no throw, no process.exitCode. A deleted docs/ tree, a renamed fence
language, or a walk that stopped descending all reported success.

It now exits 1, takes a CODE_SAMPLES_ROOT override so a fixture tree can be pointed at it, and
refuses a scan root that does not exist. Seven-case self-test, four of them red.

Eight reported success on an empty corpus

lint-license-headers, lint-no-regions, lint-csharp-naming, lint-asmdef and
lint-conditional-call-chains share if (-not (Test-Path $root)) { continue }, so renaming
Runtime/ turned five checks off at once and every one printed a success message. In four of them
the skip was visible only under -VerboseOutput. Beyond that:

• lint-conditional-call-chains exited 0 with "OK: no [Conditional] methods found" — in a
  repository that declares them.
• lint-duplicate-usings exited 0 on "No C# files to scan."
• lint-dependabot exited 0 when its config was deleted.
• audit-license-years.sh printed "All files have correct copyright years!" for zero files.

All eight now fail closed. An explicit filter that matches nothing is still a pass — agent:preflight
passes the changed files and there may be none — so only the repository-wide walk finding nothing is
an error.

PWS005 — the rule written to prevent this — was itself untested

scripts/tests/test-lint-pwsh-invocations.ps1 covered PWS001-PWS004 and contained zero
occurrences of PWS005. It also matched the parameter named Paths, and the name is not what
makes pwsh -File drop arguments — the array-ness is. Four live scripts carried the same defect
under other names:

scripts/run-unity-tests.ps1               Platforms, AdditionalArguments
scripts/unity/ensure-editor.ps1           RequiredEditorPayloadRelativePath
scripts/unity/maintain-windows-runner.ps1 RunnerMaintenanceUnityVersions
scripts/unity/run-ci-tests.ps1            AdditionalScriptingDefines

maintain-windows-runner.ps1 is already invoked through -File by a contract test with a single
version token — one token away from biting. Each now catches the remainder and refuses it rather
than guessing which array parameter it belonged to; with two array parameters there is no safe guess.

And a script the AST parser could not read was continued rather than reported, so a syntax error
bought exemption from the whole rule. That is now a PWS005 violation naming the parse error.

Each red half was read, not just counted

A red half that is red for the wrong reason proves nothing, so every new guard was run and its
output inspected:

[lint-license-headers] Source root not found: Runtime. If it moved, update $sourceRoots in the same commit.
[lint-no-regions] ERROR: source root not found: Runtime. ...
[lint-csharp-naming] ERROR: source root not found: Runtime. ...
[lint-asmdef] ERROR: Source root not found: Runtime. ...
[lint-conditional-call-chains] ERROR: scan root not found: Runtime. ...
[lint-duplicate-usings] ERROR: the repository-wide scan found no C# files, so a pass here would mean nothing.
[audit-license-years] ERROR: no .cs files were audited, so a pass here would mean nothing.
[lint-dependabot] ERROR: dependabot.yml not found at: .../.github/dependabot.yml. ...

Fixtures added: 7 for the extractor (4 red), 6 for PWS005 (3 red), 8 for the empty-corpus class.

The new guards then reddened two things, both mine

Worth recording because it is the same lesson: the csharp-naming fixture repo contained only
Runtime/, so the missing-root refusal broke its own self-test. The fixture was made
representative rather than the guard relaxed. And csharp-naming/no-regions guarded root
existence but not file count, so all three roots present and empty would still have passed; both now
refuse a zero-file repository-wide walk.

Still class C

The audit that produced this list found roughly 25 gates with no self-test at all. The ten fixed
here are the ones where a silent pass was demonstrable. verify-shipped-analyzers.js is the next
one worth doing: moved iterates only after.keys(), so a deleted shipped DLL is structurally
invisible, and a missing Runtime/Analyzers makes the whole check a permanent no-op.

[wallstop]

The meta-check, and it nearly shipped with this issue's own defect

PR #563 delivers ask 3 ("consider a meta-check") and one of the missing red halves from ask 2.

The check

test-run-repo-lint.js gains no linter in scripts/ has been left unfalsifiable: every
scripts/lint-* and scripts/validate-* must have a self-test that invokes it and that a
registered suite runs. Seven are carried on an explicit allowlist, each naming
#562, and the list cannot rot
in either direction — an entry whose file is gone fails, an entry that has since grown coverage
fails, and an entry without a #NNN reference fails.

A predicate I wrote and deleted

The first version also required the self-test to name a red half — a case description matching
fail|reject|violation|.... Every one of the 32 self-tests that exist already matches it. It
discriminates nothing, so it would have been one more check that cannot go red, inside the test
whose entire subject is checks that cannot go red. Removed rather than kept as decoration.

The defect it nearly had

The first draft scanned every file under scripts/tests/ for a mention of the linter's file
name — including test-run-repo-lint.js itself, which names linters in two allowlists rather
than running them. Every allowlisted linter read as covered by the very list excusing it, and
lint-unity-test-modules.ps1 read as covered because the pre-existing runElsewhere map mentions
it. Excluding the two registry files moved the real gap from 6 to 7.

It was caught only because the allowlist carries a red half of its own — "an entry that has since
gained coverage must leave" — which fired on all six entries at once. Without that assertion the
check would have been green, wrong, and indistinguishable from correct.

.llm/context.md now carries the general form: a gate that asks "is this covered" must exclude
the files that merely name the thing.

One red half added rather than allowlisted

lint-changelog.ps1 was the eighth uncovered linter, and the reason was structural: it read
CHANGELOG.md from $PSScriptRoot with no way to hand it anything else. It takes -ChangelogPath
now, and scripts/tests/test-lint-changelog.ps1 drives ten red halves — one per rule that exits
non-zero — each asserting its own message rather than the exit code alone, so a fixture cannot
pass by tripping a different rule than the one it is named for.

All four branches of the meta-check were proved red on the real tree and then green.