Skip to content

feat(claim-evidence)!: refuse calibrated false-refutation shapes at intake - #145

Merged
drewstone merged 3 commits into
mainfrom
feat/evidence-intake-integrity
Aug 17, 2026
Merged

feat(claim-evidence)!: refuse calibrated false-refutation shapes at intake#145
drewstone merged 3 commits into
mainfrom
feat/evidence-intake-integrity

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

The hole

The record-time gate already refuses three evidence shapes that a blind grader cannot use: no check, a constant-emitter check, and no expected value. A downstream discovery fleet measured five more failure shapes after that boundary shipped:

  • 4 mathematically correct claims died because a multiline expectation can never match one contiguous-substring comparison.
  • 2 correct claims died because a multi-value expectation became false when the checker inserted one token between values.
  • 1 retained campaign recorded literal \\n sequences as an enormous first shell word.
  • Bash syntax errors were admitted and discovered only after the author disappeared.
  • A 299-run autopsy reduced ten reported grader failures to the same root: intake stored a command it had never executed in the grader's working directory and environment.

These are evidence-integrity failures, not research policy. A gate that admits a check the grader is guaranteed to refuse is not a gate.

Change

  • Extend assertGradeableEvidence and gradeFor with the same static refusal vocabulary for:
    • multiline expectations;
    • expectations carrying three or more key=value tokens;
    • the >200-character first-word escaped-newline shape.
  • Add opt-in verifyGradeableEvidence:
    • runs bash -n first;
    • executes the exact check in caller-supplied cwd and environment;
    • returns the existing calibrated ClaimGrade rather than inventing a second verdict system.
  • Dynamically import node:child_process only when the verifier is invoked, preserving importability for consumers that never execute shell evidence.
  • State the security boundary explicitly: callers must sandbox untrusted commands before opting in.

Compatibility

This is a breaking change at the record boundary: rung-4/5 evidence that was previously accepted can now throw UncheckableClaimError. That is intentional and follows the package's prior major-release convention for stronger evidence admission.

Below rung 4, behavior remains unchanged. A check that parses and genuinely refutes a claim still returns contradicted; an environment failure remains unrunnable.

Tests

The new suite proves record/grade wording parity, each static refusal, no tightening below the threshold, bash syntax refusal, exact cwd/environment execution, successful verification, and a parsed check that refutes its claim.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — b635c379

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

tangletools · auto-approval · reason: drewstone_author · 2026-08-17T01:45:40Z

@drewstone
drewstone merged commit 6a32f57 into main Aug 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants