Skip to content

Close the P0 completeness gaps: oracles, network, auth, navigation, and a re-runnable suite - #21

Open
wlu03 wants to merge 23 commits into
mainfrom
p0-completeness
Open

Close the P0 completeness gaps: oracles, network, auth, navigation, and a re-runnable suite#21
wlu03 wants to merge 23 commits into
mainfrom
p0-completeness

Conversation

@wlu03

@wlu03 wlu03 commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Implements the twelve gaps from the repo audit. Test suite goes 360 → 589 passing, ruff clean, tool reference regenerated.

What was actually broken

Three of these were features that already existed and silently did nothing:

  • ReproSpec.oracle was never populated. build_repro_spec was called in three places, always without an oracle, and report_issue had no way to accept one — so _eval_oracle always returned None and every re-verification fell back to digit-normalized summary matching. report_issue now takes assertions, and auto-filed findings inherit the last set evaluated by check_assertions.
  • audit_dom no-opped on the entire local path. LocalElectronAdapter never overrode it, so on the default no-API-key config the deterministic-evidence tool returned {} and read as a clean pass. Now implemented over CDP, with axe-core 4.10.2 vendored and injected as source text so an app's script-src CSP can't silently kill it. axe_ran / axe_error are forwarded to the MCP boundary — an empty violations list can no longer be mistaken for a pass.
  • The fix loop was hidden. update_finding_status and verify_fix were in ADVANCED_TOOLS, so under the default core profile an agent could file a finding but never close it.

New capability

  • Network — CDP Network domain captured as a bounded, failure-preferring channel; failed requests and 5xx file as findings; AssertionKind.NETWORK is a real check instead of a hardcoded inconclusive.
  • Auth — cookies and web storage captured and replayed (capture_state / seed_state), so a run starts logged in instead of burning its iteration budget on a login form.
  • Navigationnavigate / back / forward / reload, plus hover, right_click, drag destinations, and scroll direction. drag was previously unreachable (_resolve raised unconditionally) and scroll direction never left the tool boundary.
  • Viewportset_viewport, which makes the responsive checks in the attack catalog executable for the first time.
  • Re-runnable suites — plans persist per repo with per-scenario history and run_plan, so "re-run my checkout suite after this change" is one call.
  • Honest ledger — absence from the latest run is absent/not_run, not verified. Only an explicit sign-off is verified.

The bug worth reading the diff for

The new action types round-tripped into ReproStep but had no branch in replay_spec, so they counted as replayed without being performed. verify_fix could report a bug fixed while it was still there, and run_plan could report a scenario PASSED having done nothing. Fixed, with the invariant documented — plan.run_scenario and reverify.replay_spec share one executor, so any future action type must get a branch there.

Known follow-ups (not in this PR)

  1. AnthropicDriver has no stop_reason handling. Opus 5 has thinking on by default and max_tokens caps thinking + response together; at 2048 a dense screen can truncate, degrade to "{}", and be parsed as a no-op step. A refusal does the same. The brain can go quietly blind mid-run.
  2. get_adapter now raises when E2B_API_KEY is unset for sandboxed web/Electron — a deferred failure becomes a hard one at construction.
  3. Driver default claude-sonnet-4-6claude-opus-5: ~1.7x per token, and a separate rate-limit bucket. Wants a CHANGELOG line.
  4. Vendored axe-core has no NOTICE / third-party entry (MPL-2.0 header is intact in the file).
  5. The E2B sandbox path still loads axe from the CDN — it reports axe_error now, but a CSP'd app there still gets no axe results.

🤖 Generated with Claude Code

wlu03 added 23 commits July 27, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant