Close the P0 completeness gaps: oracles, network, auth, navigation, and a re-runnable suite - #21
Open
wlu03 wants to merge 23 commits into
Open
Close the P0 completeness gaps: oracles, network, auth, navigation, and a re-runnable suite#21wlu03 wants to merge 23 commits into
wlu03 wants to merge 23 commits into
Conversation
…st without E2B_API_KEY
…icit unsupported signal
…and scroll direction intact
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the twelve gaps from the repo audit. Test suite goes 360 → 589 passing, ruff clean, tool reference regenerated.
What was actually broken
Three of these were features that already existed and silently did nothing:
ReproSpec.oraclewas never populated.build_repro_specwas called in three places, always without an oracle, andreport_issuehad no way to accept one — so_eval_oraclealways returnedNoneand every re-verification fell back to digit-normalized summary matching.report_issuenow takesassertions, and auto-filed findings inherit the last set evaluated bycheck_assertions.audit_domno-opped on the entire local path.LocalElectronAdapternever overrode it, so on the default no-API-key config the deterministic-evidence tool returned{}and read as a clean pass. Now implemented over CDP, with axe-core 4.10.2 vendored and injected as source text so an app'sscript-srcCSP can't silently kill it.axe_ran/axe_errorare forwarded to the MCP boundary — an empty violations list can no longer be mistaken for a pass.update_finding_statusandverify_fixwere inADVANCED_TOOLS, so under the defaultcoreprofile an agent could file a finding but never close it.New capability
Networkdomain captured as a bounded, failure-preferring channel; failed requests and 5xx file as findings;AssertionKind.NETWORKis a real check instead of a hardcodedinconclusive.capture_state/seed_state), so a run starts logged in instead of burning its iteration budget on a login form.navigate/back/forward/reload, plushover,right_click, drag destinations, and scroll direction.dragwas previously unreachable (_resolveraised unconditionally) and scroll direction never left the tool boundary.set_viewport, which makes the responsive checks in the attack catalog executable for the first time.run_plan, so "re-run my checkout suite after this change" is one call.absent/not_run, notverified. Only an explicit sign-off isverified.The bug worth reading the diff for
The new action types round-tripped into
ReproStepbut had no branch inreplay_spec, so they counted as replayed without being performed.verify_fixcould report a bug fixed while it was still there, andrun_plancould report a scenario PASSED having done nothing. Fixed, with the invariant documented —plan.run_scenarioandreverify.replay_specshare one executor, so any future action type must get a branch there.Known follow-ups (not in this PR)
AnthropicDriverhas nostop_reasonhandling. Opus 5 has thinking on by default andmax_tokenscaps thinking + response together; at 2048 a dense screen can truncate, degrade to"{}", and be parsed as a no-op step. A refusal does the same. The brain can go quietly blind mid-run.get_adapternow raises whenE2B_API_KEYis unset for sandboxed web/Electron — a deferred failure becomes a hard one at construction.claude-sonnet-4-6→claude-opus-5: ~1.7x per token, and a separate rate-limit bucket. Wants a CHANGELOG line.axe_errornow, but a CSP'd app there still gets no axe results.🤖 Generated with Claude Code