P0.1: evidence-based verification (typed assertions, ReproSpec, honest reverify) - #20
Merged
Merged
Conversation
Add inspector/assertions.py: an Assertion (text, role, value, count, url, state, network, screenshot) evaluates to pass / fail / inconclusive with evidence. The DOM/state/url channels are evaluated against the observation; unavailable channels (network, screenshot, missing element, no URL) return inconclusive with a reason rather than silently passing, so a pass always means a real check ran. Pure and unit-tested; the session-wiring tool follows.
Wire the assertions into a check_assertions(session_id, list[Assertion]) core tool that gathers the observation (DOM text/elements, URL on CDP surfaces, control-state for referenced elements) and evaluates each assertion to pass/fail/inconclusive with evidence. Typed input schema (the Assertion enums) + AssertionsOut output. Verified live against a real web app: text/url/count pass, network returns inconclusive rather than a false pass. Updated the tool-registry tests + reference.
Add ReproStep/ReproSpec models (surface, route, preconditions, semantic steps by element label, and a typed oracle) and attach one to every finding at creation (the log tap, the DOM audit, and report_issue) via build_repro_spec, which parses the session's semantic action log rather than raw coordinates, and degrades gracefully on a partial session. The durable scenario re-verification needs; the reverify/dashboard rewiring to use it (and report not_run) follows next.
reverify replayed raw coordinates and called a bug fixed when a similar summary did not reappear. Add replay_spec (re-find each element by its semantic label, not raw coords) and verify_fix_spec, which replays the finding's ReproSpec, evaluates its typed oracle (pass=fixed, fail=still_present, inconclusive=not_run), and returns not_run when the scenario diverges (a locator can't be found) rather than assuming fixed. The verify_fix tool now prefers the ReproSpec, falling back to the coordinate replay. Add a shared session.observation_context used by both the assertion tool and the oracle.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P0.1 evidence-based verification (from your roadmap). Off
main; four commits, each unit-tested, the tool live-verified against a real app.Delivered
inspector/assertions.py) — an Assertion (text, role, value, count, url, state, network, screenshot) evaluates to pass | fail | inconclusive with evidence. Unavailable channels (network/screenshot, a missing element, no URL on the surface) returninconclusivewith a reason, never a false pass.check_assertionstool — a core tool takinglist[Assertion](so the enums are in the input schema) that gathers the observation and evaluates. Live-verified against a real web app: text/url/count pass with evidence, network returns inconclusive.ReproSpecon every finding — surface, route, preconditions, semantic steps (by element label, not raw coordinates), and a typed oracle. Built at finding creation (log tap, DOM audit,report_issue).verify_fixnow replays the ReproSpec by semantic locator (not raw coordinates) and judges by the explicit oracle: pass = fixed, fail = still_present, inconclusive / diverged scenario = not_run — instead of assuming fixed because a summary did not reappear.Verification
pytest -qgives 360 passed; ruff clean;check_assertionsverified end to end onsample-buggy-app.Remaining P0 (next)
aggregate.py) — relabel absent vs actually re-verified.get_capabilities+ stable observation refs; network / app-state tools; privacy / secret handling.This branch is off current
main(which still tracksdocs/); PR #19 removes the docs machinery for the DeepWiki-only decision.