evals: port the act/extract/observe bench suite to the v4 SDK - #2494
Open
shriyatheunicorn wants to merge 24 commits into
Open
evals: port the act/extract/observe bench suite to the v4 SDK#2494shriyatheunicorn wants to merge 24 commits into
shriyatheunicorn wants to merge 24 commits into
Conversation
|
shriyatheunicorn
marked this pull request as ready for review
July 29, 2026 13:13
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
6 issues found across 81 files
Confidence score: 3/5
- In
packages/evals/tasks/bench/act/heal_simple_google_search.ts, the v4 benchmark currently keepsselfHeal: false, so the invalid selector fails and the task submits Enter without filling the field, which can invalidate the healing benchmark outcome — enable healing for this task (or otherwise ensure the act step can recover before submit). - In
packages/evals/tasks/bench/extract/extract_memorial_healthcare.tsandpackages/evals/tasks/bench/extract/extract_baptist_health.ts, custom grading logic can drift from prior comparison behavior (whitespace/punctuation normalization, prefix weighting, transpositions, threshold behavior), risking silent score changes — add parity-focused regression tests against the established utility around edge cases. - In
packages/evals/tasks/bench/observe/observe_iframes2.ts,packages/evals/tasks/bench/observe/observe_yc_startup.ts, andpackages/evals/tasks/bench/observe/observe_file_uploads.ts, new selector-resolution and XPath/CSS identity paths lack focused executable coverage, so malformed or nonmatching selectors could silently alter pass/fail scoring — add deterministic matching/nonmatching tests including iframe XPath rejection.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/tasks/bench/extract/extract_memorial_healthcare.ts">
<violation number="1" location="packages/evals/tasks/bench/extract/extract_memorial_healthcare.ts:19">
P3: Evaluator pass/fail scoring can drift from the established comparison utility without detection because this hand-written Jaro-Winkler replacement has no parity coverage for whitespace, punctuation, prefix, and transposition cases. Add focused tests comparing it with `utils.ts` behavior before relying on the port.
(Based on your team's feedback about adding unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>
<file name="packages/evals/tasks/bench/observe/observe_yc_startup.ts">
<violation number="1" location="packages/evals/tasks/bench/observe/observe_yc_startup.ts:39">
P3: This new XPath/CSS identity check has no focused regression coverage, so malformed selectors or the expected XPath match can silently change benchmark scoring. Add deterministic coverage for matching XPath, nonmatching CSS, and resolver failures.
(Based on your team's feedback about unit tests for changed logic.) [FEEDBACK_USED]</violation>
</file>
<file name="packages/evals/tasks/bench/observe/observe_file_uploads.ts">
<violation number="1" location="packages/evals/tasks/bench/observe/observe_file_uploads.ts:28">
P3: The new selector-resolution path has no regression coverage, so malformed/nonmatching XPath results can silently change this benchmark's pass/fail outcome. Add focused matching and nonmatching XPath cases around the identity check.
(Based on your team's feedback about unit tests for new behavior.) [FEEDBACK_USED].</violation>
</file>
<file name="packages/evals/tasks/bench/act/heal_simple_google_search.ts">
<violation number="1" location="packages/evals/tasks/bench/act/heal_simple_google_search.ts:12">
P1: This v4 healing benchmark will not exercise healing: its invalid selector produces a failed `ActResult` when v4 uses its default `selfHeal: false`, then the task presses Enter without filling the field. Enable `selfHeal` in the v4 eval initialization (or configure this task's v4 client) so this remains a healing parity test.</violation>
</file>
<file name="packages/evals/tasks/bench/extract/extract_baptist_health.ts">
<violation number="1" location="packages/evals/tasks/bench/extract/extract_baptist_health.ts:19">
P3: This new grading implementation has no focused regression coverage for normalization, prefix weighting, and transpositions, so parity with the prior evaluator can regress unnoticed. Add focused cases around the 0.85 boundary before relying on it for bench pass/fail results.
(Based on your team's feedback about unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>
<file name="packages/evals/tasks/bench/observe/observe_iframes2.ts">
<violation number="1" location="packages/evals/tasks/bench/observe/observe_iframes2.ts:49">
P3: This new selector-resolution path has no executable coverage; add focused cases for matching CSS/XPath candidates and rejecting an iframe XPath so the benchmark's v3-equivalence criterion cannot silently regress.
(Based on your team's feedback about unit tests for changed behavior.) [FEEDBACK_USED]</violation>
</file>
Architecture diagram
sequenceDiagram
participant Runner as Bench Runner
participant Task as Task Function
participant SDK as Stagehand v4 SDK
participant Page as Page (Browser)
participant Replay as observeReplay Shim
Note over Runner,Replay: NEW: v4 Bench Task Execution Flow
Runner->>Task: invoke with {stagehand, page, ...}
Task->>SDK: stagehand.act("instruction")
SDK-->>Task: result (success/failure)
Task->>SDK: stagehand.extract("instruction", z.object({...}))
SDK-->>Task: { data }
alt Observe + Replay Flow
Task->>SDK: stagehand.observe("instruction")
SDK-->>Task: { data: ObservedAction[] }
Task->>Replay: replayObservedAction(page, action)
Replay->>Page: locator.click() / fill() / etc.
Page-->>Replay: done
end
Task->>Page: await page.url()
Page-->>Task: current URL
alt Success
Task-->>Runner: { _success: true, ... }
else Failure
Task->>SDK: stagehand.close()
Task-->>Runner: { _success: false, error }
end
Task->>SDK: stagehand.close()
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| // Self-healing act(Action) replay (V4_API_LOGS.md #1, restored by | ||
| // stagehand#2427): same intentionally invalid selector as the v3 | ||
| // twin — healing must re-locate "The search bar" and fill it. | ||
| await stagehand.act({ |
Contributor
There was a problem hiding this comment.
P1: This v4 healing benchmark will not exercise healing: its invalid selector produces a failed ActResult when v4 uses its default selfHeal: false, then the task presses Enter without filling the field. Enable selfHeal in the v4 eval initialization (or configure this task's v4 client) so this remains a healing parity test.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/act/heal_simple_google_search.ts, line 12:
<comment>This v4 healing benchmark will not exercise healing: its invalid selector produces a failed `ActResult` when v4 uses its default `selfHeal: false`, then the task presses Enter without filling the field. Enable `selfHeal` in the v4 eval initialization (or configure this task's v4 client) so this remains a healing parity test.</comment>
<file context>
@@ -1,25 +1,27 @@
+ // Self-healing act(Action) replay (V4_API_LOGS.md #1, restored by
+ // stagehand#2427): same intentionally invalid selector as the v3
+ // twin — healing must re-locate "The search bar" and fill it.
+ await stagehand.act({
description: "The search bar",
selector: "/html/not-the-search-bar",
</file context>
| .trim(); | ||
| } | ||
|
|
||
| function jaroSimilarity(first: string, second: string): number { |
Contributor
There was a problem hiding this comment.
P3: This new grading implementation has no focused regression coverage for normalization, prefix weighting, and transpositions, so parity with the prior evaluator can regress unnoticed. Add focused cases around the 0.85 boundary before relying on it for bench pass/fail results.
(Based on your team's feedback about unit tests for new behavior.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/extract/extract_baptist_health.ts, line 19:
<comment>This new grading implementation has no focused regression coverage for normalization, prefix weighting, and transpositions, so parity with the prior evaluator can regress unnoticed. Add focused cases around the 0.85 boundary before relying on it for bench pass/fail results.
(Based on your team's feedback about unit tests for new behavior.) </comment>
<file context>
@@ -1,15 +1,92 @@
+ .trim();
+}
+
+function jaroSimilarity(first: string, second: string): number {
+ // string-comparison initParams: strip all whitespace + lowercase
+ const s1 = first.replace(/\s+/g, "").toLowerCase();
</file context>
| matchedLocator = candSel; | ||
| break; | ||
| } | ||
| const matched = await page.evaluate( |
Contributor
There was a problem hiding this comment.
P3: This new selector-resolution path has no executable coverage; add focused cases for matching CSS/XPath candidates and rejecting an iframe XPath so the benchmark's v3-equivalence criterion cannot silently regress.
(Based on your team's feedback about unit tests for changed behavior.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/observe/observe_iframes2.ts, line 49:
<comment>This new selector-resolution path has no executable coverage; add focused cases for matching CSS/XPath candidates and rejecting an iframe XPath so the benchmark's v3-equivalence criterion cannot silently regress.
(Based on your team's feedback about unit tests for changed behavior.) </comment>
<file context>
@@ -34,35 +33,65 @@ export default defineBenchTask(
- matchedLocator = candSel;
- break;
- }
+ const matched = await page.evaluate(
+ ({
+ observedSelector,
</file context>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
…elity, per-page extract targeting, deterministic dropdown check
The fixture contains a cross-origin OOPIF, so the outer page cannot access its contentDocument. Validate the form through v4 cross-frame locators (iframe >> selector) with inputValue()/isChecked() instead.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 67 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
1 issue found across 81 files
Confidence score: 5/5
- In
packages/evals/tasks/bench/extract/extract_public_notices.ts, the local copy of the scoring algorithm can drift from the other three copies, so later fixes may change comparison semantics in one place but not this benchmark, leading to silent inconsistency in eval results—move the logic to a shared framework utility (allowed by this task) to keep behavior aligned.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/tasks/bench/extract/extract_public_notices.ts">
<violation number="1" location="packages/evals/tasks/bench/extract/extract_public_notices.ts:10">
P3: Four copies of this scoring algorithm must stay aligned; a future comparison fix can silently leave this benchmark using different semantics. A shared framework utility would remain within this task's permitted import boundary and avoid that drift.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| * string-comparison@1.3.0 — v4 eval tasks may only import "zod" and | ||
| * "../../../framework/*.js". Pure computation, no behavior change. | ||
| */ | ||
| function normalizeString(str: string): string { |
Contributor
There was a problem hiding this comment.
P3: Four copies of this scoring algorithm must stay aligned; a future comparison fix can silently leave this benchmark using different semantics. A shared framework utility would remain within this task's permitted import boundary and avoid that drift.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/extract/extract_public_notices.ts, line 10:
<comment>Four copies of this scoring algorithm must stay aligned; a future comparison fix can silently leave this benchmark using different semantics. A shared framework utility would remain within this task's permitted import boundary and avoid that drift.</comment>
<file context>
@@ -1,17 +1,94 @@
+ * string-comparison@1.3.0 — v4 eval tasks may only import "zod" and
+ * "../../../framework/*.js". Pure computation, no behavior change.
+ */
+function normalizeString(str: string): string {
+ return str
+ .toLowerCase()
</file context>
Contributor
There was a problem hiding this comment.
5 issues found across 80 files
Confidence score: 4/5
- The biggest risk is duplicated selector resolver/equality logic across
packages/evals/tasks/bench/observe/observe_yc_startup.ts,observe_github.ts, andobserve_file_uploads.ts: future XPath/CSS/frame fixes can land in one task but not others, causing inconsistent benchmark behavior and hard-to-track regressions — extract a shared eval helper and route all six observe tasks through it. packages/evals/tasks/bench/observe/observe_iframes1.tsandobserve_iframes2.tsnow carry effectively the same iframe selector-matching implementation, so frame-handling or selector-format changes are likely to drift between the two and skew iframe-specific eval results — centralize the iframe resolver/comparison path so both tasks inherit the same semantics and error handling.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/tasks/bench/observe/observe_iframes2.ts">
<violation number="1" location="packages/evals/tasks/bench/observe/observe_iframes2.ts:49">
P3: Iframe selector matching now has two identical implementations in `observe_iframes1.ts` and this task. Extract the resolver/comparison into a shared helper so XPath and frame-handling fixes apply consistently to both benches.</violation>
</file>
<file name="packages/evals/tasks/bench/observe/observe_github.ts">
<violation number="1" location="packages/evals/tasks/bench/observe/observe_github.ts:45">
P3: Selector-identity matching is now duplicated across multiple observe tasks. A shared helper would keep XPath/selector semantics fixes consistent instead of requiring edits to each copied evaluator.</violation>
</file>
<file name="packages/evals/tasks/bench/observe/observe_file_uploads.ts">
<violation number="1" location="packages/evals/tasks/bench/observe/observe_file_uploads.ts:36">
P3: Selector-resolution logic is now duplicated across six v4 observe tasks, so fixes to XPath/CSS handling can silently diverge between benchmarks. A shared page-evaluation helper would keep these identity checks consistent.</violation>
</file>
<file name="packages/evals/tasks/bench/observe/observe_yc_startup.ts">
<violation number="1" location="packages/evals/tasks/bench/observe/observe_yc_startup.ts:47">
P3: Selector parsing is now duplicated across six observe tasks, so XPath or frame-selector fixes will drift between benchmarks. Extract the shared resolver/equality check into an eval helper and reuse it here.</violation>
</file>
<file name="packages/evals/tasks/bench/observe/observe_iframes1.ts">
<violation number="1" location="packages/evals/tasks/bench/observe/observe_iframes1.ts:46">
P3: Future selector-format changes must be copied across several bench tasks because this resolver duplicates existing v4 observe validation logic. A shared helper would keep XPath/CSS resolution and error behavior consistent.</violation>
</file>
Architecture diagram
sequenceDiagram
participant T as Bench Task (v4)
participant EV as Eval Runner (--sdk v4)
participant SH as Stagehand v4 SDK
participant P as v4 Page (RPC-backed)
participant B as Browserbase Session
participant LLM as LLM Provider
participant L as Eval Logger
Note over T,L: NEW: v4 bench task execution flow (defineBenchV4Task)
EV->>EV: Select task via --sdk v4 flag
EV->>T: invoke fn(ctx) with BenchV4TaskContext
alt ctx.stagehand is undefined (v3 runner mistaken)
T-->>EV: Throw "requires v4 harness" error
else ctx.stagehand is defined
T->>SH: stagehand.act("click the button")
SH->>LLM: Resolve instruction to action
LLM-->>SH: Action (selector, method, args)
SH->>P: Execute resolved action
P-->>SH: Action result
SH-->>T: { success: boolean }
T->>SH: stagehand.extract("extract page text", z.object({...}))
SH->>P: Extract current DOM context
SH->>LLM: Send DOM snapshot + extraction schema
LLM-->>SH: Structured data
SH-->>T: { data: { extracted fields } }
T->>SH: stagehand.observe("find the main header")
SH->>P: Observe interactable elements
SH->>LLM: Process observation instruction
LLM-->>SH: Action[] observations
SH-->>T: { data: Action[] }
alt Multi-tab flow
T->>SH: stagehand.context.activePage()
SH->>B: Poll for active tab
B-->>SH: Active page reference
SH-->>T: v4 Page (active tab)
T->>SH: stagehand.context.pages()
SH->>B: List all pages
B-->>SH: Page references
SH-->>T: v4 Page[]
T->>SH: stagehand.context.setActivePage(page1)
SH->>B: Switch active tab
B-->>SH: Done
end
alt Cross-frame assertions (OOPIF)
T->>P: page.locator('iframe >> input[placeholder="Jane"]').inputValue()
P->>B: Resolve >> cross-frame selector server-side
B-->>P: Input value from OOPIF
P-->>T: "Stagehand"
else Same-origin iframe assertions
T->>P: page.evaluate(() => document.querySelector("iframe")?.contentDocument...)
P-->>T: Evaluated result
end
T->>P: await page.url()
P-->>T: current URL (async)
T->>P: page.locator(selector).inputValue()
P-->>T: Field value
T->>P: page.evaluate(() => window.scrollY + window.innerHeight / 2)
P-->>T: Scroll position
alt Error in task
T-->>EV: { _success: false, error: error.message }
else Success
T-->>EV: { _success: true, ... }
end
T->>SH: stagehand.close()
SH->>B: Cleanup session
B-->>SH: Closed
end
EV->>L: Log result (status, debugUrl, sessionUrl)
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| matchedLocator = candSel; | ||
| break; | ||
| } | ||
| const matched = await page.evaluate( |
Contributor
There was a problem hiding this comment.
P3: Iframe selector matching now has two identical implementations in observe_iframes1.ts and this task. Extract the resolver/comparison into a shared helper so XPath and frame-handling fixes apply consistently to both benches.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/observe/observe_iframes2.ts, line 49:
<comment>Iframe selector matching now has two identical implementations in `observe_iframes1.ts` and this task. Extract the resolver/comparison into a shared helper so XPath and frame-handling fixes apply consistently to both benches.</comment>
<file context>
@@ -34,35 +33,65 @@ export default defineBenchTask(
- matchedLocator = candSel;
- break;
- }
+ const matched = await page.evaluate(
+ ({
+ observedSelector,
</file context>
| matchedLocator = candSel; | ||
| break; | ||
| } | ||
| const matched = await page.evaluate( |
Contributor
There was a problem hiding this comment.
P3: Selector-identity matching is now duplicated across multiple observe tasks. A shared helper would keep XPath/selector semantics fixes consistent instead of requiring edits to each copied evaluator.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/observe/observe_github.ts, line 45:
<comment>Selector-identity matching is now duplicated across multiple observe tasks. A shared helper would keep XPath/selector semantics fixes consistent instead of requiring edits to each copied evaluator.</comment>
<file context>
@@ -33,35 +32,62 @@ export default defineBenchTask(
- matchedLocator = candSel;
- break;
- }
+ const matched = await page.evaluate(
+ ({
+ observedSelector,
</file context>
| observedSelector: string; | ||
| candidateSelectors: string[]; | ||
| }) => { | ||
| const resolve = (selector: string): Element | null => { |
Contributor
There was a problem hiding this comment.
P3: Selector parsing is now duplicated across six observe tasks, so XPath or frame-selector fixes will drift between benchmarks. Extract the shared resolver/equality check into an eval helper and reuse it here.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/observe/observe_yc_startup.ts, line 47:
<comment>Selector parsing is now duplicated across six observe tasks, so XPath or frame-selector fixes will drift between benchmarks. Extract the shared resolver/equality check into an eval helper and reuse it here.</comment>
<file context>
@@ -28,35 +27,61 @@ export default defineBenchTask(
+ observedSelector: string;
+ candidateSelectors: string[];
+ }) => {
+ const resolve = (selector: string): Element | null => {
+ const raw = selector.startsWith("xpath=")
+ ? selector.slice("xpath=".length)
</file context>
| observedSelector: string; | ||
| candidateSelectors: string[]; | ||
| }) => { | ||
| const resolve = (selector: string): Element | null => { |
Contributor
There was a problem hiding this comment.
P3: Future selector-format changes must be copied across several bench tasks because this resolver duplicates existing v4 observe validation logic. A shared helper would keep XPath/CSS resolution and error behavior consistent.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/observe/observe_iframes1.ts, line 46:
<comment>Future selector-format changes must be copied across several bench tasks because this resolver duplicates existing v4 observe validation logic. A shared helper would keep XPath/CSS resolution and error behavior consistent.</comment>
<file context>
@@ -24,35 +23,64 @@ export default defineBenchTask(
+ observedSelector: string;
+ candidateSelectors: string[];
+ }) => {
+ const resolve = (selector: string): Element | null => {
+ const raw = selector.startsWith("xpath=")
+ ? selector.slice("xpath=".length)
</file context>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
This reverts commit e817832.
Contributor
There was a problem hiding this comment.
2 issues found across 79 files
Confidence score: 4/5
- In
packages/evals/tasks/bench/act/multi_tab.ts, the finalstagehand.extractcan read from the wrong tab if focus shifts after the URL check, which can produce flaky or incorrect multi-tab eval results even whenactivePagewas captured correctly — pass{ page: activePage }to lock extraction to the intended tab. - In
packages/evals/tasks/bench/act/multi_tab.ts, the V4 GAP note saysextractlacks a{ page }option, but nearby usage intab_handling.tsshows it does; this mismatch can mislead future edits and preserve the tab-selection bug path — update the comment to reflect actual API support and intended usage.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/tasks/bench/act/multi_tab.ts">
<violation number="1" location="packages/evals/tasks/bench/act/multi_tab.ts:53">
P3: The V4 GAP comment claims that `extract` has no `{ page }` option, but `tab_handling.ts` in the same directory already uses it successfully: `stagehand.extract(instruction, z.object({...}), { page: page1 })`. This comment is factually inaccurate and could mislead future maintainers into thinking they need to work around a nonexistent limitation. Consider updating the comment to reflect the actual v4 API or removing it.</violation>
<violation number="2" location="packages/evals/tasks/bench/act/multi_tab.ts:58">
P2: The final extraction can target a different active tab if focus changes after the URL check, even though `activePage` already identifies page2. `stagehand.extract` supports `{ page }`; pass the captured page to preserve the multi-tab assertion.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Runner as V4 Task Runner
participant TaskDef as defineBenchV4Task
participant Ctx as BenchV4TaskContext
participant Stagehand as Stagehand (V4)
participant Page as V4Page
participant Target as Evaluation Site
Note over Runner,Target: NEW: V4 Bench Task Execution Flow
Runner->>TaskDef: load task definition (default export)
TaskDef-->>Runner: { meta, fn: wrapperFn }
Runner->>TaskDef: invoke wrapperFn(ctx)
alt ctx.stagehand is undefined
TaskDef->>TaskDef: throw "requires v4 harness (--sdk v4)"
else stagehand available
TaskDef->>Ctx: cast ctx to BenchV4TaskContext
Note over Ctx: NEW: provides stagehand, page, logger<br/>page.url() is async
Ctx->>Page: await page.goto(siteUrl)
Page-->>Target: navigate
Ctx->>Stagehand: stagehand.act(instruction)
Stagehand->>Target: perform action (click/fill/scroll)
Target-->>Stagehand: result
Stagehand-->>Ctx: { message, ... }
alt Extract instruction
Ctx->>Stagehand: stagehand.extract(instruction, zodSchema)
Note over Stagehand: NEW: schema required<br/>returns { data: ... }
Stagehand->>Target: observe & extract
Target-->>Stagehand: data
Stagehand-->>Ctx: { data: ... }
end
alt Observe instruction
Ctx->>Stagehand: stagehand.observe(instruction)
Stagehand->>Target: generate observation candidates
Target-->>Stagehand: Action[]
Stagehand-->>Ctx: { data: Action[] }
Ctx->>Stagehand: stagehand.act(observation)
Note over Stagehand: replay Action from observe
end
Ctx->>Page: await page.url() (async)
Page-->>Ctx: current URL
Ctx->>Ctx: evaluate pass/fail
Ctx-->>Runner: return TaskResult
end
alt Error in task
Ctx->>Ctx: catch (error) → error.message or String(error)
Ctx-->>Runner: return { _success: false, error: ... }
end
Note over Ctx,Stagehand: Cleanup (finally)
Ctx->>Stagehand: stagehand.close()
Stagehand-->>Ctx: done
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| // active page here, so extract operates on it. v3 also used schemaless | ||
| // extract; v4 requires a schema. Single-word key to | ||
| // stay clear of the snake_case wire-casing bug (#14). | ||
| const { data: page2text } = await stagehand.extract( |
Contributor
There was a problem hiding this comment.
P2: The final extraction can target a different active tab if focus changes after the URL check, even though activePage already identifies page2. stagehand.extract supports { page }; pass the captured page to preserve the multi-tab assertion.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/act/multi_tab.ts, line 58:
<comment>The final extraction can target a different active tab if focus changes after the URL check, even though `activePage` already identifies page2. `stagehand.extract` supports `{ page }`; pass the captured page to preserve the multi-tab assertion.</comment>
<file context>
@@ -45,10 +50,18 @@ export default defineBenchTask(
+ // active page here, so extract operates on it. v3 also used schemaless
+ // extract; v4 requires a schema. Single-word key to
+ // stay clear of the snake_case wire-casing bug (#14).
+ const { data: page2text } = await stagehand.extract(
+ "extract the entire page text",
+ z.object({ extraction: z.string() }),
</file context>
| } | ||
|
|
||
| const page2text = await v3.extract({ page: activePage }); | ||
| // V4 GAP: extract has no { page } option (v3: |
Contributor
There was a problem hiding this comment.
P3: The V4 GAP comment claims that extract has no { page } option, but tab_handling.ts in the same directory already uses it successfully: stagehand.extract(instruction, z.object({...}), { page: page1 }). This comment is factually inaccurate and could mislead future maintainers into thinking they need to work around a nonexistent limitation. Consider updating the comment to reflect the actual v4 API or removing it.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tasks/bench/act/multi_tab.ts, line 53:
<comment>The V4 GAP comment claims that `extract` has no `{ page }` option, but `tab_handling.ts` in the same directory already uses it successfully: `stagehand.extract(instruction, z.object({...}), { page: page1 })`. This comment is factually inaccurate and could mislead future maintainers into thinking they need to work around a nonexistent limitation. Consider updating the comment to reflect the actual v4 API or removing it.</comment>
<file context>
@@ -45,10 +50,18 @@ export default defineBenchTask(
}
- const page2text = await v3.extract({ page: activePage });
+ // V4 GAP: extract has no { page } option (v3:
+ // v3.extract({ page: activePage })) — the target page is already the
+ // active page here, so extract operates on it. v3 also used schemaless
</file context>
Replaced stagehand act with replayObservedAction for flight observation.
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Added note about additional instructions for Stagehand.
shriyatheunicorn
left a comment
shriyatheunicorn
left a comment
Contributor
Author
There was a problem hiding this comment.
resolved
Added a comment placeholder in the publications schema.
Removed unnecessary closing brackets and whitespace.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This will port the act/extract/observe bench suite (77 tasks) from the v3 API over to the v4 SDK, converting each task in place on its existing
tasks/bench/path so file history and blame stay continuous.Tasks compile clean against the in-tree v4 SDK, including the #2460 result shapes and #2427 Action inputs. The execution harness (v4 init, task worker, run isolation) follows as a stacked change.
Note: this PR also includes Sam's functionality from 2431 to enable the usage of cross-frame locators for iframe form assertions
Scope
act (40) · extract (25) · observe (12). Combination tasks can be added if needed for a more expansive task suite.
Summary by cubic
Ports the act/extract/observe bench suite (77 tasks) to the v4 SDK with behavior parity. Uses
@browserbasehq/stagehandactions,zodextracts, cross-frame assertions (OOPIF/CSR), safer errors/clicks, and cleans upextract_csa.ts.New Features
defineBenchV4TaskandBenchV4TaskContextwith a fail-fast, null-safe guard for v3 runners; documentedBenchTaskMeta.systemPrompt; folded v4 types intotypes.tsand refactored imports.stagehand.act/extract/observe, native observed-action replay viastagehand.act(Action), and the v4Page(asyncurl()); multi-tab viastagehand.context.activePage()/pages().zodschemas with per-page targeting; inlined string-similarity helpers for checks; expanded and cleaned the CSA publications schema inextract_csa.ts.iframe >> selectorfor iframe/OOPIF cases; added deterministic dropdown checks.Migration
--sdk v4; paths and task names are unchanged.Written for commit 8b68d62. Summary will update on new commits.