Add back in extract({screenshot:true}) functionality

[monadoid]

Summary

Until now, extract({ screenshot: true }) was just a no-op returning an error - this brings over the functionality from v3 exactly, and we base64 encode the image to serialize it over the wire.

Summary by cubic

Adds screenshot-backed extraction in v4: when extract({ screenshot: true }) is used, we capture a viewport PNG and send it to the LLM alongside the accessibility tree using protocol-native LLMImageContent. Works with direct and client-provided LLMs, keeps cache/timeout behavior, and exposes image content in the Python SDK.

• New Features
  • Capture a viewport PNG and include it with extraction prompts via LLMImageContent.
  • Update prompts and AI SDK conversion to handle image blocks.
  • Support both server and client LLM flows without changing cache/timeout semantics.
  • Export LLMImageContent from stagehand Python SDK and add cross-stack tests.

^{Written for commit 16f6ba4. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: fdd3731

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 8 files

Architecture diagram

sequenceDiagram
    participant Client as Client (TS/Python SDK)
    participant ExtractSvc as extractService
    participant Page as Page (Browser)
    participant Inference as inference.ts
    participant LLM as LLM Service (server or client)
    participant AISDK as AI SDK Converter
    participant Cache as Cache Service

    Note over Client,Cache: NEW: Screenshot-backed extraction flow

    Client->>ExtractSvc: extract({ screenshot: true, ... })
    ExtractSvc->>ExtractSvc: check timeout (start)
    ExtractSvc->>Cache: withCache (check cache)
    Cache-->>ExtractSvc: cache miss (or skip)
    ExtractSvc->>Page: captureSnapshot()
    Page-->>ExtractSvc: accessibility tree
    ExtractSvc->>Page: screenshot({ fullPage: false, type: "png" })
    Page-->>ExtractSvc: Uint8Array (PNG bytes)
    ExtractSvc->>ExtractSvc: ensureTimeRemaining()  -- NEW: check timeout after screenshot
    alt Timeout exceeded
        ExtractSvc-->>Client: throw TimeoutError("extract() timed out")
    else Still time remaining
        ExtractSvc->>ExtractSvc: convert PNG bytes to LLMImageContent (base64 data)
        ExtractSvc->>Inference: extract({ ..., screenshot: LLMImageContent })
        Inference->>LLM: generateStructured() with systemPrompt + userPrompt (including screenshot)
        Note over Inference,LLM: NEW: userPrompt is now an LLMMessage with content array (text + image)
        alt Client-provided LLM (clientLLMGenerate)
            LLM->>AISDK: generateWithAiSdk() (converts protocol image block to AI SDK image part)
            AISDK->>AISDK: map image.type -> image.image, mimeType -> mediaType
            AISDK-->>LLM: AI SDK call result
        else Server-side LLM (direct model)
            LLM->>LLM: call model with image content
        end
        LLM-->>Inference: structured result + metadata
        Inference-->>ExtractSvc: extracted data + metadata
        ExtractSvc->>Cache: store result in cache
        ExtractSvc-->>Client: final result (data + metadata)
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>