Skip to content

Standardize act, observe, and extract result shapes - #2460

Merged
monadoid merged 5 commits into
v4-spikefrom
standardize-ai-result-shapes
Jul 28, 2026
Merged

Standardize act, observe, and extract result shapes#2460
monadoid merged 5 commits into
v4-spikefrom
standardize-ai-result-shapes

Conversation

@monadoid

@monadoid monadoid commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • give act(), observe(), and extract() the same { data, metadata } protocol and SDK result shape
  • share one StagehandResultMetadata schema containing optional cacheStatus and actionId
  • stop the TypeScript and Python SDKs from unwrapping all three AI method results and discarding metadata
  • replace the inner extract result field with JSON-validated data and remove x-stainless-any
  • simplify cache status assignment now that every AI method stores it at metadata.cacheStatus
  • regenerate protocol/Python models and update V4 tests, examples, and documentation

Why

The protocol already carried cache status and action ID for act, observe, and extract, but the SDKs returned only each method's inner payload. That made metadata inaccessible and left three inconsistent public contracts.

All three methods now return:

{
  data: T,
  metadata: {
    cacheStatus?: "HIT" | "MISS";
    actionId?: string;
  }
}

act().data is ActResultData, observe().data is Action[], and extract().data is refined by the caller's Zod or Pydantic schema.

User impact

This is a V4 API-shape change:

  • actResult.success becomes actResult.data.success
  • observeResult[0] becomes observeResult.data[0]
  • extracted fields move under extractResult.data
  • cache status is consistently available at result.metadata.cacheStatus in TypeScript and result.metadata.cache_status in Python

Cache field names, hit/miss values, key calculation, thresholds, reads, and writes are unchanged.


Summary by cubic

Standardized act(), observe(), and extract() to return a consistent { data, metadata } shape across the protocol and all SDKs (TypeScript, Python, Go). Metadata now carries cache status and action ID; docs, wire casing, server caching, examples, and tests are updated.

  • New Features

    • All AI methods return typed data plus shared metadata (ActResult, ObserveResult, ExtractResult).
    • Exported StagehandResultMetadata (actionId?, cacheStatus?) and CacheStatus in protocol/types and all SDKs.
    • extract() returns JSON-validated data; the protocol preserves extract().data keys on the wire, while casing structured act/observe data.
  • Migration

    • TypeScript
      • act().successact().data.success
      • observe()[0]observe().data[0]
      • extract() result moved under .data with type ExtractResult<Schema>; read cache via .metadata.cacheStatus
    • Python
      • act() now returns ActResult; result.successresult.data.success
      • observe() returns ObserveResult; actions[0]actions.data[0]
      • extract() returns ExtractResult[Model]; read model via .data and cache via .metadata.cache_status
    • Go
      • Act() now returns ActResult; result.Successresult.Data.Success
      • Observe() returns ObserveResult; actions[0]result.Data[0]
      • Extract() returns ExtractResult; decode from result.Data and read cache via result.Metadata.CacheStatus

Written for commit b96d99b. Summary will update on new commits.

Review in cubic

@changeset-bot

changeset-bot Bot commented Jul 27, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: b96d99b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@monadoid
monadoid marked this pull request as ready for review July 27, 2026 21:44
@monadoid
monadoid requested a review from a team as a code owner July 27, 2026 21:44

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/sdk-python/src/stagehand/stagehand.py Outdated
Comment thread packages/docs/v4/reference/stagehand.mdx

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found and verified against the latest diff

Confidence score: 3/5

  • In packages/server/services/cacheService.ts, the cache result-shape change appears to be a breaking Stagehand REST API contract update without the required integration coverage, which raises real regression risk at the client/server boundary and could break cache consumers at runtime — add an integration test under packages/server/test that exercises the new shape end to end.
  • In packages/server/services/actService.ts, cache persistence behavior after the shape migration is not covered, so miss/hit flows could silently stop writing or returning cached action data/metadata and degrade correctness or performance — add focused cache-enabled miss and hit tests that assert the expected action payload structure.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/server/services/cacheService.ts">

<violation number="1" location="packages/server/services/cacheService.ts:215">
P2: Custom agent: **Any breaking changes to Stagehand REST API client / server implementation must be covered by an integration test under packages/server/test**

Breaking change to cache result shape without integration test coverage for the new `metadata.cacheStatus` path

The `withCache` utility now writes `cacheStatus` into `result.metadata.cacheStatus` instead of `result.cacheStatus`. This changed code path is exercised by act, observe, and extract, yet none of the server integration tests verify that a cache hit or miss correctly populates `metadata.cacheStatus`. Add coverage in `packages/server/tests/` that exercises `withCache` (or the services that use it) and asserts the `metadata.cacheStatus` value for both HIT and MISS scenarios.</violation>
</file>

<file name="packages/server/services/actService.ts">

<violation number="1" location="packages/server/services/actService.ts:96">
P3: Cache persistence after the result-shape migration lacks coverage, so a future data/metadata regression can silently stop act cache writes or hits. Add focused cache-enabled miss and hit tests asserting the action array is persisted/replayed and `metadata.cacheStatus` is set.

(Based on your team's feedback about adding unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/server/services/extractService.ts
Comment thread packages/protocol/schemas.ts
Comment thread packages/protocol/schemas.ts
Comment thread packages/sdk-python/src/stagehand/_generated/models.py
Comment thread packages/docs/v4/first-steps/ai-rules.mdx Outdated
Comment thread packages/server/services/cacheService.ts
Comment thread packages/docs/v4/reference/stagehand.mdx
Comment thread packages/server/services/actService.ts
@monadoid
monadoid force-pushed the standardize-ai-result-shapes branch from 94ba308 to 06d0e91 Compare July 28, 2026 12:33

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 15 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread packages/sdk-python/src/stagehand/client_models.py Outdated
@monadoid
monadoid merged commit 6f3f959 into v4-spike Jul 28, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants