Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/browser-screenshot-jpeg.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@cloudflare/computer": patch
---

Show `codemode:browser`'s screenshot example as a JPEG, since a PNG of a whole window is often over the 1 MiB one call can return.
2 changes: 1 addition & 1 deletion docs/17_isolate_javascript.md
Original file line number Diff line number Diff line change
Expand Up @@ -806,7 +806,7 @@ The function may throw instead, when a run without that metadata shouldn't brows

Runs in one browser scope take turns, like `browserTool` calls, because they would otherwise see each other's navigation: pi and the AI SDK run a message's tool calls in parallel. A run takes its turn at its first browser call and keeps it until it ends, and runs that never call the module never wait. A run gives up after `lockTimeoutMs` (45 seconds by default) with an error saying why; keep it under the run's timeout and the backend's `maxHostCallMs`. A run that browsed and is waiting for a person keeps its browser scope while it waits: its session is closed, and other runs' first calls in that browser scope fail at once, saying the browser is held by a run that's waiting for a person. A run that took its turn while one ahead of it paused fails the same way. The module remembers which runs browsed in memory, so after the Durable Object restarts, a waiting run holds nothing until it resumes. Once that run resumes, it takes its turn again only at its first new browser call, since replay answers its earlier calls from the journal.

Don't expose `browserTool` to code through `codemode:tools` alongside this module. A run calling it would wait for its own turn until the host call timed out. The module's description says so, naming `toolsSpecifier` (`codemode:tools` by default) as it names `specifier` (`codemode:browser` by default) in its import line, and also tells the model to save screenshots to a file with `node:fs` and open them with `read`, to take about eight per run (each host call returns at most 1 MiB, and a run's calls share 8 MiB each way), and to work on several pages inside one run rather than across runs started together. The backend adds that description to its own, which the `exec` tool shows. An app that writes `exec`'s description itself must include it, from `backend.description`.
Don't expose `browserTool` to code through `codemode:tools` alongside this module. A run calling it would wait for its own turn until the host call timed out. The module's description says so, naming `toolsSpecifier` (`codemode:tools` by default) as it names `specifier` (`codemode:browser` by default) in its import line, and also tells the model to take screenshots as JPEG, save them to a file with `node:fs` and open them with `read`, to take about eight per run (each host call returns at most 1 MiB, and a run's calls share 8 MiB each way), and to work on several pages inside one run rather than across runs started together. The backend adds that description to its own, which the `exec` tool shows. An app that writes `exec`'s description itself must include it, from `backend.description`.

The module is experimental and may change in any release.

Expand Down
5 changes: 5 additions & 0 deletions packages/computer/src/modules/browser.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,11 @@ describe("createBrowserModule", () => {
expect(module.description).toContain('import * as browser from "codemode:browser"');
expect(module.description).toContain("written to stderr after the run ends");
expect(module.description).toContain("open the file with `read`");
// A PNG of a whole window is often over what one call returns, so the example takes a JPEG.
expect(module.description).toContain(
'browser.cdp({ method: "Page.captureScreenshot", params: { format: "jpeg", quality: 80 }, sessionId: "active" })',
);
expect(module.description).toContain("Take it as a JPEG and name the file `.jpg`");
expect(module.description).toContain("about eight screenshots per run");
expect(module.description).toContain("Target.createTarget");
expect(module.description).toContain("Don't call the browser tool from code");
Expand Down
2 changes: 1 addition & 1 deletion packages/computer/src/modules/browser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ const DEFAULT_TOOLS_SPECIFIER = "codemode:tools";
const workspaceRules = (specifier: string, toolsSpecifier: string) => [
`Import the module as a namespace, \`import * as browser from ${JSON.stringify(specifier)}\`, and call \`browser.<name>({ ... })\` as above.`,
"Notices about the browser (it restarted, your tab was closed, the page opened tabs) are written to stderr after the run ends.",
'To look at a page, save a screenshot and open the file with `read`: `const { data } = await browser.cdp({ method: "Page.captureScreenshot", sessionId: "active" })`, then `await fs.writeFile(path, Uint8Array.from(atob(data), (c) => c.charCodeAt(0)))` with `node:fs/promises`.',
'To look at a page, save a screenshot and open the file with `read`. Take it as a JPEG and name the file `.jpg`, since a PNG of a whole window is often over the 1 MiB one call can return: `const { data } = await browser.cdp({ method: "Page.captureScreenshot", params: { format: "jpeg", quality: 80 }, sessionId: "active" })`, then `await fs.writeFile(path, Uint8Array.from(atob(data), (c) => c.charCodeAt(0)))` with `node:fs/promises`.',
"One call returns at most 1 MiB, and a run's calls share 8 MiB each way, so take about eight screenshots per run at most, and print PDFs in a run of their own.",
"Runs that use the browser take turns, even when started together: a run waits until the runs ahead of it end. To work on several pages, do it in one run: open a tab per page with Target.createTarget and drive them with Promise.all.",
`Don't call the browser tool from code through \`${toolsSpecifier}\`: it waits for this run's turn, which only ends when this run does.`,
Expand Down
Loading