Repository navigation
Commit 900afc3
feat(tests): workflow tests as a workspace resource (#8773)
* fix(chat): send entitlements and agent mode from the public chat API
`/api/v2/chat` (what `sim chat` uses) stopped sending `entitlements` and `mode`
in #8208, so CLI and API chats got no entitlement-gated tools and every CLI
service refused them with "CLI services require agent mode". The route now
computes entitlements per turn like the workspace chat and sends
`mode: 'agent'`. Its test still mocked the old entitlements function and
listed `mode` as a forbidden legacy field; both are updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* feat(tests): workflow tests as a workspace resource
Workflow tests are a workspace resource whose source is a plain vitest file,
`tests/<name>.test.js`, owned by the test (workspace_files.context = 'test').
Sim creates a test's metadata with the tests tool and writes its cases with
the file tools; every write is collected in the sandbox and refused if the
file does not load.
- Runner: test files run in the isolated-vm sandbox against draft or
deployed workflows. `runWorkflow` executes real runs; `mockBlock`,
`mockTool` (Agent tool calls) and `spyOnBlock` reach blocks in the tested
workflow and in every child workflow it runs, matched by name as each
workflow starts. `.mockSampleOutput()` builds outputs shaped like the real
block or tool. `toMatchRubric` asks a model judge for pass or fail.
- Runs record live per-case progress, the source hash, and the deployment of
every workflow they ran, so results show as out of date once the test or
a workflow changes.
- UI: Tests page and test page (Edit / Split / Preview over the file, the
preview a dashboard of the selected run), a test resource type in chat,
and a Tests sidebar entry behind the `workflow-tests` flag.
- Owned files never open as file tabs in chat: only workspace files and
chat uploads do.
- Migration 0400 adds workflow_test and workflow_test_run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* feat(tests): pick a run from a dropdown and open what each run ran against
The test page shows one run at a time, chosen from a run picker with status
dots and Draft / Out of date chips. Case statuses use the Badge status chip.
Each ran-against entry records one execution, so a draft row opens the
workflow snapshot from that run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* fix(tests): type errors and tests broken by workflow tests
- Narrow the test principal to the kinds workflow_tests.run admits before
handing it to executeWorkflow.
- Select progress with the latest-run rows, guard file upsert ids in the
tab filter, and set the sandbox Event polyfills through Reflect.
- Cover the tests tool in the management tool contract, expect content
writes to reach test files, and stub test availability in the payload test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* fix(tests): address review on redaction, staleness, and the harness
- Redact each run's resolved secrets from what returns to the sandbox
(output, errors, mocked tool inputs); mocked tools get only declared params.
- Custom blocks no longer receive the consumer's test hooks.
- Draft runs go stale when the draft changes; children a run calls are
recorded in ran-against.
- Test cases commit in the same transaction as the source file write.
- Harness: runWorkflow is rejected in suite hooks, a timed-out case stops
the file, and expect.assertions/hasAssertions are supported.
- Insert run rows in one statement and start each run's clock with its file;
check bans before each workflow run; restrict owned-file access to Copilot
delegation; validate names in the tool contract.
- Delete soft-deletes the test file and removes the chat tab; a finished run
shows its own cases; polling at 3s on a separate read bucket; list error
state; store reset; tests stay in the org Add Resource picker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* fix(tests): reserve run slots, await pending assertions, keep dynamic tool args
- Each test workflow run reserves and releases an execution slot.
- A case waits for assertions it did not await and fails if one fails.
- Mocked MCP and custom tools keep the arguments their schema declares.
- A closed session refuses starts still awaiting their lookups.
- Stable refresh callback; scroll fade on the results pane.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* fix(tests): keep test sources out of file tabs, refresh after runs, reopen tests
- File-edit tool results mark a non-tab file `fileTab: false`, and the browser
skips promoting it.
- Idle test pages poll every 15s so runs started elsewhere appear; a Mothership
run returns its tests as resource changes.
- open_resource accepts test resources through an authorized read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* fix(tests): refresh test tabs after a run, keep saved edits successful
- A finished Mothership run refreshes its tests instead of upserting tabs, so a
test deleted mid-run does not come back.
- A failed file-tab lookup after a saved edit opens no tab instead of
reporting the edit as failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* fix(tests): starter source imports every test helper
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* build(tests): add @vitest/expect and @vitest/spy for the sandbox bundle
The vitest-expect sandbox bundle builds from these packages; rebuilt with the
Reflect-based event polyfills.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* build(tests): tell knip the sandbox bundle uses @vitest/expect and @vitest/spy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* feat(tests): name MCP and custom tool mocks by server and title
MCP tool ids embed the server's database id, which changes when a server is
re-added or a workspace is forked, so a stale mock silently stopped matching
and the real server was called. Tests now name workspace tools the way the
workspace does: mockTool({ mcp: 'Server', tool: 'name' }) resolved per run
(failing on an unknown or ambiguous server), and mockTool({ customTool:
'Title' }) matched case- and space-insensitively. Raw mcp- and custom_ ids
are rejected; built-in catalog ids are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* fix(tests): reserve the run name, hold Run for unsaved edits, cancel judges on close
- A test named "run" collided with the static run endpoint, so its detail
page got a 405; the name is now reserved.
- Run is disabled while the open editor holds edits the server has not
saved (including a refused save), so a run never uses the previous source.
- toMatchRubric model calls are aborted when the sandbox run ends, so a
stopped test no longer keeps calling or billing the judge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
* fix(tests): fail a run whose selected case names no test in the file
A renamed or misspelled `only` path skipped every case, and the run was then
saved as passing. The harness now rejects unknown names, so the run is
recorded as an error with the names it could not find.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EHGBgHrtePpi7KMrEfy41R
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>1 parent 66e535d commit 900afc3
144 files changed
Lines changed: 37125 additions & 230 deletions
File tree
- apps/sim
- app
- api
- v2/chat
- workspaces/[id]/tests
- [name]
- runs/[runId]
- run
- o/[organizationId]
- home/components/composer
- workspace/[workspaceId]
- files/components/file-viewer
- home
- components
- mothership-view
- components
- add-resource-dropdown
- resource-content
- resource-registry
- user-input/components
- hooks/stream
- providers
- tests
- [name]
- components
- w/components/sidebar
- executor
- execution
- handlers/workflow
- hooks/queries
- lib
- api/contracts
- billing
- core/config
- execution
- sandbox
- bundles
- internal/file
- logs
- mothership
- agent-cli
- application
- chat
- generated
- request
- go
- tools
- resources
- tools/server
- files
- vfs
- uploads
- contexts/workspace
- workflow-tests
- application
- workflows
- application
- executor
- workspace-files
- application
- providers
- sandbox-tasks
- stores
- logs/filters
- workflow-tests
- packages
- db
- migrations
- meta
- testing/src/mocks
- scripts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | | - | |
| 87 | + | |
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| |||
417 | 417 | | |
418 | 418 | | |
419 | 419 | | |
| 420 | + | |
420 | 421 | | |
421 | | - | |
| 422 | + | |
422 | 423 | | |
423 | 424 | | |
424 | 425 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| 39 | + | |
39 | 40 | | |
40 | 41 | | |
41 | 42 | | |
| |||
343 | 344 | | |
344 | 345 | | |
345 | 346 | | |
| 347 | + | |
| 348 | + | |
346 | 349 | | |
347 | 350 | | |
348 | 351 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
Lines changed: 27 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
Lines changed: 3 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
232 | 232 | | |
233 | 233 | | |
234 | 234 | | |
| 235 | + | |
235 | 236 | | |
236 | 237 | | |
237 | 238 | | |
| |||
309 | 310 | | |
310 | 311 | | |
311 | 312 | | |
| 313 | + | |
312 | 314 | | |
313 | 315 | | |
314 | 316 | | |
| |||
403 | 405 | | |
404 | 406 | | |
405 | 407 | | |
| 408 | + | |
406 | 409 | | |
407 | 410 | | |
408 | 411 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
55 | 56 | | |
56 | 57 | | |
57 | 58 | | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
69 | 72 | | |
70 | 73 | | |
71 | 74 | | |
72 | 75 | | |
73 | 76 | | |
74 | 77 | | |
75 | 78 | | |
| 79 | + | |
76 | 80 | | |
77 | 81 | | |
78 | 82 | | |
| |||
Lines changed: 13 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
123 | 129 | | |
124 | 130 | | |
125 | 131 | | |
| |||
175 | 181 | | |
176 | 182 | | |
177 | 183 | | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
178 | 187 | | |
179 | 188 | | |
180 | 189 | | |
| |||
253 | 262 | | |
254 | 263 | | |
255 | 264 | | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
256 | 268 | | |
257 | 269 | | |
258 | 270 | | |
| |||
Lines changed: 29 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
405 | 406 | | |
406 | 407 | | |
407 | 408 | | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
408 | 412 | | |
409 | 413 | | |
410 | 414 | | |
| |||
422 | 426 | | |
423 | 427 | | |
424 | 428 | | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
425 | 432 | | |
426 | 433 | | |
427 | 434 | | |
| |||
503 | 510 | | |
504 | 511 | | |
505 | 512 | | |
| 513 | + | |
506 | 514 | | |
507 | 515 | | |
508 | 516 | | |
| |||
608 | 616 | | |
609 | 617 | | |
610 | 618 | | |
611 | | - | |
| 619 | + | |
| 620 | + | |
612 | 621 | | |
613 | 622 | | |
614 | 623 | | |
| |||
626 | 635 | | |
627 | 636 | | |
628 | 637 | | |
629 | | - | |
| 638 | + | |
630 | 639 | | |
631 | 640 | | |
632 | 641 | | |
| |||
639 | 648 | | |
640 | 649 | | |
641 | 650 | | |
642 | | - | |
| 651 | + | |
| 652 | + | |
643 | 653 | | |
644 | 654 | | |
645 | 655 | | |
| |||
789 | 799 | | |
790 | 800 | | |
791 | 801 | | |
792 | | - | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
793 | 806 | | |
794 | 807 | | |
795 | 808 | | |
| |||
847 | 860 | | |
848 | 861 | | |
849 | 862 | | |
850 | | - | |
851 | | - | |
852 | | - | |
853 | | - | |
854 | | - | |
855 | | - | |
856 | | - | |
857 | | - | |
858 | | - | |
859 | | - | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
860 | 875 | | |
861 | 876 | | |
862 | 877 | | |
| |||
0 commit comments