Skip to content

Preserve Stirrup code-execution workspaces for scenario-suite runs#454

Merged
DhavalRepo18 merged 2 commits into
mainfrom
stirrup_agent
Jul 15, 2026
Merged

Preserve Stirrup code-execution workspaces for scenario-suite runs#454
DhavalRepo18 merged 2 commits into
mainfrom
stirrup_agent

Conversation

@ChathurangiShyalika

Copy link
Copy Markdown
Collaborator

This PR adds opt-in preservation for Stirrup code-execution workspaces so benchmark artifacts can be inspected after each scenario run.

Changes:

  • Adds --workspace-dir and --preserve-workspace to stirrup-agent.
  • Passes the workspace path into Stirrup’s Docker/local code backend.
  • Copies final code_exec files back into the requested workspace before Stirrup cleanup.
  • Adds scenario-suite flags: --stirrup-workspace-root and --preserve-workspaces

Keeps per-scenario workspaces under:
<root>/stirrup_agent/<model-dir>/stirrup_agent_<scenario_id>/

Example:

uv run python -m benchmark.scenario_suite_runner \
  --scenario-ids benchmarks/scenario_suite/scenarios.txt \
  --scenario-root /.../scenarios_data \
  --agent_name stirrup_agent \
  --model-id tokenrouter/z-ai/glm-5.2 \
  --trajectory-root /tmp/assetopsbench-trajectories \
  --reports-root /tmp/assetopsbench-reports \
  --stirrup-workspace-root /tmp/assetopsbench-stirrup-workspaces \
  --preserve-workspaces \
  --continue-on-error

Validation:

  • Added focused tests for Stirrup workspace preservation.
  • Confirmed CLI help exposes the new flags.
  • Verified focused tests pass.

Signed-off-by: Chathurangi Shyalika <chathurangishyalika@dhcp-9-31-97-67.watson.ibm.com>
@DhavalRepo18 DhavalRepo18 merged commit a826537 into main Jul 15, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants