Skip to content

feat(inference): add per-sandbox gateway routes#1132

Closed
shayanjm wants to merge 1 commit intoNVIDIA:mainfrom
shayanjm:776-per-sandbox-inference-sjm
Closed

feat(inference): add per-sandbox gateway routes#1132
shayanjm wants to merge 1 commit intoNVIDIA:mainfrom
shayanjm:776-per-sandbox-inference-sjm

Conversation

@shayanjm
Copy link
Copy Markdown

@shayanjm shayanjm commented May 2, 2026

Summary

Add gateway-managed per-sandbox inference overrides so multiple sandboxes can share one gateway while each keeps using the stable https://inference.local endpoint. The gateway resolves a sandbox-specific provider/model route when configured and otherwise falls back to the gateway default.

Related Issue

Closes #776

Changes

  • Added sandbox-scoped inference RPCs for setting, reading, and clearing one sandbox inference override.
  • Updated bundle resolution so sandboxes pass their sandbox ID and receive the matching gateway-owned route without changing sandbox-local policy or hostname behavior.
  • Added openshell inference sandbox set/get/clear CLI commands that resolve sandbox names to IDs before calling the inference service.
  • Added Python SDK parity for the new sandbox inference APIs.
  • Cleaned up sandbox inference overrides when a sandbox is deleted.
  • Updated architecture and user-facing inference docs to describe gateway defaults and per-sandbox overrides.
  • Added unit coverage and a Docker e2e test proving OpenAI-compatible and Anthropic-compatible sandboxes remain isolated behind inference.local.
  • Fixed the forwarding lifecycle test to allocate an available loopback port instead of assuming port 8080 is free.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Additional validation run locally:

  • cargo test -p openshell-cli --test sandbox_create_lifecycle_integration sandbox_create_keeps_sandbox_with_forwarding
  • env OPENSHELL_E2E_DOCKER_TEST=multi_sandbox_inference mise run e2e:docker
  • mise run ci
  • git diff --check

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: shayanjm <shayan.mohanty@gmail.com>
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 2, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 2, 2026

Thank you for your submission! We ask that you sign our Developer Certificate of Origin before we can accept your contribution. You can sign the DCO by adding a comment below using this text:


I have read the DCO document and I hereby sign the DCO.


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the DCO Assistant Lite bot.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 2, 2026

Thank you for your interest in contributing to OpenShell, @shayanjm.

This project uses a vouch system for first-time contributors. Before submitting a pull request, you need to be vouched by a maintainer.

To get vouched:

  1. Open a Vouch Request discussion.
  2. Describe what you want to change and why.
  3. Write in your own words — do not have an AI generate the request.
  4. A maintainer will comment /vouch if approved.
  5. Once vouched, open a new PR (preferred) or reopen this one after a few minutes.

See CONTRIBUTING.md for details.

@github-actions github-actions Bot closed this May 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Support multiple sandboxes with different inference providers

1 participant