Skip to content

fix(reward): add resource limits to the untrusted-code grader (refs #71)#111

Open
shin-core wants to merge 1 commit into
mini-router:mainfrom
shin-core:sn74-shin-core-sandbox-home
Open

fix(reward): add resource limits to the untrusted-code grader (refs #71)#111
shin-core wants to merge 1 commit into
mini-router:mainfrom
shin-core:sn74-shin-core-sandbox-home

Conversation

@shin-core

@shin-core shin-core commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

⚠️ Scope changed since your approval — please re-review

@tmimmanuel — since you approved this, the core secret-leak fix for #71 landed on main (isolated throwaway HOME/cwd, scrubbed env, python -I). I've rebased this PR down to only the part that main still lacks: resource limits. It no longer touches the HOME/env logic and does not re-implement what's already merged.

What this now adds (purely additive — 73 insertions, 0 deletions)

Main's sandbox closes the exfiltration vector but has no resource caps, so an untrusted candidate can still exhaust host memory or fork-bomb the eval box within its wall-clock timeout.

  • _rlimit_preexec() wired as preexec_fn on the grading subprocess: best-effort POSIX setrlimit for address space (2 GiB), CPU (30s), and process count (64), each guarded so a platform rejecting one still applies the others; returns None (no hook) off POSIX.

Tests

POSIX-gated tests appended to main's tests/test_reward_sandbox.py:

  • a 4 GiB memory-bomb candidate is now killed instead of passing (verified it does pass on main without this change),
  • ordinary grading is unaffected.

Still out of scope (flagged for validator maintainers)

OS-level isolation — network namespace, filesystem confinement, unprivileged user (issue #71 items #2–4) — belongs at the validator host/deploy layer, not a portable code change. The grader should still run on an isolated host with no live credentials.

Refs #71

Notes

A corresponding lab-notebook entry was added to docs/JOURNAL.md per repository protocol (AGENTS.md §6).

🤖 Generated with Claude Code

@github-actions github-actions Bot added docs Documentation changes miner Miner contribution PR train Training or routing changes labels Jul 11, 2026
@tmimmanuel tmimmanuel added the bug Something isn't working label Jul 12, 2026

@tmimmanuel tmimmanuel left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The patch does what the issue asks for: the grader now runs each candidate in a throwaway home/cwd with a hard allowlist env and adds POSIX rlimits. The new tests cover both the leak PoC and normal grading, so the secret-exposure path is addressed cleanly.

The HOME/secrets leak from mini-router#71 is now fixed on main (isolated HOME/cwd,
scrubbed env, python -I), but the grading subprocess still has no resource
caps, so an untrusted candidate can exhaust host memory or fork-bomb the
eval box within its timeout. Verified: a 4 GiB allocation runs to
completion and 'passes' on main.

Add _rlimit_preexec() and wire it as preexec_fn: best-effort POSIX
setrlimit for address space (2 GiB), CPU (30s), and process count (64),
each guarded, returning None (no hook) off POSIX. Purely additive on top
of main's sandbox.

Append POSIX-gated tests to tests/test_reward_sandbox.py: the memory-bomb
candidate is now killed instead of passing; ordinary grading unaffected.

Refs mini-router#71

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shin-core
shin-core force-pushed the sn74-shin-core-sandbox-home branch from 5b09e2f to 80d8121 Compare July 13, 2026 09:17
@shin-core shin-core changed the title fix(reward): code grader leaks HOME/secrets to untrusted candidate code (#71) fix(reward): add resource limits to the untrusted-code grader (refs #71) Jul 13, 2026
@shin-core

Copy link
Copy Markdown
Contributor Author

Heads-up: the core #71 fix (isolated HOME/scrubbed env/-I) landed on main after your approval, so I rebased this down to only the resource-limit layer main still lacks (RLIMIT_AS/CPU/NPROC via preexec_fn). It's now purely additive (73 insertions, 0 deletions) and no longer overlaps the merged fix. Re-review would be appreciated since the scope changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working docs Documentation changes miner Miner contribution PR train Training or routing changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants