The orchestrator is a .NET BackgroundService that drives work items through
the pipeline, running several at once under a set of admission limits. This
page covers sizing those limits, detecting a hung agent, and pausing dispatch —
globally, per project, or per agent.
Three independent knobs control worker admission and sandbox pressure:
| Config key | Type | Default | Purpose |
|---|---|---|---|
CodeyBox:WorkerPool:MaxConcurrentWorkers |
int |
1 |
Hard cap on simultaneous in-flight work items |
CodeyBox:WorkerPool:MaxConcurrentSandboxes |
int |
2 * MaxConcurrentWorkers |
Global cap on live sandboxes/VMs across every phase |
CodeyBox:WorkerPool:MinSpawnInterval |
string (TimeSpan) |
"00:00:00" (none) |
Minimum wall-clock gap between consecutive spawns |
CodeyBox:WorkerPool:PreferInFlightOverFresh |
bool |
true |
Rank items past the work phase ahead of equal-priority fresh starts in dispatch ordering, tie-broken by queue position |
CodeyBox:WorkerPool:DispatchGateAcquisitionBackoff |
string (TimeSpan) |
"00:00:01" |
Backoff between dispatch pickups after a SQLite write-gate acquisition timeout |
CodeyBox:WorkerPool:MaxConsecutiveDispatchGateTimeoutsBeforeEscalation |
int |
10 |
Consecutive pickup gate timeouts before fatal escalation (host stops, non-zero exit). Waits on an in-budget SQLite maintenance hold do not count. |
Dispatch pickup reads run on dedicated SQLite connections and never contend
for the write gate, so a slow writer only delays pickup when pickup itself
needs to write. When a pickup attempt cannot acquire the gate, the loop logs
a Warning naming the current holder, waits
DispatchGateAcquisitionBackoff, and retries on the next dispatch turn —
the timeout never faults the host by itself. If timeouts persist for
MaxConsecutiveDispatchGateTimeoutsBeforeEscalation consecutive attempts,
the loop escalates fatally: the host stops with a non-zero exit code so a
supervisor (Restart=on-failure) restarts the process instead of sitting
idle. An intentional shutdown still exits 0.
Periodic SQLite maintenance (VACUUM) holds the write gate while it rewrites
the state database file. That hold is announced up front with an expected
budget derived from SqliteMaintenance:VacuumTimeout, and waits caused by
it do not count toward the escalation threshold while the hold stays inside
budget — a routine VACUUM cannot stop the host. A maintenance hold that
outlives its budget counts like any other stuck holder and still escalates.
Startup validation rejects a VacuumTimeout larger than
SqliteWriteGate:AcquisitionTimeout x
MaxConsecutiveDispatchGateTimeoutsBeforeEscalation, so lengthening the
maintenance budget for a larger database requires explicitly widening the
escalation window to match.
Controls how many work items run their full pipeline at the same time.
The orchestrator uses a SemaphoreSlim of this size: once the cap is reached
the dispatch loop blocks on the semaphore until a slot is released by a
finishing worker.
Set this to match how many independent pipeline items should make progress at once. It is not a VM ceiling: one item can create additional sandboxes during audit, merge/rebase, required-build verification, smoke checks, or security review.
Controls sandbox admission for the placement path and the non-placement
fallback. Work-phase acquisitions route through sandbox placement, where each
SandboxClass member owns its own admission gate sized from the member's
Capacity, and the process-wide ceiling is derived as the sum of member
capacities. Permits are held for the sandbox's whole lifetime and
released on disposal. Each registry provider kind keeps its admission wrapper
for lifecycle tracking and metrics, but that kind gate is derived from the
same catalog (kind target = sum of member capacities naming it) so it never
clamps what the member gates admit. The legacy single-provider fallback (and
phase acquisitions that bypass placement) still share an admission gate of
this size, so every CreateAsync call path remains budgeted.
When unset, the default is 2 * MaxConcurrentWorkers: every worker can hold
its phase sandbox while acquiring the next phase's sandbox (work → audit,
audit → merge handoffs) without deadlocking the pool. It also seeds the
capacity of the default single-member class synthesized when no
SandboxClasses are configured, so single-member behaviour matches the
former process-wide gate. Set it explicitly on hosts with a known VM
capacity. Values below 2 * MaxConcurrentWorkers are rejected at startup
and on hot-reload, and every member capacity is re-checked against twice the
worker count on every reload: a resize dropping a member below the minimum
is refused with an error naming the member and the minimum, leaving prior
values in force.
This value is hot-reloadable: editing MaxConcurrentSandboxes resizes the
live admission gate without restarting CodeyBox. Raising it admits queued
sandbox creations immediately; lowering it never aborts in-flight sandboxes —
new admissions above the new target stay blocked until holders dispose and the
count converges down. Every change is logged as
Hot-reloaded WorkerPool:MaxConcurrentSandboxes: <old> → <new> (admitted=<in-flight>) so operators can confirm the effective VM ceiling.
Member capacity edits log per member as
Hot-reloaded SandboxClasses member '<id>' capacity: <old> → <new> (in-flight=<in-flight>).
MaxLlmAuditorParallelism remains a per-project audit policy. It bounds how
many LLM auditors one item may try to run concurrently, but those auditor
sandbox creates still queue behind the sandbox admission budget. The effective VM
ceiling on the placement path is:
live sandboxes <= sum of SandboxClasses member capacities
not:
MaxConcurrentWorkers * MaxLlmAuditorParallelism
A worker may transiently hold its phase sandbox while acquiring the next
phase's sandbox (work → audit, audit → merge handoffs), so one worker can
hold 2 permits at once. Startup validation therefore rejects any
MaxConcurrentSandboxes below 2 * MaxConcurrentWorkers with an error
naming both keys and the minimum — the 2026-09-11 incident ran 6 workers
against 6 permits, and every worker ended up holding one permit while
waiting for another that never freed.
Beyond that structural bound, the pipeline surrenders its work-phase reusable
sandbox before phases that provision their own sandboxes (audit fan-out,
merge, conflict rework), so a worker never idles on a permit wait while
holding its work permit. Phase sandbox handles are still await using, so a
phase releases its token when its sandbox is disposed.
A permit wait that outlasts
CodeyBox:PipelineTuning:SandboxPermitWaitWarningThreshold (default 5
minutes) logs a Warning naming the waiting work item and phase with the
current admitted/max counts, so a saturated ceiling is diagnosable instead
of silent. Auditor creation remains cancellable and queued FIFO at the
provider boundary.
Enforces a minimum wall-clock delay between two successive worker spawns. This is unrelated to how long items take to run — a worker that finishes does not reset or consume the interval; only new spawns are paced.
Use this when several work items share a rate-limited API quota (e.g. Claude Pro or Codex Plus subscription tokens). Firing all workers at once saturates the quota immediately; pacing them spreads the API calls out over time so each agent actually gets useful work done before hitting a limit.
The value is a .NET TimeSpan string, e.g. "00:00:30" for 30 seconds.
It is hot-reloadable: the new floor applies to the next worker spawn (a spawn
already waiting out the old interval keeps the bound it started with).
MaxConcurrentWorkers, MaxConcurrentSandboxes, MinSpawnInterval, and
PreferInFlightOverFresh are re-bound live by AgentConfigHotReload — no
restart needed, in-flight work is never aborted by a resize, and an ordering
preference flip applies to the next pickup. The remaining WorkerPool knobs
(DispatchGateAcquisitionBackoff,
MaxConsecutiveDispatchGateTimeoutsBeforeEscalation, NoProgressBackoffBase,
NoProgressBackoffMax, MaxNoProgressRedispatches) are captured into the
orchestrator's startup options snapshot and still require a restart, because
no live reload bridge re-binds them yet. The exact split is codified in
WorkerPoolHotReloadPolicy next to WorkerPoolOptions.
t=0s spawn worker 1 ─────────┐
t=10s spawn worker 2 ───────┐ │ (MinSpawnInterval=10s, MaxConcurrent=4)
t=20s spawn worker 3 ─────┐ │ │
t=30s spawn worker 4 ───┐ │ │ │
│ │ │ │
▼ ▼ ▼ ▼ (workers finish at varying times)
t=35s worker 3 done → spawn worker 5 at max(t=35s, t=40s) = t=40s
The pacing floor and the concurrency cap interact: if MaxConcurrentWorkers=1
and MinSpawnInterval=30s, at most one item runs at a time and the next item
waits at least 30 s after the previous spawn (not after it finishes).
Single solo developer
"WorkerPool": {
"MaxConcurrentWorkers": 1,
"MaxConcurrentSandboxes": 2,
"MinSpawnInterval": "00:00:00"
}One item at a time, no pacing. Simplest setup for a single-user instance.
Small team, isolated projects
"WorkerPool": {
"MaxConcurrentWorkers": 4,
"MaxConcurrentSandboxes": 6,
"MinSpawnInterval": "00:00:00"
}Up to four items run concurrently, with six total sandboxes/VMs admitted across work, audit, and merge. Suitable when each project uses its own API key and there is no shared quota.
Shared Claude Pro / Codex Plus quota
"WorkerPool": {
"MaxConcurrentWorkers": 4,
"MaxConcurrentSandboxes": 6,
"MinSpawnInterval": "00:00:30"
}Up to four workers, but each new spawn is at least 30 s after the last. This gives each agent a head-start before the next one calls the API, reducing the chance of hitting the subscription rate limit on the first request of every session.
"CodeyBox": { "Concurrency": 4 }is still read. Resolution order is WorkerPool:MaxConcurrentWorkers, then
Concurrency, then 1, so an existing deployment keeps working untouched —
but every other pool knob lives under WorkerPool, so set it there.
A worker slot must be released whichever way the worker exits — success,
exception, cancellation, or an early return before any phase begins. If a slot
is ever left occupied with no worker behind it, the pool reports itself at
capacity while nothing runs: no VM exists, no work item is in a running state
(Working, Merging, Auditing, Reworking), and dispatch stalls with
every failure counter reading zero. That exact shape stalled dispatch for
hours in September 2026.
Two mechanisms close it:
- Structural release. The dispatch loop holds each acquired gate permit in
a scope that releases it on every exit (including a store failure during
pickup or a rejected worker task), and the worker task releases its lease in
a
finallythat thestartedaudit event can no longer bypass. - Reconciliation. On every watchdog sweep where the pool reports itself at
capacity, the orchestrator checks for the orphan signature: no running-state
item and no live sandbox. When it holds, each held slot whose worker has
already exited is reclaimed immediately; a slot whose worker task is still
running is reclaimed only past
CodeyBox:WorkerPoolHealthWatchdog:OrphanedSlotMaxAge(default 30 minutes, well above legitimate provisioning). Reclaimed slots are logged at Warning with the worker/item identities, published as theworker_pool.slot_reclaimedwebhook, and refilled via the normal dispatch wake.
GET /workers/status exposes the current occupancy (occupiedSlots: worker
index, work item, registry id, acquisition time) so a suspected leak is
confirmable directly instead of inferred from unmatched started/finished
log lines.
The orchestrator monitors each running agent for liveness. If an agent stops consuming CPU and has no open network sockets for a configurable period, it is classified as stuck (deadlocked, blocked on a closed socket, blocked on stdin, etc.) and killed so the slot can be recovered.
While a work, rework, or merge phase is running, a background probe samples agent activity every 30 seconds:
| Dimension | What is measured | "Active" if … |
|---|---|---|
| CPU | utime + stime from /proc/<pid>/stat |
tick counter increased between two samples |
| Network | Open socket file-descriptors in /proc/<pid>/fd |
at least one socket FD exists |
The agent is classified stuck when both dimensions show zero activity for
StuckThresholdMinutes consecutive minutes (default 10 min = 20 samples ×
30 s). Once stuck is detected:
- Audit event
agent.stuck_detectedis logged. - The phase's
CancellationTokenis cancelled, causing the sandbox to kill the agent process. - Audit event
agent.killed_by_stuck_probeis logged. - Webhook event
work_item.agent_stuckis fired (see webhooks.md). - The work item transitions to Failed (or is re-queued if
AutoRetryOnStuckis enabled — see projects.md).
It reads /proc on the host, so it only sees an agent that is a host process
in the orchestrator's own process tree:
| Provider | CPU + socket probe |
|---|---|
process |
yes — the agent is a direct child |
bubblewrap |
yes — host-visible PIDs, per-process netns, and the orchestrator owns the process |
multipass, incus, multipass-remote, sprites |
no — the agent runs inside a VM, invisible to host /proc |
Inside a VM TryRead() returns null every sample, no stuck classification
happens, and the phase timeout is the only protection. The same is true on
macOS and Windows, where the source is null.
The ancestor filter matters on a workstation: only processes whose parent chain
reaches the orchestrator PID count, so an operator's own interactive claude or
codex session sitting idle cannot be mistaken for a stuck agent.
Default: 10 minutes (20 × 30 s samples).
Choosing a threshold involves two failure modes:
| Too low | Too high |
|---|---|
Kills agents that are just slow (large LLM response streaming, big git clone) |
Wastes compute and human time on a genuinely deadlocked agent |
Rules of thumb:
- Set to at most half of your phase timeout. If
WorkTimeout = 30 min, a threshold ≤ 15 min ensures the probe fires before the timeout. - For phases that routinely produce large diffs or audit many files, start with the default (10 min) and only reduce if you're confident your slowest legitimate run completes within that window.
- Set
StuckThresholdMinutes = 0to disable the probe entirely for a project (e.g. when you want to investigate hangs manually via the VM console rather than auto-recovering).
| Audit event | Logged when |
|---|---|
agent.stuck_detected |
Probe classified the agent as stuck (includes phase, agent kind, stuck duration in seconds) |
agent.killed_by_stuck_probe |
Phase CTS was cancelled and the agent was killed |
Both events are at Warning level. Per-sample CPU-delta and socket-count readings are logged at Debug level (30 s × multi-hour runs → very noisy at higher levels).
Operators can pause and resume the global pickup queue without restarting the orchestrator. A paused queue blocks all new work-item pickup; in-flight items continue normally.
Running ──(POST /queue/pause)──► Paused
Paused ──(POST /queue/resume)──► Running
State is persisted to the same SQLite database as work items (queue_state
table, single row). A restart with the queue in Paused state logs a
Warning audit event (queue.started_while_paused) so operators don't
forget they left it paused.
- Incident response: a project is generating runaway items. Pause the queue, cancel the offending items via the API, then resume.
- Maintenance window: pause before a dependency upgrade, resume after.
- Investigations: stop the dispatch loop to inspect state before it changes.
| What | Behaviour |
|---|---|
| New item pickup | Blocked — items stay in the Queued state |
| In-flight workers | Unchanged — run to completion normally |
| Webhook delivery | Unchanged — audit logging and webhooks continue |
| State persistence | Yes — survives a restart |
Pausing is not the same as cancelling. Items blocked by the pause gate remain Queued and are picked up automatically on resume.
Pause alone does not wait: it returns immediately while in-flight workers
keep running, so restarting right after pausing still interrupts running work
(which recovery then re-queues). POST /queue/drain closes that gap — it
pauses new pickup and then blocks until no workers are running or the deadline
elapses:
POST /queue/pause |
POST /queue/drain |
|
|---|---|---|
| New item pickup | Blocked | Blocked (pauses first if still running) |
| In-flight workers | Unaffected, keeps running | Waits until none are running |
| Returns | Immediately | When quiescent or timeoutSeconds elapses |
| Queue state after | Paused | Paused (resume it, or restart, when ready) |
Safe-restart sequence: drain → restart → resume. When drained is true,
every worker has reached a safe boundary and the restart disturbs nothing.
When it is false, some workers were still running at the deadline — drain
again or restart anyway and let recovery re-queue the interrupted items.
GET /queue/status → { state, pausedAt, pausedReason, refactorGates }
POST /queue/pause body: { "reason": "..." } → { state, pausedAt }
POST /queue/resume → { state }
POST /queue/drain body: { "reason": "...", "timeoutSeconds": 300 }
→ { state, drained, currentlyRunning, pausedAt, pausedReason }
Operators must supply a non-empty reason when pausing. The reason is stored in the audit log and shown in the admin dashboard banner.
refactorGates lists project-scoped refactor drains and locks. A
draining entry means a queued refactor has reached its normal dispatch turn
for that project and is holding all fresh same-project non-refactor starts,
including later higher-priority items, while existing in-flight work completes.
A locked entry means the refactor itself is in flight.
| Event | Payload |
|---|---|
queue.paused |
{ pausedAt, reason, pausedBy } |
queue.resumed |
{ resumedAt } |
The queue index page shows a coloured banner at the top:
- Paused (red): "QUEUE PAUSED — reason: … — paused at …" + Resume queue button.
- Running (green): subtle dot + Pause queue button (opens a modal asking for a reason).
Operators can pause and resume one agent kind without stopping the whole
queue. The pause is a pickup gate only: in-flight runs are not killed, and all
other agents continue dispatching normally.
When an agent class pools multiple subscriptions for one kind, pause the
specific route key (claude/acct-a) to leave its siblings dispatching.
GET /agents/paused
POST /agents/{kind}/pause body: { "reason": "...", "durationSeconds": 21600 }
POST /agents/{kind}/resume body: { "reason": "..." }
POST /agents/{kind}/instances/{instanceId}/pause
POST /agents/{kind}/instances/{instanceId}/resume
codeybox agents pause claude --reason "reserve quota" --for 6h
codeybox agents pause claude/acct-a --reason "account flagged today"
codeybox agents resume claude
codeybox agents paused
The paused set is persisted in SQLite and survives orchestrator restart. A pause can be indefinite or have an expiry; expired pauses auto-resume on the next pause-state read.
| What | Behaviour |
|---|---|
| New work/rework/audit/merge dispatch | Paused agent is excluded from eligible candidates |
| One pooled instance is paused | That route key is excluded; same-kind siblings remain eligible |
| Only eligible agent is paused | Item parks at WaitingForAgentResume and resumes automatically on unpause |
| In-flight run on paused agent | Continues normally |
/quota and dashboard |
Show paused status separately from quota exhaustion |
| Queued control work item | agentControl work items bypass agent routing so they can resume a paused agent |
The audit log records agent.paused, agent.resumed,
agent.pause_dispatch_deferred, and
agent.pause_waiting_item_resumed entries for operator traceability.
| Event name | Logged when |
|---|---|
worker_pool.spawn_throttled |
A spawn waited non-zero time due to MinSpawnInterval (includes actual wait ms) |
worker_pool.worker_started |
A worker task starts (worker index + work item ID) |
worker_pool.worker_finished |
A worker task completes (worker index + work item ID) |
queue.paused |
Operator paused the queue (includes reason) |
queue.resumed |
Operator resumed the queue |
queue.started_while_paused |
Orchestrator started with queue already paused |
budget.deferred |
Work item deferred by a per-project budget cap |
GET /workers/status
Returns a JSON snapshot of the pool:
{
"maxConcurrent": 4,
"currentlyRunning": 2,
"queuedCount": 3,
"lastSpawnAt": "2026-04-30T12:34:56.789+00:00",
"occupiedSlots": [
{
"workerIndex": 7,
"workItemId": "fdc4158c-...",
"registryWorkerId": null,
"acquiredAt": "2026-04-30T12:34:50.123+00:00"
}
]
}lastSpawnAt is null if no worker has been spawned since startup.
occupiedSlots lists every slot the pool currently believes is occupied (worker index, work item id, registry worker id once registered, acquisition time); empty when no slots are held.