Skip to content

feat(pod): per-pod memory and CPU limits, restart that keeps home - #24652

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:feat/pod-config-limits
Nov 1, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:feat/pod-config-limits

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let a developer or agent give one dev pod more memory and CPU, and apply the change with a restart that keeps the pod's home.

Every pod runs under MemoryMax=4G and CPUQuota=200% from the template unit, with no flag to change them. Agents inside a pod read the 4G ceiling as low memory and stopped spawning subagents, and local speech-to-text took 30 to 60 seconds per utterance at two CPUs. The only restart an agent can reach is pod_down then pod_up, which deletes the pod home with its apps, config and models.

Why it matters

App and agent testing in pods needs more than 4G and two cores, and rebuilding a pod home after every limit change costs a model download and a manual app and voice setup each time. Agents behind a sandbox cannot reach the systemd user bus, so a hand-written drop-in is not something they can apply.

Not a goal

  • Sizing pods automatically from the host. The defaults stay 4G and 200%.
  • App install, voice setup, or kirocrew pod api from an agent shell inside pods. Those are separate issues.
  • A CPU ceiling on macOS or Windows, which have no matching mechanism.

What changed (motivation → approach → change)

A pod can now get its own memory and CPU ceilings, and a restart applies them without deleting the pod's sessions, apps and models. A pod already keeps its per-pod settings (PORT=, SEED=, APPROVAL=) in its env file, and pod up already rewrites a per-pod systemd drop-in on every start. The ceilings follow that path instead of a new store.

  • kirocrew pod up <wt> --memory 16G --cpu 800% validates both values in the new pod/limits.py and records MEMORY= / CPU= in the env file. --cpu 8 means eight cores. The floor is 512M and 10%. Memory is capped at 80% of host RAM, the same _CGROUP_TOTAL_MEMORY_FRACTION the agents slice uses, and CPU at 100% per host CPU. A bad value is refused before anything starts.
  • The drop-in appends MemoryMax= / CPUQuota= only for values the pod set, so the template stays the one home of the defaults. A start re-validates the env file and refuses a bad hand edit rather than falling back to 4G.
  • New kirocrew pod restart <wt>: halt_pod then start_pod under one hold of the name mutex, so the home is never reclaimed and a concurrent down cannot interleave. It refuses a pod with no pinned checkout. If the restarted gateway fails its health wait, the command stops the unit with halt_pod and keeps the home, so a crash-looping unit is not left behind. On Windows it refuses before stopping anything, because the Windows start refuses the home and run record the Windows stop keeps; the Dev Fleet button keeps its down-then-up there, and the agent pod_restart route refuses with pod_restart_unsupported before any pod work, because that down-then-up would delete the home the tool promises to keep.
  • pod status, pod ls and the pod up handle print memory and cpu, in text and --json.
  • Agent side: pod_up takes optional memory / cpu; new pod_restart tool and POST /api/apps/dev-fleet/pod/restart, admitted as a fifth exact strict internal path with the same local-caller and internal-secret checks as its siblings.
  • The Dev Fleet page's Restart button now runs pod restart and keeps the home. Before this it ran down then up, which wiped it.
  • macOS and Windows record MEMORY= / CPU=, and the CLI says they are not enforced there. A Windows pod keeps the Job object ceilings from resource_limits. Passing --memory into that Job object would change sandbox_slice.py, a sensitive path a fork PR cannot pass, so it is left to a follow-up.

Where it lives: src/kiro_crew/pod/ (limits, unit, lifecycle, CLI), cli_parser_admin.py (flags and the restart verb), src/kiro_crew/apps/builtins/dev_fleet/agent_pod_api.py and worktree_ops_pods.py (routes and pod ops), mcp_tools/apps.py and data/mcp_tool_titles.json (the tools), validation.py, src/kiro_crew/validation_parts/app_schemas.py and server_registries.py (tool schemas), dashboard/server_internal_paths.py (the strict path), and channel.py and session_environment.py, where pod_restart joins pod_up on the channel-blocked and local-only lists because it boots a gateway on this machine. Docs: the pod README, the Dev Fleet spec, mcp.md, src/kiro_crew/apps/builtins/dev_fleet/skills/pod-e2e/SKILL.md and src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-worktree-dev/SKILL.md. .github/black-baseline.txt drops mcp_tools/apps.py, which is black-clean now.

Backwards compatibility

Compatible: every new flag and field is optional, and a pod whose env file has no MEMORY= or CPU= renders the same drop-in as before.

Removes nothing: the Dev Fleet Restart button no longer deletes the pod home, and pod down remains the one verb that reclaims it.

Tests

  • test/test_pod_limits.py (new): parsing and normalisation, the floor and host caps, unit and drop-in rendering, a start that writes the ceiling and one that refuses a bad hand edit, restart order (halt then start, never stop), a restart that keeps a file in the home, the CLI handles for up, restart, status and ls, and the parser flags.
  • test/test_dev_fleet_pod_agent_api.py: the route forwards memory / cpu, a malformed value is a 400 before any pod work, the restart route and tool, and its 409 code.
  • test/test_dev_fleet_server_coverage.py: the Dev Fleet restart runs pod restart with the limits and never calls _pod_down.
  • Updated pins for the new tool and route: channel dispatch list, tool gate, validation facade, route set, schema fields.

Proof: not provable -- other: feature PR, the new tests have no pre-fix behaviour to fail against

Manual verification

Run on Linux with systemd from a pod of this branch, so its Dev Fleet agent API ran this branch's CLI on the host bus:

  • pod/up for a second pod with memory: 6G, cpu: 300%. The pod's cgroup read memory.max=6442450944 and cpu.max=300000 100000.
  • Wrote a marker file into that pod's home, then pod/restart with 8G, 400%. The cgroup read memory.max=8589934592 and cpu.max=400000 100000, the gateway PID changed, health was 200, and the marker file was still there.
  • pod/down removed the home and drop-in.

Related Issues

Closes #24651

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and KiroCrewCaptain as code owners October 27, 2026 00:02
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 1384c3874b42ca455496961656243d1b1e877727 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 1384c3874b42ca455496961656243d1b1e877727 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: the ceilings reuse the per-pod env file and drop-in that already exist, and restart is halt-then-start under the reentrant name mutex, so the home is kept.

Notes

  • The manual check covered only Linux. restart_pod on macOS (launchd halt then start) has not been tried by hand, so one macOS round-trip that keeps the home would back up the README claim.

[DESIGN-REVIEWED] 1384c38

Description read

[DESCRIPTION-READ] 953ac2b7716540ee8ea8d69062503850e9f8c24ce2620e4ce4bc4d9619458b53

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 1384c3874b42ca455496961656243d1b1e877727 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item traces to the defect in #24651: a fixed 4G/200% ceiling, and a restart path that wipes the home. Nothing here blocks the merge.

Notes

  • Item 6 — inherited: on macOS and Windows, --memory/--cpu are written to the env file but nothing enforces them. The only stated reason is "so the pod keeps it if the checkout later moves to a host that can", which is a "so we can later" premise. ## Not a goal already rules out ceilings on those platforms. Subtraction: in _limit_updates, refuse the flags when rt.IS_MACOS/rt.IS_WINDOWS instead of recording them, and delete both "recorded but not enforced" notes.

What this change ships

Inventory (9 items) — 7 justified

Intent: let a developer or agent give one pod more memory and CPU, and apply the change without losing the pod's home. This is an ADDITION.

  1. pod up --memory/--cpu sets per-pod ceilings, checked against a floor and host caps — justified
  2. New pod restart verb restarts a pod and keeps its home — justified
  3. New pod_restart MCP tool and strict internal route; pod_up takes memory/cpu — justified
  4. The Dev Fleet Restart button now keeps the home (down-then-up only on Windows) — justified
  5. pod status, pod ls and the up handle show memory and cpu — justified
  6. macOS and Windows record ceilings they do not enforce, with a warning — inherited
  7. A start refuses a bad hand-edited ceiling instead of falling back to 4G — justified
  8. The pod exec restart hint now suggests pod restart — justified
  9. Black reflow of unrelated code in mcp_tools/apps.py, and its baseline row dropped — rides along

[FIRST-PRINCIPLES-REVIEWED] 1384c38

Description read

[DESCRIPTION-READ] 7ddd9bb91414d8fa839091b31ea9252e2ceddbd762bedb36652e571a80b7c2e4

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 1384c3874b42ca455496961656243d1b1e877727 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1384c38

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 1384c3874b42ca455496961656243d1b1e877727 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1384c38

@Pearcekieser
Pearcekieser force-pushed the feat/pod-config-limits branch from fd95bb9 to 828f488 Compare October 27, 2026 00:41
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@Pearcekieser
Pearcekieser force-pushed the feat/pod-config-limits branch from 828f488 to 9f0e15c Compare October 27, 2026 01:01
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

issue (blocking): Windows restart stopped the pod and could not start it again

Fixed in 9f0e15c.

restart_pod now returns a refusal on Windows before halt_pod runs, so nothing is stopped and the home is kept. The message names pod down then pod up as the way to restart with a fresh home.
The Dev Fleet Restart button keeps its down-then-up on Windows (_pod_restart checks platform_compat.IS_WINDOWS), so the button never reaches the refused path.
A Windows restart that keeps the home needs a restart transition in pod/windows.py that consumes the terminal run record. That is new scope and will be a follow-up issue after this PR.
Test: test/test_pod_limits.py::TestRestartEdges::test_windows_restart_refuses_before_stopping_anything.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

issue (blocking): pod restart left a crash-looping unit after a failed health wait

Fixed in 9f0e15c.

On a failed health wait, _restart now calls rt.halt_pod inside the same name-mutex hold before _die. halt_pod stops the unit and keeps the home, the same call _up makes for a home that held state before its failed start.
A failed halt puts its return code and diagnostic in the message, as _up's halted branch does. stop_pod is never called, so nothing is reclaimed.
Test: test/test_pod_limits.py::TestRestartHealthFailure::test_a_failed_health_wait_halts_the_unit_and_keeps_the_home asserts the order restart then halt, and fails if stop_pod runs.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

issue (blocking, security): Windows agent restart deleted the home before validating limits

Fixed in b5a452f.

handle_pod_restart now refuses on Windows with 409 pod_restart_unsupported before _ops()._pod_restart is reached, so _pod_down never runs from the agent route and nothing is stopped or deleted.
The destructive down-then-up fallback stays only on the dashboard Restart button (http_api.api_dev_fleet_pod_restart), which a person clicks and which behaved that way before this PR.
The pod_restart tool description says it is not available on Windows.
Test: test/test_dev_fleet_pod_agent_api.py::test_pod_restart_route_refuses_on_windows_before_any_pod_work fails if any pod operation is reached.

@Pearcekieser
Pearcekieser force-pushed the feat/pod-config-limits branch 2 times, most recently from b5a452f to 17d9510 Compare October 27, 2026 02:46
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

chore (non-blocking): rebased on main, no behaviour change

Rebased onto e086a08 to restart the review lanes that hit the CI-app rate limit on b5a452f. The diff is unchanged and the targeted pod, Dev Fleet, tool and facade tests pass (391).

@Pearcekieser
Pearcekieser force-pushed the feat/pod-config-limits branch from 17d9510 to ef356a1 Compare October 27, 2026 03:32
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

issue (blocking): restart tests depended on the runner's platform

Fixed in ef356a1.

Every test that drives the non-Windows restart path now pins platform_compat.IS_WINDOWS = False. In test_dev_fleet_pod_agent_api.py that is the unusable-checkout table, the handle test and the 409 refusal test. In test_dev_fleet_server_coverage.py it is the keeps-the-home and CLI-refusal tests.
The Windows refusal test keeps its pin to True.
Checked by running these tests locally with IS_WINDOWS forced to True for the whole session: all pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@Pearcekieser
Pearcekieser force-pushed the feat/pod-config-limits branch from 16b0197 to 3b82a20 Compare October 28, 2026 01:17
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 145 files this PR also touches after 16b0197 went green, so 3b82a20 is the same diff on current main.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 28, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

GPT 6.1 Review ended as review incomplete on 3b82a20 without a verdict; could a writer rerun that lane?

@Pearcekieser
Pearcekieser force-pushed the feat/pod-config-limits branch from 3b82a20 to d938d59 Compare October 28, 2026 23:16
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 305 files this PR also touches after 3b82a20 went green, so d938d59 is the same diff on current main.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 28, 2026
@Pearcekieser
Pearcekieser force-pushed the feat/pod-config-limits branch from d938d59 to 8b6b7fa Compare October 29, 2026 02:31
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 29, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Please re-run GPT 6.1 Review, Opus 5.5 Review and Security Scope Review on 8b6b7fa: each reports "could not run" with no verdict.

@Pearcekieser
Pearcekieser force-pushed the feat/pod-config-limits branch from 8b6b7fa to c0ec985 Compare October 29, 2026 10:16
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Please re-run the failed jobs of run 40856317362: Backend Tests (3.12, 4) failed only on test_remote_workspace_file_integrity.py::test_a_regex_that_overflows_is_a_tool_error_not_a_dead_server, which this PR does not touch and which passes locally.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 29, 2026
@Pearcekieser
Pearcekieser force-pushed the feat/pod-config-limits branch from c0ec985 to 5f1e0f5 Compare October 30, 2026 07:02
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 431 files this PR also touches after c0ec985 was tested, so 5f1e0f5 is the same diff on current main.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 30, 2026
kirocrew pod up --memory/--cpu records MEMORY=/CPU= in the pod env
file and renders them into the per-pod drop-in, capped against the
host. kirocrew pod restart stops and starts the unit without
reclaiming the pod home. The same knobs reach the pod_up tool, a new
pod_restart tool and the Dev Fleet agent pod API.
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD (policy-sensitive) at 1384c38.

This PR adds a new network route, POST /api/apps/dev-fleet/pod/restart, and admits it as a strict internal path in src/kiro_crew/dashboard/server_internal_paths.py. It also adds a pod_restart tool that stops and restarts a pod gateway. New routes and new process-stop paths are reserved for a human reviewer, so I am not approving.

I found no concrete defect in the code I read. The route is machine-only and gated on the app being enabled, and it refuses on Windows before any pod work. restart_pod validates the ceilings before halting, holds the name mutex across halt and start, and starts nothing after a failed halt. The limits module enforces a floor and host caps and re-validates on every start. Each earlier blocking finding has a fix and a test, and Opus and GPT report no findings on this head.

AI findings judged non-blocking: Design Note (macOS restart not tried by hand) and First Principles Note (macOS/Windows record ceilings they do not enforce; the CLI and README say so).

Before approval, please also open and link issues for the two promised Windows follow-ups: a Windows restart that keeps the home, and --memory in the Windows Job object.

To clear: a maintainer reviews the new route and the restart path, and the follow-up issues are linked.

— crew-pr-reviewer (r136)

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on behalf of the watch holder (bolichen). Lanes pass on 1384c38, checks green, no conflict, no open change request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(pod): per-pod memory and CPU limits, and a restart that keeps the home

3 participants