Repository navigation
feat(pod): per-pod memory and CPU limits, restart that keeps home - #24652
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns: the ceilings reuse the per-pod env file and drop-in that already exist, and restart is halt-then-start under the reentrant name mutex, so the home is kept. Notes
[DESIGN-REVIEWED] 1384c38 Description read[DESCRIPTION-READ] 953ac2b7716540ee8ea8d69062503850e9f8c24ce2620e4ce4bc4d9619458b53 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Every item traces to the defect in #24651: a fixed 4G/200% ceiling, and a restart path that wipes the home. Nothing here blocks the merge. Notes
What this change shipsInventory (9 items) — 7 justifiedIntent: let a developer or agent give one pod more memory and CPU, and apply the change without losing the pod's home. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 1384c38 Description read[DESCRIPTION-READ] 7ddd9bb91414d8fa839091b31ea9252e2ceddbd762bedb36652e571a80b7c2e4 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
fd95bb9 to
828f488
Compare
828f488 to
9f0e15c
Compare
|
issue (blocking): Windows restart stopped the pod and could not start it again Fixed in 9f0e15c.
|
|
issue (blocking): pod restart left a crash-looping unit after a failed health wait Fixed in 9f0e15c.
|
|
issue (blocking, security): Windows agent restart deleted the home before validating limits Fixed in b5a452f.
|
b5a452f to
17d9510
Compare
17d9510 to
ef356a1
Compare
|
issue (blocking): restart tests depended on the runner's platform Fixed in ef356a1.
|
16b0197 to
3b82a20
Compare
|
GPT 6.1 Review ended as review incomplete on 3b82a20 without a verdict; could a writer rerun that lane? |
3b82a20 to
d938d59
Compare
d938d59 to
8b6b7fa
Compare
|
Please re-run GPT 6.1 Review, Opus 5.5 Review and Security Scope Review on 8b6b7fa: each reports "could not run" with no verdict. |
8b6b7fa to
c0ec985
Compare
|
Please re-run the failed jobs of run 40856317362: Backend Tests (3.12, 4) failed only on test_remote_workspace_file_integrity.py::test_a_regex_that_overflows_is_a_tool_error_not_a_dead_server, which this PR does not touch and which passes locally. |
c0ec985 to
5f1e0f5
Compare
kirocrew pod up --memory/--cpu records MEMORY=/CPU= in the pod env file and renders them into the per-pod drop-in, capped against the host. kirocrew pod restart stops and starts the unit without reclaiming the pod home. The same knobs reach the pod_up tool, a new pod_restart tool and the Dev Fleet agent pod API.
|
HOLD (policy-sensitive) at 1384c38. This PR adds a new network route, POST /api/apps/dev-fleet/pod/restart, and admits it as a strict internal path in src/kiro_crew/dashboard/server_internal_paths.py. It also adds a pod_restart tool that stops and restarts a pod gateway. New routes and new process-stop paths are reserved for a human reviewer, so I am not approving. I found no concrete defect in the code I read. The route is machine-only and gated on the app being enabled, and it refuses on Windows before any pod work. restart_pod validates the ceilings before halting, holds the name mutex across halt and start, and starts nothing after a failed halt. The limits module enforces a floor and host caps and re-validates on every start. Each earlier blocking finding has a fix and a test, and Opus and GPT report no findings on this head. AI findings judged non-blocking: Design Note (macOS restart not tried by hand) and First Principles Note (macOS/Windows record ceilings they do not enforce; the CLI and README say so). Before approval, please also open and link issues for the two promised Windows follow-ups: a Windows restart that keeps the home, and --memory in the Windows Job object. To clear: a maintainer reviews the new route and the restart path, and the follow-up issues are linked. — crew-pr-reviewer (r136) |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved on behalf of the watch holder (bolichen). Lanes pass on 1384c38, checks green, no conflict, no open change request.
Problem / Motivation
Goal: Let a developer or agent give one dev pod more memory and CPU, and apply the change with a restart that keeps the pod's home.
Every pod runs under
MemoryMax=4GandCPUQuota=200%from the template unit, with no flag to change them. Agents inside a pod read the 4G ceiling as low memory and stopped spawning subagents, and local speech-to-text took 30 to 60 seconds per utterance at two CPUs. The only restart an agent can reach ispod_downthenpod_up, which deletes the pod home with its apps, config and models.Why it matters
App and agent testing in pods needs more than 4G and two cores, and rebuilding a pod home after every limit change costs a model download and a manual app and voice setup each time. Agents behind a sandbox cannot reach the systemd user bus, so a hand-written drop-in is not something they can apply.
Not a goal
kirocrew pod apifrom an agent shell inside pods. Those are separate issues.What changed (motivation → approach → change)
A pod can now get its own memory and CPU ceilings, and a restart applies them without deleting the pod's sessions, apps and models. A pod already keeps its per-pod settings (
PORT=,SEED=,APPROVAL=) in its env file, andpod upalready rewrites a per-pod systemd drop-in on every start. The ceilings follow that path instead of a new store.kirocrew pod up <wt> --memory 16G --cpu 800%validates both values in the newpod/limits.pyand recordsMEMORY=/CPU=in the env file.--cpu 8means eight cores. The floor is 512M and 10%. Memory is capped at 80% of host RAM, the same_CGROUP_TOTAL_MEMORY_FRACTIONthe agents slice uses, and CPU at 100% per host CPU. A bad value is refused before anything starts.MemoryMax=/CPUQuota=only for values the pod set, so the template stays the one home of the defaults. A start re-validates the env file and refuses a bad hand edit rather than falling back to 4G.kirocrew pod restart <wt>:halt_podthenstart_podunder one hold of the name mutex, so the home is never reclaimed and a concurrentdowncannot interleave. It refuses a pod with no pinned checkout. If the restarted gateway fails its health wait, the command stops the unit withhalt_podand keeps the home, so a crash-looping unit is not left behind. On Windows it refuses before stopping anything, because the Windows start refuses the home and run record the Windows stop keeps; the Dev Fleet button keeps its down-then-up there, and the agentpod_restartroute refuses withpod_restart_unsupportedbefore any pod work, because that down-then-up would delete the home the tool promises to keep.pod status,pod lsand thepod uphandle printmemoryandcpu, in text and--json.pod_uptakes optionalmemory/cpu; newpod_restarttool andPOST /api/apps/dev-fleet/pod/restart, admitted as a fifth exact strict internal path with the same local-caller and internal-secret checks as its siblings.pod restartand keeps the home. Before this it ran down then up, which wiped it.MEMORY=/CPU=, and the CLI says they are not enforced there. A Windows pod keeps the Job object ceilings fromresource_limits. Passing--memoryinto that Job object would changesandbox_slice.py, a sensitive path a fork PR cannot pass, so it is left to a follow-up.Where it lives:
src/kiro_crew/pod/(limits, unit, lifecycle, CLI),cli_parser_admin.py(flags and therestartverb),src/kiro_crew/apps/builtins/dev_fleet/agent_pod_api.pyandworktree_ops_pods.py(routes and pod ops),mcp_tools/apps.pyanddata/mcp_tool_titles.json(the tools),validation.py,src/kiro_crew/validation_parts/app_schemas.pyandserver_registries.py(tool schemas),dashboard/server_internal_paths.py(the strict path), andchannel.pyandsession_environment.py, wherepod_restartjoinspod_upon the channel-blocked and local-only lists because it boots a gateway on this machine. Docs: the pod README, the Dev Fleet spec,mcp.md,src/kiro_crew/apps/builtins/dev_fleet/skills/pod-e2e/SKILL.mdandsrc/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-worktree-dev/SKILL.md..github/black-baseline.txtdropsmcp_tools/apps.py, which is black-clean now.Backwards compatibility
Compatible: every new flag and field is optional, and a pod whose env file has no
MEMORY=orCPU=renders the same drop-in as before.Removes nothing: the Dev Fleet Restart button no longer deletes the pod home, and
pod downremains the one verb that reclaims it.Tests
test/test_pod_limits.py(new): parsing and normalisation, the floor and host caps, unit and drop-in rendering, a start that writes the ceiling and one that refuses a bad hand edit, restart order (halt then start, never stop), a restart that keeps a file in the home, the CLI handles forup,restart,statusandls, and the parser flags.test/test_dev_fleet_pod_agent_api.py: the route forwardsmemory/cpu, a malformed value is a 400 before any pod work, the restart route and tool, and its 409 code.test/test_dev_fleet_server_coverage.py: the Dev Fleet restart runspod restartwith the limits and never calls_pod_down.Proof: not provable -- other: feature PR, the new tests have no pre-fix behaviour to fail against
Manual verification
Run on Linux with systemd from a pod of this branch, so its Dev Fleet agent API ran this branch's CLI on the host bus:
pod/upfor a second pod withmemory: 6G, cpu: 300%. The pod's cgroup readmemory.max=6442450944andcpu.max=300000 100000.pod/restartwith8G, 400%. The cgroup readmemory.max=8589934592andcpu.max=400000 100000, the gateway PID changed, health was 200, and the marker file was still there.pod/downremoved the home and drop-in.Related Issues
Closes #24651
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)