bug(kubernetes): stale boundary reconciliation never converges for suspended sandboxes

[drew]

This was generated by AI during triage.

Agent Diagnostic

• Skills loaded: diagnose, OpenShell repository create-github-issue, and cluster inspection workflows
• OpenShell version tested: 0.0.117-dev.211+g4cd5e5478
• Latest release checked: v0.0.116; the issue reproduces on a newer development build containing the RFC 0012 architecture
• Known fixes reviewed: release notes for v0.0.116, RFC 0012 implementation PR feat(isolation): implement the RFC 0012 sandbox architecture #2942, and the stale Suspended phase fix in fix(gateway): let Ready win over stale Suspended in derive_phase #2933
• Possible duplicates reviewed: searched all OpenShell issues and merged PRs for stale boundary, workload fence generation, and the exact warning text; no matching issue was found. Stop→start leaves sandbox stuck at Starting on Agent Sandbox v1beta1 (stale Suspended condition read as Stopped) #2932 concerns phase derivation from an upstream stale condition and is not this reconcile loop.
• Findings: seven already-suspended Kubernetes sandboxes with obsolete namespace-fence generations produced exactly seven warnings and seven suspension attempts per reconciliation interval. The gateway emitted 140 identical warnings in 20 minutes with no errors. Current main checks the namespace fence generation before calculating whether the sandbox should run, then unconditionally merge-patches the Sandbox to suspend it when the generations differ. The suspension patch does not make the generation comparison converge.
• Remaining reason for filing: the fail-closed transition succeeds, but reconciliation continues performing the same transition indefinitely for terminally suspended objects.

Description

Actual behavior: After the namespace workload fence generation changes, retained Sandbox CRs whose workloads are already suspended remain generation-mismatched. Every reconciliation pass logs:

namespace workload fence generation changed; suspending stale boundary

and calls the dependency-failure suspension path again. In the observed cluster, seven suspended sandboxes generated seven warnings per minute indefinitely—140 warnings in 20 minutes—and prompted repeated Kubernetes merge-patch requests despite already being suspended.

The relevant reconciliation order on current main is:

1. Ensure/read the workload NetworkPolicy.
2. Compare its namespace-fence generation with the Sandbox CR.
3. Suspend and continue on mismatch.
4. Only afterward calculate desired_running.

Consequently, a suspended object cannot reach the later suspended-state reconciliation path when this comparison remains false.

Expected behavior: A namespace-fence mismatch must still fail closed, but the resulting suspended state should converge. Subsequent reconciliation passes should not repeatedly patch an already-suspended Sandbox or emit transition-level warnings unless the state changes or a repair genuinely fails.

Reproduction Steps

1. Deploy the RFC 0012 Kubernetes split workload/supervisor architecture.
2. Create a sandbox and retain its Sandbox CR, boundary resources, and workspace when suspending it.
3. Roll out a namespace workload fence with a new generation, as during a gateway/chart architecture upgrade.
4. Wait for more than one gateway reconciliation interval.
5. Inspect gateway logs and Sandbox API writes.
6. Observe that every interval logs namespace workload fence generation changed; suspending stale boundary and re-enters the suspension helper for the same already-suspended Sandbox.

Environment

• OS: GKE nodes running Ubuntu 24.04, Linux amd64
• Kubernetes: v1.35.7-gke.1222000
• Agent Sandbox controller: v0.5.0, v1beta1 API
• Docker: not applicable; Kubernetes compute driver
• OpenShell: 0.0.117-dev.211+g4cd5e5478
• Deployment: Helm 0.0.0-dev, RFC 0012 separate workload and supervisor pods, Kubernetes NetworkPolicy fence enabled
• Latest release checked: yes, v0.0.116; the tested development build is newer
• Possible duplicates checked: yes; no matching issue found

Logs

# Aggregate over 20 minutes from the gateway pod:
stale_boundary_warnings=140
errors=0

# Repeats once per suspended sandbox on every interval:
WARN openshell_driver_kubernetes::driver: namespace workload fence generation changed; suspending stale boundary sandbox_id="<redacted>"

Acceptance Criteria

• A stale boundary still fails closed when a namespace workload fence generation mismatch is first detected.
• Once the Sandbox is suspended, subsequent reconciliation passes converge without repeated suspension patches.
• Transition-level warnings are emitted once per state transition or are otherwise rate-limited/aggregated, not once per object per interval forever.
• Active sandboxes with mismatched workload or supervisor generations retain the existing fail-closed behavior.
• A Kubernetes regression test runs multiple reconciliation passes against an already-suspended, generation-mismatched Sandbox and verifies stable state without repeated mutations.