write-protocol lifetime ownership and recovery barriers by ragnorc · Pull Request #378 · ModernRelay/omnigraph

ragnorc · 2026-07-20T22:06:13Z

What & why

This RFC proposes three coordinated changes to close structural defects in write-protocol cancellation safety and recovery residual handling:

W1 — Engine-owned write-protocol lifetime: Shield armed write protocols from caller cancellation by spawning protocol execution as a detached task. Caller disconnection means "stopped waiting for the result", never "abandoned the protocol".
W2 — Full recovery at the write-entry barrier: Escalate rollback-class recovery residuals to RecoveryMode::Full under exclusive admission at the write-entry heal, instead of parking them for process restart. Resolves RecoveryRequired as a transient condition.
W3 — Boot supervision: Replace silent graph exclusion on open failure with an observable quarantined registry state and a capped-backoff re-open loop, making recovery from transient substrate errors automatic.

Together these establish the invariant: Caller fate is not a protocol participant. An armed graph-write protocol completes or compensates under engine ownership regardless of caller cancellation, and resolving any recovery residual requires only exclusive gate acquisition — never process restart.

Backing issue / RFC

Implements / is an accepted RFC: docs/rfcs/0029-write-lifetime-and-recovery-barriers.md

Motivation

A validated production incident showed the chain of failures: client timeout interrupted a multi-statement delete mid-protocol, leaving an Armed sidecar; the live-handle heal is roll-forward-only and parks rollback-class residuals; the next write hit a stale manifest version; boot recovery hit a transient S3 error and silently quarantined the graph until redeploy. Every link was validated against current code (Lance 9.0.0-rc.1 at pinned rev).

The RFC documents this incident in detail (§1) and validates the mechanism at each step. The long-run liability: cancel-safety is maintained by audit today, and every write-path change must re-reason about cancellation at every await. W1 replaces that with a structural property.

Design summary

W1 (Stage 1 — server boundary): Apply the existing spawn-and-clone idiom from server_export to every _as writer handler. The HTTP response contract is unchanged; on client disconnect, the spawned task runs the protocol to terminal state (success or error) while the request future drops.

W1 (Stage 2 — engine level): Move the shield into the engine via Arc<Self> receivers or an Arc<OmnigraphInner> split (shape TBD in review), so the property holds for embedded SDK callers.

W2: When the write-entry heal encounters a rollback-class sidecar, escalate it to RecoveryMode::Full under exclusive admission + existing schema → branch → table gates, instead of parking it. The heal re-reads the sidecar under those gates (existing behavior), dispatches with Full mode (new), and the triggering write proceeds from a fresh capture.

W3: A graph whose open fails enters Quarantined { since, attempts, last_error, retry_at } in the registry. One supervision task per quarantined graph retries the full open_single_graph with capped exponential backoff (proposed: 5 s initial, ×2, 10 min cap). GET /graphs gains a per-graph status field (serving | quarantined) with quarantine detail.

Safety argument

W1: The spawned task holds the Arc and the admission permit; caller cancellation releases neither. The per-actor admission bound is preserved (admission bounds concurrent work, and the work genuinely continues).
W2: Restore needs exclusive authority over affected tables while restore-and-publish completes. In-process writers are excluded by exclusive admission + table gates (stronger than open-time). In-process readers are safe by construction: reads pin manifest-published versions; restore appends a new version; read caches are keyed by version/e_tag. Foreign processes are exactly as (un)protected as at open time (documented single-writer-process boundary unchanged).
W3: The retry unit is the whole idempotent open; recovery

https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

Note

Medium Risk
Changes write-handler lifetime, admission timing, multi-graph boot, and recovery reopen behavior on the server; coverage is heavy (failpoints, multi_graph) but this sits on the commit/recovery boundary.

Overview
Implements RFC-030 at the HTTP server (W1 stage 1, W2(b), W3): armed writes are no longer tied to the client connection, failed graph opens become visible and self-healing, and RecoveryRequired can trigger an in-process reopen instead of leaving the process wedged until restart.

Cancellation-safe writes (shielded_write). Mutating handlers (mutate, load, schema apply, branch create/delete/merge) run their engine protocols inside a detached tokio task while the handler still awaits the result. The workload admission permit moves into that task and is released only when the protocol finishes. If a write returns RecoveryRequired, request_reopen runs inside the shielded task so a disconnected client cannot drop the reopen signal. Merge with delete_branch: true is one shielded composite (merge plus optional source delete under one admission slot).

Quarantine and supervision (W3 + W2(b)). Boot open failures in lenient mode quarantine graphs instead of hiding them: the registry keeps serving handles and a quarantined map; GET /graphs adds status (serving | quarantined) and optional quarantine metadata; graph routes return 503 for quarantined ids (404 only for never-configured graphs). Lenient boot may start with zero serving graphs if every graph is quarantined. A GraphSupervisor loop retries open_single_graph with capped backoff and **publish**es healed handles; it also reacts to reopen notifications from shielded writes.

Supporting changes: omnigraph-api-types / OpenAPI updates, registry publish / set_quarantined, failpoints feature and HTTP failpoint tests, engine rfc030_probe, and doc/RFC/CI updates. Engine-level write shielding (W1 stage 2) and write-entry Full heal (W2(a)) are not in this diff.

^{Reviewed by Cursor Bugbot for commit e112091. Bugbot is set up for automated code reviews on this repo. Configure here.}

Greptile Summary

This PR adds cancellation-safe writes and supervised graph recovery. The main changes are:

Runs HTTP write protocols in detached tasks that retain admission guards.
Reopens graphs after recovery-required write failures.
Tracks failed graph opens as visible quarantine entries.
Retries quarantined graphs with capped backoff.
Exposes serving and quarantine status through GET /graphs.

Confidence Score: 5/5

This looks safe to merge.

Admission guards remain owned until detached write work completes.
Merge and optional branch deletion run as one admission-controlled operation.
Recovery notifications occur before delete errors are converted to response text.
All-quarantined lenient startup now reaches the active retry loop.
No blocking issues were found in the updated code.

Important Files Changed

Filename	Overview
crates/omnigraph-server/src/handlers.rs	Moves write protocols and their admission guards into detached tasks, including the merge-and-delete composite.
crates/omnigraph-server/src/lib.rs	Starts graph supervision and permits lenient startup when every valid configured graph is quarantined.
crates/omnigraph-server/src/registry.rs	Adds atomic serving and quarantine state transitions with URI collision checks.
crates/omnigraph-server/src/supervisor.rs	Adds immediate and backoff-driven graph reopen attempts with recovered-handle publication.
crates/omnigraph-api-types/src/lib.rs	Adds backward-compatible graph status and quarantine details to the management API.

_{Reviews (22): Last reviewed commit: "Merge origin/main (0.9.0 release, Lance ..." | Re-trigger Greptile}

Context used:

Context used - CLAUDE.md (source)
Context used - AGENTS.md (source)

…n barriers Proposes three coordinated, independently landable changes closing the class behind the interrupted-mutation availability incident: engine-owned write-protocol lifetime (cancellation shielding, staged server-first), full recovery escalation at the write-entry heal under exclusive gates (restart barrier -> gate barrier), and server boot supervision that makes quarantine an observable, converging state. The RFC-022 commit protocol, Armed/EffectsConfirmed classification, fail-closed ambiguity, and the single-writer-process boundary are explicitly unchanged. Every motivating claim is validated against current code and Lance upstream source at the pinned rev, including decoding the incident's _versions/18446744073709551611.manifest as an ordinary V2-named commit of table version 4 and locating Lance's verbatim orphaned-transaction- file warning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bb55888a10

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-20T22:08:56Z

+let task = tokio::spawn(async move {
+    engine.mutate_as(&branch, &query, &name, &params, actor_id.as_deref()).await
+});


Move admission guard into the spawned task

In the current run_mutate path the per-actor permit is acquired before this call (handlers.rs:686-689). If Stage 1 follows this snippet, the spawned task owns only the engine and inputs; _admission stays in the handler and is dropped when the client disconnects, releasing the slot while the write continues. That violates this RFC's admission invariant and lets disconnecting clients bypass per-actor concurrency limits, so the spawned future needs to own the guard or the whole admitted operation must be spawned.

Useful? React with 👍 / 👎.

Addressed: the implementation moves the admission guard into the spawned task (533c167), and f85ef06 updates the RFC snippet to show it plus a paragraph defining the shielded unit as the request's full effect envelope (admission + all implied protocols + the reopen notification).

Generated by Claude Code

chatgpt-codex-connector · 2026-07-20T22:08:56Z

+1. **PR-1 (W1 Stage 1):** server-boundary shield + the two W1 tests + stale
+   comment fix. Smallest change, kills the incident's trigger.


Move engine-cancellation test to Stage 2

PR-1 is scoped to the HTTP server-boundary shield, but requiring both W1 tests here also includes the failpoint test above that drops the engine caller future directly. That case remains cancellable until PR-4's engine-level Arc/inner-handle restructure, so PR-1 cannot turn its required red test green. Split the plan so PR-1 only tests a dropped HTTP client and reserve the direct future-drop test for Stage 2.

Useful? React with 👍 / 👎.

Agreed and addressed: PR-1 landed only the server-boundary cell, and f85ef06 annotates RFC §7 accordingly — the engine-boundary red cell is Stage-2/PR-3 scope, with rfc030_probe.rs phase 1 as the engine baseline pin until then.

Generated by Claude Code

…tations-graph-drift-njvits

Feature-gated failpoint test that pins the three behaviors RFC-029's design rests on, at the engine boundary: (1) dropping a mutation future between sidecar arm and confirmation leaks the Armed recovery sidecar (no Drop guard compensates); (2) a subsequent write on the still-live handle returns RecoveryRequired because the write-entry heal is roll-forward-only; (3) an in-process read-write Omnigraph::open runs the Full sweep under the shared root-scoped write queue, resolves the residual, and the previously wedged handle becomes writable again without a process restart. These are assertions of current behavior (green today), recorded as the baseline the RFC's W1/W2/W3 changes will deliberately move: after W1, cell (1) inverts — cancellation can no longer create the residual. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

Code investigation strengthened both bug claims and upgraded the design: - W1 has an exact in-engine precedent: the RFC-026 B1 stream-fold writer already detaches its whole protocol via Arc<Self> + spawn_with_query_io_probes, explicitly so cancellation cannot abandon it, and that helper already solves task-local probe propagation across the spawn. Stage 2 is rescoped to bringing the established writers up to that existing standard. - W2 gains a zero-engine-change implementation, W2(b): a supervised in-process reopen + registry swap. The open-time Full sweep runs under the same root-scoped write queue live handles share and takes exclusive recovery admission, so it serializes against live traffic by construction; the open path's own comment ('only rollback-eligible sidecars wait for this open-time sweep') names it as the designed resolution. W2(b) and W3 unify into one supervision mechanism with two triggers; the heal escalation W2(a) becomes an optional end state and the only part of the RFC that relaxes a stated rule. - Evidence: reference the checked-in tests/rfc029_probe.rs baseline pin and rework the rollout into PR-1 (shield), PR-2 (unified supervision), PR-3 (engine shield), PR-4 (optional heal escalation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…idual class The first probe run refuted the naive Bug-2 reproduction: a mutation cancelled AT the confirmation failpoint on local FS still produced a durably-confirmed sidecar, because storage puts run on spawn_blocking and an already-spawned blocking write completes after the awaiting future is dropped — the residual self-healed on the next write's roll-forward-only heal. On a network object store the in-flight PUT dies with the future, leaving the Armed class that wedges; consistent with the motivating incident occurring on S3. RFC-029 now records this: the cancellation residual's class is a storage-backend-dependent race, and W1 removes the roulette rather than tuning it. The probe is reworked into three deterministic phases, all green: (1) cancellation leaks the sidecar (class-agnostic assert); (2) a failed confirmation write — the failpoint's documented crash model — manufactures Armed deterministically, and the live handle then returns RecoveryRequired with the documented reopen guidance; (3) an in-process read-write open clears __recovery/ and the wedged handle writes again without a process restart, proving the W2(b) supervised-reopen mechanism end to end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

RFC-029 PR-1 prep: expose the engine's failpoint registry to omnigraph-server's feature-gated integration tests, mirroring the omnigraph-cluster passthrough but without dep:fail since the server defines no failpoint hooks of its own. No behavior change; the feature is never enabled in production builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…idual (RFC-029 W1, red) HTTP-boundary cancellation cell: park the engine at the sidecar-confirmation failpoint, abort the task driving the oneshot request (the exact disconnect model — tower drives the handler future inline, so the abort drops it as hyper does on client disconnect), then poll ONLY the filesystem: __recovery/ must empty with no further requests and no reopen. Sidecar deletion happens strictly after the manifest publish, so an empty __recovery/ proves the whole protocol completed under engine ownership. The poll deliberately sends nothing: a follow-up write would run the write-entry heal and mask an unshielded server. RED as predicted: 'dropped mutation abandoned its armed protocol: __recovery/ still holds [<ulid>.json] after the deadline with no further requests'. Green arrives with the next commit's shield. Also: CI runs the new suite in the failpoints step, and the failpoint-names source guard now walks omnigraph-server so the compile-checked-catalog rule covers the new call sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…FC-029 W1 Stage 1) One shielded_write helper + six site conversions (mutate, load, schema apply, branch create/delete/merge, and the merge's follow-up source delete). Cedar authorization, request validation, and per-actor admission stay in the handler in their existing order; the admission guard and owned inputs then move into a spawned task that runs the engine protocol to its own terminal state regardless of the request future's fate. The handler still awaits the result, so the HTTP contract is unchanged — only abandonment semantics change: a client disconnect no longer cancels the commit protocol mid-flight, and the admission slot releases at protocol completion rather than at disconnect. Panic in the protocol surfaces as JoinError -> 500. Turns the previous commit's red HTTP-boundary cancellation cell green; full server suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…ation claims - Correct writes.rs::cancelled_mutation_future_leaves_no_state's stale doc comment: 'only orphaned Lance fragments can remain' predates the RFC-022 sidecar protocol, and the test's final open runs the Full sweep before its assertions, so it cannot observe post-arm residue; point at rfc029_probe.rs as the residue-lifecycle pin. - Qualify writes.md's cancellation paragraph the same way: the no-Lance-write guarantee holds during op execution, while post-arm drops leave recovery-covered residue; the HTTP boundary now shields cancellation entirely (RFC-029 W1 Stage 1). - server.md: document disconnect semantics (writes complete server-side; outcome ambiguous to the disconnected client — verify by read) and admission-slot-until-completion. - testing.md: register the new server failpoints suite and its passthrough feature; names-guard coverage note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…W3 prep) QuarantineInfo + a parallel quarantined map on RegistrySnapshot (a key is never in both maps; disjointness debug-asserted), a RegistryLookup::Quarantined variant, and two production writer methods serialized on the now-ungated mutate mutex — its original doc comment named this exact consumer as the ungating condition: - publish(handle): RCU install for the handle's key, clearing any quarantine entry and replacing a prior serving handle (the supervised-reopen swap; duplicate-URI check skips the replaced key). - set_quarantined(key, info): records/updates a quarantine entry, no-op while the key is serving so a failed supervised reopen of a healthy graph cannot take it down. A quarantined graph whose config declares a per-graph policy keeps any_per_graph_policy true, so bearer auth cannot flap off while the graph heals (strictly safer than today, where the graph vanishes). Nothing populates quarantine state yet: from_handles delegates to from_boot with no entries, and the routing middleware temporarily maps Quarantined to the existing 404 until the lookup-semantics change lands behind its red test. Zero behavior change; unit tests cover publish-replaces-and-clears, noop-while-serving, the auth-flap closure, and concurrent-publish serialization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…r 503 (RFC-029 W3, red) Flip cluster_boot_quarantines_graph_open_failures' lenient-mode assertions to the RFC-029 contract: a failed boot open is observable, configured state, not silent absence. GET /graphs must list both graphs with a status discriminator (serving|quarantined) and quarantine detail (last_error, attempts); routes under the quarantined graph answer 503 (configured-but-healing, retry later); a never-configured id stays 404, pinning Quarantined-vs-Gone. Strict-mode assertions unchanged. RED as predicted: GET /graphs lists only 'good' and the broken graph routes 404. Green arrives with the boot-wiring commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…on GET /graphs (RFC-029 W3) Turns the previous commit's red test green: - open_multi_graph_state keeps each failed graph's startup config, builds a QuarantineInfo entry (attempts=1, the open error verbatim, policy_configured from the config), and retains clones of EVERY configured graph's startup config on GraphRouting.startup_configs for the upcoming supervision loop (healthy graphs included — a supervised reopen needs them too). Strict --require-all-graphs still bails before any quarantine state exists; the no-healthy-graphs bail is unchanged. - New AppState::new_multi_with_boot carries the supervision state; new_multi delegates with empty state so its six existing callers are untouched. - resolve_graph_handle maps Quarantined to a new ApiError::service_unavailable (503, no closed ErrorCode — mirroring the recovery-required 503 shape): configured-but-healing is 'retry later', not 'no such resource'. Never-configured ids stay 404. - GET /graphs merges quarantined graphs into the listing with an additive status discriminator (serde-defaulted to serving for wire compat) and a quarantine detail object (Unix-second timestamps, no new dependency); openapi.json regenerated. Full server suite (failpoints included) and api-types suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…int (RFC-029 prep) Behavior-free skeleton so the upcoming red convergence tests compile: - src/supervisor.rs: SupervisorConfig (production 5s/x2/10min/10% jitter; fast_for_tests at ms scale) and GraphSupervisor with a drain-only run() (receives and discards reopen requests; no reopen yet). - AppState::spawn_supervision(config): idempotent one-shot channel setup + task spawn — the entry point both serve() and in-process tests use. Shutdown is free: the only sender lives in AppState, so when the state drops, recv() yields None and the loop exits. - shielded_write gains the (state, key) chokepoint: a shielded writer whose result carries recovery_required notifies request_reopen, arming the W2(b) supervised reopen for every write surface at one site. The merge's follow-up source delete threads state through. Full server suite green (scaffolding drains; nothing converges yet). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…pervised reopen (RFC-029 W3+W2(b), red) Two bounded-poll convergence cells against the drain-only supervision skeleton: - multi_graph::boot_quarantined_graph_converges_to_serving_without_restart: a graph whose root is missing at boot quarantines (visible in GET /graphs with the verbatim open error), the root is initialized while the server runs (the deterministic transient-fault-clears model), and the listing must converge to serving without a restart, after which the graph's routes answer 200. RED: 'quarantined graph never converged to serving without a restart' at the 10s deadline. - failpoints::recovery_required_write_triggers_supervised_reopen_and_heals: the HTTP twin of rfc029_probe phases 2-3. A confirm-failure-injected mutation leaves an Armed sidecar; the next write surfaces 503 recovery_required (arming trigger B through the shielded-write chokepoint); polling the same write must converge to 200 and __recovery/ must empty. RED: 'RecoveryRequired never healed without a restart; last status: 503' at the 15s deadline. Green arrives with the next commit's supervision loop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

… W3+W2(b)) The real supervision loop replaces the drain-only skeleton, turning both red convergence cells green: - Seeds per-graph retry state from the registry's boot quarantine entries (trigger A) and selects over reopen notifications from the shielded-write chokepoint (trigger B — retried immediately, deduped while pending) and the earliest scheduled retry. - A due graph gets one full open_single_graph; success RCU-publishes the healed handle (clearing quarantine, or swapping a serving handle after a W2(b) reopen — in-flight requests finish on their own Arc); failure reschedules with capped exponential backoff plus jitter and refreshes the quarantine record, which no-ops while the graph still serves so a failed reopen never takes a healthy graph down. - One loop, not one task per graph: dedups notification storms and guarantees at most one in-flight open per root (two concurrent read-write opens would run redundant Full sweeps). Deliberate refinement of RFC-029 §5.1's per-graph wording; observable contract identical. - serve() starts supervision with production pacing before binding the listener; shutdown is free (the sender drops with the state). Full server suite green, failpoints included. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

- server.md: quarantine as a converging observable state — supervision backoff constants, GET /graphs status schema, 503-vs-404 semantics, and the RecoveryRequired self-heal on the write path. - errors.md: RecoveryRequired guidance rewritten — against the HTTP server the residual now heals automatically via the supervised reopen; embedded SDK callers keep the read-write reopen remedy. - invariants.md: note under the in-process recovery boundary gap that RFC-029 narrows its operational cost without relaxing any recovery rule (a reopen IS the documented resolution barrier). - testing.md: register the new W3/W2(b) coverage cells; AGENTS.md GET /graphs bullet mentions per-graph status. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

Mechanical rename across the RFC file, the engine behavior probe (rfc029_probe.rs -> rfc030_probe.rs, test fn included), and every RFC-029 reference in server/api-types source comments, test doc comments, and docs. No behavior change; doc cross-links and both renamed compile targets verified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…he reopen signal (RFC-030 review, red) Two effect-envelope cells against the current per-call shield: - merge_with_delete_branch_dropped_mid_merge_still_deletes_source: a merge with delete_branch: true whose request is dropped mid-merge (parked at the post-authority-capture failpoint) must still complete BOTH halves of the composite. RED: the merge lands detached but the source branch survives — delete_merged_source_branch lived in the dropped handler. - recovery_required_write_triggers_supervised_reopen_and_heals (restructured): the residual is manufactured and the 503 wedge is asserted BEFORE spawn_supervision, so those connected RecoveryRequired responses notify into the unset OnceLock and drop — isolating the doomed request as the only possible notifier. RED: the handler-side notify dies with the dropped request and the residual survives the filesystem-only poll. Determinism note baked into both cells: abort -> await-cancelled -> release. Awaiting the join while the engine-side task is still parked proves the handler future is gone before the protocol resumes; the reverse order races (a join woken by the released task can let an aborted handler run one final poll to completion, firing the notify and still reporting cancelled — observed empirically on the first run of this red). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…l (RFC-030 review) Turns the previous commit's two red cells green by relocating what must survive client fate INTO the shielded task: - shielded_write moves the trigger-B RecoveryRequired -> request_reopen notification inside the spawned future (AppState/GraphKey cloned in), so a disconnected client's wedged write still arms the supervised reopen; the handler-side await is now pure observation. - server_branch_merge runs merge + optional delete_branch follow-up as ONE shielded composite task owning the admission guard: a disconnect can no longer split the composite (merge landing, deletion silently skipped), and the admission slot spans both protocols as it did before the shield — closing the per-actor-limit bypass the per-call shield introduced. delete_merged_source_branch reverts to a plain helper running inside the composite. Also regenerates openapi.json: the RFC-030 renumber touched api-types doc comments that utoipa embeds as schema descriptions, and the rename commit missed the regen (caught by the drift check this run). Full server suite green (84 openapi cells included). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…FC-030 review, red) The incident's own topology: a single-graph deployment whose only graph fails its boot open on a transient fault. Lenient mode must boot with zero serving graphs when quarantine entries exist, expose the state on GET /graphs (503 on the graph's routes), and converge once the fault clears. RED: open_multi_graph_state bails 'no healthy graphs opened' before supervision can run — re-creating offline-until-redeploy for single-graph deployments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…pervision (RFC-030 W3) Zero SERVING graphs is a valid lenient-mode boot state when quarantine entries exist — the supervision loop converges them. The bail now fires only when zero graphs opened AND zero quarantined (unquarantinable failures, e.g. invalid graph ids). Strict --require-all-graphs and the settings-time config-class quarantine bail (stored-query/embedding config errors, which supervision cannot heal) are deliberately unchanged. Turns the previous commit's red single-graph incident topology green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

A quarantined key is configured, so add-only insert rejects it with DuplicateKey instead of risking one key in both snapshot maps (publish is the heal path that may replace it). Unit test pins the rejection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

… alignment - RFC-030 §3.2: the snippet now shows the admission guard moving into the spawned task, and a new paragraph defines the shielded unit as the request's effect envelope (admission + every implied protocol + the reopen notification; a merge with delete_branch is one composite) — the rule the review-fix tranche enforced. - RFC-030 §7: the engine-boundary W1 red cell is annotated as Stage-2/ PR-3 scope (it can only turn green under the engine-level shield); rfc030_probe.rs stays the engine baseline pin. - RFC-030 §12: two recorded preconditions — config-generation fencing for supervised publishes before runtime registry mutation ships, and surfacing settings-time (config-class) quarantine in GET /graphs. - server.md: composite-merge disconnect semantics, and the two quarantine classes drawn explicitly (open-class = supervised, converging; config-class = fail-fast, cluster apply + restart). - AGENTS.md + server.md: 'serves N graphs (N >= 1)' precised to N >= 1 CONFIGURED — lenient mode may transiently serve none while quarantined graphs converge; strict mode keeps serving = configured. - supervisor.rs module doc: the loop may outlive serve() by one in-flight write's duration (each shielded write holds an AppState clone for the in-task notify), bounded by the engine commit timeout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…epoint (RFC-030 review) delete_merged_source_branch stringifies failures into branch_delete_error on a 200 response by contract (the merge is already durable), so the composite merge task completes Ok and the outer shielded_write chokepoint — which inspects only Err for recovery_required — could never see a delete-phase RecoveryRequired. In a quiet system the rollback-class residual then sat unresolved until the next unrelated write's 503. Route the helper's engine call through its own inner shielded_write so the typed error passes the trigger-B chokepoint (arming the supervised reopen) before being stringified for the response. The mechanism is pinned by the existing chokepoint-level trigger-B failpoint cell; a deterministic delete-phase RecoveryRequired repro is unreachable today because no delete-scoped sidecar seam exists and the sidecars heal_pending_recovery_sidecars_for_branch_delete blocks on would also block the preceding merge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

cursor

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a11ce4a. Configure here.}

…raphs (red) Pins the invariant that "distinct canonical URIs across configured graphs" holds regardless of which opens succeed at boot. Today from_boot dedups URIs only among successfully opened handles, so a colliding pair where one side fails to open boots into an endless supervised-retry loop (the reopen succeeds but publish forever returns DuplicateUri), while the same config fails boot outright when both sides open. Red: fails with "expected DuplicateUri for serving/quarantined collision, got Ok". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…raphs (RFC-030 review) Canonicalize each quarantined entry's configured URI and include it in from_boot's duplicate detection, against both serving handles and other quarantined entries. The config invariant (distinct canonical URIs across configured graphs) now fails boot uniformly instead of depending on which opens succeeded: previously a colliding pair where one side failed to open booted into an endless supervised-retry loop — the reopen succeeded but publish forever returned DuplicateUri. A quarantined URI that cannot be normalized is skipped from the check: its supervised reopen fails at the same normalization, so it can never reach publish. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…hedule A review flagged the or_insert as a bug (further RecoveryRequired notifications do not accelerate a backed-off retry). It is intended design: the schedule is attempt-driven evidence about the open path, and a notification only proves demand — resetting on demand would let sustained 503 traffic collapse exponential backoff into a tight open loop exactly when the backend is struggling. Record the rationale at the decision site. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

…) into RFC-030 branch AGENTS.md conflict resolved by taking main's updated RFC-026/B2 bullets and folding this branch's RFC-030 quarantine/supervision wording into main's HTTP-server bullet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

greptile-apps Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread docs/rfcs/0030-write-lifetime-and-recovery-barriers.md

Comment thread docs/rfcs/0030-write-lifetime-and-recovery-barriers.md

Comment thread docs/rfcs/0030-write-lifetime-and-recovery-barriers.md

chatgpt-codex-connector Bot reviewed Jul 20, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into claude/interrupted-mu…

053b37e

…tations-graph-drift-njvits