Skip to content

fix(kernel): close the provisioner's four residual mid-boot lifecycle gaps - #3303

Open
kevinjosethomas wants to merge 23 commits into
mainfrom
lane/perf-kernel-stopgate-hardening
Open

kevinjosethomas wants to merge 23 commits into
mainfrom
lane/perf-kernel-stopgate-hardening

Conversation

@kevinjosethomas

@kevinjosethomas kevinjosethomas commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Evidence

Full-suite VM gate (the receipt-of-record) — fleet_pipeline.fleet_gate

  • verdict: GREEN
  • run: gate_1790905327_154675 (host logs: /home/ubuntu/fleet-pipeline-runs/gate_1790905327_154675)
  • head b0b4b30051f526405b04bd2160c7eb85291fbe8d on org/main tip a0c131566b6fc25c832ab9b973dab37af8080da5 (merge-base a0c131566b6fc25c832ab9b973dab37af8080da5, base folded)
  • toolchain 1.98.1 pin, VM rust:1-bookworm (sandbox it4kvxkzo4mx4laket2ztbcd), hosts_guard OK, kernel_guard OK, release killverify clean
  • cargo +1.98.1 fmt --all --check: rc 0
  • cargo +1.98.1 clippy --workspace --all-targets -- -D warnings: rc 0
  • cargo +1.98.1 test --workspace --no-fail-fast: rc 0 — 80 test-result lines, 1751 passed, 0 failed, 12 ignored, zero unknown failures
  • failures classified: none (full suite green)

Crates-mode pre-check gate (same head, opened the PR earlier)

  • verdict: GREEN — run gate_1790905137_154675: cargo fmt --all --check rc 0, cargo clippy -p pa-core --all-targets -- -D warnings rc 0, cargo test -p pa-core --no-fail-fast rc 0 (21 result lines, zero failures; all 5 drain-boundary fixtures pass on the folded tree).
  • Correction note (2026-10-02): the block the PR template auto-attached at open rendered THIS crates receipt under the full-suite header with workspace-wide command labels — the actual phase commands were -p pa-core-scoped. The full-suite receipt above is the receipt-of-record; this block restates the crates receipt accurately.

What this PR does

Closes the provisioner's four residual mid-boot lifecycle gaps — the design consult's unresolved-risk list (the kill-vs-boot and atomic-settle entries) plus the two Macroscope residuals that R3 dispositioned pre-existing with this exact follow-up recommended. One commit (87d138bdb, +722/-89, 3 files) on the #3257 head, folded over the current main tip:

  1. Atomic settle/publish: the disposed check, the memo-generation check, the listener teardown, and the manager park now share ONE lock scope (TS ensure()'s managerPromise === startup guard). A stale publisher can no longer clear a NEWER memo armed underneath it, and a boot can no longer park a live kernel into a provisioner that reported itself torn down between the old separate check and publish scopes (the dispose TOCTOU).
  2. kill()-during-boot: kill() clears the startup memo (TS kill() clears managerPromise), so a boot doomed mid-flight is killed by its own settle instead of parking a resident kernel nobody owns; the next ensure() boots fresh.
  3. Concurrent stops of one in-flight boot JOIN the first stop's gate (pending_stop_for_startup): the take-losing stop task used to open a fresh gate before the winner's final snapshot flush, un-gating a revival to race that flush over the same on-disk file. The take-loser now waits the armed gate.
  4. Five drain-boundary fixtures, no sleeps: 4 in kernel_startup_memo.rs (stale-publisher generation survival; kill-during-boot; dispose-during-boot end-state pin; kill vs share one kernel startup across first use, stop, and close #3257's panic-cleanup re-arm) + the concurrent-stops fixture in kernel_stop_revive.rs. All observables are drain-structural on the single-threaded test runtime (the settle chain runs in one scheduler drain); interpreter spawns are counted by a wrapper that pins absolute counts.

Failing-first evidence (archived, sha256 10dac2c5...)

On the pre-fix tree (7ddb838a4, the #3257 head) the memo fixtures failed 4/4 — the kill fixtures delivered a LIVE resident kernel after a mid-flight kill; the dispose fixture caught dispose() returning before the in-flight boot settled; the stale-publisher fixture caught the newer memo's kernel shut down underneath a late joiner (LEG1_RC=101). On the fixed tree: the 5 fixtures + kernel_startup_join 5 + kernel_stop_revive 4 + kernel_prewarm 2 + the 9 provisioner unit tests all green; fmt/clippy rc0. Evidence: /home/ubuntu/hillclimb/vms/perf-kernel-stopgate-hardening/evidence-stopgate.tgz (sha256 10dac2c537d3f810dfbde8be94f71bda3045c563687f02e60c158c6e79b4e261).

The fold (semantic merge, hunk-verified)

Main's squashed #3257 (b28f22ded) already carries a weaker interim of items 1/3 (the separate-scope same_channel clear; the previous_stop chaining) plus the "Waiting for the previous kernel to stop..." progress stage. The fold keeps the hardening's atomic-scope mechanics (they subsume the interims) and re-grafts the progress emit — main's evolved revival_waits_for_in_flight_stop_before_booting ordering test asserts on that exact stage. Post-fold delta verified hunk-by-hunk: HEAD vs main = exactly the 3 hardening files; the 9 hardening hunks in provisioner.rs are byte-identical to the staged commit's, and the only extra delta vs the staged commit is the 7-line emit graft.

TS parity

TS anchors (carried verified by the kernelgate design consult and the #3257 review arc, TS checkout cd1f215c): tools/ipython.ts:427-437 (kill() clears/awaits managerPromise), :440-485 (the memoized managerPromise, the managerPromise === startup clear guard, joined concurrent ensure), :499-512 (new boot gated on prior dispose), :625-639 (first tool access awaits the memo). Frozen surfaces untouched: no wire/API/tool-schema, prompt, session-JSONL, or TUI bytes change (the re-grafted progress stage is main's current behavior, byte-identical).

Gate receipts (exact head b0b4b3005)

  • Full-suite GREEN — the receipt-of-record: gate_1790905327 — cargo fmt --all --check rc0, cargo clippy --workspace --all-targets -- -D warnings rc0, cargo test --workspace --no-fail-fast rc0 — 80 test-result lines, 1751 passed, 0 failed, 12 ignored, ZERO unknown failures, release killverify clean, hosts_guard OK, kernel_guard OK, base_state=folded (merge-base == org/main tip a0c131566), VM it4kvxkzo (pool-0), finished 2026-10-02T02:00:55Z.
  • Crates-mode pre-checks GREEN: gate_1790905137 — fmt rc0, clippy -p pa-core rc0, test -p pa-core rc0 (21 result lines, zero failures; all 5 drain-boundary fixtures pass on the folded tree).
  • The folded tree exercises BOTH main's evolved kernel_stop_revive ordering test (which asserts on the re-grafted progress stage) and the hardening's join mechanics — both green in the same full-suite run: the semantic merge is proven by the suite.

Known reds

kernel_restore_guards co-scheduling family: renewed 2026-09-29 (solo x3 green), expiry extended to 2026-10-06; zero pool fires since the v1.4.7 claim guard. This lane does not touch the kernel restore surface.

No self-merge: the PR stays open for the orchestrator's numeric review + both adversarial reviewers' SHA-bound APPROVEs.


Note

High Risk
Changes concurrency, snapshot flush ordering, and kernel process lifetime during boot/stop/dispose/kill—bugs could leak interpreters, corrupt session snapshots, or deadlock revival.

Overview
Hardens IpythonKernelProvisioner so ensure(), kill(), dispose(), and stop_kernel() cannot race while a kernel is still booting.

Startup memo generations tie each in-flight boot to its own watch channel. The publisher task now performs an atomic settle/publish under one lock: generation liveness, dispose flag, listener teardown, and whether the manager parks or is torn down (settle_decision prefers kill over dispose for invalidated generations). Stale publishers clear only their own memo (same_channel), and kill() parks invalidated memos in doomed_startups so dispose() waits them out.

kill() during boot invalidates the memo, clears shared progress state, releases stop arms tied to that boot, and the settle path kills the interpreter instead of leaving a resident manager. Retries, restore/on_restore, unavailable-skills callbacks, and shared progress emits are generation-gated so dead boots cannot retry, clobber snapshots, or pollute the next turn.

stop_kernel() is refactored through arm_stop() with pending_stop_for_startup: concurrent stops supersede gates but each task waits only the strictly older gate it replaced (avoids revival/snapshot flush races and wait cycles). Failed-boot teardowns use hold_snapshot_flush_gate so killed generations never flush over an on-disk snapshot; live/disposed boots can still gate revival on their flush.

Adds Unix integration tests (kernel_startup_memo.rs, extended kernel_stop_revive.rs) and unit tests for settle order, stop-arm release, and flush gating.

Reviewed by Cursor Bugbot for commit 78f038a. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Fix provisioner's four residual mid-boot lifecycle gaps

  • Closes races between ensure(), kill(), dispose(), and stop_kernel() when a kernel boot is still in flight. A boot invalidated by kill() or a racing dispose() is now torn down at settlement instead of being published, with an ordered settlement decision (settle_decision in provisioner.rs).
  • Adds a startup-memo generation so each boot, stop gate, progress listener, and replay message belongs to exactly one generation. Stale or killed generations no longer clear newer startup memos, overwrite progress state, or report unavailable skills to shared callbacks.
  • kill() now parks an invalidated startup memo in doomed_startups; dispose() waits for those doomed boots to settle before returning, and a killed boot never flushes a dispose snapshot.
  • stop_kernel() stops form an ordered chain via arm_stop(): each new stop claims the live manager or the in-flight startup atomically, and its gate opens only after the superseded stop settles.
  • Adds Unix-only integration tests in kernel_startup_memo.rs and kernel_stop_revive.rs covering kill/dispose during boot, retry backoff, listener isolation, snapshot flushing, and stop-chain revival.
  • Behavioral Change: dispose() can now block longer because it waits for all captured startup memos, including killed ones, to settle; settlement and publication in run_startup moved to the publisher task under one state lock.

Changes since #3303 opened

  • Extended kernel::provisioner::start_kernel_impl to recheck boot generation liveness immediately after acquiring the start permit and abort with a specific error before spawning the interpreter if the generation was killed while waiting, and added that error marker to the non-retryable list in startup_failure_is_retryable. [78f038a]
  • Added SupervisorChildSessionsInner::reseed_from_ledger to rebuild the in-memory child registry from the persisted RLM spawn ledger for the current parent session, and SupervisorChildSessionsInner::prime_cursor_if_lazy to initialize a reseeded child's usage cursor to the end of its history on first delivery, preventing back-billing of pre-restart entries. [78f038a]
  • Modified SupervisorChildSessionsInner child-settle handling to record the child's return on the parent's semantic-edge ledger by reading the child's last committed request id and invoking recorder.record_child_returned, and to update the child's display file from status 'running' to 'completed' when the run settled successfully. [78f038a]
  • Rewrote run_daemon_attached_acp_mode to create and attach the daemon session during startup via bind_daemon_session, fail early on create/attach errors, remove agent_start/agent_end markers, implement terminal quiescence by polling autonomous status and child roster until outstanding subagents reach zero, cancel outstanding RLM children during stop/close/teardown, and publish heartbeats_changed broadcasts as SessionInfoUpdate frames. [78f038a]
  • Removed ACP in-process fallback from print_runtime::acp_mode_main, deleted the try_daemon_attached_acp helper, and removed in-process ACP modules including acp::session, acp::autorefine, acp::compaction_arms, acp::events, acp::goal_continuation, acp::in_process_config, acp::producer, and acp::prompt. [78f038a]
  • Extended wire_events::wire_updates to map bash_start, bash_output, and bash_end kernel events into ACP tool-call frames with synthetic ids derived from runId, and modified tool_result_text to return None for empty tool results and filter out empty text blocks. [78f038a]
  • Introduced semantic edge recording via a new semantic_edges module including SemanticEdgeRecorder, SemanticEdgeIdentity, ledger append/read helpers, wrap_stream_fn to inject request ids into model stream calls, and SemanticCompaction guard for compaction lifecycle events. [78f038a]
  • Modified create_session to optionally wrap the stream function with semantic edges when semantic_identity is provided, construct a SemanticEdgeRecorder, set an RLM spawn semantic anchor on the host bridge, and pass a pre-semantic side_question_stream_fn into the session, and added a factory_host field to SessionEngine built from captured session facts. [78f038a]
  • Updated execute_compaction to use semantic_edges::summary_slice_call for each summary request, wrapping complete_summary_call to inject request-specific headers, and to commit the compaction on the recorder before persisting the compaction entry. [78f038a]
  • Extended RlmHostBridge::register_run to capture a SemanticSpawnAnchor and set request.spawned_by_request_id by consulting the anchor to resolve the parent's last turn request id if the agent is streaming, and propagated spawned_by_request_id through spawn handling to inject spawnedByRequestId into the child's session configuration. [78f038a]
  • Introduced factory machinery including rlm::factory module with validation (validate_factory_spec), run orchestration (run_factory, status_factory, stop_factory, resume_factory, graph_factory, watch_factory), CLI dispatch, and activity handling; HarnessState::create_factory, update_factory, delete_factory methods; and a new 'factory' harness kind with deep-copy semantics for arguments. [78f038a]
  • Added RefinementKind::Factory variant, extended REFINEMENT_KINDS and related utilities to support 'factory', introduced factory_enabled function reading agent_dir/settings.json, and modified apply_refinement_proposal to refuse factory create/update edits when factory_enabled is false, recording them as planned but unapplied with FACTORY_DISABLED_MESSAGE. [78f038a]
  • Added SessionEngine::factory_activity method parsing factory activity requests, performing a blocking preflight for 'run' actions via FactoryHost (model allowlist/auth/catalog checks), and delegating to the kernel manager's factory_activity, and introduced FactoryHost bridge with FactoryHostConfig capturing session facts for preflight. [78f038a]
  • Introduced /factory CLI command with subcommands list, import, export, added FactoryCommand enum, parse_factory_command parser validating subcommands and flags, and run_factory_command entrypoint delegating to the kernel's factory CLI dispatcher via a Python runner script. [78f038a]
  • Added DaemonCommand::FactoryActivity variant with fields id, active_session_id, action, run_id, spec_id, timeout_ms, and flattened rest; extended command_active_session_id, command_type_name, default_server_capabilities, and protocol tests to support factory_activity. [78f038a]
  • Introduced factory_activity module implementing factory_lane_enabled reading settings, advertised_server_capabilities conditionally excluding 'factory_activity', and Worker::handle_factory_activity handler validating payload and delegating to agent_engine.factory_activity, and integrated factory settings into handshake and worker command dispatcher. [78f038a]
  • Added TUI factory view including FactoryView component, factory_view module with diagram rendering, SessionUi::open_factory_page, handle_factory_view_key, and factory_control methods, integrated factory updates via FactoryUpdate channel, and added Factory activity group to the activity dock gated by factory_activity_supported. [78f038a]
  • Extended Settings with factory: Option<FactorySettings> field, added FactorySettings struct with optional enabled bool, and implemented SettingsManager::get_factory_enabled and set_factory_enabled methods reading/writing the global scope only. [78f038a]
  • Added /factory builtin slash command to CANONICAL_BUILTIN_SLASH_COMMANDS with client-side execution, argument hint '[on|off|status]', and implemented SessionUi::handle_command branch for '/factory' with subcommands 'on', 'off', and informational paths, guarding 'off' against live factory runs. [78f038a]
  • Modified supervisor writer select loop to add biased; directive and reorder arms to poll targeted events first, ensuring session events published before a response are observed before the reply on the wire, and extended reader task to clear all pending replies in reader_resident.pending on connection termination, causing in-flight routes to fail immediately. [78f038a]
  • Replaced LONG_ROUTE_TIMEOUT_MS with WORKER_REQUEST_TIMEOUT_MS set to 24 hours, introduced client_route_timeout function mapping DaemonCommand variants to either WORKER_REQUEST_TIMEOUT_MS for turn-long operations or ROUTE_TIMEOUT_MS for control routes, and updated route timeout usage in Supervisor::route_client_command, handshake, and worker lifecycle methods. [78f038a]
  • Modified interactive_mode::run_interactive_mode to spawn an async flush of startup telemetry via flush_startup_telemetry, defer joining the flush handle with a timeout after the interactive run completes, and updated build_tui_options to use crate::mode::create_telemetry_disabled for the telemetry_disabled option. [78f038a]
  • Updated spawn_supervisor_detached in interactive_mode::daemon and update_flow::spawn_supervisor and run_update_command to call set_new_session on Unix targets before spawning, detaching the supervisor and coordinator processes into new sessions. [78f038a]
  • Introduced HeadlessStep::SubmitAndSettle and UiInput::SubmitAndSettle variants, implemented settled_sequence helper to derive the settled event sequence after a prompt, and integrated a submit-and-settle barrier in run_interactive_surface that delays progression until the session processes events up to the settled sequence or timeout. [78f038a]
  • Modified custom_message::custom_message_entries to map RLM child failure/terminal-notice custom types to AgentMessage entries via rlm_child_status_row instead of injected-prompt rows, removed RlmChildStatus and RlmChildOutcome from injected_prompt module, and deleted rendering helpers for child status rows. [78f038a]
  • Modified print_runtime::acp_mode_main to always resolve daemon socket, ensure daemon running, construct a daemon session create command via daemon_acp_create, and run daemon-attached ACP mode; deleted try_daemon_attached_acp; updated build_headless_engine_with to ensure SessionManager is present, construct SemanticEdgeIdentity, and pass semantic_edges into engine build options; extracted select_headless_session, explicit_cwd_override, stored_session_cwd helpers; and refactored build_session_manager_with_lease to use select_headless_session. [78f038a]
  • Updated AgentSessionEngine lifecycle initialization to no longer fall back to PRIME_AGENT_MODEL_PROVIDER/PRIME_AGENT_MODEL env vars for model selection, wire children.set_semantic_edges to record returned children's committed requests, pass semantic_edges into a later build step with cloned semantic_identity, and added factory_activity method forwarding to the running kernel. [78f038a]
  • Extended Worker::create handler to construct SemanticSpawnOrigin from runtimeMetadata when kind is 'subagent', pass semantic_spawn into bind/rebind path, and invoke reseed_rlm_children after creation; added Worker::reseed_rlm_children method calling reseed_from_ledger on the children handle; and changed refresh_replaced_session_state to async, passing semantic_spawn: None and awaiting reseed_rlm_children on success. [78f038a]
  • Updated all test helpers and spawn request construction sites to set spawned_by_request_id: None in RlmSpawnRequest structs, and updated engine/options construction in tests to pass semantic_edges: None. [78f038a]
  • Added end-to-end tests including a_daemon_restart_relists_and_wakes_the_parents_child and a_daemon_restart_marks_a_still_running_child_as_failed to verify child relisting and status after daemon restart, sigkill_closes_the_spawned_child_and_passivates_the_row to assert child row appears with status 'error' post-restart, a_spawned_child_ledger_names_its_spawning_request_and_the_parent_records_the_return to verify semantic linkage and parent recording, and interactive_launcher_detaches_supervisor_from_client_session to validate supervisor session ownership and detachment. [78f038a]
  • Updated hash_runtime_source to include packaged machine library files under src/rlm/machines via collect_package_data_files in the runtime identity hash, and added a test verifying identity changes when machine files change. [78f038a]
  • Added packaged machine specifications including review-sweep/MACHINE.md, pr-manager/MACHINE.md, and builder/MACHINE.md with JSON machine-spec definitions and documentation. [78f038a]

Macroscope summarized 8d29f37.


Follow-up heads (the Macroscope round, 2026-10-02)

The five Macroscope findings on this head reduce to two root causes, fixed across four commits:

  • 4880ab3f8 — chain superseding stop gates (arm_stop captures the gate its arm REPLACED; the stop task opens its own only after that strictly-older gate settles, so a revival gated on the new gate cannot cross before the superseded stop's final snapshot flush, and the strictly-older chain cannot form wait cycles) and generation-scope the settle's listener teardown (the doomed boot's settle clears startup_listeners/last_startup_message only when its memo is still the active generation). Two new drain-boundary fixtures, fail-first (RED receipt gate_1790910402 on the fixtures-only tree e3b92c48d).

  • 24531b57a — drop the redundant gate clone (the clippy red on the CI run for 4880ab3f8).

  • 5e3037aef — generation-scope the startup progress EMIT: the settle fix left the emit unscoped, and the listener fixture proved it (the doomed boot's Preparing Python runtime... clobbered the newer generation's replay stage after kill()). The emit now writes the shared last_startup_message and fires the listener fan-out only while the emitting memo is still the ACTIVE generation (the boot's own handler still fires).

  • 5901a45a7 — kill() clears the killed generation's shared progress state (startup_listeners + last_startup_message) in the same lock scope that invalidates the memo, so the next ensure() neither replays the killed boot's stale stage to a fresh handler nor fans the newer boot's stages out to the killed generation's listeners. The listener fixture gains the transition oracles (A's joiner hears nothing after the kill; the newer boot's handler shows exactly its own stage, no stale replay first) with its own RED receipt.

  • 4777baaaf — dead generations do not retry and do not surface restores (Macroscope's follow-up finding on 5901a45a7): the retry loop checks the boot's generation (boot_generation_is_live: memo still the armed startup generation and not disposed) before each retry and again after the backoff, so a kill() during the backoff window cannot resurrect a dead generation's boot; and the restore surface — last_restore plus the on_restore callback — is generation-gated in one lock scope (check, write, and callback together, matching the listener contract), so a doomed boot can neither overwrite the fresher generation's restore result nor append a stale restore notice. Fail-first fixture kill_during_the_retry_backoff_pins_the_spawn_count (flaky-first wrapper, kill during the backoff, spawn count pinned at 2; RED receipt: the memo check removed fails 3 vs 2 — the doomed boot consumed its retry).

  • cfcd93af0 — a dead generation does not report unavailable skills (Cursor Bugbot's finding on 155a87697): on_unavailable_skills now uses the restore gate's exact one-lock-scope contract (generation-liveness check and callback share the state lock), so a boot kill() invalidated can no longer append stale skills-unavailable rows to the notice mailbox. Fail-first fixture doomed_boot_does_not_report_unavailable_skills (RED receipt: the gate reverted fails 2 vs 1 — the dead generation's report lands first).

  • f43fcffc5 — empty retrigger commit only, no code change: CI's first run for cfcd93af0 failed one unit, pa-cli's acp_threshold_auto_compaction_publishes_the_compaction_meta (stopReason null vs end_turn), a known flaky ACP e2e — the identical test binary was green on this branch at 155a87697 and fails intermittently on main's own runs at cf285dce (main CI shows both success and failure for that commit). The retrigged run is green.

  • 8884c3262 — plain fold of main's e7e27c24c (the /settings menu search fix, fix keys dying in the /settings menu search #3309; pa-core untouched by the fold). Local gates on the folded tree: fmt rc 0, clippy -D warnings rc 0, kernel fixtures 12/12, pa-tui lib 1341/1341; CI green on the head.

  • c9de47ae0 — gate the interleave race harness to unix: main's promote the windows jobs and add the contributor trust gate #3004 promoted the windows cross-check (clippy for x86_64-pc-windows-gnu), red on main's own tip — add race tests for idle session stops and fix two bugs #3195's interleave harness binds a unix-socket probe and tokio gates UnixListener behind all(unix). The module declaration carries #[cfg(unix)] (same contract as feed's gated tests; verified green with cargo check -p pa-daemon --all-targets --target x86_64-pc-windows-gnu).

  • 3e6868044 — spent stop arms release and doomed teardowns never flush (Cursor Bugbot's two Mediums on 4c74a09ef, plus two cross-model review findings in the fix shape): pending_stop_for_startup is released by both spenders (the settle and the kill()), so a spent arm's memo receiver cannot keep a parked-or-failed manager alive past stop_kernel; the four failure-teardown sites route through hold_snapshot_flush_gate — the flush decision (dispose policy while the boot's memo is armed, never a flush for a kill()-invalidated boot) and, when flushing, a pending-stop revival-gate install share ONE lock scope with the skip only for a stop actively armed for the boot, so a kill() racing the teardown leaves the replacement boot gated on the flush. The two unix-only kernel test targets (kernel_startup_memo.rs, kernel_stop_revive.rs) also carry #![cfg(unix)] — their /bin/sh wrapper fixtures never cross-compiled for the new windows job (the sibling kernel targets already carry the gate). RED receipts in the provisioner unit tests (12/12).

Cross-model pre-push review (a different model family than the lane's authors, per the repo's review bar): verdict READY-CLEAN after each fix round, including the reviewer's own two findings in the fix commits (a kill-time shared-state leak and a restore-callback race, both fixed with RED→GREEN oracles); report at /home/ubuntu/handoffs/REVIEW-3303.md. Keeper gates on the pushed tree: cargo fmt --check rc 0; cargo clippy --workspace --all-targets -- -D warnings rc 0; kernel fixtures 12/12 (kernel_startup_memo 7 + kernel_stop_revive 5); cargo test -p pa-core --lib 1043 passed / 1 git-2.34.1 sparse-checkout artifact in workspace_snapshot (root files get the skip-worktree bit after sparse-checkout set --cone on git 2.34; untouched by this PR and green on CI's newer git).

…d startup

Review follow-ups on PR #3257 (both blockers independently verified by
both adversarial reviewers at 21da7aa):

- F1 panic wedge: a panic in run_startup or a progress listener skipped
  the memo clear and the done send, which closed the watch and wedged
  every later ensure() on the dead memo. The boot now runs on its own
  JoinHandle, so the publisher always clears the memo and sends the
  result, converts a panicked boot into a "kernel startup task failed"
  failure, and clears the listener state so the next ensure() boots
  fresh (self-heal; the memo re-arms on close). The watch now carries
  the startup result itself: joined callers keep the manager even when
  a concurrent stop takes the provisioner's live owner (TS
  managerPromise), and a dispose-raced boot rejects with the honest
  disposed-startup cause instead of handing callers a dead manager.

- F2 stop-gate gap: stop_kernel took the manager and installed the
  pending_stop gate in two separate lock passes; a revival ensure in the
  gap captured the previous gate and could restore against a snapshot
  still being flushed. The gate is now installed in the same lock block
  that takes the manager.

- F3 dispose fixture vacuity: the dispose leg pins the absolute
  interpreter count at dispose-return (exactly one spawn; the aborted
  retry never ran) in addition to the stasis assert.

- Fixtures: a panicking progress handler must not poison the next boot
  (the re-arm regression leg), and a revival boot must wait for the
  in-flight stop's held host request before its first progress stage
  (exact ordering barrier, not flush timing).
… gaps

Staged hardening set for the #3257 shared-startup head (the design
consult's unresolved-risk list + the Macroscope 07:10/07:15 R3
residuals, both fresh reviewers dispositioned pre-existing and
recommended this follow-up):

- Atomic settle/publish: the publisher's disposed check, memo-generation
  check, listener teardown, and manager park now share ONE lock scope
  (TS ensure()'s `managerPromise === startup` guard). A stale publisher
  can no longer clear a NEWER memo armed underneath it - a defunct clear
  inside the park->publish->clear window used to wipe the newer memo and
  arm a transient duplicate kernel - and a boot can no longer park a
  live kernel into a provisioner that reported itself torn down between
  the old separate check and publish scopes (the dispose TOCTOU).
- kill() clears the startup memo (TS kill() clears `managerPromise`):
  a boot doomed mid-flight is killed by its own settle instead of
  parking a resident kernel nobody owns; the next ensure() boots fresh.
- Concurrent stops of one in-flight boot share the first stop's gate
  (`pending_stop_for_startup`): the take-losing stop task used to open a
  fresh gate before the winner's final snapshot flush, un-gating a
  revival to race that flush over the same on-disk file. The take-loser
  that still exists (against a direct-arm stop) now waits the currently
  armed gate before opening its own.

Fixtures: kernel_startup_memo.rs (stale-publisher generation survival;
kill-during-boot; dispose-during-boot end-state pin; kill vs #3257's
panic-cleanup re-arm) + the concurrent-stops fixture in
kernel_stop_revive.rs. All drain-boundary observables on the
single-threaded test runtime (the settle chain runs in one scheduler
drain; no sleeps, no interposition into wake chains). Failing-first
verified on the pre-fix head: the kill/stale fixtures delivered a LIVE
resident kernel; the concurrent-stops fixture caught the loser's
early gate open in the settle drain.
Semantic merge of the merged #3257 shape (b28f22d's review follow-ups:
the memo-generation clear, the stop-gate chaining, and the
"Waiting for the previous kernel to stop..." progress stage) with the
staged hardening chain at 87d138b.

- provisioner.rs: the hardening mechanics stand (the one-lock-scope
  settle/publish, kill()-during-boot generation invalidation, the
  same-boot stop JOIN via pending_stop_for_startup); the weaker interim
  fixes they subsume (the separate-scope same_channel clear, the
  previous_stop chaining) give way to them; the merged #3257
  progress-stage emit is kept - main's evolved revival-ordering test
  asserts on that exact stage.
- kernel_stop_revive.rs: main's evolved ordering barrier test plus the
  lane's concurrent-stops-share-one-gate fixture.
Comment thread crates/pa-core/src/kernel/provisioner.rs
Comment thread crates/pa-core/src/kernel/provisioner.rs

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread crates/pa-core/src/kernel/provisioner.rs
Comment thread crates/pa-core/src/kernel/provisioner.rs Outdated
Comment thread crates/pa-core/src/kernel/provisioner.rs Outdated
kevinjosethomas and others added 6 commits October 2, 2026 03:06
The Macroscope/Bugbot findings on the stopgate-hardening head, as
drain-boundary fixtures BEFORE the fix (RED by construction; the
fail-first receipts are the ffix-branch gates cited in the fix commit):

- superseding_stop_after_kill_cannot_deadlock_the_revival_gate: a stop
  armed for one boot, kill(), a revival gated on that stop's gate, and a
  second stop of the revived boot that supersedes it; the take-losing
  first stop waiting the currently-installed gate forms the wait cycle
  (first stop -> second stop's gate -> revival boot -> first stop's
  gate) - the oracle is the bounded settle, so the cycle shape wedges
  against the 30s timeout. Absolute spawn count pinned.
- doomed_settle_keeps_the_newer_boot_listener_state: a doomed boot
  settling against a newer memo generation must not wipe the newer
  boot's progress listeners/replay stage - a late joiner of the newer
  boot replays that boot's current stage to a fresh handler; the gated
  counting wrapper holds each boot's handshake at a file barrier (the
  wrapper reads its ordinal from the count file, so the count file is
  pre-created; a Drop guard opens every gate on any exit so a wrapper
  never outlives the test as an orphan), making the settle ordering a
  test-controlled barrier, never a timing race.
…ener teardown

The five Macroscope/Bugbot findings on the stopgate-hardening head reduce
to two root causes; both fixed, with the drain-boundary fixtures that
failed first on the unfixed tree (RED receipt: gate_1790910402 on the
fixtures-only tree e3b92c4 - fmt rc0, clippy rc0, test rc101 with
EXACTLY the two new fixtures failing at their designed oracles:
"a late joiner must replay the active boot's stage" and "the first stop
settled (no wait cycle)", both bounded-settle timeouts; everything else
green; the compile-caught authoring attempts gate_1790908504/
gate_1790908744/gate_1790909006/gate_1790909694 are the fixture
authoring cycle, incl. the counting-wrapper ordinal-read-before-create
bug that made one early RED vacuous):

- The stop-gate CHAINING (main's #3257 shape) dropped by the JOIN:
  arm_stop now captures the gate the arm REPLACED and the stop task
  waits THAT before opening its own - never the currently-installed
  gate. The installed-gate design missed the supersede window (the
  take-losing second stop's loser saw its OWN gate installed and opened
  before the first direct stop's final snapshot flush settled,
  un-gating a revival to race that flush) and admitted a three-task
  wait cycle (a stop waiting a superseding gate that waits a revival
  that waits the first stop's gate). The chain is strictly-older-only,
  so acyclic by construction; the same-boot JOIN is unchanged (the
  second stop of the same boot still never arms).

- The settle's listener teardown is generation-scoped: the doomed boot
  clearing startup_listeners/last_startup_message unconditionally wiped
  a NEWER memo generation's progress listeners and replayed stage. The
  clear now runs only when the settling memo is the active generation
  (or no memo is armed at all - the stale-entries case).
…y assert

- arm_stop installs the arm's own receiver directly (clippy redundant_clone:
  the clone was the value's last use after the Armed struct stopped
  carrying stop_rx).
- the listener fixture's replay oracle asserts the replay FIRES with the
  active boot's current stage; pinning the exact stage string made the
  oracle flaky against legitimate later stages the boot emits around its
  held handshake ("Preparing Python runtime...").
The 4880ab3 settle fix scoped the LISTENER TEARDOWN to the active memo
generation but left the EMIT unscoped, and its own fixture proved it:
doomed_settle_keeps_the_newer_boot_listener_state failed 3/3 on the
fixed tree (replayed "Preparing Python runtime..." instead of "Starting
Python kernel...") - a boot kill() invalidates still emitted its later
stages, overwriting the newer generation's last_startup_message and
firing the newer boot's listeners with the wrong boot's stage.

The emit now threads the boot's memo generation through
run_startup/start_kernel/start_kernel_impl and writes the shared
progress state (last_startup_message + the listener fan-out) only when
the emitting memo is still the ACTIVE generation; the boot's own
on_progress handler keeps firing unconditionally (it is its caller's,
not shared state). Also drops the redundant stop_rx.clone() that
red-inked CI's clippy job on the swarm push (redundant_clone at the
arm_stop install).

Gates (keeper worktree, CARGO_TARGET_DIR=.cargo-target-keeper-3303):
fmt rc0; clippy --workspace --all-targets -D warnings rc0; kernel
fixtures kernel_startup_memo 5/5 + kernel_stop_revive 5/5 (the
previously-red doomed_settle fixture green); pa-core lib 1043 passed, 1
failed - workspace_snapshot::head_tree_baseline_excludes_only_absent
_skip_worktree_paths, a git-2.34.1 sparse-checkout artifact reproduced
outside the lane (git 2.34.1 marks root files S after `sparse-checkout
set --cone`; CI's newer git passes it; the PR does not touch
workspace_snapshot).
… into lane/perf-kernel-stopgate-hardening-keeper
…ill()

FINDINGS-1 from the pre-push review (gpt-6-sol cross-model reviewer,
report /home/ubuntu/handoffs/REVIEW-3303.md): kill() invalidates the
startup memo but retained the killed boot's last_startup_message and
startup_listeners, so the next ensure() replayed the killed boot's stale
stage to a fresh handler and the newer boot's emits fanned out to the
killed generation's listeners - the 5e3037a emit gate stops the doomed
boot's own emissions but not the B-to-A delivery or the stale replay.

kill() now clears both shared fields in the same lock scope that
invalidates the memo (the same stale-entries reasoning as the settle's
generation-scoped teardown), and the listener fixture grows the
transition oracles the reviewer asked for: A's joiner's collector must
show nothing after the kill, and the newer boot's handler must show
exactly its own stage twice (fan-out + boot-local, the pre-existing
main shape) with no stale replay first.

RED receipt: with only the kill-clear reverted, the fixture fails at
"the killed generation's listener must not hear the newer boot"
(kernel_startup_memo.rs:531); restored, all 10 kernel fixtures pass.
Gates: fmt rc0; clippy --workspace --all-targets -D warnings rc0;
pa-core lib 1043 passed, 1 git-2.34.1 sparse-checkout artifact
(workspace_snapshot, untouched by this PR, green on CI's newer git).
Comment thread crates/pa-core/src/kernel/provisioner.rs
kevinjosethomas and others added 3 commits October 2, 2026 04:07
Macroscope's follow-up finding on 5901a45 (thread r4162712975): a
run_startup task whose memo kill() invalidated still consumed its retry
- each attempt spawns an interpreter and runs restore/bootstrap, and the
stale attempt's settle only killed its kernel, leaving duplicate kernel
work and stale restore notifications/state against the fresher
generation.

- The retry loop now checks boot_generation_is_live (memo still the
  armed startup generation AND provisioner not disposed) before each
  retry AND again after the backoff, so a kill() during the backoff
  window cannot resurrect a dead generation's boot. The disposed arm of
  the old guard folds into the same check (dispose() does not clear the
  memo, so both conditions are needed).
- The restore surface (on_restore + last_restore) is now
  generation-gated with the same predicate: a boot kill() invalidated
  restores its snapshot only for the settle to kill its kernel, so its
  restore must not fire the notification nor pollute last_restore.

Fixture: kill_during_the_retry_backoff_pins_the_spawn_count - a
fail-first flaky wrapper (first invocation exits 1 before the
handshake; later invocations exec the real kernel python), kill()
during the backoff window, and the absolute spawn count pinned at 2.
RED receipt: with the memo check removed the fixture fails 3 vs 2 (the
doomed boot consumed its retry); restored, all 11 kernel fixtures pass.

Gates: fmt rc0; clippy --workspace --all-targets -D warnings rc0;
pa-core lib 1043 passed / 1 git-2.34.1 sparse-checkout artifact
(workspace_snapshot, untouched by this PR, green on CI's newer git).

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread crates/pa-core/src/kernel/provisioner.rs
Cursor Bugbot's finding on 155a876 (thread r4162867816): a boot that
kill() invalidated still fired on_unavailable_skills after its
bootstrap - the callback shares the restore notice's mailbox, so a
discarded kernel could append stale skills-unavailable rows the next
turn showed the model, even though on_restore/last_restore are
generation-gated in the same function.

The skills report now uses the restore gate's exact contract: the
generation-liveness check and the callback share ONE lock scope (the
callbacks must not re-enter the provisioner, same as the
startup-progress listeners), so a kill() can no longer slip between the
check and the report.

Fixture: doomed_boot_does_not_report_unavailable_skills - the gated
counting wrapper holds the doomed boot's interpreter, kill() invalidates
its generation, the newer boot arms with the SAME broken skill, and the
mailbox must carry exactly the live boot's report. RED receipt: with the
gate reverted the fixture fails 2 vs 1 (the dead generation's report
lands first); restored, all 12 kernel fixtures pass.

Gates: fmt rc0; clippy --workspace --all-targets -D warnings rc0;
pa-core lib 1043 passed / 1 git-2.34.1 sparse-checkout artifact
(workspace_snapshot, untouched by this PR, green on CI's newer git).
…ction_publishes_the_compaction_meta is a known flake: same test green at 155a876 and flaky on main's own cf285dc runs)
@snimu

snimu commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

The kill-during-boot fix is real (fixtures fail on main). The rest guards against things main already handles:

  • Concurrent stops already chain on main; the concurrent-stops fixture passes without this PR. Delete StopArm::Joined and pending_stop_for_startup. As written, pending_stop_for_startup is never cleared (keeps a settled manager alive) and a joined stop drops the second caller's snapshot option.
  • The dispose-race memo clear already exists on main; that fixture never fails first.

In the kill path:

  • kill() clears the startup memo, so a later dispose() doesn't wait for the doomed boot and can orphan the kernel at worker exit.

  • The settle path checks disposed before !mine, so a killed boot can shut down with snapshot=true. Check !mine first.

  • New test files import std::os::unix without cfg(unix); breaks the Windows cross-check.

  • Four fixtures rely on timing; the gated wrapper exists.

[written by prime-agent, reviewed by snimu]

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread crates/pa-core/src/kernel/provisioner.rs
Comment thread crates/pa-core/src/kernel/provisioner.rs
Main's #3004 promoted the windows cross-check (clippy for
x86_64-pc-windows-gnu), and the new job is red on main's own tip:
#3195's interleave harness binds a unix-socket probe, but tokio gates
UnixListener behind all(unix) - the module fails to cross-compile for
windows (E0433, the single red in pa-daemon's lib-test target).

The module declaration now carries #[cfg(unix)] (the same contract as
feed's #[cfg(unix)] tests - every race test in the module rides the
unix-socket probe, so the whole module is unix-only). This is an
upstream break the lane folds and fixes so its head can be green; main
carries the same fix independently.

Gates: cargo check -p pa-daemon --all-targets --target
x86_64-pc-windows-gnu rc 0; fmt rc 0; clippy --workspace --all-targets
-D warnings rc 0; pa-daemon turn_stream_tests 42 passed on unix.
The two Cursor Bugbot findings on 4c74a09, plus the cross-model
reviewer's TOCTOU finding on the first fix shape:

1. "Stop-arm memo never released" (Medium): pending_stop_for_startup
   is written when a stop arms against an in-flight boot and never
   cleared when that boot settles or kill() invalidates it. The arm's
   memo receiver is a strong handle to the settled StartupResult, so a
   spent arm keeps a parked-or-failed manager alive past stop_kernel -
   a later failed shutdown's process would leak with the stale
   receiver as its sole owner. Both spenders now release the arm (the
   boot's settle and the kill() that invalidates it), through one
   shared release_spent_stop_arm.

2. "Doomed boot can clobber snapshots" (Medium): a boot kill()
   invalidated still runs start_kernel_impl to completion, and its
   failure teardowns shut down with the dispose snapshot policy
   (default true) against the same snapshot_dir the replacement boot
   restores from. The four failure teardown sites now route through
   hold_snapshot_flush_gate: the policy (dispose policy while the
   boot's memo is still armed, never a flush for a kill()-invalidated
   boot; dispose() does not clear the memo so a disposed boot keeps
   the dispose's own policy) AND, when the policy still flushes, a
   pending-stop revival gate for the teardown's own flush - the
   decision and the gate install share ONE lock scope, so a kill()
   landing between the decision and the flush still leaves the
   replacement boot gated on this teardown (the same revival gate
   stop_kernel arms against its own flush; a stop ACTIVELY armed for
   this boot is the one exemption - its memo wait settles only after
   this teardown, so its own gate already covers the flush - while a
   merely installed older gate, including one long settled and left in
   place, covers nothing and is replaced by the teardown gate).

Also gates the two unix-only kernel test targets
(kernel_startup_memo.rs, kernel_stop_revive.rs) with #![cfg(unix)] -
main's #3004 promoted the windows cross-check and the whole-target
/bin/sh wrapper fixtures never cross-compiled (the sibling kernel
targets and lock_compat already carry the gate).

RED receipts (provisioner unit tests): release no-op -> both arm
tests fail; flush gate install removed -> the gate assert fails;
restored -> 12/12.

Gates: fmt rc0; clippy --workspace --all-targets -D warnings rc0; the
12 kernel fixtures pass; pa-core lib 1046 passed / 1 git-2.34.1
sparse-checkout artifact (workspace_snapshot, untouched by this PR,
green on CI's newer git); clippy --workspace --all-targets --target
x86_64-pc-windows-gnu -D warnings rc0.
@sethkarten
sethkarten self-requested a review October 2, 2026 19:45
sethkarten
sethkarten previously approved these changes Oct 2, 2026

@sethkarten sethkarten left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — approved on Seth Karten's instruction (submitted via his agent).

@snimu

snimu commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Fixed: the spent-arm leak (release in settle and kill), the cfg(unix) gates.

Remaining:

  • StopArm::Joined is patched, not deleted. Your own chain comment (:601-609) shows it's redundant; it still drops caller 2's snapshot, and arm_stop writes caller 2's dispose_snapshot before returning Joined, so the two policies split. pending_stop_for_startup is now load-bearing for the flush-gate exemption — keep that field, delete Joined.
  • kill() still clears the memo, so a later dispose() doesn't wait for the doomed boot (orphan at worker exit).
  • The settle path still checks disposed before !mine — a killed+disposed boot that reaches Ok flushes with snapshot=true, contradicting this PR's own "never flush a kill()-invalidated boot" rule, and kill→ensure→dispose can overlap two flushes on one snapshot_dir. One-line swap.
  • The four memo fixtures still spin on yield_now where the gated wrapper exists.
  • Drop c9de47a: it duplicates main's let the windows test targets compile #3306 cfg(unix) on interleave.rs (clippy duplicated_attributes risk once your branch sees that merge).

[written by prime-agent, reviewed by snimu]

…-hardening-keeper

Folding this merge drops the lane's own interleave gate from
c9de47a: main's #3306 (db686e0) gates the harness whole-file
with #![cfg(unix)] inside interleave.rs, and keeping the lane's
outer #[cfg(unix)] on the mod declaration too would leave the
module with duplicated cfg attributes (clippy duplicated_attributes
risk under -D warnings). The resolution takes main's side for
turn_stream_tests.rs, so the interleave files match main exactly.
…ed gates

snimu's five remaining review items (2026-10-02T20:36:31Z comment):

1. StopArm::Joined deleted. The join made a second stop of one
   in-flight boot wait the first stop's gate after arm_stop had
   already written the second caller's dispose_snapshot, splitting the
   two snapshot policies. Concurrent stops of one boot now supersede:
   each stop arms its own gate and its task waits the strictly older
   gate it superseded before opening its own, so a revival cannot
   cross before the final flush settles; pending_stop_for_startup
   stays (the flush-gate exemption is load-bearing), now consulted by
   the failed-boot teardown instead of the join.

2. kill() parks the startup memo it invalidates (doomed_startups)
   instead of clearing it. The doomed boot's kernel exists until its
   settle, and a later dispose() must wait that settle the same way it
   waits an armed memo - without the park, a kill followed by a
   dispose skipped the doomed boot entirely and could orphan its
   kernel at worker exit. Each parked memo is spent by its own boot's
   settle.

3. The settle path checks the generation before the disposed flag
   (settle_decision): a killed+disposed boot reaching Ok tears down
   with kill() semantics and never flushes a snapshot, and
   kill->ensure->dispose cannot overlap two flushes on one
   snapshot_dir. The order is pinned by a unit test.

4. The four memo fixtures hold their boots at file gates instead of
   spinning on yield_now; the flaky fixture holds before its failure
   on the same schedule, so a kill lands mid-boot by construction.
   Both gated wrappers self-release when their fixture directory
   vanishes (the counting loop falls through to the real interpreter,
   whose stdin pipe is gone; the flaky loop exits 1), so a cancelled
   test cannot leave a polling shell behind.

5. The main merge folds #3306's #![cfg(unix)] inside interleave.rs,
   dropping c9de47a's duplicate mod gate from the resolution (the
   duplicated_attributes clippy risk).

Red-first regressions, both demonstrated failing on the pre-change
behavior: dispose_after_a_kill_waits_the_doomed_boot and
a_killed_then_disposed_boot_never_flushes_the_snapshot; the
settle-order unit pin fails with the old disposed-first order. The
joined-gate drain test is deleted with its arm.

Cross-family review (bugbot, gpt-6-sol): READY-CLEAN, no findings.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8d29f37. Configure here.

Comment thread crates/pa-core/src/kernel/provisioner.rs
cursor[bot] finding on 8d29f37 ("Killed boot still spawns
interpreter", Medium): the boot permit gate checked only the dispose
signal, so a kill() landing while a boot was still waiting - queued
behind an in-flight stop's gate or the boot permit itself - let the
doomed generation spawn an interpreter its own settle then had to kill
(no flush, no park, but a real boot racing the replacement generation
on the same snapshot_dir and kernel-stderr.log).

The permit-gate closure now rechecks boot_generation_is_live as the
last point before the interpreter spawn and fails fast with "Kernel
provisioner killed before start" - non-retryable (FATAL_MARKERS
"provisioner killed", unit-pinned). The settle stays the correctness
backstop for the post-check race.

Red-first regression
a_kill_while_the_boot_waits_the_stop_gate_spawns_no_interpreter: boot
A holds mid-handshake at a file gate; a stop arms against it; kill #1
invalidates A; a fresh ensure arms memo B (polled once on the test
thread so the arm is guaranteed, not yield-assumed); kill #2 invalidates
B while it still waits; releasing A settles the chain and B fails the
liveness check before the spawn. Red on the unfixed tree (B spawned a
183ms handshake and settled "killed during startup"); green after (the
spawn count stays at one).

Also folds pi/main c24ac22 (factory/machine lib, acp single daemon
path, semantic-edge ledger, telemetry first-frame fix, factory e2e,
subagent relisting, factory state-machine port, vouches).

Cross-family review (bugbot, gpt-6-sol): READY-CLEAN on the fix and on
the test-hardening re-review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants