Skip to content

chore(agent-data-plane): update health registry worker to allow being restarted#1176

Open
tobz wants to merge 1 commit intotobz/adp-move-int-o11y-control-plane-to-supervisorfrom
tobz/supervisor-health-registry-worker
Open

chore(agent-data-plane): update health registry worker to allow being restarted#1176
tobz wants to merge 1 commit intotobz/adp-move-int-o11y-control-plane-to-supervisorfrom
tobz/supervisor-health-registry-worker

Conversation

@tobz
Copy link
Member

@tobz tobz commented Feb 9, 2026

Summary

This PR slightly refactors HealthWorker and the underlying health registry runner code to support being able to restart the health registry worker.

Prior to this PR, spawning the health registry worker would fail un subsequent attempts since the receiver used to registry new components into the registry was already consumed by the first call to spawn the worker. We've simply added the ability to return the receiver and reset the state such that subsequent attempts to spawn the worker can take the receiver. We're still limited by only being able to have a single health registry worker at a time, but at least we can now cleanly recover from it being restarted.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

Existing and new unit tests.

References

AGTMETRICS-393

Copilot AI review requested due to automatic review settings February 9, 2026 03:42
@dd-octo-sts dd-octo-sts bot added the area/core Core functionality, event model, etc. label Feb 9, 2026
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a shutdown-aware, restartable health registry runner so the registry can be spawned again after the runner stops.

Changes:

  • Update HealthRegistry::spawn to accept a shutdown future and allow respawning after the runner ends.
  • Introduce RunnerGuard to return the liveness response receiver back into registry state on shutdown.
  • Add tests for “duplicate spawn while running” and “respawn after shutdown”.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +391 to +393
let mut inner = self.registry.lock().unwrap();
inner.responses_rx = Some(rx);
debug!("Returned response receiver to registry state.");
Copy link

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop should avoid panicking. Using lock().unwrap() inside drop() can panic if the mutex is poisoned, and a panic during drop can lead to aborts (especially if another panic is already in flight). Prefer handling poisoning explicitly (e.g., recover the inner value) and avoid unwrap() in drop().

Suggested change
let mut inner = self.registry.lock().unwrap();
inner.responses_rx = Some(rx);
debug!("Returned response receiver to registry state.");
match self.registry.lock() {
Ok(mut inner) => {
inner.responses_rx = Some(rx);
debug!("Returned response receiver to registry state.");
}
Err(poisoned) => {
let mut inner = poisoned.into_inner();
inner.responses_rx = Some(rx);
debug!("Returned response receiver to registry state after mutex poisoning.");
}
}

Copilot uses AI. Check for mistakes.
Comment on lines +611 to +617
// Take the response receiver out of the guard so we can use it in the select loop.
// It will be put back when the guard is dropped.
let mut responses_rx = self
.guard
.responses_rx
.take()
.expect("responses_rx should always be Some when Runner is created");
Copy link

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taking responses_rx out of RunnerGuard means the receiver will not be returned to the registry if the task is cancelled/aborted or unwinds before reaching the “put it back” code path. That breaks the documented goal of being restartable “after shutdown or an error”. Consider an RAII pattern that guarantees the receiver is put back even on early-exit (e.g., a small local guard whose Drop moves responses_rx back), or keep the receiver inside RunnerGuard and only borrow it mutably for recv().

Copilot uses AI. Check for mistakes.
};

for component_id in 0..component_count {
self.process_component_health_update(component_id, HealthUpdate::Unknown);
Copy link

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On runner (re)start, this forces every existing component’s health to Unknown, which can erase the last-known state and potentially emit unnecessary state transitions/notifications. If the intent is to “pick up where it left off”, consider scheduling immediate probes without overwriting existing health, or only setting Unknown for components that truly have no prior health value.

Suggested change
self.process_component_health_update(component_id, HealthUpdate::Unknown);
// Do not overwrite existing health with `Unknown` on (re)start; just schedule immediate probes.

Copilot uses AI. Check for mistakes.
@pr-commenter
Copy link

pr-commenter bot commented Feb 9, 2026

Binary Size Analysis (Agent Data Plane)

Target: d3ab905 (baseline) vs b6e0c79 (comparison) diff
Analysis Type: Stripped binaries (debug symbols excluded)
Baseline Size: 27.39 MiB
Comparison Size: 27.50 MiB
Size Change: +116.22 KiB (+0.41%)
Pass/Fail Threshold: +5%
Result: PASSED ✅

Changes by Module

Module File Size Symbols
saluki_core::runtime::supervisor +68.89 KiB 66
core +43.43 KiB 245
agent_data_plane::internal::initialize_and_launch_runtime -22.07 KiB 2
std -10.50 KiB 51
anyhow +8.26 KiB 30
agent_data_plane::internal::control_plane -8.04 KiB 26
[sections] +7.43 KiB 10
saluki_core::runtime::process +6.71 KiB 6
agent_data_plane::internal::observability +5.60 KiB 16
saluki_core::topology::running +5.52 KiB 2
saluki_app::metrics::collect_runtime_metrics -4.74 KiB 1
tokio -3.66 KiB 110
saluki_core::runtime::restart +3.50 KiB 7
tracing_core +2.24 KiB 14
saluki_health::Runner::run +1.91 KiB 8
hashbrown +1.81 KiB 8
saluki_health::RunnerGuard +1.71 KiB 3
saluki_core::runtime::shutdown +1.65 KiB 3
indexmap +1.62 KiB 2
agent_data_plane::config::DataPlaneConfiguration +1.62 KiB 1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +2.1% +83.6Ki  +1.6% +56.4Ki    [1112 Others]
  [NEW] +63.5Ki  [NEW] +63.2Ki    _<agent_data_plane::internal::control_plane::PrivilegedApiWorker as saluki_core::runtime::supervisor::Supervisable>::initialize::_{{closure}}::h01f48e1b4b1ba033
  [NEW] +21.3Ki  [NEW] +21.1Ki    _<agent_data_plane::internal::control_plane::UnprivilegedApiWorker as saluki_core::runtime::supervisor::Supervisable>::initialize::_{{closure}}::hd516f5c8e792be07
  [NEW] +18.6Ki  [NEW] +18.4Ki    saluki_app::api::APIBuilder::serve::_{{closure}}::hc67130ad013550cb
  [NEW] +16.2Ki  [NEW] +16.1Ki    saluki_core::runtime::supervisor::WorkerState::add_worker::h2f57c36c7d6a6d25
  [NEW] +15.1Ki  [NEW] +15.0Ki    _<core::pin::Pin<P> as core::future::future::Future>::poll::hf993bf6d214be6bb
  [NEW] +11.0Ki  [NEW] +10.9Ki    std::sys::backtrace::__rust_begin_short_backtrace::h7db93e33ffed5b3c
  [NEW] +10.7Ki  [NEW] +10.6Ki    <saluki_core::data_model::event::Event as core::clone::Clone>::clone.10467
  [NEW] +10.5Ki  [NEW] +10.4Ki    saluki_health::Runner::run::_{{closure}}::h99455b3fddb78a9b
  [NEW] +9.47Ki  [NEW] +9.34Ki    saluki_core::runtime::supervisor::Supervisor::run_inner::_{{closure}}::hf3a194df28b5bfa6
  [NEW] +6.80Ki  [NEW] +6.66Ki    saluki_core::runtime::supervisor::WorkerState::shutdown_workers::_{{closure}}::he6d5081c217bc272
  [NEW] +6.24Ki  [NEW] +6.10Ki    saluki_core::runtime::supervisor::WorkerState::shutdown_workers::_{{closure}}::h1d900b0791e57095
  [NEW] +5.69Ki  [NEW] +5.54Ki    saluki_core::topology::running::RunningTopology::shutdown_with_timeout::_{{closure}}::h9563a6f0d2737005
  [NEW] +5.52Ki  [NEW] +5.38Ki    <hickory_proto::rr::record_data::RData as core::clone::Clone>::clone.8629
  [DEL] -8.42Ki  [DEL] -8.33Ki    std::sys::backtrace::__rust_begin_short_backtrace::h61680b9753eb2342
  [DEL] -9.20Ki  [DEL] -9.10Ki    saluki_health::Runner::run::_{{closure}}::h36f8d77002f294fa
  [DEL] -10.7Ki  [DEL] -10.6Ki    <saluki_core::data_model::event::Event as core::clone::Clone>::clone.10695
  [DEL] -18.0Ki  [DEL] -17.8Ki    agent_data_plane::internal::control_plane::spawn_control_plane::_{{closure}}::_{{closure}}::h9b9375d6b068ec9c
  [DEL] -18.4Ki  [DEL] -18.2Ki    agent_data_plane::internal::initialize_and_launch_runtime::_{{closure}}::h3544d2e34d6be2ff
  [DEL] -18.7Ki  [DEL] -18.6Ki    saluki_app::api::APIBuilder::serve::_{{closure}}::hfe866520c248a5c1
  [DEL] -84.5Ki  [DEL] -84.4Ki    agent_data_plane::internal::control_plane::spawn_control_plane::_{{closure}}::hd6ee71eb8e3b5d42
  +0.4%  +116Ki  +0.4% +88.1Ki    TOTAL

@pr-commenter
Copy link

pr-commenter bot commented Feb 9, 2026

Regression Detector (Agent Data Plane)

Regression Detector Results

Run ID: fe1040bc-0f8d-4c43-a7d7-451fd084763b

Baseline: d3ab905
Comparison: b6e0c79
Diff

❌ Experiments with retried target crashes

This is a critical error. One or more replicates failed with a non-zero exit code. These replicates may have been retried. See Replicate Execution Details for more information.

  • otlp_ingest_logs_5mb_memory
  • quality_gates_rss_dsd_ultraheavy
  • dsd_uds_1mb_3k_contexts_cpu

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf experiment goal Δ mean % Δ mean % CI trials links
otlp_ingest_logs_5mb_throughput ingress throughput -0.01 [-0.14, +0.12] 1 (metrics) (profiles) (logs)
otlp_ingest_logs_5mb_cpu % cpu utilization -1.48 [-6.59, +3.64] 1 (metrics) (profiles) (logs)
otlp_ingest_logs_5mb_memory memory utilization -5.84 [-6.50, -5.17] 1 (metrics) (profiles) (logs)

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
otlp_ingest_metrics_5mb_memory memory utilization +3.28 [+3.07, +3.49] 1 (metrics) (profiles) (logs)
dsd_uds_500mb_3k_contexts_memory memory utilization +1.37 [+1.19, +1.56] 1 (metrics) (profiles) (logs)
dsd_uds_1mb_3k_contexts_memory memory utilization +0.97 [+0.78, +1.15] 1 (metrics) (profiles) (logs)
dsd_uds_100mb_3k_contexts_memory memory utilization +0.84 [+0.64, +1.03] 1 (metrics) (profiles) (logs)
dsd_uds_500mb_3k_contexts_throughput ingress throughput +0.73 [+0.59, +0.87] 1 (metrics) (profiles) (logs)
dsd_uds_500mb_3k_contexts_cpu % cpu utilization +0.66 [-0.63, +1.96] 1 (metrics) (profiles) (logs)
quality_gates_rss_dsd_low memory utilization +0.52 [+0.36, +0.67] 1 (metrics) (profiles) (logs)
quality_gates_rss_dsd_medium memory utilization +0.51 [+0.32, +0.70] 1 (metrics) (profiles) (logs)
dsd_uds_512kb_3k_contexts_memory memory utilization +0.43 [+0.25, +0.61] 1 (metrics) (profiles) (logs)
quality_gates_rss_dsd_heavy memory utilization +0.41 [+0.29, +0.53] 1 (metrics) (profiles) (logs)
dsd_uds_10mb_3k_contexts_memory memory utilization +0.39 [+0.20, +0.59] 1 (metrics) (profiles) (logs)
quality_gates_rss_idle memory utilization +0.11 [+0.08, +0.15] 1 (metrics) (profiles) (logs)
quality_gates_rss_dsd_ultraheavy memory utilization +0.01 [-0.13, +0.14] 1 (metrics) (profiles) (logs)
dsd_uds_1mb_3k_contexts_throughput ingress throughput +0.00 [-0.05, +0.06] 1 (metrics) (profiles) (logs)
dsd_uds_512kb_3k_contexts_throughput ingress throughput +0.00 [-0.05, +0.05] 1 (metrics) (profiles) (logs)
dsd_uds_100mb_3k_contexts_throughput ingress throughput +0.00 [-0.05, +0.06] 1 (metrics) (profiles) (logs)
otlp_ingest_traces_5mb_throughput ingress throughput -0.00 [-0.02, +0.02] 1 (metrics) (profiles) (logs)
otlp_ingest_logs_5mb_throughput ingress throughput -0.01 [-0.14, +0.12] 1 (metrics) (profiles) (logs)
dsd_uds_10mb_3k_contexts_throughput ingress throughput -0.01 [-0.17, +0.15] 1 (metrics) (profiles) (logs)
otlp_ingest_metrics_5mb_throughput ingress throughput -0.03 [-0.17, +0.11] 1 (metrics) (profiles) (logs)
otlp_ingest_traces_5mb_memory memory utilization -0.28 [-0.52, -0.03] 1 (metrics) (profiles) (logs)
otlp_ingest_traces_5mb_cpu % cpu utilization -0.51 [-2.58, +1.55] 1 (metrics) (profiles) (logs)
dsd_uds_100mb_3k_contexts_cpu % cpu utilization -0.62 [-7.08, +5.83] 1 (metrics) (profiles) (logs)
otlp_ingest_logs_5mb_cpu % cpu utilization -1.48 [-6.59, +3.64] 1 (metrics) (profiles) (logs)
dsd_uds_10mb_3k_contexts_cpu % cpu utilization -1.53 [-29.76, +26.70] 1 (metrics) (profiles) (logs)
otlp_ingest_metrics_5mb_cpu % cpu utilization -2.41 [-8.35, +3.52] 1 (metrics) (profiles) (logs)
otlp_ingest_logs_5mb_memory memory utilization -5.84 [-6.50, -5.17] 1 (metrics) (profiles) (logs)
dsd_uds_512kb_3k_contexts_cpu % cpu utilization -8.21 [-60.81, +44.39] 1 (metrics) (profiles) (logs)
dsd_uds_1mb_3k_contexts_cpu % cpu utilization -16.78 [-63.96, +30.39] 1 (metrics) (profiles) (logs)

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed links
quality_gates_rss_dsd_heavy memory_usage 10/10 (metrics) (profiles) (logs)
quality_gates_rss_dsd_low memory_usage 10/10 (metrics) (profiles) (logs)
quality_gates_rss_dsd_medium memory_usage 10/10 (metrics) (profiles) (logs)
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 (metrics) (profiles) (logs)
quality_gates_rss_idle memory_usage 10/10 (metrics) (profiles) (logs)

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

Replicate Execution Details

We run multiple replicates for each experiment/variant. However, we allow replicates to be automatically retried if there are any failures, up to 8 times, at which point the replicate is marked dead and we are unable to run analysis for the entire experiment. We call each of these attempts at running replicates a replicate execution. This section lists all replicate executions that failed due to the target crashing or being oom killed.

Note: In the below tables we bucket failures by experiment, variant, and failure type. For each of these buckets we list out the replicate indexes that failed with an annotation signifying how many times said replicate failed with the given failure mode. In the below example the baseline variant of the experiment named experiment_with_failures had two replicates that failed by oom kills. Replicate 0, which failed 8 executions, and replicate 1 which failed 6 executions, all with the same failure mode.

Experiment Variant Replicates Failure Logs Debug Dashboard
experiment_with_failures baseline 0 (x8) 1 (x6) Oom killed Debug Dashboard

The debug dashboard links will take you to a debugging dashboard specifically designed to investigate replicate execution failures.

❌ Retried Normal Replicate Execution Failures (non-profiling)

Experiment Variant Replicates Failure Debug Dashboard
dsd_uds_1mb_3k_contexts_cpu baseline 1 Failed to shutdown when requested Debug Dashboard
otlp_ingest_logs_5mb_memory comparison 6, 4 Failed to shutdown when requested Debug Dashboard
quality_gates_rss_dsd_ultraheavy baseline 8 Failed to shutdown when requested Debug Dashboard

@tobz tobz added the type/chore Updates to dependencies or general "administrative" tasks necessary to maintain the codebase/repo. label Feb 9, 2026
@tobz tobz changed the title enhancement(agent-data-plane): add restartable worker for health registry chore(agent-data-plane): add restartable worker for health registry Feb 9, 2026
@tobz tobz changed the title chore(agent-data-plane): add restartable worker for health registry chore(agent-data-plane): update health registry worker to allow being restarted Feb 9, 2026
@tobz tobz marked this pull request as ready for review February 9, 2026 14:12
@tobz tobz requested a review from a team as a code owner February 9, 2026 14:12
@tobz tobz force-pushed the tobz/adp-move-int-o11y-control-plane-to-supervisor branch from 70c54c9 to b72ece8 Compare February 11, 2026 16:24
@tobz tobz force-pushed the tobz/supervisor-health-registry-worker branch from a0e1d19 to 7ca8cbc Compare February 11, 2026 16:24
Copilot AI review requested due to automatic review settings February 20, 2026 04:26
@tobz tobz force-pushed the tobz/supervisor-health-registry-worker branch from 7ca8cbc to 7c0a3b6 Compare February 20, 2026 04:26
@tobz tobz force-pushed the tobz/adp-move-int-o11y-control-plane-to-supervisor branch from b72ece8 to 944baa2 Compare February 20, 2026 04:27
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +611 to +618
// Take the response receiver out of the guard so we can use it in the select loop.
// It will be put back when the guard is dropped.
let mut responses_rx = self
.guard
.responses_rx
.take()
.expect("responses_rx should always be Some when Runner is created");

Copy link

Copilot AI Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Runner::run takes responses_rx out of RunnerGuard into a local variable. If the task is aborted/cancelled (e.g., supervisor brutal shutdown or graceful timeout abort) or panics before the end of run, the receiver will be dropped and never returned to RegistryState, making future spawn() calls fail again. Consider keeping the receiver as a field on Runner and returning it in Drop for Runner, or using a scope guard so the receiver is always put back into registry state even on cancellation.

Copilot uses AI. Check for mistakes.
@tobz tobz force-pushed the tobz/supervisor-health-registry-worker branch from 7c0a3b6 to a4f5a2b Compare February 20, 2026 04:39
@tobz tobz force-pushed the tobz/adp-move-int-o11y-control-plane-to-supervisor branch 2 times, most recently from 3853ae9 to 8bc1907 Compare February 20, 2026 05:20
Copilot AI review requested due to automatic review settings February 20, 2026 05:20
@tobz tobz force-pushed the tobz/supervisor-health-registry-worker branch from a4f5a2b to 35cd34f Compare February 20, 2026 05:20
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@tobz tobz force-pushed the tobz/adp-move-int-o11y-control-plane-to-supervisor branch from 8bc1907 to 4d639de Compare February 21, 2026 19:52
@tobz tobz force-pushed the tobz/supervisor-health-registry-worker branch from 35cd34f to 092d5cc Compare February 21, 2026 19:52
Copilot AI review requested due to automatic review settings February 25, 2026 03:25
@tobz tobz force-pushed the tobz/adp-move-int-o11y-control-plane-to-supervisor branch from 4d639de to d6d4281 Compare February 25, 2026 03:25
@tobz tobz force-pushed the tobz/supervisor-health-registry-worker branch from 092d5cc to 4c2d4bf Compare February 25, 2026 03:25
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@tobz tobz force-pushed the tobz/adp-move-int-o11y-control-plane-to-supervisor branch from d6d4281 to c07a545 Compare February 26, 2026 15:20
@tobz tobz force-pushed the tobz/supervisor-health-registry-worker branch from 4c2d4bf to b6e0c79 Compare February 26, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/core Core functionality, event model, etc. type/chore Updates to dependencies or general "administrative" tasks necessary to maintain the codebase/repo.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants