Skip to content

fix: improve database health check error reporting and correctness#2416

Merged
ctron merged 2 commits into
guacsec:mainfrom
ctron:fix/health_check_reporting_1
Jun 26, 2026
Merged

fix: improve database health check error reporting and correctness#2416
ctron merged 2 commits into
guacsec:mainfrom
ctron:fix/health_check_reporting_1

Conversation

@ctron

@ctron ctron commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Log at warn level when the health check future exits so operators can detect a degraded state. Also fix the timeout result handling: the outer .is_ok() discarded the inner ping result, causing a fast database failure to be reported as healthy.

Summary by Sourcery

Improve health check reporting for local checks and database connectivity failures.

Bug Fixes:

  • Correct database health check result handling so that timeouts and fast failures are reported as unhealthy instead of healthy.

Enhancements:

  • Log a warning when the local health check future terminates and default subsequent health status to failure with the associated error message.

Log at warn level when the health check future exits so operators can
detect a degraded state. Also fix the timeout result handling: the outer
.is_ok() discarded the inner ping result, causing a fast database
failure to be reported as healthy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor
Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Improves the local health check’s reporting by logging when its future terminates and ensuring that database health check failures (including timeouts) are correctly propagated as unhealthy, rather than being mistakenly treated as healthy.

Flow diagram for Local health check future termination handling

flowchart TD
    A[Health check future running] --> B{Future exits}
    B --> C[log::warn
health check future returned,
will report failure
from now on]
    C --> D["state.store(false, Ordering::Release)"]
    D --> E[Subsequent health checks
report failure]
Loading

File-Level Changes

Change Details Files
Preserve and reuse the health check error message across threads and log a warning when the check future exits, marking the health check as failed thereafter.
  • Convert the incoming error into a Cow<'static, str> once at construction time and reuse it instead of converting later
  • Clone the error message into the spawned thread to allow it to be referenced in the log message
  • Change the log level and content when the health check future returns from info to warn with contextual error text
  • Maintain the behavior that when the check loop ends, the shared state flag is set to false to indicate failure
common/infrastructure/src/health/checks/local.rs
Fix database health check result handling so that timeouts and ping failures are reported as unhealthy instead of being masked as healthy.
  • Replace the outer is_ok() on the timeout-wrapped async block with unwrap_or(false) to correctly interpret timeout errors as failed checks
  • Ensure that the inner db.ping().await.is_ok() result is properly propagated to the health check outcome instead of being discarded
server/src/profile/mod.rs

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@ctron ctron requested a review from helio-frota June 26, 2026 09:16

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

With Rust edition 2021+ disjoint capture, the async move block in the
disabled path did not capture self.health, causing the Arc<HealthChecks>
to be dropped immediately. This dropped all registered health checks
(including their Shutdown handles), killing the health check loops while
the server was still running.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@ctron ctron enabled auto-merge June 26, 2026 09:54
@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 71.30%. Comparing base (b5c7762) to head (86a2207).

Files with missing lines Patch % Lines
common/infrastructure/src/infra.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2416      +/-   ##
==========================================
+ Coverage   71.28%   71.30%   +0.01%     
==========================================
  Files         449      449              
  Lines       27146    27149       +3     
  Branches    27146    27149       +3     
==========================================
+ Hits        19352    19358       +6     
+ Misses       6662     6653       -9     
- Partials     1132     1138       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ctron ctron added this pull request to the merge queue Jun 26, 2026
Merged via the queue into guacsec:main with commit e8784e5 Jun 26, 2026
9 checks passed
@ctron ctron deleted the fix/health_check_reporting_1 branch June 26, 2026 18:13
@github-project-automation github-project-automation Bot moved this to Done in Trustify Jun 26, 2026
@trustify-ci-bot

Copy link
Copy Markdown

Successfully created backport PR for release/0.5.z:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport release/0.5.z Backport (0.5.z)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants