Skip to content

Make Sparoid::Instance public-IP resolution thread-safe and self-healing#25

Merged
oskgu360 merged 3 commits into
mainfrom
fix-instance-public-ip-resolution
Jun 30, 2026
Merged

Make Sparoid::Instance public-IP resolution thread-safe and self-healing#25
oskgu360 merged 3 commits into
mainfrom
fix-instance-public-ip-resolution

Conversation

@oskgu360

@oskgu360 oskgu360 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Problem

Sparoid::Instance memoized its public IP with @public_ips ||= super, which has two defects:

  1. Empty result pinned forever. [] is truthy in Ruby, so once @public_ips is [] the ||= short-circuits and never re-resolves. A single transient empty resolution (icanhazip down or returning an empty body) poisons the instance for its whole life — every later knock then builds zero messages and auth sends zero SPA packets.
  2. Non-atomic race / thundering herd. ||= is read-modify-write. When one Instance is shared across a thread pool, the first wave of knocks all resolve concurrently (a herd of icanhazip requests) and one slow/empty response clobbers a good value via last-writer-wins — then defect Include gem name in warnings #1 pins it.

This was caught in production (ssh-monitor worker.26, 2026-06-26): a dyno sent zero-packet knocks for its whole lifetime and alarmed every host as SSH-down. ssh-monitor currently carries an app-side monkey-patch for this; this PR moves the proper fix into the gem.

Fix

  • Move the caching into Instance#cached_public_ips (calling the inherited raw public_ips fetch directly — no super, no public_ips override).
  • Resolve behind a per-instance mutex with a double-check — kills the thundering herd and the last-writer-wins clobber; concurrent callers wait and reuse the freshly cached IP.
  • Cache only a non-empty result (the stable egress IP, returned lock-free on the fast path); never memoize empty, so a transient failure self-heals on the next knock.
  • Close the silent-success hole: generate_messages now raises when an empty public-IP resolution and no IPv6 fallback would otherwise produce no messages, instead of letting auth report success having sent nothing.
  • The raise is a new Sparoid::PublicIPError — a sibling of ResolvError under Error (so existing rescue Sparoid::Error still catches it), letting callers distinguish "I can't resolve my own public IP" (a client-side failure affecting every host) from "the target host is unreachable." Notably it is not a ResolvError, so ssh-monitor's rescue Sparoid::ResolvError "machine deleted" no-op won't silently swallow it.

Tests

  • test_instance_retries_and_recovers_instead_of_pinning_empty — empty resolve is not memoized; recovers once icanhazip responds again (webmock).
  • test_auth_raises_when_no_public_ip_resolvedauth raises Sparoid::PublicIPError rather than silently sending zero packets.
  • Manual concurrency check: 40 threads on one instance → a single resolution (2 HTTP calls, not 80), all threads get the good IP.

Adds webmock as a dev dependency.

Note

Version bump + CHANGELOG and the ssh-monitor cleanup are intentionally left to separate follow-up PRs. The follow-up can rescue Sparoid::PublicIPError specifically so an icanhazip outage doesn't mark every host as SSH-down.

@oskgu360
oskgu360 force-pushed the fix-instance-public-ip-resolution branch 5 times, most recently from 653ad44 to 95a790f Compare June 29, 2026 12:30
Sparoid::Instance memoized with `@public_ips ||= super`, which is neither
thread-safe nor empty-aware: `[]` is truthy, so a transient empty resolution
(icanhazip down or empty body) was pinned for the process lifetime and every
later knock then sent zero SPA packets. Under a shared instance hit by a thread
pool, the first wave of knocks also resolved concurrently and one slow/empty
response clobbered a good value via last-writer-wins.

Move the caching into Instance#cached_public_ips: resolve behind a per-instance
mutex with a double-check (kills the thundering herd and the clobber) and cache
only a non-empty result, so a transient empty resolution self-heals on the next
knock instead of pinning the instance.

Also raise when generate_messages would otherwise produce no messages: an empty
public-IP resolution with no IPv6 fallback used to let auth report success
while sending zero packets. It now raises Sparoid::PublicIPError (a sibling of
ResolvError under Error) so the caller can tell 'I cannot resolve my own public
IP' apart from 'the target host is unreachable', instead of a silent no-op.
@oskgu360
oskgu360 force-pushed the fix-instance-public-ip-resolution branch from 95a790f to 2c1cea6 Compare June 29, 2026 12:35
@oskgu360
oskgu360 marked this pull request as ready for review June 29, 2026 12:41
@oskgu360
oskgu360 requested a review from a team as a code owner June 29, 2026 12:41
@walro
walro requested a review from Copilot June 29, 2026 12:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens Sparoid::Instance public-IP resolution by making it thread-safe, avoiding permanently caching empty resolutions, and by failing fast when an auth attempt would otherwise send zero SPA packets.

Changes:

  • Add Sparoid::PublicIPError and raise it when no public IP can be resolved for message generation.
  • Implement per-instance, mutex-guarded, non-empty memoization via Instance#cached_public_ips.
  • Add WebMock-backed tests for retry/self-healing behavior and for the new error path; add webmock to dev deps.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
lib/sparoid.rb Adds PublicIPError, raises on empty IP resolution, and introduces mutex-protected per-instance caching of public IPs.
test/test_helper.rb Enables WebMock for the suite (but currently allows real network connections).
test/sparoid_test.rb Adds WebMock stubs and tests covering non-pinning empty results and raising on zero-message auth.
Gemfile Adds webmock for development/test usage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/test_helper.rb Outdated
Comment thread lib/sparoid.rb
@oskgu360

oskgu360 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Ran ssh-monitor fix with this patch locally (just verified it were able to ssh).

Will rescue PublicIPError to handle that gracefully in case of this happening again. If we decide to merge this that is

Comment thread lib/sparoid.rb
Comment thread lib/sparoid.rb
Comment thread test/test_helper.rb Outdated
Comment thread lib/sparoid.rb
oskgu360 added 2 commits June 29, 2026 15:11
Drop WebMock.allow_net_connect! and stub icanhazip in test_it_resolves_public_ip
so the suite no longer depends on a working internet connection or icanhazip
being up (which would make CI flaky and let mistyped stubs fall through to the
network).
@public_ips is now always an array, so the empty/cached check is a plain
.any? instead of &.any?.
@oskgu360
oskgu360 merged commit 992ab16 into main Jun 30, 2026
8 checks passed
@oskgu360
oskgu360 deleted the fix-instance-public-ip-resolution branch June 30, 2026 05:52
@oskgu360 oskgu360 mentioned this pull request Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants