Skip to content

Home relay failure is a redial, not endpoint death - #91

Merged
lann merged 1 commit into
mainfrom
home-relay-reconnect
Aug 23, 2026
Merged

Home relay failure is a redial, not endpoint death#91
lann merged 1 commit into
mainfrom
home-relay-reconnect

Conversation

@lannbot

@lannbot lannbot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

A home-relay wire failure permanently bricked the endpoint (#88): the pump
treated a receive or send error on the home relay websocket as endpoint
death, so any transient loss — a mobile browser backgrounding the page, a
network blip, a relay restart — left the endpoint refusing every
operation, and the application had to rebuild it from scratch. Upstream
iroh reconnects the relay with backoff and never kills the endpoint over
it (iroh-1.0.3 src/socket/transports/relay/actor.rs); the death ruling
was an accumulated divergence with no artifact.

The fix

The pump retires the failed home conn and redials it:

  • Backoff matches upstream's shape — jittered exponential from 10ms,
    unbounded attempts, immediate-and-reset after a connection that had
    lived ≥ 10s (upstream's reset-on-established rule, approximated by
    uptime instead of relay-pong accounting). One deliberate divergence:
    the cap is 5s where upstream uses 16s, because recovery latency after
    a browser page returns to foreground dominates this design's use;
    recorded on REDIAL_MAX_DELAY's doc.
  • Drop-while-reconnecting: transmits to the home relay while it is
    down are dropped and QUIC loss recovery owns them, exactly upstream's
    semantics. Reconnect reoccupies pool key HOME_RELAY, so routes carry
    over unchanged.
  • Stale-future discipline: relay_receive returns the conn it read,
    and both re-arm and error handling gate on Rc::ptr_eq against the
    pool's current entry — a datagram received on a replaced conn is still
    handled, but stale futures neither double-arm the successor nor
    trigger spurious redials. A retired conn is closed at retirement so
    its pending receive resolves and the teardown drain terminates.
  • The redial dial is deliberately unbounded: settling a select race by
    dropping an in-flight import inside a live task wedges the polyengine
    host, so a dial timeout is not expressible here. The constraint and
    its consequence (a relay that accepts and then stalls the handshake
    pins the slot) are recorded on the pump and redial_home docs.
  • With no caller left, the endpoint-death machinery (dead,
    mark_dead) is gone.

Same decision, adjacent defect (#88): a dead foreign relay now
retires its relay_keys URL mapping along with its pool entry, so a
later dial through that relay reconnects instead of returning a key
whose transmits blackhole.

The regression guard

The deltic exam gains scenario 7, relay outage: two endpoints dial and
echo over an exam-owned relay; the relay is stopped; 8s of outage; the
relay restarts; the same connection must survive and echo (outage
plus the 5s backoff cap stays inside the 30s idle window), then a
fresh dial on the same endpoints must succeed — the anti-brick
assertion — and finally both endpoints close with redials in flight,
which must resolve without a trap. The scenario reports blocked rather
than failed when the relay was adopted instead of spawned (it cannot
restart a process it does not own).

Falsified against the unfixed endpoint: with the endpoint changes
stashed, the connections read closed after the hold —
assertion failed: both connections outlived 8000 ms without a relay (client closed, server closed).

The exam's scenario inventory in host-polyengine/README.md and the
justfile recipe comment had gone stale twice (neither listed the
stream-outcomes or idle-survival rows); both now describe the exam's
shape instead of enumerating rows, per AGENTS.md's docs-state-invariants
rule.

Gates

  • just check (fmt, clippy, validate-wit, test): pass
  • just build: pass
  • just exam-polyengine: 8/8 scenarios pass — outage row:
    survived 8 s without a relay; echo back 4471 ms after it returned, then a fresh dial
  • just matrix: all 13 pairings pass

Fixes #88

A home-relay wire failure permanently bricked the endpoint (issue #88):
the pump treated a receive or send error on the home relay as endpoint
death, so any transient loss of the websocket — a mobile browser
backgrounding the page, a network blip, a relay restart — left the
endpoint refusing every operation, and the application had to rebuild
it from scratch. Upstream iroh reconnects the relay with backoff and
never kills the endpoint over it; the death ruling was an accumulated
divergence with no artifact.

The pump now retires the failed home conn and redials it: jittered
exponential backoff from 10ms, unbounded attempts, immediate-and-reset
after a connection that had lived >= 10s (upstream's established rule,
approximated by uptime). The cap is 5s where upstream uses 16s —
recovery latency after a browser page returns to foreground dominates
this design's use, and the divergence is recorded on REDIAL_MAX_DELAY.
Transmits to the home relay while down are dropped; QUIC loss recovery
owns them. Reconnect reoccupies pool key HOME_RELAY, so routes carry
over unchanged. Stale futures from a replaced conn are gated by
Rc::ptr_eq against the pool's current entry: their datagrams are still
handled, but they neither re-arm against the successor nor trigger a
spurious redial. A retired conn is closed at retirement so its pending
receive resolves and the teardown drain terminates.

The redial dial is deliberately unbounded: settling a select race by
dropping an in-flight import inside a live task wedges the polyengine
host, so a dial timeout is not expressible here — the constraint and
its consequence are recorded on the pump and redial_home docs. With no
caller left, the endpoint-death machinery (dead, mark_dead) is gone.

Same decision, adjacent defect: a dead foreign relay now retires its
relay_keys URL mapping along with its pool entry, so a later dial
through that relay reconnects instead of returning a key whose
transmits blackhole.

The deltic exam gains scenario 7, relay outage: dial and echo over an
owned relay, stop the relay, hold 8s of outage, restart it, and require
the same connection to survive and echo (the outage plus the 5s backoff
cap stays inside the 30s idle window), then a fresh dial on the same
endpoints — the anti-brick assertion — and finally a close with redials
in flight, which must resolve without a trap. Blocked, not failed, when
the relay was adopted rather than spawned. Falsified against the
unfixed endpoint: the connections read closed after the hold and the
scenario fails.

Gates: just check, just build, just exam-polyengine (8/8),
just matrix (13/13 pairings).

Fixes #88
@lann
lann merged commit 1dfb3f0 into main Aug 23, 2026
1 check passed
@lann
lann deleted the home-relay-reconnect branch August 23, 2026 16:40
lannbot pushed a commit that referenced this pull request Aug 23, 2026
Patch release carrying the relay-robustness line landed since 0.5.0:
home-relay failure is a redial rather than endpoint death (#91), every
relay dial carries a deadline (#95), and quiet relay wires are pinged
with unanswered pings retiring the connection (#97). No WIT or JS
surface change; the published @polyengine/runtime floor is already
^0.5.1 (A23 prompt-discard of dropped import futures, polyengine#239).

Gates: build-components + deno publish --dry-run clean; full CI on the
PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

endpoint: home relay wire failure is endpoint death — reconnect with backoff like upstream

2 participants