Skip to content

Fix slot-lag reporting co-located endpoints as stale (round-trip compensation)#2

Open
jenish-25 wants to merge 3 commits into
rpc-edge:mainfrom
jenish-25:fix/slot-lag-rtt-bias
Open

Fix slot-lag reporting co-located endpoints as stale (round-trip compensation)#2
jenish-25 wants to merge 3 commits into
rpc-edge:mainfrom
jenish-25:fix/slot-lag-rtt-bias

Conversation

@jenish-25

Copy link
Copy Markdown

The bug

slot_lag currently reports co-located endpoints as stale and distant ones as fresh. For a
benchmark whose stated purpose is to surface co-located infrastructure, the metric is inverted.

An RPC server reads its slot when it processes the request — at roughly send + one_way_delay
not at the instant the client sent it. Two requests fired on the same tick therefore do not reach
their servers at the same time. A distant endpoint reads the chain later in wall-clock time and, purely
because of that head start, returns a higher slot. At Solana's ~400ms slot time an endpoint 600ms
away gains ~0.75 slots on one 4ms away, for free.

probe_all compares slots aligned on send time (leader[tick] = max(slot)), so that free advance
is scored as superior freshness — and the co-located endpoint is charged the difference as lag.

Reproduction

Two mock RPCs serving the same chain clock (identical base_slot, both reading it at request-
processing time). They are by construction equally fresh — neither node is behind the other, one is
just further away. A correct metric must report 0.0 for both. On main:

endpoint                 p50 ms  slot_lag avg   max
near (co-located)           5.5          0.67     1     <-- penalised for being close
far  (distant)            608.2          0.00     0     <-- crowned the leader

This ships as a regression test (probe::tests::distance_is_not_reported_as_slot_lag), which fails on
main and passes here.

The blast radius reaches the leaderboard: report.rs feeds this straight into the published slotLag
field. Worth saying plainly — the bias runs against rpc edge's own co-located product, so this reads
as an honest bug, not a thumb on the scale. But it means a published board would rank a laggy, far-away
RPC above a fresh, nearby one.

The fix

Timestamp every reading at its estimated server-read moment (send + rtt/2) and score it against
the best chain state anyone had demonstrably observed by that moment — a running maximum over
observations ordered by read time. Slots advance monotonically, so this is a sound lower bound on the
true chain state, and it assumes no slot rate.

The logic is pure and network-free, so per CONTRIBUTING.md it lives in solbench-core
(slotlag.rs, 9 unit tests) rather than the CLI; probe.rs just feeds it observations. Public JSON
shape (avg/max/ticks) is unchanged, so serve and report are untouched.

Honest labelling, per the README's own standard — the new docs state what it does not mean:

  • rtt/2 assumes a symmetric path and folds server processing time into the network estimate. It is an
    estimate of the read moment, not a measurement.
  • Lag is relative to the best-informed endpoint in the run, never absolute chain truth; a lone
    endpoint is always 0.0.
  • The reference is only as fresh as the most recent observation at or before that moment, so the number
    is a conservative lower bound on true staleness — it under-reports rather than over-reports.
    Shorter --interval-ms (or more endpoints) tightens it.

Verification

It strips out distance bias without zeroing genuine staleness. Live, two public endpoints:

endpoint         host                          p50ms    p99ms   jitter     lag      ok         slot
rpc edge         solana-rpc.publicnode.com    260.83   527.17   114.45     0.0   10/10    432158746
public mainnet   api.mainnet-beta.solana.com  202.21   700.87   206.05    30.6   10/10    432158715

api.mainnet-beta really is ~30 slots behind, confirmed independently with curl against both
endpoints (raw slot deltas of 33 / 33 / 32 across three rounds). The real lag survives; the fake lag
is gone.

cargo fmt --all --check, cargo clippy --all-targets -- -D warnings, cargo test --all
(19 tests) and cargo build --release all pass locally.

Two smaller things I noticed (not in this PR — happy to open issues or follow-ups)

  1. p99 and p99.9 carry no tail information at default sample counts. Nearest-rank means
    p99 == max at the probe default of --samples 20, and p99.9 == max for any n < 1000.
    Roadmap item 1 lists "jitter + p99.9 as first-class output — done", but p99.9 is currently just
    max under another name. Relatedly, the README's example table shows p99=246.25 next to
    max=251.10 at 20/20 samples, which the current code cannot produce. Suggest either gating
    percentiles on sufficient count, or the HDR histogram already on the roadmap.

  2. The probe.rs module doc claimed latency was measured "from each sample's INTENDED start time, not
    its actual issue time", while run_endpoint measured the actual send→reply round-trip. Since I was
    editing that file's docs anyway I corrected the comment to match the code (the open-loop design
    already handles coordinated omission properly; only the comment was wrong).

An RPC reads its slot when it processes the request (~send + rtt/2), not when
the request was sent. Aligning slot readings on send time therefore lets a
distant endpoint read the chain later in wall-clock time and report a higher
slot for free -- rewarding distance and penalising co-location.

SlotLagEstimator instead timestamps each reading at its estimated server-read
moment and scores it against the best chain state anyone had observed by then.
Network-free and pure, so it lives in core and is unit-tested there.
Feeds send-offset + round-trip into the core estimator instead of comparing
slots per tick against a per-tick max. Adds a regression test: two mock RPCs
serving the SAME chain clock, differing only in simulated distance (4ms vs
600ms round-trip). Before, the co-located one was reported 0.67 slots behind
and the distant one led at 0.00; both must now measure zero.

Also saturates the nanosecond cast (a wrapping 'as u64' on Duration::as_nanos
would report a huge latency as a tiny one) and corrects the module docs, which
claimed latency was measured from each sample's intended start time while the
code measured the actual send->reply round-trip.
States what the number does not mean: rtt/2 assumes a symmetric path and folds
in server processing time, and lag is relative to the best-informed endpoint in
the run (a lone endpoint is always 0), making it a conservative lower bound.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant