Fix slot-lag reporting co-located endpoints as stale (round-trip compensation)#2
Open
jenish-25 wants to merge 3 commits into
Open
Fix slot-lag reporting co-located endpoints as stale (round-trip compensation)#2jenish-25 wants to merge 3 commits into
jenish-25 wants to merge 3 commits into
Conversation
An RPC reads its slot when it processes the request (~send + rtt/2), not when the request was sent. Aligning slot readings on send time therefore lets a distant endpoint read the chain later in wall-clock time and report a higher slot for free -- rewarding distance and penalising co-location. SlotLagEstimator instead timestamps each reading at its estimated server-read moment and scores it against the best chain state anyone had observed by then. Network-free and pure, so it lives in core and is unit-tested there.
Feeds send-offset + round-trip into the core estimator instead of comparing slots per tick against a per-tick max. Adds a regression test: two mock RPCs serving the SAME chain clock, differing only in simulated distance (4ms vs 600ms round-trip). Before, the co-located one was reported 0.67 slots behind and the distant one led at 0.00; both must now measure zero. Also saturates the nanosecond cast (a wrapping 'as u64' on Duration::as_nanos would report a huge latency as a tiny one) and corrects the module docs, which claimed latency was measured from each sample's intended start time while the code measured the actual send->reply round-trip.
States what the number does not mean: rtt/2 assumes a symmetric path and folds in server processing time, and lag is relative to the best-informed endpoint in the run (a lone endpoint is always 0), making it a conservative lower bound.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
slot_lagcurrently reports co-located endpoints as stale and distant ones as fresh. For abenchmark whose stated purpose is to surface co-located infrastructure, the metric is inverted.
An RPC server reads its slot when it processes the request — at roughly
send + one_way_delay—not at the instant the client sent it. Two requests fired on the same tick therefore do not reach
their servers at the same time. A distant endpoint reads the chain later in wall-clock time and, purely
because of that head start, returns a higher slot. At Solana's ~400ms slot time an endpoint 600ms
away gains ~0.75 slots on one 4ms away, for free.
probe_allcompares slots aligned on send time (leader[tick] = max(slot)), so that free advanceis scored as superior freshness — and the co-located endpoint is charged the difference as lag.
Reproduction
Two mock RPCs serving the same chain clock (identical
base_slot, both reading it at request-processing time). They are by construction equally fresh — neither node is behind the other, one is
just further away. A correct metric must report
0.0for both. Onmain:This ships as a regression test (
probe::tests::distance_is_not_reported_as_slot_lag), which fails onmainand passes here.The blast radius reaches the leaderboard:
report.rsfeeds this straight into the publishedslotLagfield. Worth saying plainly — the bias runs against rpc edge's own co-located product, so this reads
as an honest bug, not a thumb on the scale. But it means a published board would rank a laggy, far-away
RPC above a fresh, nearby one.
The fix
Timestamp every reading at its estimated server-read moment (
send + rtt/2) and score it againstthe best chain state anyone had demonstrably observed by that moment — a running maximum over
observations ordered by read time. Slots advance monotonically, so this is a sound lower bound on the
true chain state, and it assumes no slot rate.
The logic is pure and network-free, so per
CONTRIBUTING.mdit lives insolbench-core(
slotlag.rs, 9 unit tests) rather than the CLI;probe.rsjust feeds it observations. Public JSONshape (
avg/max/ticks) is unchanged, soserveandreportare untouched.Honest labelling, per the README's own standard — the new docs state what it does not mean:
rtt/2assumes a symmetric path and folds server processing time into the network estimate. It is anestimate of the read moment, not a measurement.
endpoint is always
0.0.is a conservative lower bound on true staleness — it under-reports rather than over-reports.
Shorter
--interval-ms(or more endpoints) tightens it.Verification
It strips out distance bias without zeroing genuine staleness. Live, two public endpoints:
api.mainnet-betareally is ~30 slots behind, confirmed independently withcurlagainst bothendpoints (raw slot deltas of 33 / 33 / 32 across three rounds). The real lag survives; the fake lag
is gone.
cargo fmt --all --check,cargo clippy --all-targets -- -D warnings,cargo test --all(19 tests) and
cargo build --releaseall pass locally.Two smaller things I noticed (not in this PR — happy to open issues or follow-ups)
p99andp99.9carry no tail information at default sample counts. Nearest-rank meansp99 == maxat theprobedefault of--samples 20, andp99.9 == maxfor anyn < 1000.Roadmap item 1 lists "jitter + p99.9 as first-class output — done", but p99.9 is currently just
maxunder another name. Relatedly, the README's example table showsp99=246.25next tomax=251.10at20/20samples, which the current code cannot produce. Suggest either gatingpercentiles on sufficient
count, or the HDR histogram already on the roadmap.The
probe.rsmodule doc claimed latency was measured "from each sample's INTENDED start time, notits actual issue time", while
run_endpointmeasured the actual send→reply round-trip. Since I wasediting that file's docs anyway I corrected the comment to match the code (the open-loop design
already handles coordinated omission properly; only the comment was wrong).