Repository navigation
perf(beacon): build justified balances at import, weigh the gloas boost from the snapshot - #665
MegaRedHand wants to merge 8 commits into
Conversation
Since the validator registry moved onto a persistent tree, every `state.validator(i)` in the fork-choice vote loop is a tree descent, and `get_proposer_score` rescans the registry through `get_total_active_balance` (an active-index Vec, then one descent per active index). Both run on every `get_head`, so the head recompute went from flat-array reads to millions of descents per call at mainnet scale. Flatten the justified checkpoint state once into `JustifiedBalances`: a per-validator weight (zero when inactive or slashed) and the spec-exact total active balance (slashed-but-active validators count, floored at one increment), computed in the same in-order pass. It is keyed by the justified checkpoint itself, so every writer of that checkpoint is covered without a hook, and its only source is `checkpoint_state(justified)`, the state `get_weight` reads. `get_weight` stays spec-literal and is the oracle: the fork-choice fixture runner now compares `compute_weights` with it for every block of the filtered tree at each `checks` step. `calculate_committee_fraction` is split so `is_head_weak`, `is_parent_strong` and the boost share the snapshot's total. Adds lean_beacon_justified_balances_lookups_total and lean_beacon_justified_balances_build_seconds.
The vote loop walked a HashMap<u64, LatestMessage> in hash order and probed the equivocator set once per vote, then read the snapshot balance at a random offset. Store the votes as a Vec<Option<LatestMessage>> indexed by validator instead, so votes and snapshot balances are read in index order and the equivocator probe is skipped while the set is empty. Saturating sums of non-negative balances give the same weights in any order, so the result is unchanged. The table grows to the highest voting index, so set_latest_message asserts (debug builds) that the index is plausible; indices only come from validated attestations, and a wild one would otherwise allocate a table to match.
The first block of every epoch ran process_epoch inline on the chain actor,
then paid the post-epoch rehash for its state-root check. The transition
depends only on the parent, which is known before the block arrives.
A blocking worker now clones the head state, advances it to the first slot of
the next epoch, flushes and hashes it, and the actor stores it under
CheckpointState { epoch, root }, the key fork_choice::checkpoint_state already
derives for that checkpoint. Triggers: an import that makes a last-slot block
head, and three quarters into a last slot (covers a skipped last slot). Both
are skipped while syncing; at most one worker runs.
fork_choice::on_block resumes from the cached state for an epoch-crossing
block and applies the block with the new stf::apply_block. A miss, including
a block that races the worker, takes the previous path unchanged.
Adds lean_beacon_epoch_precompute_{lookups_total,seconds,started_total}.
Brings in the justified-balances snapshot that the import-time balances build on. Resolved as tmp/bci-...-gloas-live resolved the same merge (df472fa): gloas's `is_head_weak`/`is_parent_strong` keep their own versions, `compute_node_weights` weighs votes from the snapshot, the snapshot iterates the registry through `iter_validators`, and the gloas `LatestMessage` fields are filled in the moved tests. The metrics hunks take #649's own placement, since #648 is not on this branch.
The import-time justified balances hook into the precompute's `transition_block`. Taken from tmp/bci-...-gloas-live's resolution of the same merge (23986cf), which also enables the precompute for gloas blocks, minus the active-balance cache (#648) and the block-production operations (#646) that branch threads through `apply_block` and the precompute tests: neither is on this branch.
…st from the snapshot At each epoch-start tick the justified checkpoint moves to (E-1, R) and the next head computation rebuilt that checkpoint's state inline when it had left the state cache: a ~150 MB snapshot decode (0.25-0.5 s on Hoodi, about a third of epochs), plus a full epoch transition when R's block precedes the epoch's first slot (2-4 s). The slot-0 block queued behind it. Lighthouse avoids this by building the balances at block import, for the block's epoch-boundary checkpoint, and only looking them up at the tick. Do the same: - The store keeps a small cache (8 entries) of JustifiedBalances keyed by checkpoint instead of one slot. Eviction drops the lowest epoch first (ties: oldest insert) and never the current justified checkpoint, since a many-fork network could flush a plain FIFO within one epoch. - A block at its epoch's first slot caches balances from its post-state under (epoch, block_root). A block crossing from an earlier-epoch parent caches them from the epoch-boundary state under (epoch, parent), inside transition_block: from the precompute on a hit, or by advancing the cloned parent to the epoch start once on a miss, so process_slots still runs each slot once. The boundary state depends only on the parent, so building before the block is validated cannot be wrong. - justified_balances() looks the current checkpoint up and keeps the lazy checkpoint_state path for a miss (restart, unseen boundary import). The gloas boost gate also read the justified state on every head walk (compute_node_weights fetched it unconditionally). is_head_weak, is_parent_strong and should_apply_proposer_boost now read the snapshot: total active balance and vote weights were already there, and the equivocator term needs the raw effective balance, which the snapshot now keeps exactly as a u16 count of increments per validator (2 bytes each, asserted at build). Committees still come from the head/parent block states. A head walk on a snapshot hit reads no justified state; the spec-literal get_weight / gloas_get_weight stay state-based as references. # Conflicts: # crates/blockchain/state_transition/src/beacon/fork_choice.rs
🤖 Claude Code ReviewReview of PR 665: beacon epoch-transition precompute and justified-balances snapshotI read the diff and did not build it or run the tests. The design is sound. The precompute is a pure speed trade with an inline fallback, the cache keys match Findings
What looks good
Automated review by Claude (Anthropic) · sonnet · custom prompt |
🤖 Codex Code ReviewA couple of things look solid here: the precompute path is correctly treated as a cache-only optimization, and the new tests around epoch-boundary resume and justified-balance eviction are well targeted.
Overall, I don’t see a direct fork-choice or state-transition correctness break in the main precompute design; the biggest thing I’d want fixed before merging is the lock-order race around justified-balance eviction in Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Kimi Code ReviewI'll review this PR focusing on epoch-transition precomputation and justified balances snapshot optimization for beacon fork choice. Overall AssessmentThis is a well-structured performance optimization PR with two main components:
The code is generally correct and well-tested, but I have several concerns around edge cases, concurrency, and spec compliance. Critical Issues1. Integer overflow in
|
| Severity | Count | Categories |
|---|---|---|
| Critical | 2 | Race condition (Item 2), OOM vector (Item 6) |
| High | 2 | Cache unbounded growth (Item 5), panic on corrupt state (Item 7) |
| Medium | 3 | Worker leak on panic (Item 3), unconditional hash (Item 10), integer precision (Item 1) |
| Low | 4 | Error type ergonomics, test gaps, documentation, minor optimizations |
The PR is fundamentally sound and the optimizations are valuable. The critical issues should be addressed before merge, particularly the race condition in worker spawning and the release-build OOM vector from unbounded array growth.
Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt
…26-63-64-633-636-638-gloas-live Keeps #665's import-time balances cache and its split of process_slots at the epoch start in transition_block, and threads tmp's ActiveBalanceCache through state_transition and apply_block on both paths. The head walk takes #665's version (no justified state read). tmp's epoch-precompute tests keep their operations and sync_aggregate fields, and the one transition_block test call gains the active_balances argument. The inherited "Temporary merge branch" markers in metrics.rs and the docs are gone; the justified-balances metrics are unchanged by #665, and their docs now describe the import-built cache.
…merge/tmp-jb Picks up the wall-clock fixes for sync duties and the attester-slashings pool on top of the #665 merge; no conflicts.
Motivation
On the eth-3 followers (48h of import timings, 10-03 to 10-05), the head recompute at an epoch-start tick sometimes took 0.5-4 s, and the epoch's first block queued behind it:
At that tick the justified checkpoint moves to (E-1, R), and the first head walk builds the justified-balances snapshot from
checkpoint_state((E-1, R)). When that state has left the 32-entry state cache, the chain actor rebuilds it inline: a full snapshot decode (~0.3 s on Hoodi), plus a whole epoch transition when R's block is before the epoch's first slot (an empty first slot, or an epoch-start block orphaned by a late block). A replay of the logs through a model of the state cache puts the rebuilds at ~100 of ~410 Hoodi epoch ticks.Lighthouse never reads a state at that tick:
BeaconForkChoiceStore::on_verified_blockbuilds the justified balances at block import, for the block's epoch-boundary checkpoint, into a small cache keyed by (boundary root, epoch), from a state import already has. The tick only looks them up. Prysm keeps a per-checkpoint balances vector too, but builds it from the checkpoint block's un-advanced post-state, which is not what the specification'scheckpoint_statesholds; this PR follows Lighthouse.Changes
This branch carries two open PRs it builds on, merged in with the resolutions
tmp/bci-...-gloas-livealready uses for gloas (minus #648's active-balance cache and #646's block-production operations, which are not on this branch):86b07388)transition_block(b3017e4b)The change itself is
6badb4f3:JustifiedBalanceskeyed by checkpoint. When full it evicts the lowest epoch first (ties: oldest insert), and never the current justified checkpoint. Not Lighthouse's plain 4-entry FIFO: on a network with many forks (Plataberget), several blocks crossing one boundary from different parents could flush the canonical entry within an epoch.checkpoint_statereturns for it.transition_block: from the precomputed state on a precompute hit, or by advancing the cloned parent to the epoch start once on a miss (each slot is still processed once). The boundary state depends only on the parent, so building it before the block is validated cannot cache a wrong value.justified_balances()looks the current checkpoint up and keeps the lazycheckpoint_statepath for a miss (a restart, or a boundary this node never imported).compute_node_weightsfetched the justified checkpoint state on every head walk, only for the gloas boost gate.is_head_weak,is_parent_strongandshould_apply_proposer_boostnow use the snapshot: the total active balance and vote weights were already there; the equivocator term inis_head_weakneeds the raweffective_balanceat the justified state, which the vote weights zero for slashed validators, so the snapshot gains an exact copy as au16count ofEFFECTIVE_BALANCE_INCREMENTper validator (asserted at build). Committees still come from the head/parent block states. The spec-literalget_weight,gloas_get_weightandget_attestation_scorestay state-based as references.On a snapshot hit, no head walk (pre-gloas or gloas) reads the justified state.
Cost
lean_beacon_justified_balances_build_seconds) per epoch-boundary checkpoint, at import, on the importer, as in Lighthouse.Testing
cargo test --profile release-fast --lib --binsfor storage, state-transition, blockchain and types: 1083 passed, 0 failed.fork_choicefixtures, consensus-specs v1.7.0-beta.2, mainnet: 206 passed, 0 failed.cargo clippy --workspace --all-targets -- -D warnings: clean.Not covered
on_blockunit test of the first-slot hook needs a BLS-signed block; it is tested throughtransition_blockand the cache helper instead.