Skip to content

perf(beacon): build justified balances at import, weigh the gloas boost from the snapshot - #665

Open
MegaRedHand wants to merge 8 commits into
feat/beacon-gloas-livefrom
perf/beacon-justified-balances-gloas-live
Open

MegaRedHand wants to merge 8 commits into
feat/beacon-gloas-livefrom
perf/beacon-justified-balances-gloas-live

Conversation

@MegaRedHand

Copy link
Copy Markdown
Collaborator

Motivation

On the eth-3 followers (48h of import timings, 10-03 to 10-05), the head recompute at an epoch-start tick sometimes took 0.5-4 s, and the epoch's first block queued behind it:

network head computes > 0.5 s slot-0 block queue wait
Hoodi 8 (7 in 2-4 s, 1 in 4-12 s) up to 2.3 s
Plataberget 22 up to 1.1 s

At that tick the justified checkpoint moves to (E-1, R), and the first head walk builds the justified-balances snapshot from checkpoint_state((E-1, R)). When that state has left the 32-entry state cache, the chain actor rebuilds it inline: a full snapshot decode (~0.3 s on Hoodi), plus a whole epoch transition when R's block is before the epoch's first slot (an empty first slot, or an epoch-start block orphaned by a late block). A replay of the logs through a model of the state cache puts the rebuilds at ~100 of ~410 Hoodi epoch ticks.

Lighthouse never reads a state at that tick: BeaconForkChoiceStore::on_verified_block builds the justified balances at block import, for the block's epoch-boundary checkpoint, into a small cache keyed by (boundary root, epoch), from a state import already has. The tick only looks them up. Prysm keeps a per-checkpoint balances vector too, but builds it from the checkpoint block's un-advanced post-state, which is not what the specification's checkpoint_states holds; this PR follows Lighthouse.

Changes

This branch carries two open PRs it builds on, merged in with the resolutions tmp/bci-...-gloas-live already uses for gloas (minus #648's active-balance cache and #646's block-production operations, which are not on this branch):

The change itself is 6badb4f3:

  1. A balances cache instead of one snapshot. The store keeps up to 8 JustifiedBalances keyed by checkpoint. When full it evicts the lowest epoch first (ties: oldest insert), and never the current justified checkpoint. Not Lighthouse's plain 4-entry FIFO: on a network with many forks (Plataberget), several blocks crossing one boundary from different parents could flush the canonical entry within an epoch.
  2. Built at import.
    • A block at its epoch's first slot caches the balances from its own post-state, under (epoch, block_root): that post-state is what checkpoint_state returns for it.
    • A block crossing from an earlier-epoch parent (empty first slot) caches them from the boundary state, under (epoch, parent_root), inside transition_block: from the precomputed state on a precompute hit, or by advancing the cloned parent to the epoch start once on a miss (each slot is still processed once). The boundary state depends only on the parent, so building it before the block is validated cannot cache a wrong value.
    • justified_balances() looks the current checkpoint up and keeps the lazy checkpoint_state path for a miss (a restart, or a boundary this node never imported).
  3. The gloas boost reads the snapshot. compute_node_weights fetched the justified checkpoint state on every head walk, only for the gloas boost gate. is_head_weak, is_parent_strong and should_apply_proposer_boost now use the snapshot: the total active balance and vote weights were already there; the equivocator term in is_head_weak needs the raw effective_balance at the justified state, which the vote weights zero for slashed validators, so the snapshot gains an exact copy as a u16 count of EFFECTIVE_BALANCE_INCREMENT per validator (asserted at build). Committees still come from the head/parent block states. The spec-literal get_weight, gloas_get_weight and get_attestation_score stay state-based as references.

On a snapshot hit, no head walk (pre-gloas or gloas) reads the justified state.

Cost

  • One in-order registry pass (~25 ms on Hoodi, lean_beacon_justified_balances_build_seconds) per epoch-boundary checkpoint, at import, on the importer, as in Lighthouse.
  • 10 bytes per validator per cached snapshot (8 for the vote weight, 2 for the new raw effective balance): ~12 MB per entry on Hoodi, so ~96 MB with all 8 entries filled. Worth reviewing whether 8 entries (vs 4) earn their memory.

Testing

  • New: the cache's eviction policy (bound, lowest epoch, ties, protected justified, replace); a crossing import caches the boundary balances on both precompute paths with the block's state root still verifying; a first-slot import caches from its post-state and other slots cache nothing; a snapshot hit reads no justified state on every head path under both fork rules; a slashed equivocator still counts toward head weakness, matching the state-based reference.
  • cargo test --profile release-fast --lib --bins for storage, state-transition, blockchain and types: 1083 passed, 0 failed.
  • Beacon fork_choice fixtures, consensus-specs v1.7.0-beta.2, mainnet: 206 passed, 0 failed.
  • cargo clippy --workspace --all-targets -- -D warnings: clean.

Not covered

  • Live effect on the epoch-tick head recompute: worth re-measuring on a follower after deploy.
  • An on_block unit test of the first-slot hook needs a BLS-signed block; it is tested through transition_block and the cache helper instead.

Since the validator registry moved onto a persistent tree, every
`state.validator(i)` in the fork-choice vote loop is a tree descent, and
`get_proposer_score` rescans the registry through `get_total_active_balance`
(an active-index Vec, then one descent per active index). Both run on every
`get_head`, so the head recompute went from flat-array reads to millions of
descents per call at mainnet scale.

Flatten the justified checkpoint state once into `JustifiedBalances`: a
per-validator weight (zero when inactive or slashed) and the spec-exact total
active balance (slashed-but-active validators count, floored at one
increment), computed in the same in-order pass. It is keyed by the justified
checkpoint itself, so every writer of that checkpoint is covered without a
hook, and its only source is `checkpoint_state(justified)`, the state
`get_weight` reads. `get_weight` stays spec-literal and is the oracle: the
fork-choice fixture runner now compares `compute_weights` with it for every
block of the filtered tree at each `checks` step.

`calculate_committee_fraction` is split so `is_head_weak`,
`is_parent_strong` and the boost share the snapshot's total.

Adds lean_beacon_justified_balances_lookups_total and
lean_beacon_justified_balances_build_seconds.
The vote loop walked a HashMap<u64, LatestMessage> in hash order and probed
the equivocator set once per vote, then read the snapshot balance at a
random offset. Store the votes as a Vec<Option<LatestMessage>> indexed by
validator instead, so votes and snapshot balances are read in index order
and the equivocator probe is skipped while the set is empty. Saturating sums
of non-negative balances give the same weights in any order, so the result is
unchanged.

The table grows to the highest voting index, so set_latest_message asserts
(debug builds) that the index is plausible; indices only come from validated
attestations, and a wild one would otherwise allocate a table to match.
The first block of every epoch ran process_epoch inline on the chain actor,
then paid the post-epoch rehash for its state-root check. The transition
depends only on the parent, which is known before the block arrives.

A blocking worker now clones the head state, advances it to the first slot of
the next epoch, flushes and hashes it, and the actor stores it under
CheckpointState { epoch, root }, the key fork_choice::checkpoint_state already
derives for that checkpoint. Triggers: an import that makes a last-slot block
head, and three quarters into a last slot (covers a skipped last slot). Both
are skipped while syncing; at most one worker runs.

fork_choice::on_block resumes from the cached state for an epoch-crossing
block and applies the block with the new stf::apply_block. A miss, including
a block that races the worker, takes the previous path unchanged.

Adds lean_beacon_epoch_precompute_{lookups_total,seconds,started_total}.
Brings in the justified-balances snapshot that the import-time balances
build on. Resolved as tmp/bci-...-gloas-live resolved the same merge
(df472fa): gloas's `is_head_weak`/`is_parent_strong` keep their own
versions, `compute_node_weights` weighs votes from the snapshot, the
snapshot iterates the registry through `iter_validators`, and the
gloas `LatestMessage` fields are filled in the moved tests. The metrics
hunks take #649's own placement, since #648 is not on this branch.
The import-time justified balances hook into the precompute's
`transition_block`. Taken from tmp/bci-...-gloas-live's resolution of the
same merge (23986cf), which also enables the precompute for gloas
blocks, minus the active-balance cache (#648) and the block-production
operations (#646) that branch threads through `apply_block` and the
precompute tests: neither is on this branch.
…st from the snapshot

At each epoch-start tick the justified checkpoint moves to (E-1, R) and
the next head computation rebuilt that checkpoint's state inline when it
had left the state cache: a ~150 MB snapshot decode (0.25-0.5 s on Hoodi,
about a third of epochs), plus a full epoch transition when R's block
precedes the epoch's first slot (2-4 s). The slot-0 block queued behind it.

Lighthouse avoids this by building the balances at block import, for the
block's epoch-boundary checkpoint, and only looking them up at the tick.
Do the same:

- The store keeps a small cache (8 entries) of JustifiedBalances keyed by
  checkpoint instead of one slot. Eviction drops the lowest epoch first
  (ties: oldest insert) and never the current justified checkpoint, since
  a many-fork network could flush a plain FIFO within one epoch.
- A block at its epoch's first slot caches balances from its post-state
  under (epoch, block_root). A block crossing from an earlier-epoch
  parent caches them from the epoch-boundary state under (epoch, parent),
  inside transition_block: from the precompute on a hit, or by advancing
  the cloned parent to the epoch start once on a miss, so process_slots
  still runs each slot once. The boundary state depends only on the
  parent, so building before the block is validated cannot be wrong.
- justified_balances() looks the current checkpoint up and keeps the lazy
  checkpoint_state path for a miss (restart, unseen boundary import).

The gloas boost gate also read the justified state on every head walk
(compute_node_weights fetched it unconditionally). is_head_weak,
is_parent_strong and should_apply_proposer_boost now read the snapshot:
total active balance and vote weights were already there, and the
equivocator term needs the raw effective balance, which the snapshot now
keeps exactly as a u16 count of increments per validator (2 bytes each,
asserted at build). Committees still come from the head/parent block
states. A head walk on a snapshot hit reads no justified state; the
spec-literal get_weight / gloas_get_weight stay state-based as references.

# Conflicts:
#	crates/blockchain/state_transition/src/beacon/fork_choice.rs
@MegaRedHand MegaRedHand added performance Performance improvements or possible performance improvements beacon Ethereum Beacon Chain client labels Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 665: beacon epoch-transition precompute and justified-balances snapshot

I read the diff and did not build it or run the tests. The design is sound. The precompute is a pure speed trade with an inline fallback, the cache keys match checkpoint_state, and decide is a pure function with good unit tests. Cached states are cloned before use, so a failing block can't corrupt an entry. I found two robustness issues and two nits.

Findings

  1. A worker panic disables precompute for the rest of the process (crates/blockchain/src/epoch_precompute.rs, maybe_start_epoch_precompute).

    • epoch_precompute_in_flight is set before spawn_blocking and cleared only in the EpochPrecomputed handler.
    • If the closure panics, no message is sent. advance_to_epoch_start can panic, since it runs process_slots and hash_tree_root on a large state. The JoinHandle is dropped, so nothing observes the panic.
    • in_flight then stays Some forever, and decide returns None for every later trigger. The node silently goes back to the inline epoch path with no log line and no metric.
    • The fix is to wrap the closure in std::panic::catch_unwind and send Err(..). Alternatively, .await the JoinHandle in a small task and send the failure from there.
  2. build_justified_balances panics on the actor thread (fork_choice.rs, in build_justified_balances).

    • It uses assert_eq!(effective_balance % EFFECTIVE_BALANCE_INCREMENT, 0) and u16::try_from(..).expect(..).
    • It runs on the import and head path against states that can come from a checkpoint-sync provider. The old code only read effective_balance, so a non-conforming state was never fatal.
    • A malformed or hostile anchor state would now crash the chain actor instead of returning an error. The function is already called under Result contexts (justified_balances), so changing it to return Result with an Error::SpecAssert(..) would be cheap.
    • If you keep it as an invariant, note that with_validators test states and the minimal preset need to satisfy it.
  3. Nit: missing blank line (fork_choice.rs). There is no blank line between the end of advance_to_epoch_start and the /// Applies signed_block... doc comment on transition_block. cargo fmt won't flag it, but it reads as a merge slip.

  4. Nit: duplicated epoch computation (epoch_precompute.rs). candidate_epoch in maybe_start_epoch_precompute duplicates the epoch arithmetic inside decide. If the two drift, already_cached would check the wrong key. Consider having decide return the key before the cache check, or putting the epoch computation in one helper.

What looks good

  • The Timer and Head guards in decide are reasonable. The slot checks stop a stale timer message from precomputing against a newer head.
  • The justified-balances semantics look right.
    • Slashed-but-active validators count toward total_active_balance.
    • The total is floored at one increment.
    • A slashed validator's raw effective balance is kept separately for the is_head_weak equivocator add-back.
  • The cache is keyed by the checkpoint, so none of the writers of the justified checkpoint needs a hook. Eviction never drops the current justified entry.
  • The boundary-balances cache is filled before block validation. As the comment argues, that is safe because the boundary state depends only on the parent.
  • The tests are well targeted: precompute equals inline post-state, the gloas parent case, the crossing-import cache fill, and the equivocator and slashed case.

Automated review by Claude (Anthropic) · sonnet · custom prompt

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

A couple of things look solid here: the precompute path is correctly treated as a cache-only optimization, and the new tests around epoch-boundary resume and justified-balance eviction are well targeted.

  • crates/storage/src/store.rs:3881 — potential race in insert_justified_balances. It reads current_justified before taking the beacon lock, then uses that stale value during eviction under a later lock. If another thread advances the justified checkpoint in between, the eviction policy can discard the actual current justified snapshot, violating the “never evict current justified” invariant. This is subtle but consensus-relevant because get_head can then rebuild from disk unexpectedly and observe cache churn depending on timing. Read beacon_justified_checkpoint while holding the same lock used for justified_balances.insert, or pass the current checkpoint in from a caller that already holds the lock.

  • crates/blockchain/src/epoch_precompute.rs:230 — EpochPrecomputed completion blindly clears self.epoch_precompute_in_flight = None without checking that msg.key matches the tracked in-flight key. Today the actor tries to keep one worker at a time, but this makes the invariant fragile: any future duplicate send / retry / reordered completion can clear a newer worker and allow overlapping precomputes. Safer to clear only when self.epoch_precompute_in_flight == Some(msg.key) and ignore/log stale completions.

  • crates/blockchain/src/epoch_precompute.rs:174 and crates/blockchain/src/epoch_precompute.rs:236 — there is a benign-but-costly duplicate-work window: two triggers can both miss already_cached, one starts computing, the block imports inline and caches the same CheckpointState, then the worker stores again. That preserves correctness, but on epoch boundaries it may do an extra full process_slots + rehash. Consider re-checking cache presence in EpochPrecomputed before cache_state, or documenting that the extra write is expected and bounded.

  • crates/blockchain/state_transition/src/beacon/fork_choice.rs:1083 — using Result::inspect for cache_boundary_balances is clever, but easy to miss because it injects side effects into the slot-processing chain. For maintainability, an explicit stf::process_slots(...)?; cache_boundary_balances(...); ... style would be clearer in this consensus-critical path.

  • crates/blockchain/state_transition/src/beacon/fork_choice.rs:1019 — advance_to_epoch_start always calls apply_pending_mutations() and hash_tree_root(), even when state.slot() >= target_slot. That is correct, but it means every spawned precompute pays a full post-state flush/hash even in the no-op case. If callers can ever hand it a state already at the epoch boundary, an early return would avoid unnecessary work.

Overall, I don’t see a direct fork-choice or state-transition correctness break in the main precompute design; the biggest thing I’d want fixed before merging is the lock-order race around justified-balance eviction in crates/storage/src/store.rs:3881.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR focusing on epoch-transition precomputation and justified balances snapshot optimization for beacon fork choice.

Overall Assessment

This is a well-structured performance optimization PR with two main components:

  1. Epoch precomputation: Running process_epoch ahead of time on a blocking worker
  2. JustifiedBalances snapshot: Flattening validator balances to avoid per-vote state lookups

The code is generally correct and well-tested, but I have several concerns around edge cases, concurrency, and spec compliance.


Critical Issues

1. Integer overflow in three_quarter_slot (crates/blockchain/src/epoch_precompute.rs:117)

pub(crate) fn three_quarter_slot(slot_duration_ms: u64) -> Duration {
    Duration::from_millis(slot_duration_ms / 4 * 3)
}

Problem: slot_duration_ms / 4 * 3 performs division before multiplication. For slot_duration_ms = 6_000, this gives 1_500 * 3 = 4_500 (correct). However, for values where slot_duration_ms / 4 truncates, precision is lost. More critically, if slot_duration_ms is very large (though unlikely in practice), slot_duration_ms / 4 * 3 could theoretically overflow after the division, though u64::MAX / 4 * 3 still fits in u64.

Suggested fix: Use slot_duration_ms.saturating_mul(3) / 4 or document the truncation behavior explicitly. The current order matches the test expectation of 4_500 for 6_000, but this should be explicit.


2. Race condition in epoch_precompute_in_flight handling (crates/blockchain/src/epoch_precompute.rs:168-172)

let Ok(Some(head_state)) = self.store.get_state(&head_root) else {
    return;
};

self.epoch_precompute_in_flight = Some(key);

Problem: Between checking inputs.in_flight in decide() and setting self.epoch_precompute_in_flight = Some(key) here, another message could arrive and pass the decide() check because in_flight was still None. The decide() function is pure and reads from inputs, but the mutation happens after an async state access (get_state).

Impact: Two workers could be spawned for the same key. The second one's result would overwrite the first in the cache (harmless but wasteful), or for different keys, both run concurrently (violating "at most one").

Suggested fix: Set epoch_precompute_in_flight before the get_state call, with a rollback on failure:

// Tentatively claim the slot
let old = self.epoch_precompute_in_flight.replace(key);
debug_assert!(old.is_none(), "decide() should have prevented this");
let Ok(Some(head_state)) = self.store.get_state(&head_root) else {
    self.epoch_precompute_in_flight = old; // Release the claim
    return;
};

3. Missing epoch_precompute_in_flight cleanup on actor restart or panic path

In EpochPrecomputed::handle, the in-flight flag is cleared on both success and error. However, if the tokio::task::spawn_blocking task panics or the actor system drops the message, the flag remains set forever.

Suggested fix: Use a guard pattern or set a timeout. At minimum, document this risk. Consider wrapping the spawn in catch_unwind or using spawn_blocking with AbortHandle.


4. decide() allows precompute when head_slot == compute_start_slot_at_epoch(epoch) (crates/blockchain/src/epoch_precompute.rs:95-97)

if inputs.head_slot >= compute_start_slot_at_epoch(epoch) {
    return None;
}

Problem: This check prevents precomputing when the head is already at or past the epoch start. But for Trigger::Timer with slot = 2*SPE - 1 and head_slot = 2*SPE - 3, epoch = 2, compute_start_slot_at_epoch(2) = 2*SPE, and head_slot < 2*SPE passes. However, if head_slot = 2*SPE - 1 (last slot) and timer fires at same slot, epoch = 2, compute_start_slot_at_epoch(2) = 2*SPE, 2*SPE - 1 < 2*SPE passes. But then advance_to_epoch_start would advance from slot 2*SPE - 1 to 2*SPE, running process_slots for one slot. This seems correct.

Wait—re-reading: the comment says "A head past the slot would be a stale message racing a newer import; one already at the boundary has nothing left to advance." But the code checks head_slot > slot for timer, not head_slot >= epoch_start. The head_slot >= compute_start_slot_at_epoch(epoch) check at line 95 catches the "already at boundary" case for both triggers. This seems correct.


5. JustifiedBalancesCache eviction may retain wrong entry under high fork activity (crates/storage/src/store.rs:696-714)

fn insert(&mut self, balances: Arc<JustifiedBalances>, current_justified: &BeaconCheckpoint) {
    // ...
    if self.entries.len() >= JUSTIFIED_BALANCES_CAPACITY {
        let victim = self.entries.iter().enumerate()
            .filter(|(_, (_, entry))| entry.checkpoint() != *current_justified)
            .min_by_key(|(_, (sequence, entry))| (entry.checkpoint().epoch, *sequence))
            .map(|(index, _)| index);
        if let Some(victim) = victim {
            self.entries.swap_remove(victim);
        }
    }
    self.entries.push((sequence, balances));
}

Problem: When full and current_justified is the lowest epoch, all entries may be filtered out, leaving victim = None. The new entry is then pushed, growing beyond capacity. This is a slow leak under specific conditions (all entries are for current_justified or the filter removes everything).

Actually, re-reading: if current_justified matches all entries, victim is None, and we push anyway, exceeding capacity. Next insert will have the same problem. This could grow unbounded if the justified checkpoint doesn't change and many forks at same epoch are cached.

Suggested fix: Handle the case where all entries are protected:

if let Some(victim) = victim {
    self.entries.swap_remove(victim);
} else {
    // All entries are current_justified; still must make room
    // Evict oldest of same-epoch entries or use LRU
}

Or simpler: ensure capacity is at least 2 so there's always a victim that isn't current_justified when inserting a different checkpoint.


6. latest_messages dense array may cause memory issues with malicious indices (crates/storage/src/store.rs:3769-3785)

pub fn set_latest_message(&mut self, index: u64, message: LatestMessage) {
    debug_assert!(
        index < MAX_PLAUSIBLE_VALIDATOR_INDEX,
        "validator index {index} would grow the dense vote table far past any registry"
    );
    let slot = usize::try_from(index).expect("validator index fits in usize");
    let mut beacon = self.beacon.lock().unwrap();
    if beacon.latest_messages.len() <= slot {
        beacon.latest_messages.resize(slot + 1, None);
    }
    beacon.latest_messages[slot] = Some(message);
}

Problem: The debug_assert! is only active in debug builds. In release builds, a malicious or buggy attestation with index u64::MAX / 2 would attempt to allocate ~4 billion entries, causing OOM. The MAX_PLAUSIBLE_VALIDATOR_INDEX bound should be checked in all builds, not just debug.

Suggested fix: Use assert! or explicit bounds check with error return:

if index >= MAX_PLAUSIBLE_VALIDATOR_INDEX {
    warn!(index, "validator index exceeds plausible maximum, ignoring");
    return;
}

7. build_justified_balances uses assert_eq! that could panic on corrupt state (crates/blockchain/state_transition/src/beacon/fork_choice.rs:1480-1485)

assert_eq!(
    validator.effective_balance % preset::EFFECTIVE_BALANCE_INCREMENT,
    0,
    "effective balance is a whole number of increments"
);

Problem: This assert_eq! panics on any state where effective balance isn't a multiple of the increment. While this should never happen on a valid chain, a corrupt database or buggy state transition could crash the node. Consensus code should be defensive.

Suggested fix: Use debug_assert_eq! or handle gracefully with warn! and rounding. The spec guarantees this invariant, but defensive coding is preferable for production.


8. snapshot_attestation_score ignores get_ancestor errors silently (crates/blockchain/state_transition/src/beacon/fork_choice.rs:1370-1381)

if matches!(get_ancestor(index, message.root, block_slot), Ok(ancestor) if ancestor == root)

Problem: get_ancestor errors are silently ignored (treated as "not ancestor"). This matches the spec's behavior where votes for unknown roots are skipped, but the ? operator in get_ancestor can fail for reasons beyond "not in store" (e.g., arithmetic overflow in slot calculation).

Suggested fix: The current behavior is likely correct per spec, but document that get_ancestor errors are intentionally treated as "not on this chain".


9. transition_block clones precomputed state even when not needed (crates/blockchain/state_transition/src/beacon/fork_choice.rs:1045-1106)

let Some(precomputed) = precomputed_epoch_state(...) else { ... };
// ...
let mut state = (*precomputed).clone();

Problem: When crosses_with_empty_slots is false and precomputed is found, we still clone and then may not use the boundary balances path. Actually, re-reading: precomputed_epoch_state only returns Some when parent_slot < epoch_start && epoch_start <= block_slot, so crosses_with_empty_slots is always true when precomputed is Some. The if crosses_with_empty_slots before cache_boundary_balances is redundant.

Wait: crosses_with_empty_slots = parent_state.slot() < epoch_start && epoch_start < block_slot. Note strict < block_slot, not <=. So if block_slot == epoch_start, crosses_with_empty_slots is false, but precomputed_epoch_state requires epoch_start <= block_slot, so it could return Some when block_slot == epoch_start. In that case, precomputed is the state at epoch_start, and we clone it, then skip process_slots (since state.slot() == block_slot), and apply block directly. This is correct but the crosses_with_empty_slots check for cache_boundary_balances is false, so we don't cache. That's correct because the block itself is at the boundary—its post-state will be cached by cache_first_slot_balances in on_block.

Actually, re-reading precomputed_epoch_state: epoch_start <= block_slot (not strict). And crosses_with_empty_slots uses epoch_start < block_slot. So when block_slot == epoch_start, precomputed is Some but crosses_with_empty_slots is false. The code then clones precomputed, doesn't cache boundary balances (correct—the block is its own boundary), and applies the block. This seems correct.


10. advance_to_epoch_start hashes unconditionally (crates/blockchain/state_transition/src/beacon/fork_choice.rs:1017-1028)

pub fn advance_to_epoch_start(
    parent_state: &BeaconState,
    epoch: Epoch,
    config: &Config,
) -> Result<BeaconState> {
    let target_slot = compute_start_slot_at_epoch(epoch);
    let mut state = parent_state.clone();
    if state.slot() < target_slot {
        stf::process_slots(&mut state, target_slot, config)?;
    }
    state.apply_pending_mutations();
    let _ = state.hash_tree_root();
    Ok(state)
}

Problem: The hash_tree_root() is called even when state.slot() >= target_slot (state unchanged). This is wasteful. Also, the discard of the result with let _ = suggests this is for side effects (caching hashes), which should be documented.

Suggested fix: Only hash when we actually advanced:

if state.slot() < target_slot {
    stf::process_slots(&mut state, target_slot, config)?;
    state.apply_pending_mutations();
    let _ = state.hash_tree_root(); // pre-compute and cache tree hashes
}

Security Considerations

11. fork_choice::advance_to_epoch_start called from blocking thread may hold Arc<BeaconState> longer than expected

The precompute worker clones the head state via Arc::new in the spawn_blocking closure. The head_state is obtained from self.store.get_state(&head_root) which returns Option<Arc<BeaconState>>. The clone increments the Arc refcount. If the store evicts this state while the worker holds it, that's fine (Arc keeps it alive). But the worker could run for seconds, keeping a large state in memory.

Mitigation: Documented and acceptable for performance trade-off. The "at most one worker" limit controls concurrency.


12. decide() current_slot > head_slot.saturating_add(1) allows precompute at 1 slot behind only (crates/blockchain/src/epoch_precompute.rs:82-84)

if inputs.current_slot > inputs.head_slot.saturating_add(1) {
    return None;
}

Problem: This gates precompute to "head is at most 1 slot behind wall clock". But if current_slot == head_slot + 1 and we're in the last slot, we precompute. If the block arrives late (next slot), the precompute is wasted. This seems correct per the comment about sync tracker behavior.


Performance Concerns

13. compute_weights and compute_node_weights both call justified_balances independently

In check_weights_against_the_spec and potentially in production code paths, both weight functions may be called, each calling justified_balances which does a cache lookup. This is cheap but redundant.

Suggested fix: Not critical, but consider if compute_node_weights could reuse weights from compute_weights in some paths.


14. for_each_non_equivocating_latest_message iterates entire dense array even with few votes

for (index, message) in beacon.latest_messages.iter().enumerate() {
    let Some(message) = message else { continue; };
    // ...
}

With 1 million validators and few votes, this iterates 1 million entries. The old HashMap only iterated actual votes. This is a trade-off: cache-friendly sequential access vs sparse iteration. The PR claims this is better for the vote loop's index-order access pattern.

Validation needed: Benchmark data would be useful. For a network with low participation, this could be slower.


Code Quality Issues

15. Missing #[derive(Default)] or constructor for DecisionInputs used only in tests

The inputs() test helper constructs DecisionInputs manually. This is fine but a Default impl or builder could reduce boilerplate.


16. PrecomputeKey derives Copy but contains H256 which is Copy—verify this is intentional

H256([u8; 32]) is Copy, so this is fine. But PrecomputeKey is stored in epoch_precompute_in_flight: Option<PrecomputeKey>, and copied into the worker closure. This is correct and efficient.


17. EpochPrecomputed::result uses String for error type instead of structured error

pub(crate) result: Result<Arc<BeaconState>, String>,

Problem: String errors lose structured information and complicate error handling. The fork_choice::advance_to_epoch_start returns Result<BeaconState> with a proper error type.

Suggested fix: Consider using Arc<str> or keeping the original error type serialized. Not critical for logging-only usage.


Tests Review

The tests are comprehensive and well-designed:

  • Unit tests for decide(): Cover all branches (syncing, stale head, in-flight, cached, timer vs head triggers)
  • Integration tests for precompute correctness: Verify resumed state matches inline path
  • Snapshot tests: Verify JustifiedBalances matches spec's state-based computation
  • Cache eviction tests: Verify LRU-like behavior and current_justified protection

Test gap: No test for epoch_precompute_in_flight stuck scenario

If EpochPrecomputed never arrives (task panic, system shutdown), the flag remains set. No test covers recovery.


Documentation

The docs/beacon_stf.md and docs/metrics.md updates are excellent—clear, detailed, and include operational guidance for reading the metrics.


Summary

Severity Count Categories
Critical 2 Race condition (Item 2), OOM vector (Item 6)
High 2 Cache unbounded growth (Item 5), panic on corrupt state (Item 7)
Medium 3 Worker leak on panic (Item 3), unconditional hash (Item 10), integer precision (Item 1)
Low 4 Error type ergonomics, test gaps, documentation, minor optimizations

The PR is fundamentally sound and the optimizations are valuable. The critical issues should be addressed before merge, particularly the race condition in worker spawning and the release-build OOM vector from unbounded array growth.


Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

MegaRedHand added a commit that referenced this pull request Oct 6, 2026
…26-63-64-633-636-638-gloas-live

Keeps #665's import-time balances cache and its split of process_slots at
the epoch start in transition_block, and threads tmp's ActiveBalanceCache
through state_transition and apply_block on both paths. The head walk takes
#665's version (no justified state read). tmp's epoch-precompute tests keep
their operations and sync_aggregate fields, and the one transition_block test
call gains the active_balances argument. The inherited "Temporary merge
branch" markers in metrics.rs and the docs are gone; the justified-balances
metrics are unchanged by #665, and their docs now describe the import-built
cache.
MegaRedHand added a commit that referenced this pull request Oct 6, 2026
…merge/tmp-jb

Picks up the wall-clock fixes for sync duties and the attester-slashings
pool on top of the #665 merge; no conflicts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

beacon Ethereum Beacon Chain client performance Performance improvements or possible performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant