Conversation
eserilev
changed the base branch from
unstable
to
fork-choice-ef-test-runners
September 2, 2026 20:21
eserilev
added
fork-choice
test improvement
Lighthouse rejects valid Gloas data column batches during custody backfill and marks the peers that returned them as faulty. Gloas columns do not contain a signed block header, so the signature list is empty and the batch fails. The initial Gloas support left a TODO at this check: https://github.com/sigp/lighthouse/blob/38f4915ffeb1ba078831d4db91d4e1957ee22990/beacon_node/network/src/sync/range_data_column_batch_request.rs#L188-L198 This change requires a block-header signature only for Fulu columns. Lighthouse still checks Gloas columns against the block root and verifies their KZG proofs before import.
## Issue Addressed Part of the FOCIL implementation. Add the engine API integration for fetching inclusion lists from the execution client. [Execution API spec](https://github.com/ethereum/execution-apis/blob/main/src/engine/bogota.md#engine_getinclusionlistv1) ## Proposed Changes - Add `engine_getInclusionListV1` support to the engine API module, with serde tests for the response format - Expose the capability through `exchange_capabilities` - Use `ProgressiveTransactions` for the response type, matching the `InclusionList.transactions` spec type (sigp#9747) - Add mock execution engine support so the IL flow is testable through the harness
## Issue Addressed Reject bids whose block hash equals the parent block hash ethereum/consensus-specs#5594 Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
…(Gloas builder API 3/5) (sigp#9805) Third PR of the Gloas builder API stack: - beacon_chain: direct builder bid verification (spawned on the blocking executor), gossip-bid refinements, wei-domain bid selection (`BidCandidate`/`BidSource`), and Gloas block production that assembles local, gossip, and direct-builder candidates - network: process payload-bid and proposer-preference gossip, including the new `PayloadBidError` variants - client: construct the `Builders` service when the Gloas fork is scheduled and wire it into the beacon chain The HTTP API still serves `GET` produceBlockV4 at this point; the `POST` conversion and the `Eth-Builder-Url` round-trip land in the next PR.
## Issue Addressed Fix failing CI observed here: - https://github.com/sigp/lighthouse/actions/runs/34082051165/job/101627302362?pr=9983 ## Proposed Changes Keep Rust at 1.88.0 while moving the builder off Bullseye to avoid the failing Debian security package downloads. Update the Ubuntu runtime to 24.04 so its glibc is newer than the builder's. Co-authored-by: antondlr <anton@sigmaprime.io>
take 2 Co-authored-by: Daniel Knopik <daniel@dknopik.de>
## Issue Addressed This PR helps out in 2 ways - sigp#9527 will need the engine API methods for Gloas envelope reconstruction when finalized envelopes are pruned and need to be rebuilt on request - sigp#9652 will require it for SSZ implementation of `getPayloadBodies` V2 methods ## Proposed Changes - Add `engine_getPayloadBodiesByHashV2` engine API method to the `execution_layer` crate in accordance with the engine API [spec](https://github.com/ethereum/execution-apis/blob/main/src/engine/amsterdam.md#engine_getpayloadbodiesbyhashv2) for `glamsterdam` - Add `BlockAccessList` field to `ExecutionPayloadBodyV2` - Extend `EngineCapabilities` for the `getPayloadBodies` V2 methods - ~~Refactor `get_payload_bodies_by_*` public functions exposed through `ExecutionLayer` into a single function for `by_hash` and `by_range` each with V2 method given preference over V1 if the EL advertises it and returns an error if neither V1 nor V2 are advertised by EL~~ No longer applicable with the changes introduced in commit sigp@9feccf4 Note: we don't add `engine_getPayloadBodiesByRangeV2`, because Lighthouse is planning to stop using the range methods: - sigp#9996 Co-authored-by: hopinheimer <knmanas6@gmail.com> Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Closes sigp#9623 ## Description `import_historical_block_batch` writes across the blobs, hot, and cold databases, then updates hot-DB metadata, without sync barriers. On machine crash this can leave durable `AnchorInfo` with missing cold `BeaconBlockRoots` entries (permanent gaps). Backfill only imports slots `<= oldest_block_slot`, so those missing entries are never rewritten on a later run. This PR addresses the issue in two parts: ### Prevention Sync each DB after its write (skipping empty batches), before advancing durable metadata. This matches the freezer migration approach: durable data before durable metadata. ### Repair Add a one-shot repair at store open (`repair_cold_block_root_index`) for nodes that already have gaps from the old behavior: 1. Scan `[oldest_block_slot, split.slot)` for missing `BeaconBlockRoots` entries. 2. If holes exist, reconstruct them from hot blinded blocks by walking the parent chain (same skip-slot filling as backfill). 3. If the index is still incomplete (e.g. required hot blocks are missing), rewind `AnchorInfo` to the lowest slot with a complete suffix up to the split so backfill can re-download that range. 4. Persist a one-shot `BACKFILL_INDEX_REPAIR_KEY` marker so repair does not run again. Repair is skipped when `AnchorInfo` is still uninitialized, so it can run correctly after checkpoint sync initialization. ## Test plan - [x] `cargo check` - [x] `cargo nextest run -p store find_cold_block_root_holes` - [x] `cargo nextest run --release -p beacon_chain --test beacon_chain_tests backfill_index_repair` - [x] `cargo nextest run --release -p beacon_chain --test beacon_chain_tests weak_subjectivity_sync`
…igp#9818) ## Issue Addressed An "overloaded" lighthouse node can fail to write to the database (for example due to file descriptor exhaustion). In these conditions the recovery step `handle_import_block_db_write_error` will fail as well. This ends up putting the node in an inconsistent state where fork choice and the database have diverged (fork choice contains a block that the db does not, a "phantom" block). This PR adds tests to recreate this situation and prove that, in these conditions, fork choice can diverge from the store. In a subsequent PR I will add fixes to prevent this divergence and update these tests accordingly.
## Issue Addressed The order of fields in `PartialDataColumnGroupId` has been swapped in the spec, and need to be swapped as well in our impl. ethereum/consensus-specs@b70c9ec#diff-7dccdd0e4f706e5a9503316db2a896724035a1b6f0feec17f42b0e69d77f95a6L48 ## Proposed Changes Swap the fields so that `Encode` and `Decode` match the spec. ## Additional Info Thanks to @aarshkshah1992 for spotting this!
## Description The only remaining production caller of `engine_getPayloadBodiesByRangeV1` was `ExecutionLayer::get_payload_for_header`, always with `count = 1`. That path now uses `engine_getPayloadBodiesByHashV1` with `header.block_hash()`. Unused by-range client code is deleted: - `get_payload_bodies_by_range` / HTTP `get_payload_bodies_by_range_v1` - by-range metrics and mock RPC handler - by-range V1/V2 capability fields and `LIGHTHOUSE_CAPABILITIES` entries `BeaconChain::get_block` and `BeaconBlockStreamer` are unchanged (streamer already used by-hash). Closes sigp#9996 ## Test plan - [x] `cargo check` - [x] `cargo nextest run -p execution_layer`
## Proposed Changes Enable partial messages on custom networks by default. Notably, this includes Gloas devnets. ## Additional Info This change is also already in the Devnet 7 branch. Alternatively, we might consider just enabling partials on all networks 👀.
## Issue Addressed The [validator monitor](https://github.com/sigp/lighthouse/blob/115bd16fb565c3df2b169a740fa5754adde00762/beacon_node/beacon_chain/src/validator_monitor.rs#L484-L502) walks the already indexed registry during recent block imports, including with default monitoring and zero monitored validators. Milhouse's `iter().skip(index)` traverses the skipped prefix even when there are no new validators. ## Proposed Changes Start the iterator at the first unknown index with `iter_from`, keeping absolute validator indices. A regression test covers empty registries, growth, shorter forks, late registration and pending updates for both fixed and progressive lists. ## Additional Info A local mainnet A/B on 7 Sep 2026 used two Lighthouse followers with separate mock ELs, zero monitored validators and a registry of about 2.36 million validators. Each follower had a 3-CPU quota on the same AMD EPYC-Milan VM. The builds used baseline `1256bd99`, with only this loop changed in the treatment. Each window followed one epoch of settling, with binaries swapped between nodes for the second window. | Median `import_block` duration | Baseline | Treatment | Matched blocks | | --- | ---: | ---: | ---: | | 19:29-19:43 UTC | 199.87 ms | 38.47 ms | 66 | | 19:53-20:07 UTC, after swap | 191.60 ms | 40.56 ms | 66 | One first-window trace was unmatched and excluded; baseline logs confirm that block was received and became head. All slow outliers are included. All 132 matched imports improved. Mean paired import savings were 159.70 ms and 152.68 ms; the monitor child span accounted for 158.24 ms and 150.67 ms respectively. This measures local import completion with simulated execution validity. The monitor runs under the fork-choice write lock, normally after early attestability. These results do not establish earlier attestations, whole-client CPU savings or the same absolute saving in production.
…gp#9997) ## Issue Addressed Gloas block production was calculating the parent_root twice, once before advancing the parent state and once after. The calculation prior to the state advance **would yield the wrong block root** in the case where the parent state was not already advanced. In practice, this didn't occur very often because the state advance timer would make the advanced state available in the state cache. https://github.com/sigp/lighthouse/blob/1256bd99e1b6c5d1290241849354b7e27dbc0ed9/beacon_node/beacon_chain/src/block_production/gloas.rs#L185-L191 This potentially incorrect block root was used to calculate `should_build_on_full` on the next line: https://github.com/sigp/lighthouse/blob/1256bd99e1b6c5d1290241849354b7e27dbc0ed9/beacon_node/beacon_chain/src/block_production/gloas.rs#L193-L199 As a result, we would sometimes end up reading `should_build_on_full` for the **grandparent** block rather than the parent. Often, this wouldn't make any difference. The impact of this bug was further mitigated by the fact that we re-calculated the `parent_root` a 2nd time _after_ the state advance: https://github.com/sigp/lighthouse/blob/1256bd99e1b6c5d1290241849354b7e27dbc0ed9/beacon_node/beacon_chain/src/block_production/gloas.rs#L378-L398 This `parent_root` would always be correct, so the block would always be valid, just possibly building on the wrong empty/full variant. ## Proposed Changes Calculating the `parent_root` at all is conceptually unnecessary. It was already decided and known when we called `load_state_for_block_production`. This PR threads that value through and removes both re-calculations. A regression test is added in `gloas_block_production_parent_root_with_unadvanced_state`. I've verified that it fails on unstable and passes with this patch. ## Additional Info Codex was used, but as you can see from the commit history, there was a lot of tweaking to arrive at the simplest fix. Some more refactors and cleanups are possible (e.g. removing ReOrgInputs), but I'll do that in a separate PR to avoid muddying this bugfix. This bug is similar to another one recently found in block prod. I'm going to try to weed them all out: - sigp#9983
…#9983) ## Issue Addressed Closes: - sigp#9981 ## Proposed Changes Filter out exits for validators which _could_ be impacted by any withdrawal triggered in the parent execution payload. See linked issue for details of the bug. This fix prevents production of invalid blocks (!!) in some scenarios post-Gloas. ## Additional Info Fix & comments written manually, tests fixed by Codex. Regression test by Codex with manual review.
Add a test in operation pool about the index for the cases with and without payload_present. Written with Claude Code and did a self review Currently blocked waiting for ethereum/consensus-specs#5399 fix Update: the fix ethereum/consensus-specs#5473 is merged and included in [v1.7.0-alpha.13](https://github.com/ethereum/consensus-specs/releases/tag/v1.7.0-alpha.13), currently pending for Lighthouse update to alpha 13
…sigp#9962) Closes sigp#9544. ## Description - `find_head_walk` was filtering virtual EMPTY/FULL children of a PENDING node against `viable_nodes`. Under deep non-finality the justified seed can be non-viable, so those children were dropped and `get_head` returned Pending, which breaks block production. - Skip the filter for PENDING heads. Viability filtering still applies to real block children from EMPTY/FULL nodes. - Adds `pending_head_resolves_when_justified_subtree_non_viable` in `gloas_payload.rs`.
## Issue Addressed Fix flaky CI test: - sigp#9999 ## Proposed Changes The test was failing due to a race between importing the block on gossip vs importing the block as a result of its `getBlobs` call completing. The `getBlobs` codepath was missing a notification to the reprocess queue, which this PR adds. There's also a new test added which prevents regression on the `getBlobs` codepath. ## Additional Info Heavily Codex driven, manually reviewed. Change makes sense to me.
## Issue Addressed Each commit has a link to the relevant consensus spec PR
…builder API 4/5) (sigp#9806) Fourth PR of the Gloas builder API stack (beacon-APIs sigp#630): - convert `/eth/v4/validator/blocks/{slot}` to POST with an optional `BuilderConfig` body (min_bid, builder_boost_factor, direct builders) - add `POST /eth/v1/validator/builder_preferences` for forwarding signed builder preferences - set `Eth-Builder-Url` on produceBlockV4 responses when a direct-builder bid wins, accept it on `POST /eth/v2/beacon/blocks`, and forward the signed block to that builder The validator client still uses the legacy GET methods at this point; it migrates in the final PR of this stack.
eserilev
force-pushed
the
fc-compliance-drop-apply-invalid-block
branch
from
September 10, 2026 00:15
aff059a to
976a082
Compare
|
Some required checks have failed. Could you please take a look @eserilev? 🙏 |
mergify
Bot
added
waiting-on-author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Addressed
When a test vector says a block is invalid, our ef/compliance test runner attempts to import an already rejected block directly into
ForkChoice::on_blockto assert that fork choice rejects it too. The point is to prove fc rejects a block that was already rejected further upstream in block verification.Lighthouse's fork choice only rejects blocks under four conditions
This block re-import calls
ForkChoice::on_blockwithPayloadVerificationStatus::Irrelevant. So the block either fails import for the above four conditions OR gets rejected withInvalidPayloadStatus. For post-merge & pre-gloas blocks a block fork choice would have accepted still errors out because of this. We had to introduceblock_rejection_is_fork_choice_levelfor the post-gloas case.For the four conditions LH fork choice rejects blocks we have test cases in
consensus/fork_choice/tests/tests.rs:invalid_block_unknown_parentinvalid_block_future_slotinvalid_block_finalized_slotinvalid_block_finalized_descendantThis PR removes
apply_invalid_blockfrom the fork choice test harness and the jankyblock_rejection_is_fork_choice_levelhelper. This means ef/compliance tests no longer run checks againstForkChoice::on_block. We already have coverage in the tests I listed above, any regressions should get caught in those tests.