fix(tooling): review fixups for the real ethrex block fixture - #884
Merged
Conversation
… sha256 not length, skip heavy VM test in local target, and add offline variant
…e download, and tighten the KZG canary.
…da_vm into feat/ethrex-real-block
… changes The cache filename is keyed on the block number alone and its download rule had no prerequisites, so make treated an already-present `caches/cache_hoodi_<n>.json` as up to date across an `ETHREX_REPLAY_REV` bump and kept converting the old input. Depending on a rev-stamped marker makes a re-pin discard the stale cache and refetch. Without this the mismatch only surfaced downstream as a `conversion_is_reproducible` digest failure whose message reads "regenerate the fixture", pointing at the wrong cause. CI never hit it (fresh checkout, `caches/` is gitignored); local re-pins did.
…uild
Two independent costs in the `test-executor` job, both paid on every PR run:
- `Swatinem/rust-cache` defaulted to `. -> target`, so neither detached
ethrex workspace was cached. `tooling/ethrex-real-block` alone locks ~335
packages, including the blst, c-kzg and secp256k1-sys C builds, malachite
and ark-ff/asm. `cache-all-crates` covers the registry, not artifacts.
Adds both tool workspaces to `workspaces:`.
- k256's `expose-field` was requested only by `lambda-vm-ethrex-crypto`, a
dev-dependency. Under resolver 3 dev-dependency features are not unified
into builds that exclude them, so `cargo run` (the fixture step) and the
`cargo test` right after it resolved different k256 feature sets and
rebuilt k256 plus every ethrex crate above it. Moving that one dependency
to `[dependencies]` makes both resolutions identical:
cargo tree -e features,no-dev -i k256 # lists expose-field
cargo tree -e features -i k256 # matches
sha2 stays a dev-dependency; it has no feature delta either way.
Also corrects the rkyv sync note: it named `executor/Cargo.toml`, which
declares no rkyv. The readers are the guest and the two other detached
ethrex tool workspaces.
… digest `conversion_is_reproducible`'s sha256 is the assertion an ethrex rev bump forces someone to rewrite: the layout moves, the test goes red, a fresh digest gets pasted in. At that moment it stops covering the substituted- chain-config case the README picked it for. Assert the Hoodi chain id directly so that check survives the churn. Docs, all measured rather than estimated: - `ethrex_bench_20.bin` is 32,766 B, not 17 KB. The benchmark scripts generate it in `distinct` mode (20 senders to 20 recipients); 17 KB is the committed `ethrex_bench_4.bin`. `same` mode would be 16,811 B. Names the mode so the row is reproducible, and replaces "a handful" of trie nodes with the 40 accounts that mode actually touches. - "How to test" and the adopt-a-new-block steps omitted `make test-ethrex`, i.e. the only check that screens precompile 0x0a. This crate links a working c-kzg, so a block needing point evaluation passes here and fails in the guest — following the README alone would have missed it. - The validation list was labelled "cheapest first" but is ordered by the argument it builds, and the ~60s cold-build figure understated a ~335 package release tree. The per-test "milliseconds" claim is accurate (0.01-0.03s each) and is kept. Cache-absent failures now say to run `make ethrex-real-block-fixture` instead of surfacing a bare ENOENT from `unwrap()`.
Oppen
reviewed
Jul 31, 2026
Oppen
approved these changes
Jul 31, 2026
The real-block fixture from tooling/ethrex-real-block existed but nothing benchmarked with it: every perf measurement still ran on ethrex_bench_20, a synthetic block of 20 plain transfers. Measured on the same guest ELF, the two workloads are not the same benchmark: ethrex_bench_20 14.2M cycles 411 keccak 80 ecsm real block 168.3M cycles 9,046 keccak 44 ecsm so the synthetic block is ecrecover-bound where a real one is keccak- and trie-bound, and a prover change can move the two in opposite directions. Wire the real block in as the headline, gated exactly like the growth sweep — `/bench-real`, push to main, workflow_dispatch — and NOT on plain `/bench`. One real-block prove is ~15 min on the CPU bench runner against ~25s for the synthetic block, and that runner is single and serialized across all PRs. Because push-to-main always publishes a real-block number into the baseline artifact, `/bench-real` on a PR pays for its own side only. The synthetic block stays as the fast per-comment screen, relabelled as such; GROWTH_PROGRAMS stays synthetic because it plots heap against block size, which needs a family of blocks and cannot come from one real one. Continuations are forced for the real block rather than chosen: monolithic peak heap grows ~3.1 GB per million cycles on this workload family (2007 MB/transfer at R^2 = 0.998, from the current main baseline artifact), so 168M cycles would need ~500 GB. At epoch 2^21 it fits in ~16 GB. Supporting changes: - cli: report `Peak heap` from `prove --continuations`. The tracker only existed on the monolithic path, so the continuation path had no heap metric at all and the benchmark's parser would have failed on it. - Makefile: derive every real-block path from ETHREX_REAL_BLOCK_NETWORK + ETHREX_REAL_BLOCK, and expose `print-real-block-fixture`. Repointing the block is now those two lines; scripts and CI resolve the path from make instead of naming the fixture themselves. - benchmark-pr.yml: anchor the baseline-artifact greps. `real_time_s` contains `time_s`, so the existing unanchored greps would have matched two lines and written a multi-line step output. - benchmark-pr.yml: record which block was measured and refuse to diff against a baseline that measured a different one, so a fixture swap cannot masquerade as a regression. - bench_verify.sh / perf_diff.sh: WORKLOAD=synthetic|real, defaulting to synthetic. Both auto-generate the gitignored fixture when absent, and bench_verify.sh carries the prove flags in its proof-cache marker so a WORKLOAD switch cannot reuse the other workload's proof. - executor/.gitignore: ignore every accepted network's fixture name, not just hoodi's, so repointing cannot make a ~1 MB fixture committable.
…R gate Two changes with one root: the real-block fixture is a benchmark INPUT, and it was being treated as a build artifact and a correctness gate. Fetch it instead of building it. `make ethrex-real-block-fixture` now downloads the finished .bin from a pinned URL and verifies its sha256 before moving it into place, mirroring prepare-sysroot (same dual sha256sum/shasum handling, same hard error when neither exists). Off the fixture's path, and therefore deleted from it: the 3.8 MB ethrex-replay cache download, the converter build, the ~335-package ethrex host dependency tree, and the ETHREX_REPLAY_REV pin. That also unblocks the block we actually want. Upstream hosts an ethrex-replay cache for Hoodi and nothing else, so a mainnet block cannot be produced by the convert-locally route at all — generating its cache takes ~4 min and ~700 archive RPC calls. Fetching a verified binary makes the benchmark block independent of what lambdaclass happens to host, and independent of this crate's own pins, which stay on Hoodi because that is what conversion_is_reproducible reads. ETHREX_REAL_BLOCK_FIXTURE_URL is deliberately left EMPTY — the artifact is not hosted yet, and a guessed URL fails with a 404 nobody can act on. Unset, it fails with an actionable message, and every consumer degrades instead of breaking: benchmark-pr.yml warns and skips its real-block section rather than reddening every push to main, and the usability screen skips too. Adopting mainnet 25453112 once hosted is four Makefile lines plus REAL_BLOCK_FIXTURE in ethrex-tests; its digest and measured cost are recorded in the crate README. Take it out of the required gate. pr_main.yaml's test-executor job no longer generates the fixture, no longer builds tooling/ethrex-real-block, and skips both real-block tests. No product code reads this fixture: it has to be right when it changes, not on every PR, and it was costing every PR a network download plus a cold ~335-package build (blst, c-kzg, secp256k1-sys, malachite, ark-ff/asm) inside the branch-protection gate. Those checks are not dropped, they move to a new path-filtered workflow (.github/workflows/ethrex-real-block.yml) triggered by tooling/ethrex-real-block, tooling/ethrex-tests and the Makefile — i.e. by the things that can actually invalidate them. It runs the converter's tests (host-side parity, network rejection, reproducibility digest) and the block-usability screen (test_ethrex_real_block_native, which is what catches a block needing precompile 0x0a). no_kzg_backend_linked stays in the PR gate: it is a pure unit test and it is the property the usability screen depends on, so it should fail on the PR that breaks it. This supersedes part of 50dfca4: `tooling/ethrex-real-block -> target` is gone from test-executor's rust-cache workspaces, since that job no longer builds the workspace; the new job carries its own cache entry under its own key. `tooling/ethrex-tests -> target` stays and is still doing its job. Corrections while in here, all against a CURRENT guest ELF (counts move ~14% with ELF vintage, so every quoted figure now names its ELF): - synthetic 9.06M cycles, real 147.5M — a 16x ratio, not the 11.9x that came from comparing measurements taken on two different ELF vintages - monolithic peak heap ~4.9 GB/Mcycle from the measured growth fit, so ~700 GB for the real block rather than ~500 GB - continuation peak ~21 GB at epoch 2^21 (15.8 GB working set plus ~69 MB/epoch of accumulated proofs), not ~16 GB; the bundle is ~5 GB on disk and needs rkyv pointer_width_64 to serialize past 2 GiB - ~13 min per prove, not ~15 - bench_vs/run_ethrex.sh claimed ethrex_10_transfers is ~42M cycles and OOMs at 36 GB. It is 6.8M; the figure predates ecrecover becoming an ECSM accelerator.
…sured costs Follow-up to the fetch change. Two things. No block number appears outside the Makefile's four variables. Comments in benchmark-pr.yml and this README previously named a specific candidate block as the destination of a repoint, which would have gone stale the moment a different candidate won and read as if the wiring depended on it. Every figure that is a property of the block is now labelled "current default", and the repoint procedure shows placeholders rather than one candidate's values. Verified: no workflow, script or env var names a block. Recorded the measured cost of each candidate, since prove time is a property of the block and moves with the repoint. mainnet 25453112 is 125,932,956 cycles on the mid-July ELF against hoodi 1265656's 168,319,360 on the same ELF — ~110M vs 147.5M on a current one, so ~10 min and ~19 GB rather than ~13 min and ~21 GB. Cheaper despite the larger fixture: cycles per gas is not constant across blocks, so the earlier gas-ratio estimate for it was wrong. Costs are stated as ranges where they span the candidates.
Executes the documented repoint now that block selection is final. The chosen block was the only one in a 90-day Dune sweep matching the shape constraints (exactly 2 heavy txs, no whale tx, sane plain-transfer share) in the 1.6-2.6M gas band, so it is structurally typical rather than merely large. It is also the cheapest candidate measured, which matters because this workload sits on a single shared bench runner: ~65.6M cycles on a current guest ELF (74,819,518 on the mid-July one) against hoodi 1265656's 147.5M, i.e. ~6 min per prove rather than ~13, ~18 GB peak and a ~1.9 GB bundle. Note cycles/gas is not constant across blocks (~26-38 here), so ranking candidates by gas would have picked wrong: 25453112 carries MORE gas than the hoodi block yet costs ~25% fewer cycles. Wiring: unchanged. Nothing outside the Makefile's variables names a block, so the repoint is those variables plus the test constants that prove it self-consistent. Verified by grep that no workflow, script or env var mentions a block number. The cache is now fetched and verified like the fixture rather than pulled from raw.githubusercontent.com at a pinned ethrex-replay rev. That route only ever worked for the one block upstream publishes, which is what previously forced the converter's tests onto a different block from the benchmarks. Hosting both artifacts collapses that split: conversion_is_reproducible now pins the SAME block the benchmarks prove, so its digest assert covers the artifact in use. ETHREX_REPLAY_REV and its stamp are gone with it — this supersedes 05e0285, whose staleness guard is carried forward in a stronger form below. Fetch targets are now phony with an internal guard (the prepare-sysroot shape) instead of file rules. A file rule does not run when its target exists, so it could never catch the case that matters: an artifact already on disk that is stale after a re-upload under the same block number. The digest is now re-checked on every invocation, which also catches corruption and hand-placed copies. Validation, all run: - make test-ethrex-real-block-converter: 3/3 pass against the new constants. conversion_is_reproducible passing is the self-consistency proof — the cache, the block stats, MAINNET_CHAIN_ID and the digest all describe one block. - regenerating from the cache reproduces the fixture byte for byte (61eba49b...), so the handed-over .bin and the hosted cache agree. - test_ethrex_real_block_native and no_kzg_backend_linked pass: the block clears the guest's precompile surface with the KZG screen live. - fetch guard exercised: matching digest is a silent no-op, a tampered file is detected and reported, and an unset URL fails with an actionable message. Incidental: tooling/ethrex-tests/Cargo.lock gains ecsm's num-integer entry. That manifest has declared it all along and the detached lockfile was stale, so any build regenerates this and a --locked build in that workspace would have failed.
Repoints the benchmark fixture only. The converter's test constants stay on Hoodi 1265656 and do NOT move with it: what they exercise is the CONVERSION, which any real block demonstrates equally well, and Hoodi's is the one cache ethrex-replay publishes upstream — so pinning there costs us no hosting and cannot drift. The fetch is what makes that decoupling possible, and it means a future repoint touches the Makefile plus one constant in tooling/ethrex-tests, never this crate. That restores ETHREX_REPLAY_REV and its rev stamp for the converter's cache, so 05e0285 is no longer superseded. The benchmark block's own cache is fetched from our release alongside the fixture and read only by regen-real-block-fixture; the two caches are now separate variables (ETHREX_CONVERTER_CACHE vs ETHREX_REAL_BLOCK_CACHE) because they serve different purposes and move independently. The fixture URLs are live (bench-fixtures-v1 release), so the TBDs are filled in. The unset-URL guards stay: they now cover the window after a repoint but before the new artifact is uploaded, which is exactly when a push to main would otherwise go red over an upload nobody in CI can perform. Block: 2,428,684 gas, 29 txs, ~65.6M cycles on a current guest ELF (74,819,518 on the mid-July one), 10,478 keccak, 116 ECSM, ~6 min per prove, ~18 GB peak, ~1.9 GB bundle. Only block in a 90-day Dune sweep matching the shape constraints in the 1.6-2.6M gas band (2 heavy txs, no whale, sane transfer mix; heavy-gas 44.3%, top-tx 22.5%, p50 tx gas 41,297), and pre-FWA so transient-clean. Also the cheapest of the three measured candidates, which matters on a shared runner. Validation, all run end to end against the live release: - make ethrex-real-block-fixture from a clean slate: fetch -> checksum -> place, digest 61eba49b... matches. Same for both caches (release + upstream). - unset-URL path still reads correctly for the default block. - converter tests 3/3 against the Hoodi pins. - test_ethrex_real_block_native and no_kzg_backend_linked pass on the mainnet block.
The GPU benchmark ran the synthetic 100-transfer fixture; it now runs the real block, and unlike the CPU path the real block is the DEFAULT here rather than opt-in. The two paths differ because their scarce resource differs: /bench-gpu rents its own Vast box per run, so the cost is money and nothing queues behind it, while the CPU bench shares one runner with every /bench, /bench-abba and /bench-verify in the repo. Same reasoning, opposite conclusion. Synthetic workloads stay reachable through the existing `cont[TX]` / `mono[TX]` tokens with unchanged semantics, so every number recorded before this reproduces with the exact syntax that produced it. A new `real` token names the default explicitly. Implementation follows the pattern already used by bench_verify.sh and perf_diff.sh: scripts/bench_abba.sh gains WORKLOAD=synthetic|real (defaulting to synthetic, so CPU /bench-abba is untouched), resolves the fixture through `make -s print-real-block-fixture`, and fetches it when absent — which a rented box always needs, since it is a fresh checkout. That fetch is a ~1 MB URL + sha256 download, not a build, so it costs seconds of rental. WORKLOAD=real forces --continuations rather than offering it: monolithic would only OOM at this trace length. Deliberately NOT touched, per the GPU sweep's tuning: LAMBDA_VM_VRAM_BUDGET_MB, mempool settings, EPOCH_SIZE_LOG2, K, and the offer query's values. Changing any of them silently would break comparability with the sweep that chose them. No GPU duration is asserted anywhere, because none has been measured — the CPU rate does not transfer, and the sweep's finding that the prover is CPU-bound at the serial producer above epoch 2^21 cuts against assuming a large speedup. The "running…" comment now says the real-block runtime is unmeasured and that the run establishes the baseline, instead of quoting the synthetic workload's ~2.5 hr. Two things checked rather than assumed: - The box RAM floor stays 48 GB and needs no change. Continuation peak heap is set by the epoch size, not the block, so the real block costs the same ~10 GB working set at the default epoch 2^20 plus ~4.6 GB of accumulated epoch proofs — ~14 GB, or ~18 GB at 2^21. The comment now records that instead of leaving the next reader to re-derive it. - The rkyv pointer_width_64 fail-fast is now scoped to synthetic >=40tx. It exists because a large synthetic continuation bundle exceeds the old 2 GiB cap; the real block's bundle is ~1.9 GB, so applying the guard to it would reject PR branches over a limit they do not reach. This workflow never checks the repo out (it orchestrates over the Vast API and SSH), so the workload label is a fixed string rather than a make query; bench_abba.sh prints the resolved fixture path on the box and that lands in the run log.
…anual The knob exists (WORKLOAD=real, which benchmark-gpu.yml drives) but the CPU ABBA workflow keeps its synthetic default: 20 pairs x 2 proves x ~6 min is ~4 hours on the one shared bench runner that every other bench queues behind. Documented as an option to reach for deliberately rather than left as an undocumented capability.
…ock-fixups # Conflicts: # scripts/bench_verify.sh
The branch predated #861 (thin LTO on the ethrex guest) and #863, so every cycle count and derived cost in these docs described a guest that no longer exists. Re-measured on this branch at the merge, guest rebuilt from source: real block 74,819,518 -> 50,781,557 cycles synthetic 9,063,727 -> 8,734,622 cycles ratio ~7x -> ~5.8x so the CPU prove drops ~6 min -> ~4.5-4.8 min and the monolithic-heap figure ~330 GB -> ~240 GB (re-derived from the same measured growth fit via a same-ELF cycle ratio, not rescaled by hand). Accelerator call counts are unchanged at 10,478 keccak / 116 ECSM, as expected — they follow the workload, not codegen. Every count now names the ELF it came from. Two things move them and both have bitten this doc already: guest optimisation (#861), and the clang major on PATH, worth ~2% — the Makefile pins the guest's target flags but not its compiler, so `cc` takes whatever clang is installed. The 50,713,534 the RTX 5090 box measured for this block on the same main commit is that effect, not a regression: clang 18 there, clang 21 here, 0.13% apart. GPU is no longer unmeasured: 59.87 s wall at --epoch-size-log2 22 on an RTX 5090, which is the recommended GPU epoch because VRAM binds — 2^22 leaves 28.9% headroom and 2^23 does not fit a 32 GiB card at all. CPU is ~4.5-4.8x that wall time. The CPU-server epoch recommendation is left as "pending calibration" rather than guessed from the GPU figure. The two alternate candidate blocks keep their PRE-LTO numbers, explicitly labelled and marked for re-measurement. Removing them would lose the selection evidence; presenting them next to a main-vintage number without a label would be exactly the vintage-mixing error the rest of this change is about.
From the RTX 5090 calibration on 2026-07-31 against main @9ccdaf2 (raw traces in ~/workspace/lambda_vm_bench_cache/gpu_epoch_calib_2026-07-31/), same fixture and CLI, one prove per setting: 2^21 70.52 s 19,193 MiB VRAM (58.9%) 25 epochs 1.65 GB proof 2^22 59.87 s 23,193 MiB VRAM (71.1%) 13 epochs 1.12 GB proof 2^23 OOM at 32,079 MiB (98.4%) after 9.7 s — needs ~44 GiB VRAM is the binding constraint, so 2^22 is the largest setting that fits a 32 GiB card, and it is ~15% faster than 2^21 with 28.9% headroom left. 2^23 is unreachable for any card below 48 GiB. Scoped to the GPU flow. bench_abba.sh keeps its 2^20 default for the CPU /bench-abba, and the CLI's DEFAULT_CONTINUATION_EPOCH_SIZE_LOG2 stays 20: 2^22 needs ~28 GiB of HOST memory on a CPU build and would break laptops. VRAM binds on one path and host RAM on the other — not the same question, so not the same default. The CPU-server recommendation is left explicitly pending calibration. Applied to the real-block path only, not to the synthetic cont[TX] tokens. Those exist so a GPU number recorded before the real-block migration still reproduces with the syntax that produced it, and silently re-pointing their epoch would break exactly that. mono[TX] has no epoch.
The CPU sweep landed (2026-07-31, 124 GiB / 32-core box, real block, branch vintage
at 53,757,588 cycles):
2^20 52 ep 616.90 s 14.56 GiB RSS 2.83 GB proof
2^21 26 ep 464.26 s 18.43 GiB 1.72 GB
2^22 13 ep 397.88 s 32.21 GiB 1.15 GB
2^23 7 ep 356.47 s 60.01 GiB 0.90 GB
2^24 4 ep ~334 s ~97-105 GiB (provisional)
There is a real knee: gain per doubling shrinks 24.7 -> 14.3 -> 10.4 -> ~6% while
memory roughly doubles at each step near the top. So the choice is per-machine, and
this commit encodes three tiers rather than one number:
GPU 32 GiB card 2^22 (VRAM-bound; already set in benchmark-gpu.yml)
CPU bench runner 2^22 (host-RAM-bound: 32.2 GiB fits a >=64 GiB box with ~50%
headroom, 2^23's 60 GiB does not)
CPU 128 GiB class 2^23 (the knee; 48% of a 124 GiB box)
laptops / CLI 2^20 (unchanged)
/bench-real moves 2^21 -> 2^22, worth ~14% wall. NOT ~35% — that figure is the
2^20 -> 2^22 gap, and this path was never at the CLI default; it has been explicitly
2^21 since it was wired. scripts/perf_diff.sh moves the same way, since it targets the
same bench server and its real-block epoch should not disagree with /bench-real's.
Absolute seconds from that table are deliberately not copied into any estimate: the
calibration box runs ~8.6 s/Mcycle against this runner's 5.31-5.62, so the ratios
between epochs transfer and the wall times do not. The docs say the first /bench-real
run at 2^22 establishes the runner's real number.
Left alone on purpose: the CLI's DEFAULT_CONTINUATION_EPOCH_SIZE_LOG2 (20, so a
laptop can still prove), bench_abba.sh's 2^20 (CPU /bench-abba), and
bench_verify.sh's CONT_EPOCH_LOG2 (20, chosen by #878 for its SYNTHETIC arm to match
bench_abba's bundle shape) — that one gains a pointer to the tier table for anyone
running its real-block arm by hand.
2^24 fits a 124 GiB box but buys ~6% for ~15% headroom, so it is documented as "fits,
marginal, not recommended on shared boxes" rather than adopted. Its RSS cell is marked
provisional pending the calibration agent's formal report.
`/bench-real` is gone. The real block now runs on every invocation of plain `/bench`,
alongside the synthetic screen rather than instead of it, and push-to-main publishes a
real-block baseline the same way it publishes the synthetic one. One command instead of
two, for the number that actually means something.
Sampled 3x and summarised exactly like the synthetic screen — median headline, spread,
and every raw value — so a single slow run shows up as spread instead of silently
moving the verdict. The verdict bands are wider than the synthetic 5%: 3 runs of a
minutes-long prove resolve less than 5 runs of a 25s one, so anything from 3-10% is
reported as unresolved rather than as "fine".
The baseline fallback path samples to the same count. A 3-vs-1 comparison would put the
two sides' noise on different footings, which is the error sampling exists to avoid.
Cost, stated plainly because it lands on a shared resource: one runner carries every
/bench, /bench-abba and /bench-verify in the repo, and this takes a /bench from about a
minute of proving to ~15-20 min, on every comment and every push to main. That trade is
deliberate. BENCH_RUNS_REAL is the dial, and the comment next to it says so — the
runner's per-run time is still unmeasured (the epoch calibration ran on a different
box), so if runs land above ~6 min each, turning 3 down is the first move rather than
the last resort.
Checked rather than assumed: the proof bundle (~1.15 GB at epoch 2^22) is deleted
between every run on both the PR and baseline loops, so 3 runs do not accumulate ~3.5 GB
of disk; and peak heap is ~32 GiB against a runner floor of >=64 GiB with nothing else
scheduled, since the runner serialises.
The block-identity guard is untouched: a baseline that measured a different block still
refuses to diff.
Renderer: run count moved into the heading ("median of 3") to match the synthetic
section instead of being repeated in the subtitle, and the footer no longer offers a
command to request the real block — its absence now means the fixture could not be
fetched, so it says that and points at the URL. Exercised over 8 scenarios including
tight-spread, bad-spread, and fixture-unavailable.
Removes the synthetic sampled screen from /bench. The job now proves one workload —
3 sampled runs of the real block on the PR side against the cached 3-run baseline
push-to-main publishes — and the comment carries one table.
Why the screen went, recorded in the workflow so the next person doesn't re-add it:
1. Its only unique coverage was the MONOLITHIC prove path, which is vestigial
(reportedly slower than a single-epoch continuation). Runner time spent covering
a path we intend to delete is not a trade worth making.
2. Its crypto mix is one no real block has — 8.83 ECSM/Mcycle against a real block's
~1.3-1.6 — so screening against it tunes the prover for a worst case that cannot
occur.
Net -280 lines: the synthetic prove loop, its baseline arm, its half of the metrics
artifact, its comparison arithmetic, and its section of the renderer all go together.
Leaving the baseline arm behind would have kept proving numbers nothing reads.
`/bench N` now sets the real-block sample count, replacing the old PR-vs-baseline
split (3 and 5). Both sides use one count deliberately: an asymmetric count puts the
two sides' noise on different footings, which is the error sampling exists to prevent.
Still clamped to 5 — at ~4-5 min a run that is also the difference between a 15 and a
25 minute occupancy of the one bench runner.
Untouched, as scoped: GROWTH_PROGRAMS and the synthetic fixture generation it needs
(/bench-growth still sweeps them for a heap-vs-block-size slope, which needs a family
of blocks and cannot come from one real one), /bench-abba, /bench-verify. The
block-identity guard still refuses to diff across a repoint.
Two things the removal exposed and this fixes:
- metrics.txt was created by the synthetic step with `>` and appended to by everything
else. It is now truncated in the config step, which always runs — otherwise a
persistent self-hosted runner would have carried a previous run's values forward.
- A missing baseline is no longer an error. The old code hard-failed on a zero/empty
synthetic baseline; for the real block an absent baseline is the normal state before
main has published one, so it now emits a notice and renders the PR side alone.
Renderer wording, caught by the scenario harness: "prove time down -17.2%" and "up
+17.3%" double-signed themselves, and the no-fixture footer still said "synthetic
blocks only" — which is now simply false, since nothing else is proven. Exercised over
9 scenarios (normal, improvement, regression, unresolved band, bad spread, no
baseline, block mismatch, fixture unavailable, growth sweep).
Contributor
Author
|
/bench |
benchmark-pr.yml's comment step is ~200 lines of JS embedded in YAML, and the only
other way to see its output was to push and run a real benchmark — now ~15-20 min of
occupancy on the single shared bench runner. This extracts the script block and
renders it against synthetic env values, so a wording or formatting change is checked
in a second.
It renders rather than asserts, deliberately: the failures it has actually caught were
readability ones a human spots instantly and an assertion would have to anticipate —
double-signed percentages ("prove time down -17.2%"), and a footer that still claimed
"synthetic blocks only" after the synthetic screen was removed.
The nine scenarios are the renderer's branches: normal, improvement, regression, the
unresolved 3-10% band, bad spread, no baseline yet, block mismatch, fixture
unavailable, and the growth sweep. It lived in a scratchpad while the workflow was in
flux; committing it so the next person editing that comment has it.
Benchmark — ethrex 20 transfers (median of 3)Table parallelism: auto (cores / 3)
Commit: 7a29bb1 · Baseline: cached · Runner: self-hosted bench |
…rypto dev-dep
Review suggestion (Oppen): declaring k256 with `expose-field` in [dependencies]
unifies the feature set between `cargo run` and `cargo test` just as well as
promoting lambda-vm-ethrex-crypto out of [dev-dependencies] — resolver 3 unions
the direct declaration into both resolutions — while keeping the test-only
crypto crate where it belongs and dropping it from the converter binary's
dependency graph entirely.
Verified: `cargo tree -e features{,no-dev} -i k256` identical (both list
expose-field); `cargo tree --no-dev` no longer contains
lambda-vm-ethrex-crypto; converter tests 3/3.
The trade is a duplicated feature requirement that must track
crypto/ethrex-crypto's k256 line; the verification commands in the comment are
the guard.
"ethrex real block" read as a benchmark; it is validation for the fixture and its converter (tests, not measurements). The bench lives in benchmark-pr.yml.
The crate was named after the motivation that produced it ("get a real
block to benchmark") rather than after what it does. What it does is
convert an ethrex-replay cache JSON — of any Ethereum block — into the
rkyv ProgramInput fixture the guest deserializes. Nothing in it is
specific to the benchmark block: its own tests are pinned to Hoodi
1265656 precisely because the conversion is what is under test, not the
workload. Name it for its function.
tooling/ethrex-real-block -> tooling/ethrex-block-converter
package ethrex-real-block -> ethrex-block-converter
.github/workflows/
ethrex-real-block.yml -> ethrex-block-converter.yml
(display name "ethrex real-block fixture tests"
-> "ethrex block-converter tests")
Deliberately unchanged, because they name the BLOCK and that is still
correct: the Makefile's ETHREX_REAL_BLOCK* variables and every
`*-real-block-*` target, the `test_ethrex_real_block_{native,vm}` tests
in tooling/ethrex-tests, the fixture filenames, release URLs and
sha256s. Only path values pointing into the moved directory moved.
Also unchanged: the "lambda-vm-ethrex-real-block-usable" rust-cache key
in the renamed workflow. It keys the block-usability job, whose
workspace (tooling/ethrex-tests) did not move — renaming it would
discard a valid cache of the heavy ethrex tree for nothing. The
converter job's key and its `workspaces:` entry did both move; that
entry is load-bearing, since the crate is a detached workspace the
default `. -> target` would miss.
No behavior change. The converter's 3 tests pass and the reproducibility
digest is untouched, so the fixture bytes are identical.
… key One-time cold cache on the next run; the key now names the screen rather than the renamed crate.
MauroToscano
enabled auto-merge
July 31, 2026 22:55
MauroToscano
disabled auto-merge
July 31, 2026 23:00
The synthetic cont[TX]/mono[TX] modes are gone: the workflow takes an optional pair count and nothing else, pins bench_abba.sh's knobs explicitly (the script is shared with the CPU ABBA flow), and drops the mode input, the token parsing, the per-mode epoch conditional and the rkyv pre-flight that existed only for large synthetic continuation proofs. The running-comment ETA now scales from the measured 4-pair run instead of quoting a synthetic figure, and the Vast RAM-floor comment cites the measured real-block peak.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this became
Started as review fixups for #880; grew into the wiring that PR was missing. #880 built a validated real-block fixture but nothing benchmarked with it — every perf number still came from the synthetic 20-transfer block its own motivation section calls unrepresentative. This branch closes that gap and swaps the block for a measured, screened mainnet one. The branch includes a merge of current
main(@ 9ccdaf2), so all numbers below are main-vintage.The block: mainnet 25368371
2,428,684 gas, 29 txs — selected via Dune sweeps against measured mainnet composition (baseline: 3.5M txs + a 6-month weekly profile). It is the only block in 90 days in the 1.6–2.6M gas band with 2 heavy txs, no whale tx (top tx 22.5% of gas), and a mainnet-like transfer mix (heavy-gas 44.3% vs mainnet ~39%, p50 tx gas 41,297 vs ~38k, tx/Mgas 11.9). Verified: executes statelessly to its post-state root on the guest's precompile surface (no KZG 0x0a — the no-KZG screen was live during the check), free of transient-contract dependence (predates the FWA lottery deployment), 1 EIP-7702 authorization.
ethrex_bench_20(synthetic, growth sweep only)/benchworkload)--epoch-size-log2 22)Real/synthetic ≈ 5.8× by cycles. Both prove times are measured (see "First measured runs" below), not estimated.
Measurement rules learned the hard way, recorded in
tooling/ethrex-block-converter/README.md:Benchmark tiers
/bench— proves the real block, and nothing else: 3 sampled runs vs the cached 3-run baseline published on push to main (/bench Nadjusts, 1–5; missing baseline renders the PR side alone with a notice). Verdict bands are honest about resolution: 3–10% deltas report as unresolved. Measured job time ~12 min. The synthetic sampled screen was removed deliberately: it was the only monolithic-path coverage (a path considered vestigial) and its EC-dense mix (8.83 ECSM/Mcycle vs 1.3–1.6 on real blocks) is not a workload that will ever be real.benchmark-pr.ymlrecords which block it measured and refuses to diff across a repoint./bench-gpu— the real block (golden target), epoch 2^22 per calibration (28.9% VRAM headroom on a 32 GiB card; 2^23 needs ~44 GiB). A/B/B/A paired runs on a rented box — absolute times are per-host (the prover is partly host-CPU-bound); the deltas are what travel. Takes an optional pair count (/bench-gpu [N]) and nothing else./bench-growth,/bench-abba,/bench-verify— unchanged specialist commands (the growth sweep still uses the synthetic fixture family — a scaling curve needs a family of comparable blocks, which real blocks can't provide).bench_verify.sh/perf_diff.sh/bench_abba.sh:WORKLOAD=synthetic|real, defaulting to synthetic.bench_verify.shcomposes with Fix /bench-verify and measure continuations #878's monolithic/continuation split:WORKLOADpicks the block, Fix /bench-verify and measure continuations #878's knobs pick how it's proven; underWORKLOAD=realthe monolithic arm is skipped (a real block monolithically is ~240 GB of heap — it can only OOM).Continuations are mandatory for real blocks, not tuning: monolithic peak heap grows ~3–5 GB/Mcycle (measured, R²≈0.998).
prove --continuationsalso now printsPeak heap(it printed nothing; the benchmark parser would have hard-failed).Fixture mechanism
Fetched by URL + sha256 (the
prepare-sysrootpattern:.tmp+ verify + move, digest re-checked every invocation), from thebench-fixtures-v1pre-release — both fixtures plus their ethrex-replay provenance caches, all digest-pinned. Regenerating the fixture from the hosted cache reproduces it byte-for-byte. No binaries in git. The converter (tooling/ethrex-block-converter, renamed fromethrex-real-block— it converts any block, the name now says so) moves off the per-PR path into a path-filtered workflow (ethrex-block-converter.yml); its ~335-package build, cache download and rev pin leave the required gate.no_kzg_backend_linkedstays in the gate (microseconds, and it's the property the usability screen rests on). Converter pins stay on Hoodi — the one upstream-hosted cache — decoupled from the benchmarked block; repointing the benchmark is five Makefile lines + one test constant.Review fixups (original scope)
ETHREX_REPLAY_REVbump silently kept converting the old cache (now guarded).expose-fieldcame only from a dev-dependency, socargo runandcargo testresolved different feature sets and rebuilt the ethrex tree twice per job.HOODI_CHAIN_ID → MAINNET_CHAIN_IDflip during the repoint instead of letting a substituted config hide behind a regenerated hash.ethrex_bench_20is 32,766 B indistinctmode, not 17 KB; rkyv note; "cheapest first" ordering).Known items
/bench-abba's escalation text points from a 20-tx monolithic result to a 100-tx continuations tiebreaker — pre-existing mismatch, flagged not fixed.tooling/ethrex-tests/Cargo.lockgains one line (ecsm/num-integer): pre-existing staleness;cargo build --lockedin that workspace currently fails on main. Separable on request.clangis on PATH (moves cycle counts ~0.1–2% between clang majors); a ready pin lives on the unpushed-to-PR branchfix/pin-guest-clang— optional hygiene, the bench runner builds its own ELF so bench numbers never depended on it.First measured runs (branch dispatches, pre-merge)
CPU bench runner (run): real block, 3 sampled runs at epoch 2^22 — median 158.8 s, spread 2.8% (156.2/158.8/160.7), peak heap ~52 GB, total job ~12 min including the growth sweep. Two findings: the runner proves the real block at 3.13 s/M cycles vs 5.3–5.6 on synthetic — real-world keccak/trie work is cheaper per cycle than the synthetic EC-heavy mix, so all prior time projections were pessimistic; and peak heap is machine-dependent (52 GB here vs 32 GiB on a 32-core calibration box at the same epoch — table parallelism scales with cores), leaving ~19% headroom on the runner. Stable across all three runs.
GPU (4-pair A/B/B/A branch-vs-main dispatch, rented RTX 5090, epoch 2^22): PR 117.9 s vs main 118.3 s — −0.35%, 95% CI [−1.38%, +0.67%], verdict inconclusive-at-zero, which is the pass condition for a tooling-only branch. Full chain validated (rental → provision → dual build → real-block ABBA → paired-t/Wilcoxon) in ~20 min. Note the per-run absolute (~118 s) vs the calibration box's 59.87 s on the same card class: the prover is partly host-CPU-bound, and rented hosts vary — absolute GPU times are per-box; only same-box A/B deltas are comparable, which is what this workflow measures.
Note: PR-comment triggers run the workflow definition from main (GitHub semantics), so these numbers came from
workflow_dispatchon the branch; after merge, plain/benchcomments exercise this flow directly, and the merge push publishes the first baseline.How to test
Comment-triggered
/benchexercises this workflow only after merge (comment triggers run main's workflow definition); pre-merge validation was done viaworkflow_dispatchon the branch — both runs linked above.