fix: price public simulation at a slot L1 has not already taken by spalladino · Pull Request #25357 · AztecProtocol/aztec-packages

spalladino · 2026-08-29T01:45:05Z

Fixes #25344. Fixes A-1885. Part of A-1903.
Design: https://claude.ai/code/artifact/e9b78bd2-501d-4c30-a81a-5a10cbeed392

The symptom

On a fresh sandbox, a wallet asks the node what fee to pay, pads it by 50%, and sends the transaction to simulatePublicCalls. Sometimes the node's own simulation rejects it:

maxFeesPerGas.feePerL2Gas must be greater than or equal to gasFees.feePerL2Gas,
but got maxFeesPerGas.feePerL2Gas=1058030306 and gasFees.feePerL2Gas=3415500000

The wallet paid exactly what the node told it to pay.

Where the fee number comes from

Both the wallet quote and the simulation ask the L1 rollup contract the same question — "what is the minimum mana fee for a block in slot X?" (Rollup.getManaMinFeeAt). The answer depends on an L1 gas oracle updated when checkpoints are proposed, and the new value only kicks in a couple of slots later, so the answer is a step function of the slot. While the sandbox's anvil base fee is decaying (the first minutes after start) each step is a large drop — in the issue, 3,415,500,000 → 920,600,000.

The two sides therefore only agree if they ask about the same slot. They didn't.

The bug: the simulator could target a slot that was already taken

The fee quote (FeeProviderImpl) picks its slot as max(slot of the latest checkpoint on L1 + 1, next slot by the node clock) — anchored to L1.

The simulator (NodePublicCallsSimulator.computeTargetSlot) picked max(next slot by the node clock + pipelining offset, slot of the locally proposed checkpoint + 1) — anchored to the node clock, and that second term disappears once the archiver promotes the proposed checkpoint to checkpointed.

Worked example (72s slots, the oracle steps at slot 15):

The sandbox builds checkpoint 14, sends it to L1, anvil mines it. L1 now says: latest checkpoint is at slot 14. The node's clock has not been bumped yet — the automine sequencer only advances it at the very end of its publish routine.
Fee quote: the latest L1 checkpoint is at slot 14, so the next block is slot 15 at the earliest → fee for slot 15 → cheap (post-step) → the wallet declares 1.5x that.
Simulator: the node clock says the next slot is 13, plus the pipelining offset → slot 14. There is no proposed checkpoint any more (already promoted) → fee for slot 14 → expensive (pre-step).
expensive > 1.5 x cheap → the assertion fires.

Slot 14 is nonsense for the simulator to target: a checkpoint already exists there on L1, so the next block can only land in slot 15 or later. The simulator didn't know, because it trusted its clock over the chain.

The fix

computeTargetSlot gains a third term in its max: slot of the latest checkpointed checkpoint + 1.

The slot is read from the checkpointed tip's block header by the tip's block hash, not by its number, so a checkpoint unwind that replaces the block at that number cannot silently answer with a different block's slot. A miss means the archiver no longer holds the block its own tips name — a torn snapshot — and throws a retryable error rather than dropping the floor.
The term never lowers a correct answer: when the clock is ahead, as it normally is, the clock term is already larger. It only binds when the clock is behind the chain, which is exactly the broken case. In the example above, max(14, 15) = 15 — the same slot the quote used.
Before the first checkpoint lands the checkpointed tip is the genesis block, which the archiver does not store; there is no slot taken yet, so the floor is skipped rather than failing the simulation.

Nothing else changes: no new L1 call on any path, no fee RPC touched, and transaction admission (isValidTx, the p2p validators) is untouched.

What this does not fix

The residual L1-poller window. The fee provider and the archiver each run their own L1 poll, so they can briefly hold different views of which checkpoints exist on L1, and the quote and the simulation can still disagree across that window. A follow-up PR drives the fee provider from the archiver's L1 sync point and pins every fee read to that L1 block, which closes it.
The frozen mid-checkpoint fee (Bug 2). All blocks in a checkpoint share the fee frozen into its first block. The simulator honours that (it copies the latest proposed block's header); the quote only looks at forward-looking L1 projections and never sees the frozen value, so a correctly-priced transaction can still fail simulation on a network with multi-block checkpoints when fees are falling fast. This is not reachable on the sandbox (one block per checkpoint) and was not the reported issue. A later PR in this stack makes the quote lead with the fee the next block will actually charge, which fixes it.

Tests

aztec-node/src/aztec-node/fee_quote_vs_simulation.integration.test.ts (new): real anvil, real L1 contracts, RollupContract, EpochCache, GlobalVariableBuilder, FeeProviderImpl, AztecNodeService/NodePublicCallsSimulator, and a real PublicProcessor on a real world-state fork; only the archiver is mocked. It steps the oracle 1000 gwei → 1 gwei (~1000x fee step) and plants the L1 pending checkpoint at the slot before the step via storage cheats. Two cases: a lagging node clock (reproduces the exact assertion from the issue without the fix) and nothing lagging (quote and simulation already agree, and must keep agreeing).
Unit tests in node_public_calls_simulator.test.ts: the floor binds when the clock lags; it does not raise the slot when the clock is ahead; the checkpointed tip is read by hash and not by number; a missing tip block throws a retryable error; the floor is skipped at genesis.
RollupCheatCodes.setPendingCheckpoint extracted from fee_predictor.test.ts so both suites plant a pending checkpoint the same way.

Stack

#25384 (atomic archiver L2Frontier snapshot, A-1897) is stacked on top of this PR. It replaces the by-hash tip-header read added here with a field of the snapshot, so the slot and the overrides plan come from one atomic archiver read.

This is the bottom PR of the five-PR fee-quote / public-simulation series (GitHub stack #25385), merged in order:

fix: price public simulation at a slot L1 has not already taken #25357 — slot floor (this PR, A-1885)
fix: read archiver tips and proposed checkpoint as one atomic snapshot #25384 — atomic archiver L2Frontier snapshot (A-1897)
fix: drive the fee provider from the archiver's L1 sync point #25392 — fee provider driven by the archiver's L1 sync point (A-1904)
refactor: move next-block planning and the boundary fee cache out of the public calls simulator #25393 — next-block predictor, planning and fee cache out of the simulator (A-1905)
feat: lead the fee quote with the fee the next block will charge #25394 — quote leads with the fee the next block will charge (A-1906)

## The symptom Any wallet/PXE talking to a node with npm-installed Aztec packages dies as soon as it simulates a transaction: ``` ERROR: pxe:service RangeError: RangeError: Maximum call stack size exceeded at get schema (.../@aztec/stdlib/dest/tx/public_simulation_output.js:23:42) at Object.getter (.../@aztec/stdlib/dest/tx/public_simulation_output.js:24:66) ``` It shows up in the `aztec-up` release tests (`bridge_and_claim`) and for anyone installing the published packages. It does not show up anywhere in yarn-project CI. ## What is going on Two schemas describe recursive data and refer to themselves through `z.lazy`: - `NestedProcessReturnValues` — return values of a public call tree - `PrivateCallExecutionResult` — a private call and its nested calls Both were written as `static get schema()` that builds a brand new schema object on every access: ```ts static get schema() { return z.object({ // every dereference of this lazy calls the getter again, which builds another schema nested: z.array(z.lazy(() => NestedProcessReturnValues.schema)), }); } ``` zod ties a recursive schema together by **object identity**: the lazy target is expected to be the same schema instance each time it is dereferenced. Up to zod 4.4.x this only cost extra allocations. From **4.5.0** onwards the schema graph is walked eagerly, so a lazy target that hands back a fresh instance every time never closes the loop and recurses until the stack overflows — at *any* nesting depth, including none at all (`{ nested: [] }` is enough). Why it is invisible in CI: the monorepo's `yarn.lock` pins zod 4.4.3, but every package declares `"zod": "^4"`. The release tests and real users npm-install the published packages, so they resolve whatever is latest — zod 4.5.0 was published on 2026-08-28, and CI started failing on the first `aztec-up` run after that. ## When was this introduced The pattern is old, the breakage is new — and it is **not** the zod 4 migration. | | | |---|---| | `NestedProcessReturnValues.schema` written as a rebuilt-per-access recursive getter | #9672, reverted in #9875, relanded as #9878 — Nov 2024 | | `PrivateCallExecutionResult.schema` written the same way | #11155 — Jan 2025 | | yarn-project migrated zod 3 → zod 4 | #23410 — May 2026 | | zod 4.5.0 published, pattern becomes fatal | 2026-08-28 18:14 UTC | | first `aztec-up` CI run after that | 2026-08-29 01:55 UTC — fails | Verified directly by parsing a rebuilt-per-access recursive schema against each release: zod **3.25.76**, **4.0.0** and **4.4.3** all accept it; **4.5.0**, **4.5.2** and **4.5.4** all overflow the stack. So the schemas were latently wrong for ~21 months, survived the zod 4 migration untouched, and only started failing when 4.5.0 began walking the schema graph eagerly. Nothing in this repo changed — the dependency did. ## The fix Build each schema once on first use and hand out that same instance: ```ts let nestedProcessReturnValuesSchema: ZodFor<NestedProcessReturnValues> | undefined; function getNestedProcessReturnValuesSchema(): ZodFor<NestedProcessReturnValues> { return (nestedProcessReturnValuesSchema ??= z.object({ nested: z.array(z.lazy(() => getNestedProcessReturnValuesSchema())), }) /* ... */); } ``` Construction stays lazy (no module-eval-time schema building, so no new circular-import hazards), and the recursion now closes on a stable object. The other `z.lazy` uses in the tree (`AbiValueSchema`, `AbiTypeSchema`, `AbiDecodedSchema`, `callTraceSchema`) already point at a single module-level schema, and were verified to parse fine under zod 4.5.x. ## Pinning zod to a minor Fixing the schemas removes today's breakage; the range is what let a dependency change behaviour for users without anyone here choosing it. All 17 workspaces declared `"zod": "^4"`, so the published packages accept any 4.x while the monorepo's lockfile quietly held 4.4.3 — CI and consumers were never running the same zod, which is precisely why this reached users without a single failing test in yarn-project. The range is now `~4.4.3` everywhere: patch releases still flow, a minor bump has to be deliberate. The lockfile resolution is unchanged (4.4.3), so the only lockfile diff is the descriptor rename — no transitive churn. ## Verification With zod 4.5.2 swapped into `node_modules` locally: - before the change: 5 failures across the two test files, each `RangeError: Maximum call stack size exceeded` — the exact CI error; - after the change: all 10 pass, and the full stdlib suite is green (132 suites, 1060 tests); - with the repo's pinned zod 4.4.3 the suite is green as well. Reproduced independently against zod 4.5.0, 4.5.2 and 4.5.4 (all fail on a rebuilt-per-access lazy target, all pass on a stable one); zod 4.4.3 tolerates both. Each schema also gets a regression test asserting it hands out the same instance on every access, which is the invariant that keeps the recursion closed. ## Note Split out of #25357, whose CI surfaced this. Unrelated to that PR's changes — the failure reproduces on the base branch with zod 4.5.x installed. Fixes A-1886

The public simulator picked the next block's slot from the node clock, so a node whose clock lagged the chain could price a slot a checkpointed checkpoint already occupies. The L1 gas oracle steps between slots, so the wallet quote (anchored to L1) and the simulation could disagree by a whole fee step and reject a correctly-padded transaction. `computeTargetSlot` now takes a third `max` term, `checkpointedTipSlot + 1`, read from the checkpointed tip's header by block hash so an unwind cannot answer with a different block. It only binds when the clock is behind the chain; before the first checkpoint lands there is no checkpointed slot and the floor is skipped.

alexghr · 2026-09-07T16:35:05Z

+   * - The sequencer's exact formula, `getEpochAndSlotInNextL1Slot().slot + PROPOSER_PIPELINING_SLOT_OFFSET`.
+   * - `proposedCheckpointSlot + 1`, an RPC-side approximation of the next build: when a proposed checkpoint
+   *   is gossiped before its L1 slot starts, the next build (once its wall clock arrives) will target
+   *   `parentSlot + 1`. The sequencer never advances its own target past wall clock — it just declines to
+   *   build — so this is a prediction of inclusion globals, not literal sequencer behavior. The parent slot
+   *   comes from the proposed checkpoint header so the slot and the overrides plan cannot derive from
+   *   different snapshots.
+   * - `checkpointedTipSlot + 1`, a floor: the next block can never land in a slot already taken by a
+   *   checkpointed checkpoint. This only binds when this node's clock is behind the chain, in which case the
+   *   first term would otherwise price the next block in a slot L1 has already moved past — and the L1 gas
+   *   oracle can step between the two, so wallet quotes and simulations would disagree on the fee.


IIUC this most likely is going to be the next L2 slot at the current timestamp, which with pipelining is correct because we're building blocks for slot N+1 during slot N.

#25384) Fixes A-1897. Part of A-1903. Design: https://claude.ai/code/artifact/e9b78bd2-501d-4c30-a81a-5a10cbeed392 Planning the next block needs several facts about the *same* instant: the L2 tips, the leading proposed (not-yet-L1-confirmed) checkpoint, the pending-chain validation status, and the headers those tips name. They came from separate reads with different backing — `getL2Tips()` served from the in-memory `L2TipsCache`, `getProposedCheckpointData()` and `getPendingChainValidationStatus()` read straight from the store, headers fetched by a further lookup — so an archiver write landing between them produced a torn set. The typical shape: a checkpoint that has just been promoted is gone from the proposed map while the cache still reports the pre-promotion tips, so the next block is classified as continuing an in-progress checkpoint and gets mis-priced. ### Design - `BlockStore.getL2Frontier(genesisBlockHash)` resolves the four tips, the leading proposed checkpoint, the proposed tip's `BlockHeader`, the latest checkpointed checkpoint (`CheckpointHeader` + its `L1PublishedData`), and the pending-chain validation status inside a single `db.transactionAsync`, which is a consistent LMDB read snapshot. It replaces `getL2TipsData`, reusing its body. Every added field comes from bytes the transaction already loaded: the latest block entry carries its header, the latest checkpoint entry carries the header the call already deserialized for the tip plus its L1 buffer, and the validation status is one singleton read. - `L2TipsCache` becomes `L2FrontierCache`: it holds the whole snapshot behind one promise, refreshed at exactly the points the tips cache refreshed before (every `ArchiverDataStoreUpdater` write, after the transaction commits). `getL2Tips()` stays and derives from the snapshot. Because readers get one immutable object, refresh ordering versus commit no longer affects consistency — only freshness — so a plain post-commit refresh suffices. - The snapshot carries an `l1SyncPoint: { blockNumber, blockHash }`, the L1 block whose state its data reflects. `L1Synchronizer.syncFromL1` sets it at the top of a pass, right after the no-new-block early return and **before any of that pass's writes**, via `L2FrontierCache.setL1SyncPoint` (which rewrites the cached object with no store read). The ordering is the invariant: the pass's writes reflect L1 state up to that block, so the anchor must move ahead of the data and never behind it. A reader that saw data from L1 block N under an anchor of N-1 would price a fee at N-1 while the data already includes a checkpoint that landed at N. Data behind the anchor is harmless, because the overrides plan derived from the snapshot fully describes the parent. - `L2BlockSource.getL2Frontier()` is the new interface method, with the `L2Frontier` type and zod schema in `l2_block_source.ts` and an entry in `ArchiverApiSchema` so the whole snapshot round-trips over JSON-RPC. `getCheckpointedTipSlot(frontier)` is a free helper over the checkpointed checkpoint's header, so the slot has one source rather than a duplicated field. ### Consumers migrated - `NodePublicCallsSimulator` makes one `getL2Frontier()` call and performs **no** block lookup at all. The mid-checkpoint globals copy reads `frontier.latestBlockHeader` instead of `getBlockData({ number })`; a missing header at a non-genesis proposed tip is now an invariant violation and throws, rather than falling through to the boundary path and double-inserting the ongoing checkpoint's L1-to-L2 messages. The slot floor reads the checkpointed checkpoint's header slot from the same snapshot, deleting the by-hash header read #25357 added. `buildSimulationOverridesPlan` reads `frontier.pendingChainValidationStatus` instead of calling `getPendingChainValidationStatus()`. - The sequencer's `checkSync` takes tips, proposed checkpoint and validation status from one `getL2Frontier()` instead of three reads in the same `Promise.all`; the orphan-block guard now compares the block against a pair from the same snapshot, and the comment claiming there was no split read to reconcile is corrected. `checkSync` also now looks the tip block up **by hash** instead of by number: the tips come from the cached snapshot, so after a prune-and-replace commits (but before the cache refreshes) a by-number read could return the replacement block while every hash check in the function still described the pruned one (pre-existing, surfaced by review). The archiver's `getBlockData` resolves a hash matching the initial header to the genesis sentinel, so the genesis case still works. - `automine_sequencer.ts` replaces its unguarded `Promise.all([getL2Tips(), getProposedCheckpointData()])` with one `getL2Frontier()`. - Standalone by-number/by-slot proposed-checkpoint queries stay store-backed and unchanged. `TXEArchiver` and `MockL2BlockSource` implement the new method directly (neither ever holds a proposed checkpoint, and both report a valid pending chain and no L1 sync point). ### Tests - `block_store.test.ts`: `getL2Frontier` on an empty store (no headers, valid pending chain), the tips / proposed checkpoint / block header / checkpointed header + L1 data together, the validation status read in the same snapshot, and a promotion moving the checkpointed tip and dropping the proposed entry within one snapshot. - `data_store_updater.test.ts`: a reader that lands in the window after a promotion commits but before the cache refreshes gets a self-consistent snapshot, header included. Verified red — with the pair split back apart the frontier and the proposed tip disagree (`Expected: 1, Received: 0`) — and green with the atomic read. - `l2_frontier_cache.test.ts`: warm reads do not hit the store; a refresh reloads once; a sync point set before or after the first load is attached without a store read and survives a refresh; each update yields a fresh object, so a snapshot a reader already holds cannot change under it. - `archiver-sync.test.ts`: the anchor-before-writes invariant, asserted over a real synchronizer and store. Verified red by moving `setL1SyncPoint` to the end of `syncFromL1` (`Expected: < 18, Received: 23`) and green with the call in place. - `node_public_calls_simulator.test.ts`: rewritten around a `makeFrontier` helper. The floor tests keep their coverage but now read the slot from the snapshot; the by-hash-lookup and torn-snapshot tests are replaced by "no block read happens at all" and "a snapshot missing the proposed tip header is rejected"; genesis still lets the clock win because no checkpoint has landed yet. - `archiver.test.ts` in stdlib covers the JSON-RPC round-trip for every new field (bigint and `Buffer32` included). ### Notes - Stacked on #25357, which introduces the slot floor this PR rewires onto the snapshot. GitHub retargets this PR to `merge-train/spartan-v5` automatically once #25357 merges. - The next PR in the series drives `FeeProviderImpl` from `l1SyncPoint`: it stops polling L1 for its own head, refreshes when the archiver's anchor advances, and pins every fee read to that block, so the quote and the simulation can no longer disagree about which checkpoints exist on L1. - `l1SyncPoint` is set when a sync pass starts, before any write, so the L2 data is never ahead of its anchor. The other direction is possible while a pass is in flight, and during a same-height L1 reorg the anchor points at the new fork until that pass reconciles the frontier; a pass that fails part-way leaves the anchor at the new block, and the next pass retries since the "no new L1 block" early return is only armed on success. That window was the same before this series, when fees were read at `latest`. - Supersedes #25383 (writer-side `L2TipsCache.refreshAfter`), now closed. - Deployments with a **remote archiver**: migrated clients call `getL2Frontier` unconditionally, so a new node against an older remote archiver gets method-not-found until the archiver is upgraded. Upgrade the archiver first (or together, as our deployments do); a client-side fallback to the old read pair would reintroduce the torn snapshot this PR removes, so none is provided.

spalladino added the ci-draft Run CI on draft PRs. label Aug 29, 2026

spalladino mentioned this pull request Aug 30, 2026

fix(stdlib): keep recursive zod schemas identity-stable #25364

Merged

spalladino force-pushed the spl/a-25344-fee-slot-floor branch from beb0cae to 4574a12 Compare September 1, 2026 15:48

This was referenced Sep 1, 2026

fix: repoint the archiver tips cache before a write commits #25383

Closed

fix: read archiver tips and proposed checkpoint as one atomic snapshot #25384

Merged

spalladino force-pushed the spl/a-25344-fee-slot-floor branch from 5f76a33 to c03f640 Compare September 2, 2026 19:20

spalladino changed the title ~~fix: price public simulation and wallet fee quotes at the same slot~~ fix: price public simulation at a slot L1 has not already taken Sep 2, 2026

spalladino marked this pull request as ready for review September 2, 2026 20:29

spalladino removed the ci-draft Run CI on draft PRs. label Sep 2, 2026

alexghr approved these changes Sep 7, 2026

View reviewed changes

spalladino merged commit ca0774e into merge-train/spartan-v5 Sep 7, 2026
25 checks passed

spalladino deleted the spl/a-25344-fee-slot-floor branch September 7, 2026 17:10

Navigation Menu

Uh oh!

fix: price public simulation at a slot L1 has not already taken - #25357

fix: price public simulation at a slot L1 has not already taken#25357
spalladino merged 1 commit into
merge-train/spartan-v5from
spl/a-25344-fee-slot-floor

spalladino commented Aug 29, 2026 •

edited

Loading

Uh oh!

alexghr Sep 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

spalladino commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The symptom

Where the fee number comes from

The bug: the simulator could target a slot that was already taken

The fix

What this does not fix

Tests

Stack

Uh oh!

alexghr Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

spalladino commented Aug 29, 2026 •

edited

Loading