feat(fast-inbox): make the inbox message rollback content-aware

docs/docs-operate/operators/reference/changelog/v6.md

@@ -16,7 +16,7 @@ displayed_sidebar: operatorsSidebar
 
 ### Choosing a value
 
-`0` consumes a message as soon as your archiver sees it, so a bridged message reaches L2 in the next block your node builds. The cost is that an L1 reorg touching an L1 block whose messages your node has already consumed costs your node that slot, whether or not the messages themselves changed: the archiver rewinds the messages from that block on and drops the blocks built on them, and the checkpoint is abandoned. Your node also re-resolves the bucket it consumed by its contents before publishing, but that only prevents a false rejection when a reorg elsewhere renumbers buckets without touching their messages. At ten one-block L1 reorgs a day, with messages present in 30% of L1 blocks, that puts an upper bound of roughly 0.13% on the slots your node loses — about three a day across the whole network. Nothing else is affected: the chain carries on and the message is consumed by the next proposer.
+`0` consumes a message as soon as your archiver sees it, so a bridged message reaches L2 in the next block your node builds. The cost is that an L1 reorg which changes the messages your node has already consumed costs your node that slot: the archiver drops the messages from the first one whose contents changed, along with the blocks built on them, and the checkpoint is abandoned. A reorg that re-mines the same messages in a different L1 block costs nothing — the archiver replaces only the bucket metadata derived from that block, the blocks that consumed the messages stand, and your node re-resolves the bucket it consumed by its contents before publishing. The exception is a re-mine that lands two buckets' messages in a single L1 block: the boundary between them stops existing, and the block that consumed through it goes, along with the blocks after it. At ten one-block L1 reorgs a day, with messages present in 30% of L1 blocks, that puts an upper bound of roughly 0.13% on the slots your node loses — about three a day across the whole network, and only for the reorgs that reorder a message, drop one, or merge two buckets into one. Nothing else is affected: the chain carries on and the message is consumed by the next proposer.
 
 `1` consumes a message only once a child block is observed on top of the one carrying it, or the following L1 slot was missed and the block is still canonical after it. That removes the lost slots above and adds roughly one Ethereum slot (~12s) of latency to every bridged message.
 

yarn-project/archiver/src/modules/data_store_updater.ts

@@ -24,6 +24,7 @@ import type { UInt64 } from '@aztec/stdlib/types';
 
 import type { ArchiverDataStores } from '../store/data_stores.js';
 import type { L2TipsCache } from '../store/l2_tips_cache.js';
+import type { InboxMessageReplacement } from '../store/message_store.js';
 
 /** Operation type for contract data updates. */
 enum Operation {
@@ -291,39 +292,71 @@ export class ArchiverDataStoreUpdater {
    *
    * @returns The pruned blocks.
    */
-  public async rewindMessagesAndPruneProposedBlocks(
+  public rewindMessagesAndPruneProposedBlocks(
     messagesSyncPoint: L1BlockId,
     firstRemovedIndex: bigint,
+  ): Promise<L2Block[]> {
+    return this.writeMessagesAndPruneProposedBlocks(
+      () => this.stores.messages.rewindMessagesTo(messagesSyncPoint, firstRemovedIndex),
+      firstRemovedIndex,
+    );
+  }
+
+  /**
+   * Replaces the messages above the last canonical Inbox bucket with the ones the canonical chain delivers, and
+   * prunes the proposed blocks that consumed a leaf the swap changed or ended on a bucket boundary it took away, in
+   * a single transaction.
+   *
+   * Splitting the two would make the prune unrecoverable, for the same reason
+   * {@link rewindMessagesAndPruneProposedBlocks} keeps them together: the next sync pass would find the local
+   * messages consistent with L1 and never come back to the blocks built on the leaves that are gone.
+   *
+   * @returns The pruned blocks.
+   */
+  public replaceMessagesAndPruneProposedBlocks(args: InboxMessageReplacement): Promise<L2Block[]> {
+    return this.writeMessagesAndPruneProposedBlocks(
+      () => this.stores.messages.replaceMessagesAboveBucket(args),
+      args.firstInvalidatedIndex,
+    );
+  }
+
+  /**
+   * Commits a write to the message store and the prune of the proposed blocks that consumed a message from
+   * `firstInvalidatedIndex` on, refreshing the tips cache only once both have landed.
+   */
+  private async writeMessagesAndPruneProposedBlocks(
+    writeMessages: () => Promise<void>,
+    firstInvalidatedIndex: bigint | undefined,
   ): Promise<L2Block[]> {
     const prunedBlocks = await this.stores.db.transactionAsync(async () => {
-      await this.stores.messages.rewindMessagesTo(messagesSyncPoint, firstRemovedIndex);
-      return await this.removeProposedBlocksConsumingMessagesFrom(firstRemovedIndex);
+      await writeMessages();
+      return firstInvalidatedIndex === undefined
+        ? []
+        : await this.removeProposedBlocksConsumingMessagesFrom(firstInvalidatedIndex);
     });
     await this.l2TipsCache?.refresh();
     return prunedBlocks;
   }
 
   /**
    * Removes the proposed (not yet L1-checkpointed) blocks that consumed an L1-to-L2 message at or after
-   * `firstRemovedIndex`, along with every block after them. A block's L1-to-L2 tree leaf count is the cumulative
+   * `firstInvalidatedIndex`, along with every block after them. A block's L1-to-L2 tree leaf count is the cumulative
    * count of messages consumed through it, so a leaf count above the index means the block consumed a message the
-   * local chain no longer has, and one equal to it means the block stopped below it.
+   * local chain no longer backs, and one equal to it means the block stopped below it.
    *
-   * The index is where the rollback rewound to, not the first leaf whose value actually changed, so a reorg that
-   * re-mines the same messages in a different L1 block also prunes blocks whose trees are unchanged and which L1
-   * would still accept. That over-pruning is accepted for now: the rollback rewinds before it knows what the
-   * canonical chain re-delivers, so it cannot tell a re-mine from a content change, and a prune from the first
-   * differing leaf has to wait until the rollback becomes content-aware.
+   * The index is the first leaf whose value actually changed, or whose bucket boundary the reorg took away, so a
+   * reorg that re-mines the same messages in a different L1 block prunes nothing, and one that replaces the tail of
+   * a bucket keeps the blocks that stopped below the replaced leaf.
    *
    * Checkpointed blocks are never touched: a message store that disagrees with a checkpoint L1 accepted means one
    * of the two views of L1 is mid-reorg and this one is not necessarily the right one, so only the archive
    * comparison in the checkpoint step may unwind published state. For the same reason nothing is pruned when the
    * rollback reaches below the checkpointed tip's leaf count, where every proposed descendant would qualify.
    *
-   * @param firstRemovedIndex - Index of the first L1-to-L2 message that was removed from the store.
+   * @param firstInvalidatedIndex - Index of the first L1-to-L2 message the local chain no longer backs.
    * @returns The removed blocks.
    */
-  private async removeProposedBlocksConsumingMessagesFrom(firstRemovedIndex: bigint): Promise<L2Block[]> {
+  private async removeProposedBlocksConsumingMessagesFrom(firstInvalidatedIndex: bigint): Promise<L2Block[]> {
     const [checkpointedBlockNumber, latestBlockNumber] = await Promise.all([
       this.stores.blocks.getCheckpointedL2BlockNumber(),
       this.stores.blocks.getLatestL2BlockNumber(),
@@ -339,10 +372,10 @@ export class ArchiverDataStoreUpdater {
     const checkpointedTipLeafCount = BigInt(
       checkpointedTip?.header.state.l1ToL2MessageTree.nextAvailableLeafIndex ?? 0,
     );
-    if (firstRemovedIndex < checkpointedTipLeafCount) {
+    if (firstInvalidatedIndex < checkpointedTipLeafCount) {
       this.log.warn(
-        `Not pruning proposed blocks: rollback index ${firstRemovedIndex} is below the checkpointed tip's leaf count`,
-        { firstRemovedIndex, checkpointedTipLeafCount, checkpointedBlockNumber },
+        `Not pruning proposed blocks: rollback index ${firstInvalidatedIndex} is below the checkpointed tip's leaf count`,
+        { firstInvalidatedIndex, checkpointedTipLeafCount, checkpointedBlockNumber },
       );
       return [];
     }
@@ -352,7 +385,7 @@ export class ArchiverDataStoreUpdater {
       limit: latestBlockNumber - checkpointedBlockNumber,
     });
     const firstAffected = proposedBlocks.find(
-      block => BigInt(block.header.state.l1ToL2MessageTree.nextAvailableLeafIndex) > firstRemovedIndex,
+      block => BigInt(block.header.state.l1ToL2MessageTree.nextAvailableLeafIndex) > firstInvalidatedIndex,
     );
     if (firstAffected === undefined) {
       return [];