ADR-0022: meta free-list annex — per-commit free-list save into the meta page (format v2)

PROGRESS.md

@@ -153,3 +153,5 @@ ADR-0019 PARKED, 2026-10-01 — durable meta write via an O_DSYNC fd (2 → 1 fd
 ADR-0020 SPIKE, 2026-10-01 — 24-byte page-header spike approved for measurement only; the format change is not approved and awaits the spike's numbers. Branch `qdequele/zerodb-header24-spike`.
 
 ADR-0021 PRODUCTION PASS, 2026-10-02 — true in-place WRITE_MAP hardened on the spike branch (not merged): B1 unsafe-fn broker with the sole sanctioned call in zerodb-core::dirty; B2 superseded by a stronger M1 miri finding (under Stacked Borrows a cached whole-map reference is UB after any in-place write — the in-map writer now borrows its map view lazily per access, like readers; spill-time re-derivation kept for the heap-staged paths); B3 map-aware fault backend (brokered regions journaled, sealed per sync, image cuts open the real writable map; vacuousness tripwire + smoke pin); B4 loom justified as no-new-model (ADR-0021 §hardening), 180 s stress writemap-in-place variant, M1.13 erased-cursor audit + miri battery in heed-zerodb; M1 zerodb_io::testmap::TestWriteMap (test-backing feature) + writemap_in_place_miri battery; M2 release-mode TXN-62 typed guard in allocate; M3 abort-after-spill / nested-fanout / put_reserved batteries parameterized over WRITE_MAP. SPEC 04 TXN-71/C5a/TXN-45b and SPEC 06 REC-20 amended in the same change.
+
+ADR-0022 SPIKE, 2026-10-03 — meta free-list annex (format v2): the per-commit freed PIL rides inside the CRC-covered meta page; the GC tree is the spill/cold path (over-cap sets, reader-gated carries). Steady-state freelist_save drops from ~2 GC-tree puts + 1 delete + 1 leaf COW per commit to zero tree ops (~1.2–1.5 µs → ~0.1 µs local 4 KiB census; one fewer dirty page per commit), and churn steady-state files shrink ~12% vs v1. First-cut lesson: folding the carried annex into `freed` up front starved the save's own allocations into per-commit extends (unbounded ratchet, caught by churn_parity_general); the carried remainder must stay a gated in-save pool (GC-12 source 4) merged at placement. Pinned crash seed re-pinned (198) per its documented protocol. Implemented on coordinator-relayed approval; ratified directly 2026-10-05 (format v2 + seed 198), re-gated on main after ADR-0021 merged (test 653/0, miri 206/0, loom 9/0, stress 3/0, crash-test-quick clean, fuzz-quick clean).

README.md

@@ -60,11 +60,33 @@ bytes. Full list: [`docs/COMPATIBILITY.md`](docs/COMPATIBILITY.md).
 
 ## Performance
 
-On its real consumers, ZeroDB runs at **LMDB-level performance** (Meilisearch
-indexing 1.00×, search 1.03×; hannoy search 0.95×). Against the field — the
+On its real consumers, ZeroDB runs at **LMDB-level performance**: Meilisearch
+indexing 1.01× (movies) and 0.99× (incremental hackernews additions), search
+0.94× — time relative to LMDB, lower is better; hannoy search 0.95×.
+
+**ZeroDB vs LMDB, current tree** — YCSB on a Graviton4 with local NVMe,
+10 M × 128 B, kops/s (higher is better). `WRITE_MAP` is opt-in on both engines
+(LMDB's `MDB_WRITEMAP`, ZeroDB's in-place `WRITE_MAP`):
+
+| Workload | LMDB | zerodb | LMDB `WRITE_MAP` | zerodb `WRITE_MAP` |
+|---|---:|---:|---:|---:|
+| YCSB A (no-sync, 2 GiB cap) | 256 | 211 | 545 | 408 |
+| YCSB B (no-sync, 2 GiB cap) | 490 | 355 | 1,199 | 847 |
+| YCSB B (fsync) | 238 | **258** | — | — |
+
+Durable writes are ahead of LMDB (1.08×); no-sync writes trail it by 18–27%, and
+`WRITE_MAP` makes ZeroDB 1.6–1.7× faster than default LMDB, still behind LMDB's
+own `WRITE_MAP`. Details:
+[`benches/results/2026-10-05-meta-annex-real-case.md`](benches/results/2026-10-05-meta-annex-real-case.md).
+_(2026-10-05, 2 reps.)_
+
+**Against the field** — the
 [rust-storage-bench](https://github.com/marvin-j97/rust-storage-bench) suite
 behind the *fjall 3* article, on a Graviton4 with local NVMe — throughput
-in kops/s (higher is better; per-row winner in **bold**):
+in kops/s (higher is better; per-row winner in **bold**). This run predates
+in-place `WRITE_MAP` and the meta free-list annex, and uses a different setup
+(8 GiB cap, 2 GiB cache, 100 B values), so its numbers don't compare with the
+table above:
 
 | Workload | LMDB | zerodb | fjall 3 | rocksdb | redb | sqlite |
 |---|---:|---:|---:|---:|---:|---:|
@@ -74,7 +96,7 @@ in kops/s (higher is better; per-row winner in **bold**):
 | feed | **55** | 42 | 37 | 31 | 16 | 38 |
 | 100 M keys | **93** | 79 | 76 | 65 | 15 | 36 |
 
-ZeroDB tracks LMDB closely — at parity on durable writes (YCSB B), a little ahead
+In that run ZeroDB tracked LMDB closely — at parity on durable writes (YCSB B), a little ahead
 on 4 KB values, behind on the write-heavy no-sync mix (its weakest path). The LSM
 engines (fjall, rocksdb) take the raw write-throughput rows; the B-trees (LMDB and
 ZeroDB) keep read p99 in microseconds where the LSMs run to hundreds. Per-engine

benches/results/2026-10-05-meta-annex-real-case.md

@@ -0,0 +1,109 @@
+# Meta free-list annex (ADR-0022) — real-case three-column check — 2026-10-05
+
+The commit-ladder A/B for ADR-0022 only showed a win on tiny no-sync commits
+(`commit/batch/n1` −22%; `n10k` and the fsync rungs flat). This run checks it on
+real workloads: YCSB through rust-storage-bench, and Meilisearch's own
+`cargo xtask bench` workloads.
+
+Three columns throughout:
+
+| column | tree |
+|---|---|
+| **LMDB** | heed 0.22.1 (the Meilisearch LMDB fork) — drift control |
+| **ZeroDB before** | `8b066a6` — main with ADR-0021 (in-place `WRITE_MAP`), no annex |
+| **ZeroDB after** | `b01b75b` — `8b066a6` + ADR-0022 (the annex), nothing else |
+
+`*-wm` rows open the environment with `WRITE_MAP` (LMDB `MDB_WRITEMAP`, ZeroDB
+in-place `WRITE_MAP`).
+
+## YCSB — Graviton4 NVMe (the target hardware)
+
+AWS `m8gd.xlarge` (Graviton4, 4 vCPU, 16 GiB), Ubuntu 24.04 aarch64, local NVMe
+instance store, ext4 `noatime`. rust-storage-bench (with the `zerodb` backend),
+10 M items × 128 B, json corpus, 8 MiB engine cache, 60 s per run; no-sync runs in
+a `MemoryMax=2G` scope, durable runs uncapped. Fresh data dir and dropped page
+cache before every run. 2 reps, the second in reverse order.
+
+### YCSB A (50/50 read/update), no-sync
+
+| config | ops/s (reps) | median | ÷ LMDB | write p50 | write p99 |
+|---|---|---:|---:|---:|---:|
+| LMDB | 255k / 257k | 256k | 1.00 | 6.4 µs | 13.9 µs |
+| ZeroDB before | 207k / 207k | 207k | 0.81 | 14.0 µs | 27.7 µs |
+| **ZeroDB after** | 213k / 209k | 211k | 0.82 | **12.0 µs** | **24.8 µs** |
+| LMDB-wm | 545k / 545k | 545k | 2.13 | 1.3 µs | 4.1 µs |
+| ZeroDB-wm before | 421k / 404k | 412k | 1.61 | 4.4 µs | 9.3 µs |
+| **ZeroDB-wm after** | 412k / 405k | 408k | 1.60 | **3.3 µs** | **7.8 µs** |
+
+after ÷ before: default **1.018** (rep range 1.008–1.028); wm 0.991 (0.963–1.020).
+
+### YCSB B (95/5 read/update), no-sync
+
+| config | ops/s (reps) | median | ÷ LMDB | write p50 | write p99 |
+|---|---|---:|---:|---:|---:|
+| LMDB | 492k / 487k | 490k | 1.00 | 7.1 µs | 15.4 µs |
+| ZeroDB before | 368k / 358k | 363k | 0.74 | 15.1 µs | 31.9 µs |
+| **ZeroDB after** | 372k / 339k | 355k | 0.73 | **13.1 µs** | 54.0 µs ¹ |
+| LMDB-wm | 1209k / 1190k | 1199k | 2.45 | 1.4 µs | 6.4 µs |
+| ZeroDB-wm before | 819k / 810k | 814k | 1.66 | 4.6 µs | 11.6 µs |
+| **ZeroDB-wm after** | 845k / 848k | 847k | 1.73 | **3.5 µs** | **10.3 µs** |
+
+after ÷ before: default 0.980 (0.922–1.040, noisy); wm **1.040** (1.033–1.047).
+
+¹ One rep only: rep 1 was 29.4 µs (better than before's 31.9 µs); rep 2 was
+78.5 µs during a run that read 382 MiB from disk vs ~210 MiB for the others — a
+page-cache eviction episode under the 2 GiB cap, not the annex.
+
+### YCSB B, durable (every commit fsynced)
+
+| config | ops/s (reps) | median | ÷ LMDB | write p50 | write p99 |
+|---|---|---:|---:|---:|---:|
+| LMDB | 238k / 239k | 238k | 1.00 | 167.8 µs | 178.1 µs |
+| ZeroDB before | 238k / 241k | 239k | 1.00 | 161.2 µs | 178.1 µs |
+| **ZeroDB after** | 258k / 258k | **258k** | **1.08** | **129.4 µs** | **148.8 µs** |
+
+after ÷ before: **1.078** (rep range 1.070–1.085). One page fewer written and
+flushed per commit shows up directly when the flush is cheap (NVMe).
+
+## YCSB — x86-64 bench server (confirmation)
+
+Xeon E3-1230 v2 (4C/8T, turbo off, performance governor), 2× SATA SSD in
+mdraid RAID 1, Debian 12. Same method, 45 s per run, 2 reps (durable: 1 rep).
+
+| workload | before → after ops/s | after ÷ before | write p50 before → after |
+|---|---|---:|---|
+| A no-sync | 254k → 261k | **1.027** (1.019–1.034) | 17.2 → 13.8 µs |
+| A no-sync, wm | 336k → 357k | **1.065** (1.022–1.110) | 8.7 → 8.0 µs |
+| B no-sync | 421k → 448k | **1.064** (1.046–1.083) | 18.2 → 13.9 µs |
+| B no-sync, wm | 661k → 696k | **1.053** (1.052–1.054) | 8.9 → 6.9 µs |
+| B durable | 188k → 189k | 1.006 | 1398 → 1316 µs |
+
+The durable row is flat here: this box's fsync is a ~1.4 ms SATA RAID flush,
+which drowns per-commit CPU (the same reason the ladder's `commit/sync/*` rungs
+measured flat on it).
+
+## Meilisearch — x86-64 bench server
+
+Meilisearch v1.53.1 built three times (stock LMDB, ZeroDB before, ZeroDB after),
+`cargo xtask bench` on `movies.json`, `hackernews-add-new-documents.json` and
+`search/movies.json`, 3 rounds with the starting engine rotated each round.
+Median server-side time (`::meta::total` span) over all runs:
+
+| workload | LMDB | ZeroDB before | ZeroDB after | after ÷ before | after ÷ LMDB |
+|---|---:|---:|---:|---:|---:|
+| movies indexing (30 runs) | 4.850 s | 4.865 s | 4.879 s | 1.00 | 1.01 |
+| hackernews incremental additions (9 runs) | 40.25 s | 40.74 s | 40.01 s | 0.98 | 0.99 |
+| — of which `indexing::scheduler::commit` | 11.03 s | 10.86 s | 10.21 s | **0.94** | 0.93 |
+| movies search (30 runs) | 15.4 ms | 14.8 ms | 14.4 ms | 0.97 | 0.94 |
+
+No regression. Bulk indexing is flat as expected (few, large commits); the
+incremental-additions workload, which commits more often, gains 6% on its commit
+span.
+
+## Verdict
+
+ADR-0022 is a real-case win where commits are frequent or durable on fast
+storage, and neutral for Meilisearch bulk indexing: YCSB +2–8% throughput, write
+p50 −14% to −25% in every configuration on both machines, no regression in any
+Meilisearch workload. It is not a Meilisearch indexing speed-up and should not be
+described as one.

crates/zerodb-core/src/builder.rs

@@ -355,6 +355,7 @@ fn finalize_image(
         last_pg,
         free_db: DBRecord::empty(),
         main_db,
+        fl_count: 0,
     };
     m0.encode(&mut buf[0..psize as usize])?;
     let mut m1 = m0;
@@ -894,6 +895,7 @@ impl<S: PageSink> EnvStream<S> {
             last_pg,
             free_db: DBRecord::empty(),
             main_db,
+            fl_count: 0,
         };
         meta.encode(&mut frame)?;
         sink.emit(0, &frame)?;

crates/zerodb-core/src/check.rs

@@ -39,9 +39,13 @@ struct Checker<'a> {
     /// otherwise choose ids that all collide (HashDoS). The engine's
     /// dirty-store hasher is unaffected — its keys are engine-authored.
     visited: HashSet<u64>,
-    /// Free page ids collected from every GC PIL → occurrence count
-    /// (INV-22/INV-24; SPEC 05 §9). Randomly seeded, see `visited`.
+    /// Free page ids collected from every GC PIL **and the selected meta's
+    /// free-list annex** → occurrence count (INV-22/INV-24/INV-28; SPEC 05
+    /// §9/§2a). Randomly seeded, see `visited`.
     free: HashMap<u64, u64>,
+    /// The selected meta's annex id count (INV-28/GC-29: `> 0` forbids a GC
+    /// tree entry keyed `BE(meta_txnid)`).
+    annex_count: usize,
     violations: Vec<String>,
 }
 
@@ -353,6 +357,15 @@ impl<'a> Checker<'a> {
         if txnid > self.meta_txnid {
             self.fail("INV-23", format!("GC entry keyed by future txn {txnid}"));
         }
+        // INV-28/GC-29 (format v2): a meta with a non-empty annex holds the
+        // newest freeing-txn's PIL itself — a tree entry under the same
+        // txnid would be the forbidden split placement (GC-32).
+        if txnid == self.meta_txnid && self.annex_count > 0 {
+            self.fail(
+                "INV-28",
+                format!("GC entry {txnid} coexists with a non-empty meta annex (GC-29/GC-32)"),
+            );
+        }
         let pil: Vec<u8> = match val {
             LeafValue::Inline(v) => v.to_vec(),
             LeafValue::Overflow { head_pgno, dsize } => {
@@ -522,9 +535,9 @@ pub fn check_image(bytes: &[u8], psize: u32) -> Vec<String> {
         (Ok(a), Ok(b)) => (a, b),
         _ => return vec!["INV-1: meta slots undecodable".into()],
     };
-    let meta = match select_meta(&v0, &v1, false) {
-        crate::page::MetaChoice::Both { meta, .. }
-        | crate::page::MetaChoice::OnlyOne { meta, .. } => meta,
+    let (meta, chosen) = match select_meta(&v0, &v1, false) {
+        crate::page::MetaChoice::Both { meta, chosen }
+        | crate::page::MetaChoice::OnlyOne { meta, chosen } => (meta, chosen),
         crate::page::MetaChoice::None => {
             return vec!["INV-2: no valid meta slot (both torn/foreign)".into()]
         }
@@ -557,13 +570,34 @@ pub fn check_image(bytes: &[u8], psize: u32) -> Vec<String> {
             return violations;
         }
     }
+    // Format v2 (ADR-0022): the selected meta's free-list annex ids are free
+    // pages (SPEC 05 §2a). Validate their GC-29 shape (the INV-25/26
+    // analogues, labelled INV-28) and count them into the free set for the
+    // INV-22/24 partition.
+    let annex = MetaPage::read_annex(&bytes[chosen * ps..(chosen + 1) * ps], psize)
+        .expect("annex count was bounds-checked by MetaPage::validate");
+    let mut free: HashMap<u64, u64> = HashMap::new();
+    let mut prev: Option<u64> = None;
+    for &id in &annex {
+        if id < FIRST_DATA_PGNO || id > meta.last_pg {
+            violations.push(format!("INV-28: annex free id {id} out of range"));
+        }
+        if let Some(p) = prev {
+            if p >= id {
+                violations.push("INV-28: annex ids not strictly ascending".into());
+            }
+        }
+        prev = Some(id);
+        *free.entry(id).or_insert(0) += 1;
+    }
     let mut checker = Checker {
         bytes,
         psize,
         meta_txnid: meta.txnid,
         last_pg: meta.last_pg,
         visited: HashSet::new(),
-        free: HashMap::new(),
+        free,
+        annex_count: annex.len(),
         violations,
     };
     checker.check_catalog_record("main_db", &meta.main_db);
@@ -607,8 +641,10 @@ mod tests {
             let base = slot * PS as usize;
             let e_off = base + 120 + 32; // main_db record + entries offset
             img[e_off] = 99;
-            let crc = crate::page::crc32c(&img[base..base + 168]);
-            img[base + 168..base + 172].copy_from_slice(&crc.to_le_bytes());
+            // Format v2 CRC: fl_count (offset 168) is 0 in builder images,
+            // so the coverage is exactly [0, 172); the CRC field sits at 172.
+            let crc = crate::page::crc32c(&img[base..base + 172]);
+            img[base + 172..base + 176].copy_from_slice(&crc.to_le_bytes());
         }
         let v = check_image(&img, PS);
         assert!(

crates/zerodb-core/src/env.rs

@@ -189,7 +189,7 @@ pub trait Backing: Send + Sync {
 /// state (SPEC 04 TXN-18). Readers `Arc`-clone the env's published snapshot at
 /// begin and never re-read a durable meta page (the slot a pinned txnid lived
 /// in is overwritten two commits later, TXN-63).
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+#[derive(Debug, Clone, PartialEq, Eq)]
 pub struct Snapshot {
     /// The commit point of this snapshot.
     pub txnid: u64,
@@ -199,17 +199,28 @@ pub struct Snapshot {
     pub main_db: DBRecord,
     /// Root/stats of the free (GC) DB.
     pub free_db: DBRecord,
+    /// This snapshot's meta free-list annex (SPEC 02 §3 format v2, ADR-0022;
+    /// SPEC 05 §2a): the pages freed by txn `txnid`, exactly the `fl_ids` of
+    /// its meta. Pinned **in the snapshot** — the meta slot `txnid & 1` is
+    /// overwritten by txn `txnid + 2` even while this snapshot stays pinned
+    /// (TXN-63), so holders (`copy`, the next writer) must never re-read the
+    /// slot. `Arc<[u64]>` keeps `Snapshot` cheap to clone.
+    pub free_annex: std::sync::Arc<[u64]>,
 }
 
 impl Snapshot {
-    /// The snapshot a validated meta page describes.
+    /// The snapshot a validated meta page describes, with `annex` as the
+    /// meta's free-list annex ids (read via [`MetaPage::read_annex`] from the
+    /// same validated slot buffer; `annex.len()` must equal `meta.fl_count`).
     #[must_use]
-    pub fn from_meta(meta: &MetaPage) -> Snapshot {
+    pub fn from_meta(meta: &MetaPage, annex: Vec<u64>) -> Snapshot {
+        debug_assert_eq!(annex.len(), meta.fl_count as usize);
         Snapshot {
             txnid: meta.txnid,
             last_pg: meta.last_pg,
             main_db: meta.main_db,
             free_db: meta.free_db,
+            free_annex: annex.into(),
         }
     }
 }
@@ -1635,20 +1646,27 @@ pub fn open_with_backing_policy(
     let slot1 = read_slot(bytes, META_B_PGNO, ps, page_size)?;
 
     // Select the live snapshot (SPEC 02 §3.2 / SPEC 06 REC-2..5).
-    let meta = match select_meta(&slot0, &slot1, prev_snapshot) {
-        MetaChoice::Both { meta, .. } => meta,
-        MetaChoice::OnlyOne { meta, .. } => {
+    let (meta, chosen_slot) = match select_meta(&slot0, &slot1, prev_snapshot) {
+        MetaChoice::Both { meta, chosen } => (meta, chosen),
+        MetaChoice::OnlyOne { meta, chosen } => {
             if prev_snapshot {
                 // REC-2† (ratified 2026-07-16): one valid slot + PREV_SNAPSHOT is
                 // a hard error — there are not two committed snapshots to pick an
                 // older from.
                 return Err(Error::Mdb(MdbError::Invalid));
             }
-            meta
+            (meta, chosen)
         }
         // REC-3: both invalid → unrecoverable.
         MetaChoice::None => return Err(Error::Mdb(MdbError::Invalid)),
     };
+    // Format v2 (ADR-0022): read the selected slot's free-list annex ids —
+    // the one and only meta read, alongside the roots (TXN-18); the slot is
+    // overwritten two commits later, so the ids are pinned in the Snapshot.
+    // `fl_count` passed the §3.2 rule-5 bound + CRC; the ids' GC-29 shape is
+    // validated by the consumer before any id is handed out (GC-33).
+    let annex = MetaPage::read_annex(&bytes[chosen_slot * ps..(chosen_slot + 1) * ps], page_size)
+        .ok_or(Error::Mdb(MdbError::Invalid))?;
 
     // SPEC 06 REC-1a / SPEC 02 §3.2 step 6 (geometry validation): a slot can
     // carry a valid CRC and still name geometry the real file cannot back — a
@@ -1699,7 +1717,7 @@ pub fn open_with_backing_policy(
         map_size,
         // Seed the published-snapshot cell from the durable meta — the one
         // and only time a meta *page* is read for roots (SPEC 04 TXN-18).
-        snap_cell: SnapshotCell::new(Arc::new(Snapshot::from_meta(&meta))),
+        snap_cell: SnapshotCell::new(Arc::new(Snapshot::from_meta(&meta, annex))),
         write_mutex: WriterLock::new(),
         commit_hook: Mutex::new(None),
         poisoned: AtomicBool::new(false),

crates/zerodb-core/src/nested.rs

@@ -171,6 +171,9 @@ impl TxnRead for NestedRoTxn<'_> {
     fn free_record(&self) -> &DBRecord {
         self.parent.free_record()
     }
+    fn free_annex_count(&self) -> u64 {
+        self.parent.free_annex_count()
+    }
     fn page_size(&self) -> u32 {
         self.parent.page_size()
     }