Skip to content

Add DBOptions::partition_wal_usage for per-record WAL ordering numbers (wal_index) - #15151

Closed
JustinGaoF wants to merge 1 commit into
facebook:mainfrom
JustinGaoF:export-D117410129
Closed

Add DBOptions::partition_wal_usage for per-record WAL ordering numbers (wal_index)#15151
JustinGaoF wants to merge 1 commit into
facebook:mainfrom
JustinGaoF:export-D117410129

Conversation

@JustinGaoF

Copy link
Copy Markdown

Summary:
Adds a new DBOptions::partition_wal_usage (PartitionWALUsage enum) that
controls whether each logical WAL record carries a monotonically increasing
ordering number (wal_index / LSN) persisted on disk.

  • kNone (default) keeps the WAL bit-for-bit compatible with the previous
    format for downgrade safety.
  • kPartitionByWALIndexHash enables the feature: log::Writer prepends a
    fixed64 wal_index to every logical record (starting at 1, incremented per
    AddRecord), and emits a leading marker record (kWALIndexMarkerType /
    kRecyclableWALIndexMarkerType, magic WAL_LSN_V1) that identifies the file
    as carrying wal_index. The marker type is < 128 without the safe-ignore
    bit, so older binaries report corruption instead of misreading the file.

On the read side, log::Reader detects the marker, strips the wal_index from
each logical record so upper layers observe the original payload, recomputes the
per-record checksum over the stripped payload, and reports a corruption (with a
sticky gap flag) when the observed wal_index is not the expected successor. The
index survives block fragmentation, recyclable headers, and WAL compression.
The marker and file-level meta records do not consume a wal_index.

The option is parseable from an options string (both partition_wal_usage and
Partition_WAL_Usage, values kNone/None and
kPartitionByWALIndexHash/partition_by_wal_index_hash), and is wired into
db_bench, db_stress, and db_crashtest.py.

Differential Revision: D117410129

…s (wal_index)

Summary:
Adds a new `DBOptions::partition_wal_usage` (`PartitionWALUsage` enum) that
controls whether each logical WAL record carries a monotonically increasing
ordering number (wal_index / LSN) persisted on disk.

- `kNone` (default) keeps the WAL bit-for-bit compatible with the previous
  format for downgrade safety.
- `kPartitionByWALIndexHash` enables the feature: `log::Writer` prepends a
  fixed64 wal_index to every logical record (starting at 1, incremented per
  `AddRecord`), and emits a leading marker record (`kWALIndexMarkerType` /
  `kRecyclableWALIndexMarkerType`, magic `WAL_LSN_V1`) that identifies the file
  as carrying wal_index. The marker type is `< 128` without the safe-ignore
  bit, so older binaries report corruption instead of misreading the file.

On the read side, `log::Reader` detects the marker, strips the wal_index from
each logical record so upper layers observe the original payload, recomputes the
per-record checksum over the stripped payload, and reports a corruption (with a
sticky gap flag) when the observed wal_index is not the expected successor. The
index survives block fragmentation, recyclable headers, and WAL compression.
The marker and file-level meta records do not consume a wal_index.

The option is parseable from an options string (both `partition_wal_usage` and
`Partition_WAL_Usage`, values `kNone`/`None` and
`kPartitionByWALIndexHash`/`partition_by_wal_index_hash`), and is wired into
`db_bench`, `db_stress`, and `db_crashtest.py`.

Differential Revision: D117410129
@meta-cla meta-cla Bot added the CLA Signed label Aug 25, 2026
@meta-codesync

meta-codesync Bot commented Aug 25, 2026

Copy link
Copy Markdown

@JustinGaoF has exported this pull request. If you are a Meta employee, you can view the originating Diff in D117410129.

@meta-codesync

meta-codesync Bot commented Aug 25, 2026

Copy link
Copy Markdown

@JustinGaoF has imported this pull request. If you are a Meta employee, you can view this in D117410129.

@JustinGaoF JustinGaoF closed this Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant