Switch to using byte offsets as LSNs

[fabubaker]

What changes were proposed in this pull request?

Log Sequence Numbers used in the WAL are currently simple sequential counters.
By switching to using byte offsets into the underlying WAL file, we can allow for simpler and faster recovery.

Why are the changes needed?

Currently, we always cleanup the WAL file on a clean shutdown, so that there is no need to parse through the full WAL file on reboot. This is unnecessary as WAL files only need to be removed when they reach a particular size on disk.

This PR changes clean shutdowns to record the latest LSN in the WAL stream to disk and seek to that position on reboot. This allows for faster reboot without needing to always remove the WAL file on clean shutdowns.

Does this PR introduce any user-facing change? If yes is this documented?

No

How was this patch tested?

A mix of unit tests and proptests, most of which is in https://github.com/Pometry/pometry-storage/pull/223.

Are there any further changes required?

We will need some sort of background thread that goes in and cleans up outdated WAL files.

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite	Current: eeddacb	Previous: 9823ef7	Ratio
lotr_graph/num_edges	5 ns/iter (± 0)	0 ns/iter (± 0)	+∞
lotr_graph/num_nodes	133 ns/iter (± 26)	1 ns/iter (± 0)	133
lotr_graph/graph_latest	4 ns/iter (± 0)	0 ns/iter (± 0)	+∞
lotr_graph_window_100/num_edges	28 ns/iter (± 0)	8 ns/iter (± 0)	3.50
lotr_graph_window_100/num_nodes	143 ns/iter (± 26)	5 ns/iter (± 0)	28.60
lotr_graph_window_10/has_node_existing	130 ns/iter (± 8)	62 ns/iter (± 11)	2.10
lotr_graph_window_10/iterate nodes	34954 ns/iter (± 67)	11339 ns/iter (± 40)	3.08
lotr_graph_window_10/iterate edges	108328 ns/iter (± 249)	48684 ns/iter (± 211)	2.23
lotr_graph_subgraph_10pc/has_edge_existing	232 ns/iter (± 10)	93 ns/iter (± 1)	2.49
lotr_graph_subgraph_10pc/num_nodes	136 ns/iter (± 24)	4 ns/iter (± 0)	34
lotr_graph_subgraph_10pc/has_node_existing	104 ns/iter (± 1)	34 ns/iter (± 0)	3.06
lotr_graph_subgraph_10pc/iterate nodes	2466 ns/iter (± 123)	839 ns/iter (± 5)	2.94
lotr_graph_subgraph_10pc_windowed/has_node_existing	128 ns/iter (± 9)	62 ns/iter (± 14)	2.06
lotr_graph_subgraph_10pc_windowed/iterate nodes	4503 ns/iter (± 95)	1365 ns/iter (± 3)	3.30
lotr_graph_window_50_layered/num_edges	78270 ns/iter (± 1671)	38714 ns/iter (± 1017)	2.02
lotr_graph_window_50_layered/num_nodes	67009 ns/iter (± 2147)	21435 ns/iter (± 536)	3.13
lotr_graph_window_50_layered/has_node_existing	1001 ns/iter (± 77)	129 ns/iter (± 12)	7.76
lotr_graph_window_50_layered/max_id	73554 ns/iter (± 886)	25556 ns/iter (± 252)	2.88
lotr_graph_window_50_layered/iterate nodes	145169 ns/iter (± 625)	19308 ns/iter (± 47)	7.52
lotr_graph_window_50_layered/iterate edges	214135 ns/iter (± 939)	83616 ns/iter (± 1318)	2.56
lotr_graph_window_50_layered/max_neighbour_degree	773569 ns/iter (± 2582)	386651 ns/iter (± 3853)	2.00
lotr_graph_window_50_layered/graph_latest	111539 ns/iter (± 2167)	36649 ns/iter (± 916)	3.04
lotr_graph_persistent_window_50_layered/num_edges_temporal	593669 ns/iter (± 4712)	192686 ns/iter (± 1569)	3.08
lotr_graph_persistent_window_50_layered/num_nodes	86651 ns/iter (± 2521)	31517 ns/iter (± 779)	2.75
lotr_graph_persistent_window_50_layered/has_node_existing	1164 ns/iter (± 184)	174 ns/iter (± 83)	6.69
lotr_graph_persistent_window_50_layered/max_id	95840 ns/iter (± 2184)	38024 ns/iter (± 490)	2.52
lotr_graph_persistent_window_50_layered/iterate nodes	195950 ns/iter (± 993)	35886 ns/iter (± 191)	5.46
lotr_graph_persistent_window_50_layered/iterate edges	186248 ns/iter (± 781)	84161 ns/iter (± 596)	2.21
lotr_graph_persistent_window_50_layered/graph_latest	253643 ns/iter (± 26808)	57549 ns/iter (± 4809)	4.41
lotr_graph_persistent_window_50_layered_materialise/materialize	11523045 ns/iter (± 59475)	5298035 ns/iter (± 147912)	2.17
lotr_graph/proto_encode	6006571 ns/iter (± 214784)	1157897 ns/iter (± 73709)	5.19

This comment was automatically generated by workflow using github-action-benchmark.