Skip to content

fix(ann): retire dormant consumer registrations - #1626

Draft
ohdearquant wants to merge 1 commit into
mainfrom
codex/ann-dormant-consumer-compaction
Draft

fix(ann): retire dormant consumer registrations#1626
ohdearquant wants to merge 1 commit into
mainfrom
codex/ann-dormant-consumer-compaction

Conversation

@ohdearquant

Copy link
Copy Markdown
Owner

Summary

  • give first-use ANN consumers a bounded pending lifecycle so abandoned registrations can no longer pin ann_write_log forever
  • fence watermark publication by consumer state, retire only expired never-activated rows during compaction, and preserve active or recovering consumers regardless of age
  • apply the shared registry contract to memory and knowledge ANN paths, including safe in-memory publication and transaction-compatible lifecycle helpers

Correctness details

  • register new consumers at a closed pending watermark and allow only an authoritative first full checkpoint to activate them
  • condition watermark raises on the caller's authority and current durable state so retirement, recovery, and competing checkpoints cannot be overwritten
  • retire pending rows only after a 24-hour grace period and only inside the same compaction unit that derives the remaining consumer floor
  • make full in-memory scans inherit the consumer's compacted active floor, wait for an in-flight pathless checkpoint when a captured bridge is pending, and evict candidates after registration loss
  • keep pathless lifecycle mutations single-statement so callers already holding the writer transaction cannot open a nested transaction
  • serialize file-backed publication under the segment lock, reject stale publishers before persistence, and preserve the protected incumbent when replacement persistence fails
  • add the pending-state migration plus regression coverage for retirement, compaction floors, nested transactions, registration loss, publication races, and persistence failures

Validation

  • cargo test --manifest-path crates/Cargo.toml --workspace
  • cargo check --manifest-path crates/Cargo.toml --workspace
  • cargo clippy --manifest-path crates/Cargo.toml --workspace --all-targets -- -D warnings
  • cargo fmt --manifest-path crates/Cargo.toml --all -- --check
  • RUSTDOCFLAGS=-Dwarnings cargo doc --manifest-path crates/Cargo.toml --workspace --no-deps

Closes #1479

AI-assisted contribution: Codex prepared this change and PR description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prevent dormant ANN consumers from blocking write-log compaction

1 participant