Skip to content

perf(reindex): reduce canonical embedding overhead - #1632

Draft
ohdearquant wants to merge 1 commit into
mainfrom
codex/reindex-allocation-hygiene
Draft

perf(reindex): reduce canonical embedding overhead#1632
ohdearquant wants to merge 1 commit into
mainfrom
codex/reindex-allocation-hygiene

Conversation

@ohdearquant

Copy link
Copy Markdown
Owner

AI-assisted contribution: Codex prepared this change and PR description.

Summary

  • avoid constructing canonical embedding strings during FTS-only reindex runs while preserving embeddable-record accounting
  • keep the public owned note_embedding_text API compatible, add an internal borrowed form, and share one Arc<str> across multi-model spawned embedding tasks instead of cloning content per model
  • make same-identity vector replacement use an identity-constrained delete fast path, reserving the delete-log identity scan for actual metadata repair
  • add parity, allocation/borrowing, same-identity statement-count, mismatch-repair, and rollback coverage

Closes #1338.

Test plan

  • cargo test --manifest-path crates/Cargo.toml --workspace
  • cargo check --manifest-path crates/Cargo.toml --workspace
  • cargo clippy --manifest-path crates/Cargo.toml --workspace --all-targets -- -D warnings
  • cargo fmt --manifest-path crates/Cargo.toml --all -- --check
  • RUSTDOCFLAGS=-Dwarnings cargo doc --manifest-path crates/Cargo.toml --workspace --no-deps

All commands passed at 83f827aeaff2ec4556ce340bb9e0976bc8835eb3 on current main (a152ead3e577b8c01b03024687fdb66d94a00cac).

ADR

n/a — this removes avoidable allocation and probe work without changing the embedding text, vector identity, transaction, or public API contracts. The vector API documentation is updated to describe the optimized DML path.

AI-assisted contribution checklist

  • Every claim in this PR description matches the actual diff
  • Any agent-authored comment / PR body starts with an attribution line
  • Full workspace behavior, lint, formatting, and documentation gates passed
  • Existing public helper signatures and vector rollback/log ordering remain covered

Out of scope

  • changing canonical embedding content or truncation behavior
  • changing FTS document shape or reindex result accounting
  • changing vector primary keys, ANN identity, or transaction boundaries

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(reindex): minor allocation and probe overhead in the canonical embedding path

1 participant