Skip to content

test(vector): assert PQ quantizer is honored on-disk through an Engine commit#809

Merged
mosuka merged 1 commit into
mainfrom
test/798-pq-engine-commit
Jun 15, 2026
Merged

test(vector): assert PQ quantizer is honored on-disk through an Engine commit#809
mosuka merged 1 commit into
mainfrom
test/798-pq-engine-commit

Conversation

@mosuka

@mosuka mosuka commented Jun 15, 2026

Copy link
Copy Markdown
Owner

Summary

Adds the missing behavioral test that Product Quantization (PQ) is honored end-to-end through a VectorStore/Engine commit, asserting against a deterministic on-disk observable rather than only that search succeeds.

Follow-up of #790: that change propagated HnswOption::quantizer into HnswIndexConfig via from_hnsw_option, but the only coverage was converter-level unit tests. The existing pq_* tests build HnswIndexConfig directly and so bypass the from_hnsw_option path. A regression dropping quantizer from that converter (while keeping rerank_storage) would have gone unnoticed by any on-disk/behavioral test.

What the test does

test_pq_quantizer_honored_through_engine_commit (in laurus/tests/vector_segment_test.rs):

  1. Builds a VectorIndexConfig whose HNSW field carries quantizer: QuantizationMethod::ProductQuantization { subvector_count: 2 } (dim = 4, so dim % subvector_count == 0).
  2. Commits a dense, well-separated 16-document corpus (two clusters of 8, mirroring the stable test_hnsw_pq_search_returns_corpus_neighbour setup) through VectorStore — exercising from_hnsw_option.
  3. Locates the committed .hnsw segment via Storage::list_files(), skips the 20-byte HNSW preamble (num_vectors:u64 + dim/m/ef:u32×3; asserts num_vectors == 16), and reads the LVS1 VectorSegmentHeader back.
  4. Asserts QuantHeader::ProductQuantization with params.m == subvector_count and params.sub_dim == dim / subvector_count.

Why it is a real regression guard

The default QuantizationMethod is Scalar8Bit (quant_kind = 1). If from_hnsw_option dropped quantizer, the segment header would report quant_kind = 1 and the match arm would panic!. So the test fails iff PQ is not honored through the commit.

Scope

  • Test-only. No production code, on-disk format, or public API change.
  • No docs change (no behavior/feature change).

Verification

  • cargo test -p laurus --test vector_segment_test — 2 passed (1 new)
  • cargo clippy -p laurus --tests -- -D warnings — 0 warnings
  • cargo fmt --check -p laurus — clean

Closes #798

🤖 Generated with Claude Code

…e commit (#798)

Adds an end-to-end integration test that commits a dense, well-separated
16-document corpus through VectorStore with
HnswOption::quantizer = ProductQuantization, then reads the produced
on-disk LVS1 segment header back and asserts QuantHeader::ProductQuantization
(quant_kind = 2), with the configured subvector_count and sub_dim.

This is the only behavioral assertion that PQ is honored through a store
commit: it exercises the from_hnsw_option converter path (#790). A
regression dropping quantizer from that converter would fall back to the
default Scalar8Bit (quant_kind = 1) and fail the test. The existing pq_*
tests build HnswIndexConfig directly and bypass from_hnsw_option.

Closes #798
@mosuka mosuka merged commit ec137e2 into main Jun 15, 2026
22 checks passed
@mosuka mosuka deleted the test/798-pq-engine-commit branch June 15, 2026 05:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(vector): add an end-to-end test that a schema quantizer (PQ) is honored through an Engine commit

1 participant