feat(index)!: covering ("included") columns for IVF vector indexes

[vivek-bharathan]

Description

Lets a vector index materialize extra columns alongside its quantization codes, so a query whose projection is fully covered is answered from the index with no take against the base table.

ds.create_index("vec", "IVF_PQ", include_columns=["price"])
ds.to_table(nearest={"column": "vec", "q": q, "k": 10}, columns=["price"])  # no TakeExec

The index metadata gains included_fields (table.proto field 11), and a covered query drops the base-table take from the plan.

Scope

Covering works on all seven IVF variants — IVF_PQ, IVF_SQ, IVF_RQ, IVF_FLAT, IVF_HNSW_PQ, IVF_HNSW_SQ, IVF_HNSW_FLAT. Each quantizer storage carries its own covering_field_indices() override so its internal code/factor columns are never mistaken for payload.

Creation is available from Rust, Python, and Java; validation stays centralized in the Rust core.

Supported across the index lifecycle: append/optimize (merge, append, retrain), compaction and row-id remap, partition split/join, distributed sharded builds (per-shard commit and cross-shard merge), schema evolution, and concurrent-commit conflict resolution.

Rejected up front, with a clear error

• covering the indexed vector column itself (it is stored as the quantization code)
• nested/dotted columns — cover the whole top-level struct instead
• reserved storage names (_rowid, _distance, __ivf_part_id, quantizer code/factor columns), duplicates, blob columns
• merge_insert of a covered column on a stable-row-id dataset, combined with inserts, or on legacy v1 blob columns — provide the full target schema
• include_columns together with precomputed shuffle buffers (those buffers do not carry the payload)

Format change

New optional proto field IndexMetadata.included_fields (field 11) plus a writer feature flag, FLAG_COVERED_INDEX_METADATA (bit 128).

The flag exists because prost silently drops unknown fields: a writer predating this change would re-serialize the index section without included_fields while the index files still physically hold the payload columns, leaving storage the manifest no longer describes. The flag is writer-only on purpose — a reader that ignores included_fields simply falls back to a base-table take, which is correct, so fencing readers would turn a read-side optimization into a fleet-wide break.

Breaking changes

• IndexMetadata gains a required included_fields field
• VectorIndexParams gains a required include_columns field
• apply_feature_flags keeps its name but takes (&mut Manifest, &FeatureFlagsConfig) instead of (&mut Manifest, bool, bool)
• init_writer_for_{flat,pq,sq,rq} in lance-index gain a trailing covering_fields: &[FieldRef] parameter (pass &[] for the previous behavior)

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR adds included_fields metadata for covering columns, supports their persistence through IVF_PQ storage and optimization, returns them from ANN queries, and prevents schema or transaction updates from serving stale covered values.

Changes

Covering-column support

Layer / File(s)	Summary
Included-field metadata contract protos/table.proto, rust/lance-table/..., java/lance-jni/..., python/..., rust/lance/...	Index metadata stores included field IDs, with serialization, binding, constructor, and fixture updates.
Covering-column storage rust/lance-index/src/vector/{storage.rs,pq/storage.rs}	Storage aligns schemas, preserves extra columns through remapping, and exposes covering batches and schemas.
Index build and optimization propagation rust/lance/src/index/{vector.rs,vector/builder.rs,vector/ivf.rs,append.rs,create.rs}	IVF_PQ parameters accept included columns and propagate them through build, shuffle, partition joins, incremental creation, optimization, and index initialization.
Covered ANN query output rust/lance/src/index/vector/ivf/v2.rs, rust/lance/src/io/exec/knn.rs, rust/lance/src/dataset/scanner.rs	ANN search paths and fallback scans emit covering columns, including late-search, parallel, filtered, and zero-partition cases.
Index integrity and lifecycle handling rust/lance/src/dataset/{schema_evolution.rs,transaction.rs,write/merge_insert.rs}, rust/lance/src/dataset/optimize/remapping.rs	Covered fields participate in stale-index pruning, remapping, and schema-change validation. Tests cover drops, alters, partial updates, and metadata preservation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Dataset
  participant IvfIndexBuilder
  participant ProductQuantizationStorage
  participant ANNIvfSubIndexExec
  Dataset->>IvfIndexBuilder: configure include_columns
  IvfIndexBuilder->>ProductQuantizationStorage: persist row_id, codes, covering columns
  ANNIvfSubIndexExec->>ProductQuantizationStorage: search partitions
  ProductQuantizationStorage-->>ANNIvfSubIndexExec: covering batch
  ANNIvfSubIndexExec-->>Dataset: distance, row_id, covering columns

Loading

Possibly related issues

• Refactor IVF index build/optimize into a unified pipeline #6686 — Both changes concern consolidating IvfIndexBuilder and IVF optimization paths that preserve covering columns.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: added covering/included columns support for IVF_PQ vector search.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance/src/dataset/scanner.rs`:
- Around line 4012-4027: Add a regression test near the existing IVF_PQ scanner
tests that configures a non-empty include_columns payload field, creates indexed
data, appends additional unindexed rows, and runs the combined ANN/flat scan.
Assert the result schema includes the covering column and verify its values for
both indexed and unindexed rows, exercising the columns logic before
topk_appended projection.

In `@rust/lance/src/dataset/schema_evolution.rs`:
- Around line 3247-3255: Update the negative tests
test_drop_columns_fails_on_covered_column and
test_alter_columns_rename_nullable_fail_on_covered_column to assert the returned
error matches Error::InvalidInput { .. } before checking its message contents.
Preserve the existing assertions for the column, index, and remediation text,
covering each failure case rather than relying solely on err.to_string().

In `@rust/lance/src/index/vector.rs`:
- Around line 1901-1910: Update the source-field resolution in the index-copy
logic around included_columns to fail when any source_index.included_fields ID
is absent from source_dataset.schema(), rather than silently dropping it with
filter_map. Mirror the target-side resolution and error handling used in the
corresponding target-field block around the existing target resolution logic,
preserving the resolved field names only when all advertised fields are valid.

In `@rust/lance/src/index/vector/ivf.rs`:
- Line 577: Restrict included_columns propagation in the IVF rebuild paths
around the with_include_columns calls to only (SubIndexType::Flat,
QuantizationType::Product), passing an empty list or rejecting validation for
all other index formats, including binary IVF_FLAT, IVF_SQ, IVF_RQ, IVF_HNSW_SQ,
and IVF_HNSW_PQ. Add a regression test covering that non-IVF_PQ rebuilds do not
retain covering columns.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 66b72a0d-503d-4280-9d5b-e80f0194841d

📥 Commits

Reviewing files that changed from the base of the PR and between 5ba1e20 and 6be3150.

📒 Files selected for processing (32)

• java/lance-jni/src/transaction.rs
• java/lance-jni/src/utils.rs
• protos/table.proto
• python/src/indices.rs
• python/src/transaction.rs
• rust/lance-index/src/vector/pq/storage.rs
• rust/lance-index/src/vector/storage.rs
• rust/lance-namespace-impls/src/dir/manifest.rs
• rust/lance-table/src/format/index.rs
• rust/lance/src/dataset/cleanup.rs
• rust/lance/src/dataset/mem_wal/index.rs
• rust/lance/src/dataset/mem_wal/memtable/flush.rs
• rust/lance/src/dataset/optimize.rs
• rust/lance/src/dataset/optimize/remapping.rs
• rust/lance/src/dataset/scanner.rs
• rust/lance/src/dataset/schema_evolution.rs
• rust/lance/src/dataset/transaction.rs
• rust/lance/src/dataset/write/merge_insert.rs
• rust/lance/src/index.rs
• rust/lance/src/index/append.rs
• rust/lance/src/index/create.rs
• rust/lance/src/index/frag_reuse.rs
• rust/lance/src/index/mem_wal.rs
• rust/lance/src/index/scalar.rs
• rust/lance/src/index/scalar/btree.rs
• rust/lance/src/index/vector.rs
• rust/lance/src/index/vector/builder.rs
• rust/lance/src/index/vector/details.rs
• rust/lance/src/index/vector/ivf.rs
• rust/lance/src/index/vector/ivf/v2.rs
• rust/lance/src/io/commit/conflict_resolver.rs
• rust/lance/src/io/exec/knn.rs

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The change closes the base-table take at the right boundary, and its metadata fencing plus field-aware lifecycle handling preserve covered values across updates, remaps, rebuilds, and distributed merges.

ANN execution currently loads and materializes every declared included field on every query, even when the projection does not use it. Wide string or struct payloads can therefore regress I/O and memory for unrelated searches. Keep include_columns limited to small, frequently returned fields and benchmark both used and unused projections; if broader payloads are expected, pass the requested field subset into partition loading.