Implement determinant-diversity search in the new post-processing pipeline

[narendatha]

Add Determinant-Diversity Search Support

Summary

This PR implements diversity-maximizing search using determinant-diversity post-processing.
Instead of returning only the k nearest neighbors by distance, determinant-diversity search selects k vectors that maximize the volume spanned by query-scaled vectors, producing more diverse and complementary results.

Motivation

Standard nearest neighbor search often returns highly similar or redundant results.
Determinant-diversity search helps retrieve complementary results that cover different aspects of a query.
This PR uses determinant maximization of the Gram matrix of query-scaled vectors.

Algorithm

Each candidate vector is scaled by (similarity_to_query)^power, then vectors are greedily selected to maximize the determinant of the Gram matrix (equivalently maximizing spanned volume).

Two variants are implemented:

• eta = 0: pure greedy orthogonalization (pivoted QR style)
• eta > 0: ridge-regularized variant balancing diversity and numerical stability

Both variants run in O(n * k * d) where n is number of candidates, k is number of selected results, and d is vector dimension.

Performance Characteristics

In benchmark comparison runs (normal search vs determinant-diversity search), we observe the expected trade-off:

• lower QPS due to diversity optimization overhead
• recall trade-off depending on search parameters

Parameter Tuning Guide

determinant_diversity_power (default: 2.0)

Controls relevance vs diversity:

• Higher (e.g., 3.0+): stronger relevance weighting, typically less diversity
• Lower (e.g., 1.0): more diversity emphasis
• 0: pure diversity maximization without relevance weighting

determinant_diversity_eta (default: 0.01)

Ridge regularization for numerical robustness and relevance bias:

• Higher (e.g., 0.1+): more robust, typically more relevance-biased
• Lower (e.g., 0.001): closer to pure determinant maximization
• 0: pure greedy orthogonalization (no ridge regularization)

Recommended Settings

Use Case	determinant_diversity_power	determinant_diversity_eta
Balanced (default)	2.0	0.01
High diversity	1.0	0.001
High relevance	3.0	0.1

Usage

Benchmark Configuration (disk-index)

{
  "search_phase": {
    "search_list": [100, 200, 400],
    "recall_at": 10,
    "is_determinant_diversity_search": true,
    "determinant_diversity_eta": 0.01,
    "determinant_diversity_power": 2.0
  }
}

Testing

• unit tests for determinant-diversity post-processing and parameter validation
• cargo fmt --all
• cargo clippy --workspace --all-targets -- -D warnings
• cargo check --package diskann-benchmark --features disk-index

How to try

• build this branch
• set benchmark input paths to your dataset/index
• run diskann-benchmark with determinant-diversity parameters

[codecov-commenter]

Codecov Report

❌ Patch coverage is 55.43478% with 287 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.04%. Comparing base (b616aa3) to head (58e8917).

Files with missing lines	Patch %	Lines
diskann-disk/src/search/provider/disk_provider.rs	26.31%	84 Missing ⚠️
...vider/async_/determinant_diversity_post_process.rs	83.42%	58 Missing ⚠️
diskann-benchmark-core/src/search/graph/knn.rs	0.00%	47 Missing ⚠️
diskann-benchmark/src/inputs/disk.rs	0.00%	31 Missing ⚠️
diskann-benchmark/src/inputs/async_.rs	22.85%	27 Missing ⚠️
diskann-benchmark/src/backend/index/benchmarks.rs	29.41%	24 Missing ⚠️
diskann-benchmark/src/backend/index/search/knn.rs	50.00%	14 Missing ⚠️
diskann-tools/src/utils/search_disk_index.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #858      +/-   ##
==========================================
- Coverage   90.45%   89.04%   -1.41%     
==========================================
  Files         442      443       +1     
  Lines       83248    83872     +624     
==========================================
- Hits        75301    74683     -618     
- Misses       7947     9189    +1242     

Flag	Coverage Δ
miri	89.04% <55.43%> (-1.41%)	⬇️
unittests	88.88% <55.43%> (-1.53%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark-runner/src/any.rs	100.00% <100.00%> (ø)
diskann-benchmark/src/backend/index/spherical.rs	100.00% <ø> (ø)
diskann-disk/src/build/builder/core.rs	95.26% <100.00%> (+0.01%)	⬆️
diskann-tools/src/utils/search_disk_index.rs	0.00% <0.00%> (ø)
diskann-benchmark/src/backend/index/search/knn.rs	66.66% <50.00%> (-9.90%)	⬇️
diskann-benchmark/src/backend/index/benchmarks.rs	44.52% <29.41%> (-2.05%)	⬇️
diskann-benchmark/src/inputs/async_.rs	36.83% <22.85%> (-0.90%)	⬇️
diskann-benchmark/src/inputs/disk.rs	3.87% <0.00%> (-0.60%)	⬇️
diskann-benchmark-core/src/search/graph/knn.rs	75.00% <0.00%> (-19.92%)	⬇️
...vider/async_/determinant_diversity_post_process.rs	83.42% <83.42%> (ø)
... and 1 more

... and 39 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds determinant-diversity (DPP-like) reranking as an explicit search post-processing option, wiring it through the async provider pipeline, the disk-index searcher, and benchmark configuration/execution paths.

Changes:

• Introduces a determinant-diversity post-processor (with params + validation) for async graph search.
• Adds an optional determinant-diversity rerank path to the disk-index searcher API and benchmark “disk-index” runner.
• Extends benchmark input schemas and search harnesses to support running KNN with an explicit post-processor.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tmp/wiki_compare_determinant_diversity.json	Adds a sample config comparing baseline vs determinant-diversity for async index build/search.
diskann-tools/src/utils/search_disk_index.rs	Updates disk-index search invocation for the expanded search() signature.
diskann-providers/src/model/graph/provider/async_/mod.rs	Exposes the new determinant-diversity post-processing module and types.
diskann-providers/src/model/graph/provider/async_/determinant_diversity_post_process.rs	Implements determinant-diversity reranking + parameter validation + unit tests.
diskann-disk/src/search/provider/disk_provider.rs	Adds determinant-diversity rerank post-processing to disk-index search via search_with().
diskann-disk/src/build/builder/core.rs	Updates tests/call sites for the expanded disk-index search API.
diskann-benchmark/src/inputs/disk.rs	Adds disk-index benchmark config flags/params + validation + display output.
diskann-benchmark/src/inputs/async_.rs	Adds async topk benchmark params for determinant-diversity reranking + validation.
diskann-benchmark/src/backend/index/spherical.rs	Enables determinant-diversity post-processing in spherical benchmark backend via KNNWithPostProcessor.
diskann-benchmark/src/backend/index/search/knn.rs	Refactors KNN runner to support both plain KNN and KNN-with-postprocessor runners.
diskann-benchmark/src/backend/index/benchmarks.rs	Enables determinant-diversity reranking in the general benchmark backend.
diskann-benchmark/src/backend/disk_index/search.rs	Threads determinant-diversity parameters into disk-index benchmark search execution and stats output.
diskann-benchmark/example/openai-disk-determinant-diversity-compare.json	Adds an example config comparing disk-index baseline vs determinant-diversity.
diskann-benchmark-runner/src/any.rs	Updates a dispatch error test string.
diskann-benchmark-core/src/search/graph/knn.rs	Adds KNNWithPostProcessor helper to run graph KNN searches using search_with().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.