feat(index): add RaBitQ encoder and multibit-aware scorer contract

[chrevanthreddy]

Describe the issue this Pull Request addresses

Resolves #19098 (umbrella: #19094, RFC-109)

Third slice of RFC-109 (Hudi Native Vector Index): the RaBitQ encoder and metric-aware scorer contract with multibit-aware support.

Stacked PR. This branch is stacked on #19310 (options, #19096) and #19317 (schema/payload, #19097). Until those merge, the diff here shows their commits too; the only new commit in this PR is feat(index): add RaBitQ encoder and multibit-aware scorer contract. Review/merge order: #19310 → #19317 → this.

Summary and Changelog

• RaBitQEncoder: 1-bit and multibit (extended-plane) encoding contract, residual and non-residual encoding, L2/IP query-state preparation, and packed-binary/multibit dot-term scoring primitives. Encoding-version-aware so readers dispatch on the metadata written per generation.
• VectorQuantizer: quantized-vector container (QuantizedVector) and quantizer contract shared by encoder and posting builders.
• MetricQueryState: metric-aware query rotation/normalization state; enforces metric compatibility explicitly.
• RaBitQPlaneKernel / VectorQueryPlanes: bit-plane scoring kernel (Long.bitCount-based inner products) and query-plane representation.
• RaBitQByteLutScorer: byte-LUT scorer for multibit codes.
• RaBitQNeutralFactors / RaBitQFactorConfig: neutral/rescale factor computation and configuration.
• Tests (59, all passing): TestRaBitQEncoder (20), MetricEstimatorIdentityTest (14), TestVectorDistanceMetric (9), TestRaBitQNeutralFactors (6), TestMetricQueryStateRotation (5), TestRaBitQPlaneKernel (3), TestRaBitQResidualHypothesis (1), TestRaBitQResidualRecall (1).

Impact

Additive: new engine-neutral encoder/scorer classes under hudi-common. No wiring into existing write/read paths in this PR (that happens in the bootstrap #19099 and read-path #19100 slices). No behavioral change to existing tables.

Risk Level

low

Pure additive numeric/encoding logic with dedicated unit coverage; nothing references these classes from existing runtime paths yet.

Documentation Update

Encoder/scorer contracts and the multibit decomposition are documented as class/method Javadoc. User-facing RFC document tracked in #19309.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact described above
• Necessary doc changes done or have another open PR
• For large changes, please consider breaking it into sub-tasks under an umbrella issue (RFC-109: MDT-backed vector index umbrella tracking issue #19094)

[hudi-bot]

CI report:

• c4189f9 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build