Supporting spherical quantization builds for the disk index

[juchen-ms (partychen)]

Summary

• add spherical quantization as a disk index build option
• train and wire the 1-bit spherical quantizer into one-shot and sharded in-memory builds
• account for spherical vector storage in build RAM estimation
• validate the currently supported 1-bit configuration
• add disk build and search coverage for L2, inner product, and cosine metrics

Validation

• cargo test -p diskann-disk (250 unit tests and 2 doc tests passed)
• cargo test -p diskann-disk test_spherical_disk_index_builder_with_metric (2 passed)
• cargo fmt --all --check
• git diff --check

[Copilot]

Pull request overview

Adds spherical quantization as an additional quantization mode for disk index builds, wiring 1-bit spherical quantization through the disk build pipeline (training, in-memory graph construction, and RAM estimation) and extending builder tests to cover additional metrics.

Changes:

• Extend disk build quantization configuration to support SPHERICAL_<nbits> and add serialization/parse tests.
• Train and use a 1-bit spherical quantizer during disk index in-memory build (one-shot and merged/sharded paths).
• Account for spherical vector storage in build RAM estimation and add builder test coverage for L2/IP/Cosine.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
diskann-quantization/src/spherical/quantizer.rs	Adds a try_clone() convenience API for independently allocated quantizer copies.
diskann-disk/src/build/configuration/quantization_types.rs	Adds QuantizationType::Spherical plus parsing/formatting and tests for the new variant.
diskann-disk/src/build/builder/tests.rs	Extends integration tests to build/search spherical 1-bit indexes, including metric variants.
diskann-disk/src/build/builder/quantizer.rs	Trains a 1-bit spherical quantizer for disk builds and stores it in the build quantizer enum.
diskann-disk/src/build/builder/inmem_builder.rs	Plumbs spherical quantization into the async in-memory index builder via a spherical insert/prune strategy.
diskann-disk/src/build/builder/core.rs	Updates build RAM estimation to account for spherical quantized vector storage and adds validation coverage.

Suppressed comments (3)

diskann-disk/src/build/configuration/quantization_types.rs:186

• QuantizationType::SQ { standard_deviation: None } currently formats as SQ_<nbits>_None, but FromStr only accepts SQ_<nbits> for the default stddev. Because serde Serialize uses to_string() and Deserialize uses from_str, SQ-with-default cannot roundtrip (e.g. bincode serialize then deserialize fails). Consider emitting SQ_<nbits> when standard_deviation is None to keep Display/parse/serde consistent.

This issue also appears in the following locations of the same file:

• line 253
• line 312

            QuantizationType::Spherical(nbits) => write!(f, "SPHERICAL_{}", nbits),
            QuantizationType::SQ {
                nbits,
                standard_deviation,
            } => {

diskann-disk/src/build/configuration/quantization_types.rs:257

• The fmt_quantization_type test currently asserts SQ_8_None for the default-SQ formatting, which encodes the same Display/parse mismatch described above. If Display is changed to emit SQ_<nbits> when standard_deviation is None, update this expectation accordingly so the test enforces a roundtrippable representation.

    #[case(QuantizationType::Spherical(1), "SPHERICAL_1")]
    #[case(
        QuantizationType::SQ { nbits: 8, standard_deviation: None },
        "SQ_8_None"
    )]

diskann-disk/src/build/configuration/quantization_types.rs:316

• test_roundtrip_serialization documents (and works around) the fact that SQ-with-default-stddev doesn't roundtrip. If Display/parse are made consistent for standard_deviation: None, it would be valuable to include that case in this roundtrip test to prevent regressions.

                nbits: 8,
                standard_deviation: Some(Positive::new(1.5).unwrap()),
            },
            QuantizationType::Spherical(1),
        ];

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

❌ Patch coverage is 97.70115% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.57%. Comparing base (3218478) to head (f4bc552).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
diskann-disk/src/build/builder/quantizer.rs	96.87%	1 Missing ⚠️
diskann-quantization/src/spherical/quantizer.rs	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1331      +/-   ##
==========================================
+ Coverage   91.55%   91.57%   +0.01%     
==========================================
  Files         522      522              
  Lines       99541    99602      +61     
==========================================
+ Hits        91139    91206      +67     
+ Misses       8402     8396       -6     

Flag	Coverage Δ
miri	91.57% <97.70%> (+0.01%)	⬆️
unittests	91.24% <97.70%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark/src/inputs/disk.rs	1.44% <ø> (ø)
diskann-disk/src/build/builder/build.rs	92.82% <100.00%> (ø)
diskann-disk/src/build/builder/core.rs	97.28% <100.00%> (+<0.01%)	⬆️
diskann-disk/src/build/builder/inmem_builder.rs	86.61% <100.00%> (+0.90%)	⬆️
...disk/src/build/configuration/quantization_types.rs	98.11% <100.00%> (+0.45%)	⬆️
diskann-disk/src/build/builder/quantizer.rs	93.81% <96.87%> (+0.95%)	⬆️
diskann-quantization/src/spherical/quantizer.rs	98.06% <87.50%> (+0.28%)	⬆️

... and 45 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Mark Hildebrand (hildebrandmw)]

Added some comments on the use of spherical quantization and some maintenance suggestions. Please get a review from the maintainers of diskann-disk regarding how this feature fits in architecturally.

[Wei Wu (wuw92)]

Thanks for your change to support spherical in disk path. Is spherical disk-index support intentionally limited to 1-bit quantization, or is support for other bit widths planned?

[juchen-ms (partychen)]

Thanks for your change to support spherical in disk path. Is spherical disk-index support intentionally limited to 1-bit quantization, or is support for other bit widths planned?

The 1-bit limitation is intentional for this PR. The immediate goal is to provide a spherical replacement for the 1-bit scalar-quantized disk builds used in production. The underlying async spherical provider already supports 1, 2, and 4 bits, and the typed SphericalBits configuration leaves room to add those widths later, but enabling them would expand the build configuration and test matrix beyond the current request.

[Aditya Krishnan (arkrishn94)]

Thanks Junkui, it mostly looks good to me. Just had a couple of clarifying questions for my understanding.