feat(index): add index format 2 for every index by gouyt13 · Pull Request #222 · VectorDB-NTU/RaBitQ-Library

gouyt13 · 2026-11-06T12:45:25Z

Summary

Part of Index file format v2 with a written specification #212 (with 64-bit IDs, with -1 for missing results #211)

This draft adds index file format 2 and moves all three indexes onto it: IVF-RaBitQ, HNSW-RaBitQ and SymphonyQG. Each index also records its construction seed, so Python keeps seed after load. It also adds rabitqlib.FormatError for rejected files and moves Python point IDs to int64 with -1 for missing results (#211). The cosine metric (#212) is not in this PR yet.

Commits

docs(formats): the specification. It is in docs/formats.md, next to the rabitqlib.quant byte formats. It covers:
- a 128-byte header and a section table, with 64-byte-aligned sections;
- ext4-style incompat / ro_compat / compat feature masks and required/optional sections;
- the section kinds and record layouts, including the HNSW routing graph field by field;
- each index's reader rules;
- companion files for RaBitQ-SSD;
- the reading order and the compatibility rules.
feat(index): the container layer (src/index/format2.*, not installed). It writes files front to back without seeking. The reader validates the expected index kind before any section, and reads through the stream buffer.
feat(ivf): IVF-RaBitQ saves and loads format 2.
- One file. HNSW centroid routing is section 260, so save writes no <file>.hnsw sidecar.
- Earlier files still load, including their sidecars.
- New frozen fixtures, tests/fixtures/persistence/v0.8.3/. They were generated from an isolated v0.8.3 checkout and cover the RABQIDX1 v2 layout, which had no frozen fixture before.
- Rotator blob in C++. The codec and a zlib-compatible CRC-32 move to private core sources. rabitqlib.quant is unchanged; the C++ codec reads and rewrites its golden files byte for byte.
fix(ivf): two misses that CI found.
- The HNSW load benchmark still called the removed Initializer::save/load.
- A Windows test path was written as a narrow string literal; it is now a UTF-8 literal.
test(ivf): reproducible HNSW routing in a byte comparison. A test trained its routing graph with several threads, whose insertion order is nondeterministic. It now trains with one thread, and the docs say the graph is reproducible only when built with one thread.
test(persistence): frozen v0.8.3 HNSW-RaBitQ and SymphonyQG fixtures.
- HNSW at 1, 5 and 9 bits, with removals and upper layers.
- SymphonyQG raw, 4- and 8-bit, with FHT/Kac and matrix rotation.
- A separate generator writes them, so the committed IVF fixtures keep their bytes. The current readers loaded them before the writers changed.
feat(index): HNSW-RaBitQ and SymphonyQG save and load format 2.
- Sections: HNSW uses 512–516; SymphonyQG uses 768–770, and the raw mode stores no centroid.
- Shared graph checks: HNSW's earlier-layout reader keeps all of its checks, and the format-2 reader uses the same validate_graph.
- Seed and settings: every index records its seed. SymphonyQG also records ef_construction and init, which QGBuilder records on the graph.
- Read-only loading works for all three indexes:
  - IVF-RaBitQ: refuses add, reserve, remove and save.
  - HNSW-RaBitQ: refuses add, resize, remove and save.
  - SymphonyQG: refuses save until a builder rebuilds it.
feat(index): rabitqlib.FormatError.
- C++ loads throw rabitqlib::FormatError (new public header rabitqlib/index/format_error.hpp, a std::runtime_error) for a truncated, corrupt, mismatched or unsupported file, in format 2 and every earlier layout. A file that cannot be opened stays a plain std::runtime_error.
- Python raises rabitqlib.FormatError, a subclass of both ValueError and RuntimeError, so except RuntimeError keeps working. rabitqlib.quant.FormatError is the same class.
- This resolves the spec's open question on the Python error type.
feat(python): int64 point IDs (64-bit IDs, with -1 for missing results #211).
- search returns int64 IDs with -1 for a missing result, instead of 2**32 - 1; distances stay inf. add returns int64; remove already took any integer IDs and rejects -1.
- C++ point IDs stay 32-bit PID values with kPidMax. The bindings widen each search batch once, mapping kPidMax to -1.
fix(index): the remaining rejected headers. A fourth independent review found loads that escaped the FormatError contract:
- SymphonyQG records too large for size_t threw std::length_error, in format 2 and the earlier layouts.
- Earlier IVF-RaBitQ and SymphonyQG layouts with dimensions the FHT/Kac rotation does not support threw std::invalid_argument.
- The earlier IVF-RaBitQ reader caught std::ios_base::failure, which libstdc++ before GCC 13 throws with an ABI the catch does not match. It now checks each group of reads.

Breaking changes (0.9, release-noted in `docs/docs/releases/unreleased.md`)

File format: releases up to 0.8 cannot read files saved by this branch. IVF-RaBitQ files are rejected as Invalid IVF index metadata rather than misread; this was inferred from the v0.8.3 reader, not run.
ivf::Initializer (installed header): save(std::ofstream&, const char*) and load(std::ifstream&, const char*) become write(std::ostream&) and read(std::istream&, uint64_t).
New accessors (additive):
- IVF::seed();
- HierarchicalNSW::seed();
- QuantizedGraph<float>::seed(), ef_construction() and initialization().
symqg::QGInitialization moves from qg_builder.hpp to qg.hpp, which qg_builder.hpp includes.
Vendored hnswlib: a patch, marked // RaBitQ:, adds saveIndex(std::ostream&) and loadIndex(std::istream&, size_t, ...). The path overloads delegate to them.
Error type: rejected files raise rabitqlib.FormatError (C++ rabitqlib::FormatError) instead of RuntimeError/std::runtime_error, which still catch it. Earlier-layout files that threw std::invalid_argument or std::length_error for invalid contents also throw it. quant.FormatError was a ValueError only.
Python IDs (64-bit IDs, with -1 for missing results #211): search and add return int64; a missing result is -1, not 2**32 - 1. np.isfinite(distances) works before and after. txtai 9.14.0 and earlier filter with uid != 0xFFFFFFFF, so 0.9 waits for a compatible txtai release (Remove the deprecated 0.7 Python call forms #213).
Python:
- After load, seed and SymphonyQG's ef_construction and init are kept. They are None only for files from releases up to 0.8.
- cp.seed follows a recorded seed: the seed itself for IVF-RaBitQ, its low 32 bits for HNSW-RaBitQ.
Reproducible files: saved files include the IVF HNSW routing graph, which is reproducible only when built with num_threads=1.
Error messages:
- Damaged format-2 files get container messages.
- HNSW routing-graph errors say graph instead of sidecar.
- A failed HNSW-RaBitQ write raises HNSW index file cannot be written.
- A leftover .hnsw beside a resaved IVF file is not removed.

Review questions

RaBitQ-SSD:
- The companion-file design: a 4 KiB header page, then pages from offset 4,096.
- The FNV-1a fingerprint over the rotated centroids.
- The coarse quantizer recording the base fingerprint instead of being a companion.
Section 260: whether it should keep hnswlib's native layout. It is now specified field by field, as 64-bit little-endian.

Known limitations

Version stamp. Files from this branch record writer version 0.8.3 until the version is bumped.
HNSW-RaBitQ level generator. It is not reseeded on load (unchanged from before), so points added after a reload do not follow the recorded seed. HNSW-RaBitQ also cannot be rebuilt in place, so a read-only HNSW index stays read-only.
hnswlib loadIndex integrity loop. It calls tellg() twice per centroid, as upstream does; this also applies to legacy sidecars.
Allocation before the graph check. IVF's HNSW routing storage is allocated before its graph is validated; the parameter sections are checksummed.
Rotator copies. Rotator sections are copied whole before decoding. A SymphonyQG load with a matrix rotator briefly holds about three copies of it.
Dimension limit. IVF-RaBitQ save rejects more than 65,536 dimensions, the rotator blob's limit.

Testing

All runs: Linux x86-64, GCC 14.3, Release, RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF.

C++ suite: 546/546 pass on 5bf7b1f.
Core-only build: index, FastScan and OpenMP all off; 153/153 pass (run on d2b25a1, before the last three commits, which change only index and binding code).
Benchmark targets (-DRABITQ_BUILD_BENCHMARKS=ON): they build, and their 15 smoke tests pass (run on d2b25a1).
Python: the package was reinstalled from 5bf7b1f. The suite, plus tests/test_mypy.py and tests/test_pyright.py: 1169 passed, 2 skipped.
Static checks: check-format.sh and check-python.sh pass. The full check-tidy.sh with the CI configuration reports no findings. mkdocs build --strict passes.
Coverage:
- Frozen fixtures. The v0.5.2 and v0.8.3 fixtures of all three indexes load and match their reference results. Migrated files are format 2, keep an unknown seed (and, for SymphonyQG, unknown settings), and saving them again reproduces them byte for byte.
- Earlier layouts. Their checks run on copies of the v0.8.3 fixtures. A test-only converter that reproduces the v0.8.3 IVF fixtures byte for byte builds the layouts no fixture covers.
- Rejections. Every container rule and every per-index reader rule has a rejection test with its exact message.
- Other cases: read-only loading for each index, unknown optional sections, write failures, and the seed and settings round trips.
- FormatError: truncated and damaged headers of every earlier layout and format 2 raise it in C++ and Python, with exact messages; a missing file does not. The tests of the last commit fail without it.
- IDs: int64 dtypes, -1 padding for every index, and search output passed back to remove.
Load time (smoke test, not deployment performance):
- Setup: an IVF index of 1M × 64 vectors in 100k clusters, 4 bits, a 126 MB file. Xeon Gold 6418H, warm page cache, an unisolated host.
- Result: format 2 takes 138.2 ms median and the 0.8 layout 136.6 ms, with about the same read and lseek counts.
Review: four independent reviews, of the container, the IVF changes, the HNSW-RaBitQ/SymphonyQG changes, and FormatError with the int64 IDs. Their findings are addressed, including:
- a 2.5× IVF load regression;
- HNSW-RaBitQ and SymphonyQG ignoring unknown read-only features;
- several tests that patched new files at old offsets and passed without exercising their check.
CI: all checks passed on d2b25a1. This push re-runs them.

Add a draft specification of the shared index container planned for 0.9 (#212): header, section table, feature flags, record layouts, companion files, reading order and compatibility rules. Nothing in it is implemented yet.

Add the container layer of index file format 2 (#212): the header and section table encoding, a zlib-compatible CRC-32, a writer that emits the file front to back without seeking, and a reader that validates the header, feature flags, expected index kind and section table in the order the specification gives. No index uses it yet. Update the draft specification with the rules the implementation settled: empty sections, the check order, padded_dim at most 65,536, reserved section kind 0, and what a resave drops.

IVF-RaBitQ save writes index format 2 (#212): one file whose sections hold the rotator blob, the routing parameters and data, the codes, the rerank data and the IDs. HNSW centroid routing becomes a section, so save no longer writes a <file>.hnsw sidecar. Load dispatches on the magic and keeps reading every earlier layout, including their sidecars. - ivf::Initializer replaces save/load on file streams with write(std::ostream&) and read(std::istream&, bytes). - The vendored hnswlib gains stream overloads of saveIndex and loadIndex; the path overloads delegate to them. - The rotator blob codec and a zlib-compatible CRC-32 move to private core sources for C++ readers and writers. - The format-2 reader reads each bulk section front to back, and the HNSW graph check tracks a byte budget instead of querying positions. - Freeze IVF files written by v0.8.3 (RABQIDX1 v2, Flat RaBitQ and HNSW routing with sidecars) as release-produced fixtures. - Document the IVF sections, the HNSW graph layout and the IVF reader rules in the specification, and note the changes for users.

The HNSW load benchmark still called the removed Initializer save and load; write and read the graph file through the stream interface, with the file open inside the timing as before. The legacy-sidecar test named its directory with a narrow literal, which MSVC encodes in the code page; use a UTF-8 literal.

The first-add test compares saved files, which now include the HNSW routing graph. Training the explicit index with several threads built that graph in a nondeterministic order, so the files differed on CI; train it with one thread, like the implicit first add. Document that the routing graph is reproducible only when built with one thread.

Before HNSW-RaBitQ and SymphonyQG move to index format 2, freeze files their v0.8.3 writers produce: HNSW at 1, 5 and 9 bits with removals and upper layers, and SymphonyQG raw, 4- and 8-bit with FHT/Kac and matrix rotation. A separate generator writes them, so the committed IVF fixtures keep their bytes.

HNSW-RaBitQ (sections 512-516) and SymphonyQG (sections 768-770) now save index format 2, like IVF-RaBitQ. Loading dispatches on the magic and keeps reading their earlier layouts; HNSW-RaBitQ shares its graph checks between both readers. - Every index records its construction seed, and SymphonyQG also its ef_construction and initialization, so Python keeps them after load. New C++ accessors report them: IVF::seed(), HierarchicalNSW::seed(), QuantizedGraph<float>::seed(), ef_construction() and initialization(). QGInitialization moves to qg.hpp. - Unknown read-only-compatible features make every index read-only: HNSW-RaBitQ refuses add, resize, remove and save; SymphonyQG refuses save until a builder rebuilds it. - A failed HNSW-RaBitQ write reports "HNSW index file cannot be written". - Tests that patched freshly saved files at old offsets now damage format-2 fields, and the earlier layouts' checks run on copies of the v0.8.3 fixtures. - Document the HNSW-RaBitQ and SymphonyQG reader rules and the seed of each index in the specification, and format the fixture generator.

MSVC requires a lambda to capture a local constexpr variable it reads, which GCC and Clang do not.

Every index load throws rabitqlib::FormatError, a std::runtime_error in the new public header rabitqlib/index/format_error.hpp, for a truncated, corrupt, mismatched or unsupported file in format 2 and every earlier layout. Failing to open a file stays a plain std::runtime_error. Python raises it as rabitqlib.FormatError, a subclass of both ValueError and RuntimeError, so existing except RuntimeError handlers keep working. rabitqlib.quant.FormatError becomes the same class; it was a ValueError only. An earlier-layout IVF file with an invalid metric byte and an earlier-layout SymphonyQG file with invalid settings threw std::invalid_argument; both now throw FormatError. This resolves the spec's open question on the Python error type.

Closes the Python half of #211. search returns int64 IDs, and slots without a neighbor hold -1 instead of 2**32 - 1, with distance inf as before. add returns int64 IDs; remove already took any integer IDs and still rejects -1 and other out-of-range values. C++ point IDs stay 32-bit PID values with kPidMax for a missing result. The bindings search into a PID buffer and widen it once per batch to int64, mapping kPidMax to -1. That adds one pass and a temporary buffer of num_queries * k IDs per search call; this change makes no performance claim. txtai 9.14.0 and earlier filter with uid != 0xFFFFFFFF; the release notes point to #213 for the first compatible txtai release.

An independent review found loads that still escaped the FormatError contract: - A format-2 SymphonyQG file whose point count and degree bound fit their limits but whose records exceed size_t threw std::length_error. Earlier SymphonyQG layouts with huge stored dimensions did the same. - Earlier IVF-RaBitQ and SymphonyQG layouts with a dimension the stored FHT/Kac rotation does not support threw std::invalid_argument from the rotator. - The earlier IVF-RaBitQ reader relied on the stream exception mask and caught std::ios_base::failure. libstdc++ before GCC 13 throws the old ABI type from its iostreams, which that catch does not match. The reader now checks each group of reads, which also undoes the reindentation of the previous commit. Each case has a C++ test that fails without this change and a Python boundary test. The docs say that a failed read after opening raises FormatError, the release notes list every converted exception, the IVF parameter table names -1, and the txtai compatibility test notes that its filter no longer matches a missing result.

gouyt13 added 2 commits November 6, 2026 19:54

docs(formats): draft index file format 2

c1d3707

Add a draft specification of the shared index container planned for 0.9 (#212): header, section table, feature flags, record layouts, companion files, reading order and compatibility rules. Nothing in it is implemented yet.

github-actions Bot added the release:added Release category managed from the PR title label Nov 6, 2026

gouyt13 mentioned this pull request Nov 6, 2026

Index file format v2 with a written specification #212

Open

gouyt13 changed the title ~~feat(index): add the format-2 index container~~ feat(index): add index format 2 and save IVF-RaBitQ in it Nov 7, 2026

gouyt13 added 4 commits November 7, 2026 16:36

gouyt13 changed the title ~~feat(index): add index format 2 and save IVF-RaBitQ in it~~ feat(index): add index format 2 for every index Nov 7, 2026

gouyt13 added 4 commits November 7, 2026 18:32

test(symqg): capture the point count for MSVC

d2b25a1

MSVC requires a lambda to capture a local constexpr variable it reads, which GCC and Clang do not.

Navigation Menu

Repository navigation

feat(index): add index format 2 for every index - #222

feat(index): add index format 2 for every index#222
gouyt13 wants to merge 11 commits into
mainfrom
feat/format2-container

gouyt13 commented Nov 6, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

gouyt13 commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Commits

Breaking changes (0.9, release-noted in docs/docs/releases/unreleased.md)

Review questions

Known limitations

Testing

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

gouyt13 commented Nov 6, 2026 •

edited

Loading

Breaking changes (0.9, release-noted in `docs/docs/releases/unreleased.md`)