feat(index): add index format 2 for every index

CMakeLists.txt

@@ -84,6 +84,7 @@ endfunction()
 
 set(RABITQ_CORE_SOURCES
     src/core/api.cpp
+    src/core/blob.cpp
     src/core/fast_scale_table.cpp
     src/core/provisional.cpp
     src/utils/cpu_features.cpp
@@ -247,6 +248,7 @@ if(RABITQ_BUILD_FASTSCAN)
 endif()
 if(RABITQ_BUILD_INDEX)
     add_library(rabitq_index STATIC
+        src/index/format2.cpp
         src/index/hnsw.cpp
         src/index/ivf.cpp
         src/index/ivf_initializer.cpp
@@ -261,6 +263,10 @@ if(RABITQ_BUILD_INDEX)
     )
     add_library(rabitqlib::index ALIAS rabitq_index)
     target_link_libraries(rabitq_index PUBLIC rabitq_fastscan)
+    # Index files record the library version that wrote them.
+    target_compile_definitions(rabitq_index PRIVATE
+        RABITQ_LIBRARY_VERSION="${PROJECT_VERSION}"
+    )
     set_target_properties(rabitq_index PROPERTIES
         EXPORT_NAME index
         POSITION_INDEPENDENT_CODE ON

benchmarks/hotpaths/load.cpp

@@ -37,18 +37,31 @@ void run(int argc, char** argv) {
         if (!std::filesystem::exists(rabitqlib::io_impl::filesystem_path(path + ".hnsw"))) {
             rabitqlib::ivf::HNSWInitializer index(dim, count);
             index.add_vectors(data.data(), 1);
-            std::ofstream unused;
-            index.save(unused, path.c_str());
+            std::ofstream output(
+                rabitqlib::io_impl::filesystem_path(path + ".hnsw"), std::ios::binary
+            );
+            output.exceptions(std::ios::failbit | std::ios::badbit);
+            index.write(output);
             return;
         }
     }
     bench::metadata(dim, count, 1);
     uint64_t expected = 0;
     for (size_t rep = 0; rep <= reps; ++rep) {
         rabitqlib::ivf::HNSWInitializer index(dim, count);
-        std::ifstream unused;
         const auto start = bench::Clock::now();
-        index.load(unused, path.c_str());
+        // Open the graph file inside the timing, as loading an index does.
+        std::ifstream input(
+            rabitqlib::io_impl::filesystem_path(path + ".hnsw"),
+            std::ios::binary | std::ios::ate
+        );
+        if (!input) {
+            throw std::runtime_error("Cannot open the HNSW centroid graph file");
+        }
+        const auto bytes =
+            static_cast<uint64_t>(static_cast<std::streamoff>(input.tellg()));
+        input.seekg(0);
+        index.read(input, bytes);
         const double seconds =
             std::chrono::duration<double>(bench::Clock::now() - start).count();
         uint64_t h = 14695981039346656037ULL;

docs/development.md

@@ -505,12 +505,13 @@ in place of extra-bit codes.
 
 #### IVF formats and initialization
 
-New IVF files have a magic/version header, an explicit padded dimension, and a resolved initializer type (v2); v1 files infer the historical
-Flat/HNSW choice at 20,000 clusters, independently of current auto selection.
-New IVF auto routing uses Flat below 5,000 clusters, Flat RaBitQ below 60,000,
-and HNSW otherwise. Flat RaBitQ persists its mean and codes in the main file;
-HNSW retains its sidecar. Legacy quantized files and raw v1 files infer
-64-dimension padding.
+New IVF files use index format 2 (`docs/formats.md`): the resolved initializer type
+is in `ivf_params`, and the routing data, including the HNSW graph, is a section of
+the one file. Earlier `RABQIDX1` v2 files store the initializer type; v1 files infer
+the historical Flat/HNSW choice at 20,000 clusters, independently of current auto
+selection. Those layouts keep HNSW routing in the `<file>.hnsw` sidecar. New IVF auto
+routing uses Flat below 5,000 clusters, Flat RaBitQ below 60,000, and HNSW otherwise.
+Legacy quantized files and raw v1 files infer 64-dimension padding.
 
 #### Search defaults
 
@@ -532,7 +533,7 @@ and buffer changes must also preserve the [ID marker](#search-buffer-id-marker)
 
 Filesystem path strings are UTF-8 on Windows and native path bytes on POSIX. Use
 `io_impl::filesystem_path` from `utils/path.hpp` before opening files or querying
-filesystem metadata, including IVF's HNSW sidecar. It converts UTF-8 to native Unicode
+filesystem metadata, including the HNSW sidecar of earlier IVF files. It converts UTF-8 to native Unicode
 on Windows and preserves existing path bytes on Linux/macOS.
 
 For path-only changes, check path conversion and the affected I/O operations; the
@@ -550,15 +551,20 @@ keep their provenance and hashes, and do not regenerate them with the current wr
 Index files are trusted input (see the usage guide): validate structure and sizes so
 that corrupt or mismatched files are reported before allocation, without treating a
 crafted file as a supported input.
+Readers throw `rabitqlib::FormatError` (`include/rabitqlib/index/format_error.hpp`) for
+any truncated, corrupt, mismatched or unsupported content, including errors raised by
+shared validators while loading; only a file that cannot be opened throws a plain
+`std::runtime_error`. Python maps it to `rabitqlib.FormatError`; rejection tests assert it.
 
 Do not silently reinterpret an old file. Add a magic/version discriminator, use fixed-width
-serialized fields for new formats (the current IVF, HNSW and QG bodies still store several
-`size_t` fields; change those only with a format version bump), validate sizes before
-allocation, check every read, and retain a
-compatibility test fixture or an explicit rejection path. Review each affected format;
-changes to shared serialization helpers require auditing every format using them.
-IVF-RaBitQ, HNSW-RaBitQ, and SymphonyQG have separate formats. SymphonyQG includes a versioned quantized format plus a legacy
-raw-format fallback; preserve both unless a breaking change is explicitly requested.
+serialized fields for new formats (earlier layouts store several `size_t` fields; keep their readers unchanged), validate sizes before allocation, check every
+read, and retain a compatibility test fixture or an explicit rejection path. Review each affected
+format; changes to shared serialization helpers require auditing every format using them.
+Every index writes index format 2 ([`docs/formats.md`](formats.md#index-format-2), container in
+`src/index/format2.*`) and still reads its earlier layouts: IVF-RaBitQ's `RABQIDX1` versions and
+unversioned files, HNSW-RaBitQ's unversioned layout, and SymphonyQG's versioned quantized layout
+and raw fallback. Preserve those readers unless a breaking change is explicitly requested. When a writer moves to
+a new format, freeze the last files of the old writer as release-produced fixtures first.
 
 ### Change Python bindings
 
@@ -840,8 +846,8 @@ Centroids, data, and queries compared by the same estimator must be in the same
 
 Rotator state is part of persisted index state. Loading an index must restore the exact rotation
 used at construction; generating a new random rotator produces silently incorrect distances.
-Honor stored padded dimensions instead of recomputing them from current defaults. IVF `RABQIDX1`
-v1 and v2 files explicitly store this field; preserve it for both versions. Only legacy unversioned
+Honor stored padded dimensions instead of recomputing them from current defaults. Index format 2
+and IVF `RABQIDX1` v1 and v2 files explicitly store this field; preserve it. Only legacy unversioned
 quantized IVF files and old raw v1 files omit it and infer historical 64-dimension padding.
 Unreleased pad16 files whose stored dimension is not divisible by 32 are rejected.
 
@@ -875,9 +881,9 @@ updates or persistence. These markers affect live search as well as saved files.
 IVF-RaBitQ `remove` stores a removed point as `f_add = +inf` inside the existing batch data, so it needs no
 discriminator. No index written before `remove` existed has an infinite `f_add`, so nothing in an old
 file is reinterpreted. In the legacy file formats readable by release 0.5.0, that reader also
-excludes removed points. This compatibility of the removal marker does not extend to the current
-writer's `RABQIDX1` v2 format: older releases cannot read current saves, even when resaving a loaded
-legacy index. Keep this convention when touching IVF-RaBitQ code:
+excludes removed points. This compatibility of the removal marker does not extend to index
+format 2, which the current writer uses: releases up to 0.8 cannot read current saves, even when
+resaving a loaded legacy index. Keep this convention when touching IVF-RaBitQ code:
 
 - Every estimator and scan path must turn `f_add = +inf` into an infinite estimated distance and an
   infinite lower bound, and never into NaN. A NaN distance would enter the result buffer.

docs/docs/index/hnsw.md

@@ -82,7 +82,7 @@ available CPU, so **default builds are not reproducible**: pass `num_threads=1`
 still change results. Loading restores the saved rotation.
 
 Results use squared L2 or `1 - dot` estimates, ordered from smallest to largest.
-Python fills missing results with ID `2**32 - 1` and distance `+inf`; C++ returns
+Python returns `int64` IDs and fills missing results with ID `-1` and distance `+inf`; C++ returns
 only the available pairs. See [result handling](../quick_start.md#read-the-results)
 and the shared [input, count and concurrency contracts](usage.md).
 
@@ -271,8 +271,8 @@ removed = index.remove(new_ids[:10])   # IDs in [0, num_stored_points)
 A removed point keeps its codes, its links, and its place in the graph. Search
 still walks through it and `add` may link new points to it, which keeps the graph
 connected, as in hnswlib. Search never returns it, so a query can get fewer than
-`k` results once points are removed: in Python the missing slots hold `kPidMax`
-(`2**32 - 1`) with an infinite distance. Removed points still count in
+`k` results once points are removed: in Python the missing slots hold ID `-1`
+with an infinite distance. Removed points still count in
 `num_stored_points()` and cannot be restored. Removal requires the same
 exclusive access as other updates.
 
@@ -316,6 +316,10 @@ through 0.5.1 check every cluster ID on `load` and reject the file as invalid
 instead of returning removed points. Releases before 0.3.7 do not check cluster
 IDs, so do not open such a file with them.
 
+`save` writes [index format 2](https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/docs/formats.md#index-format-2), which also records the construction seed, so
+`seed` survives a reload. Releases up to 0.8 cannot read these files; every earlier
+file still loads, without a recorded seed, and saving it writes format 2.
+
 New HNSW-RaBitQ indexes pad dimensions to multiples of 32 for all supported bit widths.
 Loading preserves the saved padded dimension and rotation, so older indexes
 using multiples of 64 remain readable. Older library versions cannot load new

docs/docs/index/ivf.md

@@ -52,11 +52,11 @@ See [C++ integration](../cpp_integration.md) for build setup and
 | `seed` | Rotation seed in `[0, 2**32 - 1]` and the initial `cp.seed`. Python's default `None` draws one at random; `index.seed` reports it, and passing it back reproduces the index. |
 | `metric` | Python `"l2"` or `"ip"`; defaults to `"l2"`. |
 | `initializer` | Defaults to `"auto"`; see [centroid routing](#centroid-routing). |
-| `k`, `nprobe` | Positive result count and cluster probe count. `k` may exceed the stored point count; missing results are padded with `kPidMax` (`2**32 - 1` in Python) and `+inf`. Python's `nprobe` defaults to `num_clusters / 16`, rounded and at least 1, when omitted, `None` or 0; C++ requires a positive `nprobe`. |
+| `k`, `nprobe` | Positive result count and cluster probe count. `k` may exceed the stored point count; missing results are padded with `kPidMax` (`-1` in Python) and `+inf`. Python's `nprobe` defaults to `num_clusters / 16`, rounded and at least 1, when omitted, `None` or 0; C++ requires a positive `nprobe`. |
 | `high_accuracy` | Python defaults to `None`: HACC for 4–9 bits, standard FastScan otherwise. |
 
 Results contain IDs and squared L2 or `1 - dot` distances, ordered from smallest
-to largest. Quantized distances are estimates. Unfilled slots use ID `2**32 - 1`
+to largest. Quantized distances are estimates. IDs are `int64` in Python. Unfilled slots use ID `-1`
 and distance `+inf`; see [result handling](../quick_start.md#read-the-results).
 See the shared [input and concurrency contracts](usage.md).
 
@@ -450,15 +450,15 @@ enters the result buffer and never triggers reranking. Any code that estimates
 IVF-RaBitQ distances must keep this behavior and must never turn `+inf` into NaN.
 
 Search may return fewer than `k` results once points are removed: the missing slots
-hold `kPidMax` (`2**32 - 1` in Python) with an infinite distance, as when the probed
+hold `kPidMax` (`-1` in Python) with an infinite distance, as when the probed
 clusters hold fewer than `k` points.
 
 The removal marker lives in the ordinary batch data and needs no additional
 format change; removal survives `save` and `load`. No file written before
 `remove` existed has an infinite `f_add`, so nothing in an old file is
 reinterpreted. Release 0.5.0 also excludes removed points when reading its
-supported legacy formats. However, current saves always use `RABQIDX1` v2,
-including when resaving a loaded legacy index. Older releases cannot read these
+supported legacy formats. However, current saves always use index format 2,
+including when resaving a loaded legacy index. Releases up to 0.8 cannot read these
 files, regardless of whether they contain removals.
 
 ## Persistence
@@ -468,19 +468,23 @@ Save a constructed index with:
 ```c++
 ivf.save(output_index_file);
 ```
-New files use the `RABQIDX1` magic, version 2, storage flags, an explicit
-padded dimension, and a uint32 initializer type (1 = Flat, 2 = Flat RaBitQ,
-3 = HNSW) before the index metadata. `Auto` is resolved before saving. Loading
-restores the routing type, storage mode, rotation state, and padded dimension.
-Raw files also contain the original vectors, so querying does not require an external dataset.
-
-Version-1 `RABQIDX1` files remain readable with their stored padding. Legacy
-unversioned quantized files and raw v1 files retain their original 64-dimension
-padding. All three historical formats infer routing with the original 20,000
-cluster threshold. HNSW routing still requires the accompanying `.hnsw` file.
-Flat RaBitQ loads its saved codes without random re-quantization. Saving a loaded
-legacy index writes the new header without changing its padded dimension or encoded data. Older library
-versions cannot read the new format.
+Files are written in
+[index format 2](https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/docs/formats.md#index-format-2):
+one file with a header, a section table, and sections for the rotation, the routing
+data, the codes and the IDs. HNSW routing is a section of that file, so there is no
+separate sidecar. `Auto` is resolved before saving. Loading restores the routing type,
+storage mode, rotation state, and padded dimension. Raw files also contain the original
+vectors, so querying does not require an external dataset. Releases up to 0.8 cannot
+read format 2. Indexes of more than 65,536 dimensions cannot be saved.
+
+Files from earlier releases remain readable. `RABQIDX1` version 2 files store their
+routing type; version-1 `RABQIDX1` files keep their stored padding; legacy unversioned
+quantized files and raw v1 files retain their original 64-dimension padding. These three
+older layouts infer routing with the original 20,000 cluster threshold. All of these
+layouts keep HNSW routing in the accompanying `.hnsw` file, which loading reads.
+Flat RaBitQ loads its saved codes without random re-quantization. Saving a loaded file
+writes format 2 without changing its padded dimension or encoded data; a `.hnsw` file
+left beside it is not removed and is no longer used.
 
 ## Implementation details
 

docs/docs/index/qg.md

@@ -77,7 +77,7 @@ qg.save("qg_example.index");
 | `init` | Python keyword: `"pipnn"` (default) or `"random"`. |
 | `metric` | Python keyword: `"l2"` (default) or `"ip"`. |
 | `seed` | Python keyword: rotation and graph-construction seed in `[0, 2**32 - 1]`. The default `None` draws one at random; `index.seed` reports it, and passing it back with `num_threads=1` rebuilds the same graph from the same data. |
-| `k`, `ef` | `k` is at least 1 and may exceed the point count; slots the graph cannot fill are padded with ID `2**32 - 1` and distance `+inf`. Omitted or zero `ef` uses `max(k, 64)` in Python, and smaller values are raised to `k`, as for HNSW-RaBitQ. |
+| `k`, `ef` | `k` is at least 1 and may exceed the point count; slots the graph cannot fill are padded with ID `-1` in Python (`kPidMax` in C++) and distance `+inf`. Omitted or zero `ef` uses `max(k, 64)` in Python, and smaller values are raised to `k`, as for HNSW-RaBitQ. |
 | `num_threads` | Python `add` defaults to every available logical CPU, capped by `OMP_NUM_THREADS`, and `search` to 1; `None` selects these defaults. |
 
 Python search returns `(ids, distances)` arrays. Distances are squared
@@ -209,6 +209,10 @@ Python counterparts for complete examples.
 
 ## Persistence
 
+`save` writes [index format 2](https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/docs/formats.md#index-format-2), which also records the seed, `ef_construction`
+and `init`, so they survive a reload. Releases up to 0.8 cannot read these files; every
+earlier file still loads, without these settings, and saving it writes format 2.
+
 All three modes pad the rotated domain to a multiple of 32. Loading an older
 index preserves its saved padded dimension, including historical multiples of 64.
 Older library versions that enforce 64-dimension padding cannot load new files