Index file format v2 with a written specification #212

Problem

The current index files grew one field at a time and have no written specification. IVF-RaBitQ
(format 2) and SymphonyQG (format 1) start with a magic number and version; HNSW-RaBitQ files
start directly with max_elements, and the hnswlib sidecar that IVF-RaBitQ writes for HNSW
routing (<file>.hnsw) has no version either. None of them can store what 0.9 needs: the
cosine metric, the rotation/graph seed and the ID width (#211).

RaBitQ-SSD, planned to move into this library,
adds requirements the format should allow from the start, so it changes only once:

• 64-bit point IDs for more than 2^32 - 1 vectors. RaBitQ-SSD selects them at compile time
  (RABITQ_PID64, its base format version 3) and each build reads only its own files; here the
  ID width must be a field of the file.
• Companion files. Its index is a metadata file plus <base>.ssd, a page-packed inverted
  list read with O_DIRECT (offsets, lengths and buffers aligned to 4,096 bytes), plus a
  swappable coarse-quantizer file paired with the base by a fingerprint of the rotated
  centroids.
• Raw element type. Raw records can be float32 or uint8; IVF-RaBitQ's raw mode
  (nbits=32) stores float32 today.

Proposal

A version-2 format shared by all three indexes:

• a fixed-width header: magic, format version, index kind, metric (including cosine), original
  and padded dimensions, nbits, rotator type, seed, ID width (32 or 64), raw element type
  and point count;
• a section table (kind, offset, size) followed by sections aligned to 64 bytes;
• room for an external-ID section, so caller-provided IDs can be added later without another format change;
• companion files recorded in the main file by name, size and a fingerprint, so a mismatched
  or missing file is rejected on load. The IVF HNSW-routing sidecar becomes a section or a
  recorded companion. Page alignment applies inside a companion file only (RaBitQ-SSD's
  .ssd), not to the main file;
• a written specification in the docs.

Files from every earlier released version must stay readable (frozen fixtures already cover
them); new files are written as v2. A memory-mapped loader is not planned; 64-byte section
alignment keeps it possible later.

Also in this issue

metric="cosine": normalize data and queries internally and report 1 - cos distances. It
must be persisted, or a reloaded index would silently stop normalizing.
tests/python/test_import.py::test_invalid_metric_raises currently asserts that cosine is
rejected and changes with it.

Open questions

• Should the IVF HNSW-routing graph be embedded as a section (one file to copy) or stay a
  companion file?
• Fingerprint: RaBitQ-SSD uses 64-bit FNV-1a over the rotated centroids. Reuse it, or hash
  each companion file's header?

Done when

The spec page exists, v2 round-trips are tested for all indexes and metrics (including 64-bit
IDs and companion-file mismatches), and the historical-persistence tests still load every older
fixture.