Skip to content

Add a per-row/per-column statistics sidecar to VCS arrays - #70

Merged
aarmey merged 1 commit into
mainfrom
stats-sidecar
Oct 14, 2026
Merged

aarmey merged 1 commit into
mainfrom
stats-sidecar

Conversation

@aarmey

@aarmey aarmey commented Oct 14, 2026

Copy link
Copy Markdown
Member

Implements the statistics sidecar proposed in #26, shaped around the normalization recipe from #40 (y = (g(x·a·b) − c)·s).

What's in the sidecar

VCSCArray/VCSRArray now carry arr.stats, a vsparse.VCSStats. It is computed on first access and then kept:

field axis meaning
row_sums cells total counts (obs["total_counts"])
row_nnz cells stored nonzeros (n_genes_by_counts; explicit zeros not counted)
col_sums genes total counts
col_nnz genes n_cells_by_counts
col_depth_sums genes Σᵢ x[i,j] / row_sums[i]

These are recipe-independent, and they determine a and b for every recipe:

  • a = target / row_sums, where target is a constant or median(row_sums).
  • b = 1 / (target · col_depth_sums), or 1 / col_sums when there's no depth normalization, because target factors out of the sum.

Normalization now always takes a and b from the sidecar, so a view built from a restored sidecar matches a freshly computed one exactly. That makes raw and cp10k_log1p views free once the sidecar exists. c and s are moments of the nonlinear g, so there is no recipe-independent summary of them, and they still take one pass per recipe (for VCSC that pass no longer computes b too).

The stats live on the plain array, so they exist (and get written) whether or not a normalized view was ever built.

On disk

  • write_h5ad/write_zarr always write the sidecar as X/stats (and raw_X/stats), plus arrays nested in uns through the registered VCSC/VCSR codec. The group carries encoding-type: vsparse-stats and encoding-version: 0.1.0.
  • read_h5ad/read_zarr/load_packed attach it when it's usable. On read it degrades gracefully:
    • A missing sidecar (older files, other tools), an unknown version, or a missing field is silently ignored.
    • A sidecar whose shape doesn't match the array is discarded with a UserWarning.
    • In both cases the array recomputes the stats only when they're first needed.
  • Older vsparse versions read the arrays explicitly by key, so they ignore the extra subgroup.

Numerical notes

  • b is now computed as target · Σ x/depth rather than Σ x/(depth/target). Results differ at the ulp level; all existing tolerance-based tests pass.
  • IVCS packing sorts each value group's indices. To keep a sidecar written before a round trip bit-identical to one recomputed after it, VCSC depth sums add each group in row order. _compress now uses a stable sort, so groups built from scipy input are usually already sorted and the sort is skipped.

Not in this PR

Checks

  • pytest: 1615 passed (27 new in tests/test_stats_sidecar.py). They cover agreement with dense recomputation (Hypothesis), exact round trips for h5ad/zarr × VCSC/VCSR (X, raw_X, nested in uns), load_packed, invariance to order within groups, every degrade-gracefully path, and a check that restored stats are never recomputed.
  • ruff, codespell, ty: clean. Fast benchmarks: no regressions.
  • xenon/radon: no new C-or-worse blocks; _column_stats_major_is_col dropped out of rank C. jscpd: no new clones. vulture: only a false positive on numpy's flags.writeable.

https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM

VCSCArray/VCSRArray now carry a lazily computed `stats` (VCSStats):
row_sums/row_nnz, col_sums/col_nnz, and col_depth_sums
(sum_i x[i, j] / row_sums[i]). These are recipe-independent, and every
recipe's per-cell/per-gene scales (a/b in y = (g(x*a*b) - c) * s) follow
from them without touching the data, so normalization now reads a/b from
the sidecar and only c/s still take a pass over the nonzeros.

write_h5ad/write_zarr (and the registered VCSC/VCSR codec) always write
the sidecar as a versioned `stats` subgroup of each array's group.
Readers attach it when present and valid; a missing, unknown-version or
incomplete sidecar is ignored, and one whose shape doesn't match is
discarded with a warning, leaving the array to recompute on first use.

VCSC depth sums add each value group in row order so they don't depend
on in-memory index order (IVCS storage re-sorts groups), and compression
now uses a stable sort so groups built from scipy input are usually
sorted already.

Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM
@aarmey
aarmey merged commit 57b4166 into main Oct 14, 2026
7 of 8 checks passed
@aarmey
aarmey deleted the stats-sidecar branch October 14, 2026 17:23
aarmey added a commit that referenced this pull request Oct 14, 2026
Resolve README/usage conflicts by keeping both new sections. Reuse
_io._read_shape from #70 in the subset reader, and document and test
that a subset load recomputes its stats rather than inheriting the
file's sidecar, which describes the whole matrix.

Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant