Repository navigation
Add a per-row/per-column statistics sidecar to VCS arrays - #70
Merged
Merged
Conversation
VCSCArray/VCSRArray now carry a lazily computed `stats` (VCSStats): row_sums/row_nnz, col_sums/col_nnz, and col_depth_sums (sum_i x[i, j] / row_sums[i]). These are recipe-independent, and every recipe's per-cell/per-gene scales (a/b in y = (g(x*a*b) - c) * s) follow from them without touching the data, so normalization now reads a/b from the sidecar and only c/s still take a pass over the nonzeros. write_h5ad/write_zarr (and the registered VCSC/VCSR codec) always write the sidecar as a versioned `stats` subgroup of each array's group. Readers attach it when present and valid; a missing, unknown-version or incomplete sidecar is ignored, and one whose shape doesn't match is discarded with a warning, leaving the array to recompute on first use. VCSC depth sums add each value group in row order so they don't depend on in-memory index order (IVCS storage re-sorts groups), and compression now uses a stable sort so groups built from scipy input are usually sorted already. Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM
aarmey
added a commit
that referenced
this pull request
Oct 14, 2026
Resolve README/usage conflicts by keeping both new sections. Reuse _io._read_shape from #70 in the subset reader, and document and test that a subset load recomputes its stats rather than inheriting the file's sidecar, which describes the whole matrix. Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the statistics sidecar proposed in #26, shaped around the normalization recipe from #40 (
y = (g(x·a·b) − c)·s).What's in the sidecar
VCSCArray/VCSRArraynow carryarr.stats, avsparse.VCSStats. It is computed on first access and then kept:row_sumsobs["total_counts"])row_nnzn_genes_by_counts; explicit zeros not counted)col_sumscol_nnzn_cells_by_countscol_depth_sumsΣᵢ x[i,j] / row_sums[i]These are recipe-independent, and they determine
aandbfor every recipe:a = target / row_sums, wheretargetis a constant ormedian(row_sums).b = 1 / (target · col_depth_sums), or1 / col_sumswhen there's no depth normalization, becausetargetfactors out of the sum.Normalization now always takes
aandbfrom the sidecar, so a view built from a restored sidecar matches a freshly computed one exactly. That makesrawandcp10k_log1pviews free once the sidecar exists.candsare moments of the nonlinearg, so there is no recipe-independent summary of them, and they still take one pass per recipe (for VCSC that pass no longer computesbtoo).The stats live on the plain array, so they exist (and get written) whether or not a normalized view was ever built.
On disk
write_h5ad/write_zarralways write the sidecar asX/stats(andraw_X/stats), plus arrays nested inunsthrough the registered VCSC/VCSR codec. The group carriesencoding-type: vsparse-statsandencoding-version: 0.1.0.read_h5ad/read_zarr/load_packedattach it when it's usable. On read it degrades gracefully:UserWarning.Numerical notes
bis now computed astarget · Σ x/depthrather thanΣ x/(depth/target). Results differ at the ulp level; all existing tolerance-based tests pass._compressnow uses a stable sort, so groups built from scipy input are usually already sorted and the sort is skipped.Not in this PR
Selectionto filter from the sidecar (that piece of Create a statistics sidecar as an on-disk format feature #26's acceptance criteria).arr.statsexposes everything needed for it..T. Subsets recompute lazily; full-slice copies share the parent's stats.Checks
pytest: 1615 passed (27 new intests/test_stats_sidecar.py). They cover agreement with dense recomputation (Hypothesis), exact round trips for h5ad/zarr × VCSC/VCSR (X, raw_X, nested in uns),load_packed, invariance to order within groups, every degrade-gracefully path, and a check that restored stats are never recomputed.ruff,codespell,ty: clean. Fast benchmarks: no regressions.xenon/radon: no new C-or-worse blocks;_column_stats_major_is_coldropped out of rank C.jscpd: no new clones.vulture: only a false positive on numpy'sflags.writeable.https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM