layout="cell" archives exceed source size for very-low-cell-count files (0.84x on 9-cell bulk RNA-seq)

[sullivanj91]

Summary

layout="cell" archives come out larger than the source .h5ad when a file
holds very few cells across a wide gene axis. Measured ratio 0.84× (i.e. a
16% size increase) on bulk RNA-seq files averaging 9 cells × 61,044 genes.

Low priority — the format behaves exactly as designed for its target workload,
and we're proceeding anyway for format uniformity across a corpus. Filing so the
boundary is written down rather than rediscovered.

Measurements

Same converter, same settings (write_cell_archive(src_path, dst, codec=None)),
across a range of single-cell and bulk datasets:

cells per file	n_vars	ratio (src ÷ archive)
~2.9 M	18,533	11.18×
~376 k	18,151	~11.9×
~140 k	61,044	~6.9×
~160 k	18,151	2.29×
~378 k	36,601	1.90×
9	61,044	0.84×

The last row is 60 files sampled from a 35,457-file bulk corpus: 351 MB of
.h5ad became 419 MB of .csad. A 10-file sample independently gave 0.82×.

Compression tracks cells per file, not total corpus size or file size — the
214 GB bulk corpus compresses worse than a 7.5 GB single-cell file.

Why this is expected, and where the boundary sits

A cell-layout archive stores one compressed frame per cell, which is what makes
an O(rows) scattered gather cheap — that property held up beautifully elsewhere
in the same sweep (a 1024-row gather stayed ~1.0 s on a 214 GB archive while the
h5ad equivalent took 11.9 s). With ~9 rows per file there is almost nothing for
frame-of-reference delta coding to amortise against, and per-frame plus
container/header overhead dominates.

So the crossover is somewhere between 9 and ~140 k cells per file. Narrowing it
is not worth much on its own; knowing it exists is.

Suggestion (low priority)

Purely documentation, if you think it's worth it: a line in the README noting
that layout="cell" targets many-cell matrices and can exceed the source size
for very-low-cell-count files. Optionally, write_cell_archive could emit a
UserWarning when n_obs is below some threshold — though a caller who checks
the resulting size learns the same thing, so a doc note may be sufficient.

No behaviour change requested. Happy to contribute the doc line if useful.

Measured against cellstream 0.10.0 with the Rust extension active.