Loose ends found while preparing the recent PRs (not fixed; for triage) #790

Please note: this list was generated by Claude Code (Anthropic) from the notes of our investigation, not written by me item by item. Each entry was found and checked during that work (entries found only by reading the code are marked), but please verify any item yourself before spending time on it.

While working on the recent c-blosc2 and python-blosc2 PRs (stress tests, fuzzing, sanitizer runs, reviews) we ran into more problems than we fixed. This is the list of what we left alone, so you can decide what is worth your time; none of it needs an answer. Everything predates our PRs (reproduced on 4.14.1, or traced with git blame to older code) unless marked. There is a repro script for each reproduced item, and I can share any of them or send a PR for the ones you want fixed. c-blosc2 items are included here to keep it in one place; happy to move them.

Already filed separately: #764, #776; Blosc/c-blosc2#848, Blosc/c-blosc2#863, Blosc/c-blosc2#870.

Probably worth a look (wrong data, unreadable data or crashes in normal use)

• c-blosc2: chunks compressed with ZSTD use_dict=True and stored in a frame file cannot be read through partial reads: blosc2_schunk_get_slice_buffer() / getitem and python a[...] fail with "Dictionary mismatch"; decompress_chunk() works.
• DictStore("x.b2d", mode="w") keeps the external leaves of the store that was there before, so a new store silently lists old keys (a .b2z in mode "w" starts empty).
• A symlinked .b2z is replaced by a regular file when it is rewritten (since 4.8.0, os.replace() onto the link; 4.7.0 wrote through it).
• SWMR (locking=True): a reader in another process gets "Error while getting the buffer" and rows of zeros while the writer shrinks and then grows the array (in holding_lock() blocks). Overwrite-only, grow-only and shrink+overwrite writers are fine; the file is correct afterwards.
• SChunk(urlpath, mode="r") overwrites the chunksize with 2**24, so get_slice() on it fails.
• An SChunk with a short chunk in the middle (append_data() of less than chunksize, twice) makes s[:] raise (4.14.1 died with SIGFPE in the same probe).
• A Python prefilter with BLOSC_NTHREADS > 1 segfaults and a Python postfilter deadlocks (Python codecs/filters are handled by Keep codecs and filters registered from Python on one thread #785, pre/postfilters are not).
• BLOSC_NTHREADS set to 0, abc or an empty string makes blosc2.decompress() fail.
• NDArray.__setitem__ with a LazyExpr value and an array or boolean key fails (AttributeError / "truth value of an array is ambiguous"): the fancy path indexes the value with fancy sub-indices.
• matmul(): when the fast path fails after the prefilter is set, the chunked fallback runs with the prefilter still attached and fails the same way.
• slice(key_with_step, urlpath=p) first writes the intermediate to p: without mode="w" it raises (file exists), with it the file is removed under the intermediate.
• c-blosc2: blosc2_vlmeta_add() accepts a name longer than BLOSC2_METALAYER_NAME_MAXLEN on an in-memory super-chunk; afterwards vlmeta[:], repr() and to_dict() raise KeyError.
• SChunk.get_slice(out=...) and NDArray.get_slice_numpy(arr, key) write into read-only buffers (PyBUF_SIMPLE), e.g. a bytes object; Bound decompress2() and decompress() by the sizes of dst and src #780 fixes this for decompress()/decompress2()/decompress_chunk() only.
• oindex / get_oselection_numpy() accept float index lists (truncated) and boolean lists (taken as indices 0/1); NumPy raises for floats and treats booleans as masks.
• main only (Introduce Menudet 0.1 draft for portable array and table computation #741): the native graph span readers (execute_sources(..., native_spans=True)) leave an exception raised in a Python postfilter pending after the C call instead of re-raising it (code reading); aux_miniexpr() with more than 256 C-Blosc2 threads lets threads share worker 0's buffers and decoders.

Memory safety on unusual input (C API, edge cases)

• c-blosc2: get_set_slice()'s 0-d branch ignores buffersize (set reads, get writes typesize bytes whatever the buffer size). Check the value's shape in NDArray.__setitem__ and broadcast it as NumPy does #784 keeps __setitem__ from reaching it; the raw set_slice() and C callers still can.
• c-blosc2: b2nd_get_slice_cbuffer() / b2nd_set_slice_cbuffer() with a buffershape wider than the slice and only the slice's bytes read/write past the buffer (the check counts items, the copy uses buffershape strides; Check the indexes and the buffer of orthogonal selections c-blosc2#864 fixes the same thing for orthogonal selections).
• blosc2.asarray(np.array(5)).oindex[()] crashes: orthogonal_selection() with ndim 0 reads entry 0 of an empty selection.
• c-blosc2: compare_selection() truncates int64 differences to int, so on axes longer than 2**31 the selection sort gets an inconsistent comparator (code reading).
• c-blosc2: blosc2_bitshuffle() with typesize 0 divides before its check (SIGFPE).

Threads

• c-blosc2: blosc2_free_resources() releases the global context's pool without global_comp_mutex, so a concurrent blosc2_compress()/decompress() can use it while it is cleared.
• c-blosc2, caused by our Keep the shared thread pools usable in a fork()ed child c-blosc2#852: its fork handler takes global_comp_mutex, and with glibc < 2.29 popen() forks (newer glibc uses posix_spawn, which runs no fork handlers). So a plugin found through Python and loaded while blosc2_compress()/blosc2_decompress() hold that lock (global context, 1 thread) would deadlock in the fork handler (code reading, not reproduced).
• c-blosc2: blosc2_register_codec()/filter()/tuner() write the registries without a lock; registering while other threads compress is presumably unsupported, but nothing says so.

Leaks, error paths, inconsistencies

• c-blosc2: b2nd functions leak their output array when a later step fails (b2nd_from_cbuffer(): 74 KiB per failing blosc2.frombuffer(); b2nd_concatenate(), also its chunk buffer on the unaligned path; b2nd_create_ctx() on the ZFP + shuffle check); b2nd_squeeze_index() ignores failures of b2nd_create_ctx()/view_new().
• c-blosc2: a prefilter's error return is ignored when the output is disposable (reductions), so the error is lost (python-side workaround in Make the matmul, miniexpr and Menudet prefilters unable to need the GIL #774).
• _set_pref_expr() after a failed me_compile leaks and leaves the prefilter set without its params; SChunk.update_cparams() writes the new values into the storage cparams before _check_cparams() raises, so a rejected update leaves storage and context inconsistent.
• memoryview values take the scalar path of __setitem__ (np.isscalar(memoryview(...)) is True) and raise.
• SChunk.decompress_chunk() with a dst over 2 GiB wraps the length negative; vldecompress() raises OverflowError for a src over 2 GiB.
• c-blosc2: blosc2_schunk_copy() recompresses with the blocksize of the last (possibly short) chunk, so the copy gets smaller blocks and bigger chunks.
• python-blosc2 never calls blosc2_set_nthreads() at import, so blosc2.compress()/decompress() (global context) run single-threaded while compress2()/decompress2() use blosc2.nthreads. Maybe intended.
• File modes: to_b2d() of a .b2z opened in mode "a"/"w" creates a 0700 directory; some overwrites (RemoteArray export with overwrite=True, b2view downloads) reset a chmod-ed file to the umask mode while others keep it (Write .b2z stores and downloads with the umask's permissions again, not 0600 #789 makes .b2z writes keep it).
• Small: blosc2_get_slice_nchunks() is declared int in blosc2_ext.pyx (it returns int64_t); normalize_1d_sparse_indices() is unused; the slice() docstring still says steps are unsupported; blosc2_schunk_get_vlblock() leaks the chunk in its cbytes == 0 branch.

Tests, CI, build

• c-blosc2 CI: the "Ubuntu GCC AARCH64 (native)" job runs on an x86 runner (runs-on: ${{ matrix.os }} is ubuntu-latest; its arch: arm64 is unused), so CI has no Linux ARM64 coverage. The race fixed in Initialize the shuffle implementation once for all threads c-blosc2#872 showed up only on ARM64 and MSVC. os: ubuntu-24.04-arm would do it.
• c-blosc2: the SIMD byte-shuffle round-trip tests (tests/test_shuffle_roundtrip_{sse2,avx2,...}.c) test nothing: they check SHUFFLE_USE_*, but CMake defines SHUFFLE_*_ENABLED.
• c-blosc2: under ASan+UBSan test_fork_threads (added by our Keep the shared thread pools usable in a fork()ed child c-blosc2#852) fails on main: LSan reports in the forked children, from UBSan's symbolizer allocating while it reports the older misaligned loads in blosclz.c and fastcopy.c (Compare 64 bytes per iteration in get_run() c-blosc2#874 removes the one in get_run()).
• python-blosc2: builds with a system GCC 13 compile non-Limited-API symbols (e.g. PyGILState_Check) into the abi3 extension with only a warning; -Werror=implicit-function-declaration would catch that.

Performance ideas (measured, not done)

• name in vlmeta decodes the whole value (embedded objects included) just to answer True; set_vlmeta() reads the frame header 4 times per overwrite; vlmeta[:] re-syncs once per name.
• c-blosc2: BLOSC_TRACE_ERROR() calls getenv() on every error path, also expected ones like a vlmeta miss, and blosc2_create_dctx() reads BLOSC_NTHREADS per context.
• take(axis=...) / a[rows] on N-D arrays pass np.arange() index arrays for untouched axes; the extension copies NumPy scalars at 50-100 ns per item vs ~20 ns for list items.
• matmul()'s fast-path gate compares whole block shapes, so non-square blocks (A 32x32, B 32x16) never take it although the kernel supports them.
• c-blosc2 bitshuffle: odd element sizes (3, 5, 7) are slow in both directions (10-14 vs ~4 µs per 32 KB); 4-byte elements could get a ~30% faster unshuffle with the approach of Vectorize bitunshuffle for 1- and 2-byte elements c-blosc2#867 (estimate); the generic eightelem routine (all of ARM, since bitshuffle-neon.c is not built) writes single bytes.
• c-blosc2: with several threads, one-call contexts still pay ~10-20 µs to wake the workers after Keep unused thread pools for later contexts with the same nthreads c-blosc2#865 (64 KiB decompress2() with 4 threads: ~19 µs vs ~6 µs with 1), so small inputs are faster with 1 thread; on Windows contexts still start their own threads per call.