Bundle C2Cache, add CTable (.b2z) and hierarchical container support, chunked writes, and fancy coordinate indexing - #290
Merged
Merged
Conversation
…(), Python client Table class
…z storage handling Whole-table /api/fetch was returning the raw .b2z zip instead of a cframe, so table[:] failed client-side. Also introduces Array/Table as proper Dataset subclasses (client.py), dispatches cframe decoding by known kind instead of trial/except, adds Table.nrows/columns/head/rows, and treats .b2z as a native Blosc2 suffix for upload/load_from_url/htmx paths so tables round-trip byte-identical. Adds regression tests.
htmx_path_info/htmx_path_view: render a paged row/column preview for CTable using schema_dict(), with Filter/Sort-by hidden (filterable flag) since they don't apply to tables; also fixes a pre-existing crash in the Meta tab template for CTableMetadata (no cparams). cli.py: `info` prints table-shaped fields instead of crashing on cparams.get(None); `show` parses the optional row-slice syntax (table.b2z[start:stop]) and prints rows via the Table client class instead of calling the array-oriented fetch(). Adds regression tests for both surfaces, plus tests for nested/ non-identifier CTable column names (e.g. "trip.sec" struct leaves), now resolved natively by blosc2's CTableRow.__getitem__.
Follow-up fixes from review of the CTable support work: - client: bound Table.rows() default to [0:50) instead of the whole table, so table.rows() no longer silently fetches every row of a large table (pass stop=self.nrows for all rows). - server: fix /api/fetch CTable slice resolution to use `is None` instead of truthiness, so table[0:0] returns an empty result rather than the whole table; also normalize negative indices and clamp start/stop to [0, nrows]. - cli: coerce numpy scalars, bytes, and arrays in `show --json` via a json default, matching the web preview's cell handling. - server: return a clean htmx error (not an uncaught AssertionError) when a filter/sort is requested on a dataset type that does not support it (e.g. a .b2z). - server: drop a stray comment token in the CTable fetch branch.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
A .b2z may hold a TreeStore (a hierarchy of leaves), not just a CTable. Address leaves by path (tree.b2z/level1/ctable) without unfolding to disk: list descends, info/fetch open the leaf, the web tree expands into leaf rows, and the client dispatches by server-reported kind.
A .b2z may hold a TreeStore (a hierarchy of NDArray/CTable leaves), not just a single CTable. Address leaves by virtual path (tree.b2z/level1/ctable) without unfolding to disk: - split_container_path()/treestore_leaves() split a request path at the .b2z boundary and enumerate leaves. - API: list descends, info/fetch open the leaf; leaves inherit the container mtime. - Web: the tree expands a container into leaf rows; info/view tabs work on leaves. Unify group-like things behind models.Directory (kind="dir", mtime, size, nfiles): info returns it for a real directory, a TreeStore container (root group), and a virtual group inside one. Group size is summed cheaply from the .b2z zip index (no per-leaf open). Client: new Group class (browsable/indexable); Root.__getitem__ dispatches on server-reported kind (dir->Group, ctable->Table, shape->Array, else File), reusing already-fetched metadata to avoid a double info round-trip.
A .b2z TreeStore now shows as a single mountable row in the datasets list instead of auto-expanding into one row per leaf (which flooded the list). Clicking the plug icon "mounts" it as a virtual root alongside @personal/@shared/@public, with its own checkbox and an unmount control; checking it lists that container's leaves. Mount state lives client-side in localStorage (key caterva2:mounted), bridged to the server via an htmx:configRequest listener that adds `mounted=` params to the root-list request. No new endpoints, DB, or per-user server state. - server.py: htmx_root_list accepts `mounted` and filters it through get_rootdir_or_none; htmx_path_list renders TreeStores as single mountable rows and expands mounted containers into leaf rows. - templates: root_list.html renders mounted roots, path_list.html adds the plug button, home.html holds the mount/unmount JS. - Rename Directory.kind "dir" -> "group" (models/client/cli). Review fixes: - Avoid stored XSS: read paths from data-* attributes at click time instead of interpolating into inline handler JS source. - Don't 500 the listing on a corrupt/non-TreeStore/stale .b2z (untrusted localStorage input); skip it in both the walk and virtual-root loops. - Dedup roots in mountRoot so a repeat click can't double-list leaves. - stat() the container once per mount instead of once per leaf. - Update test_treestore.py for single-row behavior; add coverage for virtual-root leaf expansion and bogus-container safety.
Extend the .b2z TreeStore virtual-descent/mount feature to plain HDF5 files: a srv_utils.open_container() adapter (_TreeStoreAdapter / _HDF5Adapter) unifies list/info/fetch across both formats, backed by a file-less HDF5Proxy.open_leaf() (in-memory, no .b2nd written to disk). Client Group gains unfold/copy/move/remove/download, since a plain .h5 now dispatches to Group instead of File. Also fix the mounted-root unmount (x) icon: a long root name (typical for .h5 files) grew the row past the sidebar's fixed column, pushing the icon under the neighboring higher-z-index panel and eating the click. The icon now sits in a fixed, absolutely-positioned slot in the row's own gutter, so it stays clickable and its checkbox lines up with the regular-root rows above it.
htmx_path_list reused the container file's stat().st_size for every leaf inside a mounted .b2z/.h5, so all datasets showed the same size. Add a cheap leaf_size() to both container adapters (schunk cbytes for TreeStore, h5py storage size for HDF5, no full proxy needed) and use it for per-leaf rows instead.
…r 500
- get_filtered_array: accept inner_key param, open container member instead
of blosc2.open() on the whole file
- htmx_path_view: replace blanket "no filter/sort on container members" 400
with HDF5-only guard; .b2z members now flow through get_filtered_array
- Fix pre-existing 500 on 0-d container members: arr[()] returns unhashable
ndarray, broken by `value in header_sort` in template; convert to scalar
- Tests: structured & 0-d leaves in _make_tree fixture, sort asc/desc tests,
0-d view test, i4-no-fields 400 test
- get_filtered_array: HDF5Proxy branch using .indices()/.sort() (materialized,
cache-safe). Filter still blocked (needs LazyExpr plumbing on proxy).
- htmx_path_view: narrow HDF5 guard to filter-only; sort passes through.
Set filterable=False for HDF5 members (hide filter box in UI).
- hdf5.py: blosc2.asarray(self.dset) instead of self.dset[:] so ingestion
streams chunk-by-chunk from HDF5 for >16 MB datasets — no intermediate
full numpy array.
- Tests: structured HDF5 leaf in fixture, sort asc/desc, filter 400, sort
on plain-dtype 400, filterable=False assertion, 0-d scalar view fix.
1. Filter-only crash on .b2z members — root cause is a blosc2 bug: the where-fastpath re-opens the operand's urlpath, which for a TreeStore leaf is the whole .b2z. Worked around in get_filtered_array by detaching filtered members with an in-memory arr.copy() (cache-bounded, same materialization trade-off the filter path already makes). New tests cover filter-only and filter+sort on members. 2. /api/fetch silently dropping filter on members — fetch_data now routes filter requests through get_filtered_array(..., inner_key=inner_key); HDF5-member filters get a clean 400 (raised from a 2-line guard in get_filtered_array), and ValueErrors map to 400 instead of 500. Tested for both .b2z (filtered rows come back) and .h5 (400). 3. Corrupt-member 500s — added except (RuntimeError, OSError) to the htmx except chain. 4. open_container None-check divergence — new srv_utils.open_container_member() helper replaces all three copies of the open→get→validate pattern (htmx view, fetch, filtered path). The bogus-.b2z-member case now yields "Cannot open container member" instead of the nonsensical "Invalid filter" message (regression test added). 5. Double dataset ingest — HDF5Proxy now materializes once via a memoized _as_blosc2(); argsort (with indices kept as an alias) and sort share the single conversion. 6. Tiny-chunk inheritance cliff — _as_blosc2() ignores degenerate HDF5 chunks (< 1 MiB) and lets blosc2 pick its own chunking. 7. Redundant HDF5Proxy branch in the server — deleted; the argsort alias lets HDF5 members flow through the generic NDArray path. 8. 0-d comment misattribution — reworded to name blosc2.NDArray[()] as the 0-d source. Two bonus fixes along the way: the "unsupported dataset type" asserts became ValueErrors (they were uncaught 500s from /api/fetch and vanish under python -O), and running the suite exposed 4 latent test bugs from the earlier header-sort session (assertions matching row-label/y cells, and raise_for_status() on an intentional 400) — those tests were curl-verified back then because port 8000 was occupied; they're now fixed and passing under pytest.
Datasets panel: clicking a row highlights it as the keyboard cursor (separate from the teal "loaded" indicator); Up/Down move the cursor and focus its link so Enter loads it, starting from whichever dataset is already active if no cursor has been set yet. Display tab: clicking a data row highlights it; Up/Down move the highlight within the loaded window and page in the adjacent window at the edges, continuing the highlight into it. Reuses Bootstrap's border/table-active utilities, no new CSS beyond suppressing the default focus outline on dataset links in favor of the row border.
…oop hygiene Correctness: - Read dataset mtime from top-level info["mtime"] (schunk.mtime is always None for NDArray datasets), so cache invalidation on remote change works. - Read fetched data out of the proxy before the evictor runs (and inside the per-path lock, keyed by Path like partial_download), avoiding sync-HTTP refills of just-evicted chunks; verify offline cache reads after the read. - Reject filter/field and stepped slices on peer datasets with 400 instead of silently returning wrong data or tripping the offline fallback. - Only transport errors (new remote.OFFLINE_ERRORS) mark a peer offline; HTTP status errors are relayed as-is (a 404 no longer benches the peer). - Detect non-fetchable catalog entries (bare .h5/.b2z containers) via info shape/schunk and split_container_path; fail as 400. - Advertise only handshaken peers in /api/roots and the sidebar (same predicate as routing), with a background re-probe for down-at-boot peers. - Return peer listings relative to the requested path, per the contract. Event-loop hygiene: - get_known/maybe_reprobe re-handshake in a background thread; endpoints never block on peer probes. htmx_path_list catalog fetch, htmx_peers probes/scans (now parallel), and cframe packing all run off the loop. Cleanups: - Pool one caterva2.Client per peer urlbase (client_for), centralizing the trailing-slash fix; drop dead get_online(); single API_VERSION source in peers.py; single peercache.pool_dir source of truth; atomic atime writes with tolerant loads; parallel boot handshakes; consolidated imports. Tests: 3 regressions added (404 relay keeps peer online, filter/field and stepped-slice 400s, path-relative listing); full suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Builds one glibc-based image (fast prebuilt wheels for blosc2/hdf5plugin, no compiler needed) that doubles as a portable `container run` artifact and a `container machine`-bootable one (systemd baked in), for testing caterva2's remote peer mounts across two servers on one Mac.
The MVP wired peers into the API but the web UI could not actually browse them. Fixes, found by exercising a live two-server setup: - html_home forced roots=["@public"] for anonymous users, so with login=false peer roots could never be selected; now it only strips the auth-only roots (@personal/@shared). - The peers status panel was injected as a fifth child of the #page grid, shifting every cell and crushing the dataset list. It is now a Bootstrap dropdown under the Roots list (htmx-fetched per click, so badges stay live), and the grid is back to its original four children. - htmx_path_info and htmx_path_view only resolved filesystem paths and 404'd on peer datasets. Both grew a peer branch: info round-trips the peer's api/info JSON into the local pydantic models; the view prefetches exactly the visible window via afetch and reads multi-dim windows as one combined slice (the old two-step indexing would have downloaded the whole remote array). - Peer dataset sizes in the listing are real now (api/info cbytes, memoized per catalog refresh) instead of the 0 B placeholder. - Concurrent requests corrupted the sparse-frame caches: handles are not coherent under concurrent mutation (second writer or the evictor thread -> stale frame index -> "Error while getting the lazychunk"). One global peercache.io_lock now serializes all peer-cache IO (open/create, fetch, read, evict) across api/fetch, path-view and the evictor; eviction still runs only after data is read out. - A cache that fails to open (e.g. left half-written by a crashed writer) is dropped and rebuilt instead of 500ing forever. Verified live against a peer in an Apple container VM, including parallel window paging under a 20K cache quota; full suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`test_read_metadata` asserted `nbytes > cbytes` and `cratio > 1.0` of a five-row table, and had been failing for as long as anyone deselected it: 425 bytes of data go into 1405 bytes of store, because a schema, a valid-rows array and a frame per column cost more than five rows hold. Nothing was wrong except the expectation. Whether a table compresses is blosc2's business and depends on how much of it there is; what this endpoint answers for is that the two figures are reported and agree with each other, so that is what is asserted now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`get_writable_path` read the root, resolved its authorization, and then joined the rest of the request path onto it -- `..` segments and all. `Path.joinpath` keeps such a segment rather than resolving it, so nothing after that point ever asks whether the path is still under the root it was authorized against: `is_file` resolves it, the write resolves it, and `fsspec.url_to_fs` resolves it once more on the way to the publish root. So `POST api/chunk/@personal/%2E%2E/%2E%2E/public/target.b2nd` answered 200 and wrote a chunk into somebody else's public array, and the same path through `POST api/publish` would have copied it out under a key two directories above the one the server configures. The spelling matters: an HTTP client normalizes a literal `..` out of an URL before sending it, so what reaches the server is the percent-encoded form, and that arrives intact. Refused where the root is read, which is the one place that knows what the root was. `publish_destination` checks again on its own account: what it returns is an URL, and the docstring's whole subject is that a caller does not choose where a publish lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`indices` was parsed and then dropped on two paths that could not honour it.
A coordinate fetch against a peer-mounted root reached `provider.fetch(root,
rel, slice_)` with `slice_` None -- a provider fetches boxes and takes
nothing else -- and one against a `.b2z` passed a guard that asked only about
`slice_`. Both answered 200 with the whole dataset, which the client then
read as though it were the three points it asked for. A dropped key is not
a smaller answer; it is the largest one there is. Refused now, as
`filter`/`field` already are for a peer, and as `slice_` already was for a
container.
`POST api/fetch` also lifted the only bound the coordinates had. The URL's
length was doing that work, and nothing replaced it: one anonymous request
could name any number of points and have them gathered, materialized and
serialized -- on the event loop, since this branch never reached the
threadpool. `MAX_INDICES_CHARS` caps the parse and `MAX_FETCH_COORDS` the
gather, and the gather runs off the loop.
`FetchPayload` ignored unknown fields, so a body of `{"indicies": ...}`
validated with every parameter None and served the whole dataset with a 200.
Every field here narrows the answer, so ignoring one widens it: `extra`
is `forbid`, and a misspelling is the 422 it should always have been.
Finally, a malformed slice was a 500 rather than a 400: `parse_segment`
raised `TypeError` for a segment of four parts (`slice()` takes three), which
neither caller catches, and `parse_slice` was not called inside a `try` at
all. Both are `ValueError` now, and both are caught.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`api/info` stamped `accept_ranges: "bytes"` on every `models.Metadata`, including a `.b2nd` that proxies an HDF5 dataset. Such a file reads as an `NDArray` where the metadata is built, but `api/fetch` opens it as an `HDF5Proxy` and rebuilds what it sends -- the bytes on disk are a proxy's chunks, not the array's -- so it answers a `Range` with a 416. A client that took the claim on trust would find that out on its first block read, which is the request the field exists to save. Said only for a dataset that really is served from its file: an omission costs a request, a wrong answer costs correctness, which is what the field's own comment already promised. A container leaf now carries the container's `ETag`, on `api/info` and on its ranged reads alike. The two-request read -- frame header first, chunk offsets second -- is exactly what the validator was added for, and it was the one shape of read that had none: a client had nothing to tell it the container was not rewritten in between, which is when the bytes it reads as offsets are a chunk's. A `DictStore` `.b2z` answers for its groups. Its keys are paths, so a group is a prefix that some key continues and there is no object to hand back for one; `get` used to return None and `api/info` 404 a path the listing itself had just named. A prefix resolves to a group marker, which is what `is_group` has always been asked. And a model built from a peer's `api/info` survives a peer that has never heard of a field this one names: `get_model_from_obj`'s dict getter raised `KeyError` where its object getter raises `AttributeError`, and only the second was caught -- so `accept_ranges` alone was enough to 500 the web path-info panel for every dataset an older peer serves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`store_chunk` checked a chunk's nbytes and its blocksize and not its
typesize, though `Client.fill_chunk` documents all three as the contract.
The sizes do not carry it and its mismatch is silent: the shuffle filters
read and write on a stride of the typesize, so a chunk compressed against
another one is the right length, splits into the right blocks, and
decompresses to values in the wrong places -- verified, a chunk of typesize 3
written into an int32 array reads back as garbage with no error anywhere.
Read off the chunk's own header, and compared against what a chunk of this
array would carry (which is 1 for a typesize past 255, where the filters run
bytewise and the header says so).
`publish_dataset` staged every copy under one `{target}.partial`. Two
publishes of an array can overlap -- the background task the last chunk
starts, and a client calling the endpoint to finish an interrupted one -- and
neither holds the per-dataset lock across the upload, so their bytes
interleaved into a single file, one `fs.mv` moved the wreck into place and
the other raised inside a background task. A name per attempt makes them
write the same thing twice instead, and a failed attempt cleans up after
itself rather than leaving litter that nothing will ever move.
The quota check walked and stat'ed the whole state directory once per chunk,
so the walk that costs most in an array's size ran most often on the largest
arrays. The walk is kept briefly and the chunks written since are added to
it, which can only ever overstate what is on disk -- a quota bites slightly
early, never slightly late.
`SPECIAL_UNINIT` is blosc2's own value now rather than a literal, so a
renumbering of it travels; the header offset stays, that being what reading
one byte instead of walking every chunk costs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three ways a key came out meaning something else than it said. An `Ellipsis` stands for every dimension the key does not name, and the new `else: raise IndexError` in `slice_to_string` caught it: `ds[...]` had worked (it fell through to an empty slice string, which is the whole dataset and the right answer), and now raised, along with `ds[..., 0]` and `ds[..., [1,2]]`. It is expanded rather than refused, against the dataset's shape where the caller knows it -- a trailing one can be dropped without it, since the dimensions it covers are taken whole either way, and one anywhere else is refused rather than read as a different key. A bound of 0 was tested for truth: `index.start or ""` makes `0` an empty string, and an empty string in a slice string says "no bound at all", which is the whole dimension. So `ds[0:0]` asked for everything where it meant nothing, and `ds[2:0]` asked for the rest of the dimension. Written out now, in `slice_to_string` and in `key_to_indices` both. `MAX_QUERY_CHARS` was 60,000 and counted the raw string, but a coordinate list is mostly `[`, `]` and `,`, and each of those is three characters once percent-encoded. `ds[list(range(8000))]` came to 39 KB raw, stayed under the threshold, and went out as a GET whose request line was past 100 KB -- which uvicorn's h11 (16 KiB) or nginx (8 KiB) drops, so the client raised a transport error instead of quietly changing verb, which is the entire point of the threshold. Measured after encoding now, against a bound deployments actually accept. Also here: `Client.publish` goes through `_post` rather than re-implementing it, so it keeps the ReadTimeout-to-TimeoutError translation a whole-file upload is the likeliest call to need; and `_c2array` keeps its view between calls instead of spending an `api/info` per chunk written, dropping the lot whenever this client does something that could put a different array at a path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`open_cached_proxy` invalidated a cache on one thing only: the remote's mtime. That says whether what is cached went stale, and nothing about whether the code reading it did. Carrying a column's validity across the cache widened the compound dtype of every cached CTable with a masked column, so a cache written before that change is reopened against a source of the new dtype -- a `Proxy` pairing chunks of one itemsize with a cache laid out for another, and `_synth_ctable_cframe` reading `rows[name]` for a field the cached array has not got. The remote never changed, so nothing noticed. `_peer_src` records the dtype and a layout version beside the mtime, and a mismatch in any of them rebuilds. An existing cache has neither key, so it is rebuilt once, which is the wanted answer. `_ctable_fixed_dtypes` also still said a masked-null column makes a table non-cacheable, which stopped being true one commit after it was written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The entries described the features as first written; this is the same release, so they describe what ships rather than gaining a list of fixes to themselves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both test workflows were gated on `main` for push and pull_request alike, so a long-running feature branch got no CI at all -- its tests were found broken at merge time, which is the one moment there is nothing cheap to do about it. `push:` with no branch filter covers every branch here; `pull_request:` is kept for forks, whose pushes are not pushes to this repo. A concurrency group makes a new push supersede the run already going, which matters most on the paid arm64 runner; `main` is excluded from the cancelling so its history keeps a result for every commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`count_written` reads a frame's offsets through `blosc2.FsspecNDSource` and `publish_dataset` writes through `fsspec.url_to_fs`, and nothing anywhere declared fsspec: blosc2 keeps it behind an extra, so no dependency of ours pulls it in. It is on a developer's machine by accident of something else having installed it, which is why the tests pass there and would not on a clean install -- `POST api/chunk` raises ImportError on its first call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`member_window` promises None "where there is no such window" and returns it for every such case -- an embedded leaf, a C2Array reference, an HDF5 dataset, a CTable .b2z -- except one: a blosc2 without `DictStore.member_window` at all raised AttributeError, which nothing catches, so fetching any container leaf was a 500 on every blosc2 older than the one this was written against. That method is an optimization: with it a leaf is served by seeking to its frame in the file, without it the leaf is rebuilt -- the same bytes, more work. So its absence is one more way there is no window, and the caller already has the rebuild for all the others. Asked of the store rather than of a version number, so it starts being used the day it arrives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three features these tests exercise live in blosc2's `main` but in no release: `DictStore.member_window` (serve a leaf from its window), a `C2Array` that sends a fancy key as `indices`, and a `C2Array` that writes chunks. The suite has only ever been green against a working copy of blosc2, so CI on a released one would be 27 tests red with nothing to do about it. Feature-detected rather than pinned to a version: what the packaging asks for is a released blosc2, and a test that names the API it wants starts running by itself the day that API ships -- no floor to remember to raise, no red job in the meantime. The reasons say which API and which release. What is skipped is a faster path, never an answer. A leaf still comes back, rebuilt; a fancy key is still gathered by the server and still checked here through this package's own client, which needs nothing unreleased. Two tests were rewritten so they keep running everywhere -- the traversal refusal now seeds its root with an upload rather than a fill, and compares the array it protects byte for byte instead of reading the frame's offsets, since it is a security regression test and should not stand down for an optimization. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`RootProvider` names its methods after the endpoints they serve, so `list` in that class body is the method, not the builtin -- and `rows`, written below it, is annotated `list[tuple[str, int, str | None]]`. Evaluated where it stands, that subscripts a function: `TypeError: 'function' object is not subscriptable`, raised at import, which takes `server.py` and every service with it. Python 3.14 defers annotations (PEP 649) and never evaluates this one, which is why it went unnoticed: the server simply does not start on 3.12 or 3.13 -- the version CI runs and the ones most installs use. Found by the first CI run this branch has ever had. Quoted rather than renamed, since the method names are the contract with the endpoints they serve, and rather than `from __future__ import annotations`, which would stringize every annotation in the module and set this repo's ruff config asking for the imports behind them to move into a type-checking block. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The publish key came from the request path, which drops the user id `@personal` carries on disk: two users filling @personal/run.b2nd published to one destination, the second over the first's data -- and able to read it back by publishing and then fetching what is there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
Its metadata carries neither a shape nor a size, so the model chosen off the field names alone fell through to File and raised on the size it does not have: a 500 on every click on a peer table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
The peer's headers come off an httpx response, whose names are
lowercased, so setdefault("Content-Disposition", ...) matched none of
them and the answer went out naming the file twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
A handshake answer without a peer_id raised a KeyError out of a probe documented never to raise, and a null one matched every peer not yet handshaken in the dedupe check. An unreadable cache_quota raised out of a load() that says it skips invalid entries, and took the server's startup with it: parse_size raises a ValueError about the value now (and reads T, which its regex always accepted), and both callers reading a size out of a config file catch it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
It was the one Client method that no longer went through _format_paths, so a leading slash became an URL with an empty first segment and a confusing 404 instead of a clear ValueError. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
The listing strips the prefix off each key, and a prefix that names a leaf exactly is one character longer than the key it matches: the strip left [""], and a client walking it built a path this had just handed out and got a 404. A single dataset lists as [name] in the plain-file branch and in the peer provider, so HDF5 leaves (which answered []) now agree too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
The Content-Length went out with the headers, so stopping the stream on a short read hands the client a body shorter than the length it was promised -- which it waits out or takes for the whole leaf. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
A declared body is read whole before anything of ours runs, so MAX_INDICES_CHARS measured what the server had already been made to spend. Read off the request stream against MAX_FETCH_BODY instead: a 500 MB body cost 422 MB of RSS and now costs 12 MB and a 413. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
Gathering opens every cached frame in the pool and walks its chunks, and ran on every fetch against a cache with room to spare, for a list nothing then evicted from: 38.8 ms -> 1.6 ms per fetch on a pool of 200 datasets at a tenth of its budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
The pool is bounded in bytes and not in keys, so the table grew one lock per distinct path ever touched -- 2.15 MB per 10 000 of them, kept for as long as the server runs, including keys whose cache files eviction deleted. Every caller takes the lock in an async with on this call's result, so what is in use is strongly referenced anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
lru_cache keys on the shape of the call, so the fetch path (keywords, sortby=None) and the web view (positional, sortby="") asked the same question two ways: computed twice, and holding two of the sixteen entries. Normalizing before the cache makes the second call a hit (34 ms -> 0). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
pyproject registers c2cache as a caterva2.providers entry point, but the build context excluded the package: the image installed metadata pointing at a module that is not there, so provider discovery logged a ModuleNotFoundError on every boot and the image -- whose purpose is to be a caching peer -- could never act as one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1iNi3BJkAz4ibd96GS1i5
The 422 on api/fetch echoed the whole request body back, undoing the bound the endpoint had just been given: pydantic carries the input in its errors, and FastAPI's own path elides it. A bad cache_quota unmounted the peer instead of dropping the quota, and a handshake answer that was not a JSON object raised out of a probe documented never to raise, taking its thread with it. The container listing needed two mutually exclusive special cases because the three adapters disagreed on what leaves(prefix) means; they now agree on descendants, and is_leaf() answers the rest without building an HDF5Proxy to throw away. The per-dataset lock table was keyed on the request path -- which publish_key had just established does not identify an array -- and grew an entry per path ever written. Four call sites, now keyed on the resolved path and held weakly, as peercache's already were. Also: merge relayed download headers case-insensitively rather than one header at a time; walk the peer-cache pool once per fetch instead of N+1 times and again inside the eviction pass; don't copy the body that read_bounded_body just bounded; guard _model_from_info against a non-dict as get_info already does; name the server's unreadable quota setting when it refuses to boot over it; spell the 1G peer cache default once. The two standing lint warnings go with them. The RUF009 directive in test_peers is not dead: the ruff the pre-commit hook pins reports it ten times over in that file, and only a newer ruff calls the directive unused -- so RUF100 joins it rather than replacing it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018yrcUaGtr5zXfbsQbdaKXU
`extend-select` extends ruff's own default set, and 0.16 grew that set by eight linters: bumping the pin turned 48 rules nobody had opted into into errors. Spelling the selection out -- with E4/E7/E9/F, which is what the old default was -- makes a ruff upgrade change the rules and nothing else. The hook id `ruff` is a legacy alias now, TCH is spelled TC, and UP038 no longer exists to be ignored. ruff-format claims Markdown as of 0.16, where blacken-docs already formats those code blocks at black's width: left to themselves the two rewrite each other on every run, so the hook keeps to Python, notebooks and stubs. What the new formatter wanted is four files and sixteen lines, plus a notebook whose implicit string concatenations it joins. The RUF009 directive in test_peers goes with them: it was load-bearing only under the ruff that was pinned before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018yrcUaGtr5zXfbsQbdaKXU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR merges the
c2cache-monorepobranch intomain. It unifies theC2Cachepeer-caching subsystem into the Caterva2 core codebase, introduces first-class support for Blosc2CTable(.b2z) tables and hierarchical containers (TreeStore/DictStore.b2zand HDF5), adds distributed chunk-by-chunk array writes with atomic publishing, and extendsthe fetch API with server-side fancy coordinate indexing and strict HTTP byte-range semantics.
(Note: This PR also incorporates all changes from PR #288 and PR #289).
Key Features & Highlights
1. C2Cache Monorepo Integration & Remote Peer Mounts
C2Cacheis now bundled as an internalcaterva2.c2cacheprovider within the core Caterva2 wheel and container images, requiring no external package installation. It remains inert unless[[server.peer]]is configured.2. Blosc2 CTable (.b2z) & Hierarchical Container Browsing (PR #288)
.b2zstructured tables across the API, Python client (Tableclass), CLI, and Web UI.blosc2.sort_by()(ascending/descending) and condition filtering on structured fields..b2zcontainers and.h5/HDF5 files as mountable virtual roots.3. Incremental Chunk-by-Chunk Writes & Atomic Publishing
Client.lay_out()and filled chunk-by-chunk by concurrent writers viaPOST api/chunk/{path}(Client.fill_chunk()).typesize).
publish_rootdirectory upon receiving their final chunk (POST api/publish/{path}).4. Fancy Coordinate Indexing & POST Fetch API
POST api/fetchsupports JSON-encodedindicesto gather up to 1M arbitrary scattered coordinates directly on the server, only reading chunks that contain target points and drastically reducing payload sizes.ds[key]andClient.get_slice()seamlessly handle coordinate arrays and boolean masks.5. Byte Range Honesty & Frame ETags (PR #289)
Accept-Ranges: bytes(enabling block-level range reads), while dynamic on-the-fly slices respond withAccept-Ranges: noneand HTTP 416 on invalid range requests.api/inforeportsaccept_rangesmetadata ahead of fetch requests. Introduces generation-counter-awareETagcalculation.