diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 2a463b384..888938137 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -122,6 +122,23 @@ for array/table examples, persistence policy and current limitations. Release certification is tracked separately; this development entry is not an announcement that 4.15.0 has been published. +### Serializing containers that have a postfilter + +- **Behaviour change:** `SChunk.to_cframe()` and `NDArray.to_cframe()` now return + the chunks as they are stored for every storage, also when a postfilter is set + (`SChunk.postfilter`). Before, they returned the postfilter's output unless the + container was an in-memory contiguous frame (`contiguous=True` without + `urlpath`). In-memory containers are sparse by default, so the frame usually + held the postfiltered values, and setting the postfilter again on the container + loaded from it applied it twice. Code that loads such a frame without setting + the postfilter again now reads the stored values. To serialize the postfiltered + values, serialize a copy that holds them: `NDArray.copy()` applies the + postfilter, and for an `SChunk`, a new one made from its decompressed data does. + Containers without a postfilter give the same frames as before. With a + prefilter or a postfilter, `to_cframe()` no longer decompresses and recompresses + every chunk, which makes it much faster. This comes with the bundled C-Blosc2 + (Blosc/c-blosc2#858). + ## Changes from 4.14.1 to 4.14.2 XXX version-specific blurb XXX diff --git a/src/blosc2/ndarray.py b/src/blosc2/ndarray.py index bf739cb7d..f0238d0e4 100644 --- a/src/blosc2/ndarray.py +++ b/src/blosc2/ndarray.py @@ -5476,6 +5476,12 @@ def to_cframe(self) -> bytes: :func:`~blosc2.ndarray_from_cframe` This function can be used to reconstruct a NDArray from the serialized bytes. + Notes + ----- + The buffer holds the chunks as they are stored: a postfilter set on + :attr:`schunk` is not applied, and it is not part of the buffer either. + For a buffer with the postfiltered values, serialize a :meth:`copy`. + Examples -------- >>> import blosc2 diff --git a/src/blosc2/schunk.py b/src/blosc2/schunk.py index 33452acae..b48282cff 100644 --- a/src/blosc2/schunk.py +++ b/src/blosc2/schunk.py @@ -1383,6 +1383,13 @@ def to_cframe(self) -> bytes: -------- :func:`~blosc2.schunk_from_cframe` + Notes + ----- + The buffer holds the chunks as they are stored: a postfilter (see + :meth:`postfilter`) is not applied, and it is not part of the buffer + either. Set it again on the deserialized :ref:`SChunk` to read the same + values as from this one. + Examples -------- >>> import blosc2