diff --git a/docs/error-codes.md b/docs/error-codes.md index 7cf3395a..4cd8d714 100644 --- a/docs/error-codes.md +++ b/docs/error-codes.md @@ -493,7 +493,7 @@ export CACHEKIT_CONNECTION_POOL_SIZE=100 **Exception**: none raised to a `@cache`-decorated caller: the result is returned but not stored in the backend. The exception is `@cache(interop=...)`: there an unsupported return value raises `InteropError` (a `ValueError` subclass, `from cachekit.interop import InteropError`) by design, so a cross-SDK entry is never silently skipped. Calling the serializer directly raises `TypeError`, e.g. `StandardSerializer does not support custom classes (Python-specific types). ...` or `StandardSerializer does not support pandas DataFrames or Series (Python-specific types). ...` -**Cause**: The default serializer handles `None`, `bool`, `int`, `float`, `str`, `bytes`, `list`, `tuple`, `dict`, `datetime`, `date` and `time`. Custom classes, dataclasses and DataFrames are not supported. `serializer='auto'` also refuses a set or frozenset holding more than 1,024 elements that are neither strings nor bytes, or more than 8 of those that share one hash value, or two frozensets that share one, or two tuples that share one in a set where a tuple holds a frozenset (`AutoSerializer cannot store this set: it holds ...`): see [Set Size Bound](serializers/auto.md#set-size-bound). +**Cause**: The default serializer handles `None`, `bool`, `int`, `float`, `str`, `bytes`, `list`, `tuple`, `dict`, `datetime`, `date` and `time`. Custom classes, dataclasses and DataFrames are not supported. `serializer='auto'` also refuses a set or frozenset holding more than 1,024 elements that are neither strings nor bytes, or more than 8 of those that share one hash value, or two frozensets that share one, or two tuples that share one in a set where a tuple holds a frozenset (`AutoSerializer cannot store this set: it holds ...`): see [Set Size Bound](serializers/auto.md#set-size-bound). It also refuses a pandas `Timestamp` with non-zero nanoseconds anywhere in the value (`AutoSerializer cannot store this Timestamp: it has nanoseconds, ...`), and a set or frozenset holding an aware `datetime` whose UTC offset depends on `fold`, a wall time its zone repeats or skips (`... it holds an aware datetime in a wall-clock hour its zone repeats or skips, ...`), because either would read back as another value. **When it occurs**: ```python @@ -549,7 +549,7 @@ def get_json_data(): **Message** (logged): `L2 cache decrypt/integrity failure (corruption) for ...: ...`, or `(envelope_shape)` in place of `(corruption)` for the envelope-shape refusal described below -With `serializer='auto'`, a stored set or frozenset over the [Set Size Bound](serializers/auto.md#set-size-bound) is handled the same way. Under `@cache` the log line ends in `SerializationError` and names no bound, because the log carries only the exception's class; the full text, `Invalid set format: ... a hash-flood bound, not a framing error. ...`, appears only when you call `deserialize()` directly. A forged entry, an entry written by a cachekit version without the bound, or a set whose elements read back equal (pandas `Timestamp`s that differ only in nanoseconds, or an aware `datetime` at `fold=1`) causes it; the function recomputes as for any corrupt entry. +With `serializer='auto'`, a stored set or frozenset over the [Set Size Bound](serializers/auto.md#set-size-bound) is handled the same way. Under `@cache` the log line ends in `SerializationError` and names no bound, because the log carries only the exception's class; the full text, `Invalid set format: ... a hash-flood bound, not a framing error. ...`, appears only when you call `deserialize()` directly. A forged entry or an entry an earlier cachekit version wrote causes it: a version without the bound, or one that accepted a set whose elements read back equal (pandas `Timestamp`s that differ only in nanoseconds, or an aware `datetime` in a wall-clock hour its zone repeats or skips), which the writer now refuses. The function recomputes as for any corrupt entry. **Exception**: none under `@cache` — cachekit attempts to evict the entry (best effort; a failed delete is logged and the entry stays) and the function recomputes. Calling a serializer's `deserialize()` directly raises `SerializationError` (there is no separate `DeserializationError` class). diff --git a/docs/serializers/auto.md b/docs/serializers/auto.md index ee9f8884..a89e7258 100644 --- a/docs/serializers/auto.md +++ b/docs/serializers/auto.md @@ -30,13 +30,15 @@ How the default `StandardSerializer` (MessagePack) and `AutoSerializer` handle P | `tuple` | → `list` | Preserved | | `set` | Raises `TypeError` | Preserved | | `frozenset` | Raises `TypeError` | Preserved | -| `datetime` | Preserved | Preserved (ISO-8601 roundtrip) | +| `datetime` | Preserved | Preserved (ISO-8601 roundtrip, to the microsecond) | | `date` / `time` | Preserved | Preserved | | `UUID` | Raises `TypeError` | Preserved | | `numpy.ndarray` | Not supported | Preserved (zero-copy binary) | | `pandas.DataFrame` | Not supported | Preserved (columnar format), top-level value only | | `pandas.Series` | Not supported | Preserved, top-level value only | +A pandas `Timestamp` is a `datetime` and is stored as one: it reads back as a `datetime`, exact to the microsecond. A `Timestamp` with non-zero nanoseconds raises `TypeError` at write, bare or inside any container, because the read would drop them and return another value. Round it first (`ts.floor("us")`) or cache its int nanoseconds (`ts.value`). `pd.NaT` is written without an error, but no read can parse it, so every read of it raises `SerializationError`, a miss under `@cache`. + A DataFrame or Series is encoded only as the whole cached value. Nested inside a dict, list or other container it raises `TypeError`; return it on its own, or convert it first (for example with `df.to_dict("list")`). A NumPy array has no such limit and can sit at any depth. `TypeError` is what the serializer raises when you call it directly. Under `@cache` with a backend, the decorated function's result is returned but not cached, and the failure is logged: see [Serialization unsupported type](../error-codes.md#serialization-unsupported-type). @@ -71,7 +73,7 @@ Without pyarrow, DataFrames use the columnar MessagePack format. If pyarrow is i A set or frozenset holds at most 1,024 elements that are neither strings nor bytes, at most 8 of those that share one hash value, and the empty string `""` and empty bytes `b""` at most once each. No two of its frozensets may share a hash, and no two of its tuples may once a frozenset sits anywhere inside one of its tuples. Writing a set over any of these bounds raises `TypeError`; under `@cache` the result is returned but not cached, and the failure is logged (see [Serialization unsupported type](../error-codes.md#serialization-unsupported-type)). Reading one, which a forged entry or an earlier cachekit version can hold, raises `SerializationError`, which a cached read treats as a miss. -A set whose elements change their hash on the round trip can cross these bounds only once read back, so the writer accepts it and every read refuses it. pandas `Timestamp`s that differ only in nanoseconds read back as one `datetime`, so more than 8 of them, or two frozensets of them, are refused, while up to 8 bare ones collapse into one element without an error. An aware `datetime` at `fold=1` reads back with its offset but without `fold`, so it can come back equal to another element of the set. +The writer also refuses elements that the round trip could make equal to another element, because such a set could cross these bounds only once read back, or read back with fewer elements than it held. A pandas `Timestamp` with nanoseconds raises `TypeError` wherever it sits (see [What It Preserves](#what-it-preserves)): `Timestamp`s that differ only in nanoseconds would read back as one `datetime`. A set or frozenset that holds an aware `datetime` whose UTC offset depends on `fold`, as an element or anywhere inside one of its tuples, raises `TypeError` too. That is a wall time its zone repeats or skips, such as 01:30 on the night New York leaves daylight saving time, at either `fold`. Python never finds such a `datetime` equal to one in another zone, but it reads back with its offset and without `fold`, so it can come back equal to another element. Convert it to UTC first (`dt.astimezone(timezone.utc)`). The same `datetime` outside a set is stored as before. An earlier cachekit version wrote such sets without an error: reading one either refuses it as above or returns fewer elements than were written. The count takes every element that is not a string or bytes, whatever its type: an int, a float, a bool, `None`, a UUID, a tuple, a frozenset or a datetime. Ints, floats, UUIDs and tuples hash without a random seed, so a forged entry could choose colliding element hashes that make one read quadratic in the set's size. A stored set is a list, so it can also repeat an element, and every copy counts toward both bounds. Strings and bytes hash under a per-process secret, so a set of them is bounded only by the size bounds every read applies, except `""` and `b""`, which hash to 0 in every process. The secret holds while Python's hash randomization is on, its default: a process run with `PYTHONHASHSEED` set to a fixed value hashes strings and bytes predictably, so a forged entry could then flood a set of them too. diff --git a/src/cachekit/serializers/auto_serializer.py b/src/cachekit/serializers/auto_serializer.py index 6963bcc2..4284d4a7 100644 --- a/src/cachekit/serializers/auto_serializer.py +++ b/src/cachekit/serializers/auto_serializer.py @@ -30,7 +30,7 @@ import operator import sys from collections.abc import Iterable -from datetime import date, datetime, time +from datetime import date, datetime, time, timezone from functools import cached_property from importlib.util import find_spec from itertools import chain, compress, repeat @@ -82,6 +82,9 @@ _MAX_NON_STRING_SET_ELEMENTS = 1024 _MAX_SAME_HASH_SET_ELEMENTS = 8 _STR_TYPES = frozenset({str, bytes}) +_TZINFO = operator.attrgetter("tzinfo") +# Naive, or a fixed offset: no tzinfo of these types makes a datetime's UTC offset depend on fold. +_FIXED_ZONES = frozenset({type(None), timezone}) _EMPTY_TWICE = 'the empty str or bytes ("" or b"") more than once' # Error message constants for unsupported types @@ -316,6 +319,16 @@ def _auto_default(obj: Any) -> Any: # Existing: datetime/date/time support (KEEP) if isinstance(obj, datetime): + # A pandas Timestamp can hold nanoseconds: its ISO text keeps them and fromisoformat drops them, so it would + # read back as another value, and set elements 1 ns apart would read back equal. pd.NaT is a datetime but no + # Timestamp, so it never reaches the check (its nanosecond is nan). + pd_loaded = sys.modules.get("pandas") + if pd_loaded is not None and isinstance(obj, pd_loaded.Timestamp) and cast("pd.Timestamp", obj).nanosecond: + raise TypeError( + f"AutoSerializer cannot store this {type(obj).__name__}: it has nanoseconds, and a datetime is stored to " + "the microsecond, so sub-microsecond precision is not kept. Round it first (ts.floor('us')) or cache its " + "int nanoseconds (ts.value)." + ) return {"__datetime__": True, "value": obj.isoformat()} if isinstance(obj, date): return {"__date__": True, "value": obj.isoformat()} @@ -341,6 +354,14 @@ def _auto_default(obj: Any) -> Any: f"AutoSerializer cannot store this {type(obj).__name__}: it holds {reason}, which its reader refuses " "as a hash-flood bound. Cache the elements as a list instead." ) + # exposed is 0 only when every element is an exact str or bytes, which holds no datetime. + if exposed and _fold_shifts_offset(value, types): + raise TypeError( + f"AutoSerializer cannot store this {type(obj).__name__}: it holds an aware datetime in a wall-clock " + "hour its zone repeats or skips, which is stored with its UTC offset but not fold, so it can read back " + "equal to another element. Convert it to UTC first (dt.astimezone(timezone.utc)) or cache the elements " + "as a list." + ) return {"__set__": True, "value": value, "frozen": isinstance(obj, frozenset)} # NumPy array support (nested in dicts/lists via msgpack custom encoder) @@ -378,6 +399,30 @@ def _frozenset_in_tuples(values: list[Any], types: list[type]) -> bool: return True +def _fold_shifts_offset(values: list[Any], types: list[type]) -> bool: + """Whether an aware datetime whose UTC offset depends on fold (a wall time its zone repeats or skips) sits among + ``values`` or at any depth inside a tuple among them. ``types`` holds each value's type. + + At either fold, such a datetime never equals one in another zone (PEP 495), but it reads back at a fixed offset, + without fold, and can then equal one. A frozenset among ``values`` is checked when its own encode runs. Walked a + level at a time in C passes, as _frozenset_in_tuples is: only the datetimes outside a fixed-offset zone (UTC + included) reach a Python check. + """ + while values: + kinds = set(types) + if any(issubclass(t, datetime) for t in kinds): + stamps = list(compress(values, map(issubclass, types, repeat(datetime)))) + zones = list(map(type, map(_TZINFO, stamps))) + zoned = compress(stamps, map(operator.not_, map(_FIXED_ZONES.__contains__, zones))) + if any(d.utcoffset() != d.replace(fold=1 - d.fold).utcoffset() for d in zoned): + return True + if not any(issubclass(t, tuple) for t in kinds): + return False + values = list(chain.from_iterable(compress(values, map(issubclass, types, repeat(tuple))))) + types = list(map(type, values)) + return False + + def _hash_run_over(values: Iterable[Any], limit: int) -> bool: """Whether more than ``limit`` of ``values`` share one hash. diff --git a/tests/unit/test_auto_serializer_new_types.py b/tests/unit/test_auto_serializer_new_types.py index 0a855049..d3eb6833 100644 --- a/tests/unit/test_auto_serializer_new_types.py +++ b/tests/unit/test_auto_serializer_new_types.py @@ -13,8 +13,10 @@ import itertools from collections.abc import Callable -from datetime import datetime, timedelta +from datetime import UTC, datetime, timedelta +from typing import Any from uuid import UUID +from zoneinfo import ZoneInfo import msgpack import pytest @@ -228,8 +230,9 @@ class TestAutoSerializerSetHashFloodBound: """A stored set holds at most _MAX_NON_STRING_SET_ELEMENTS elements that are neither str nor bytes, at most _MAX_SAME_HASH_SET_ELEMENTS on one hash, and no two frozensets on one hash (nor two tuples, once a frozenset sits in a tuple of the set). Each refusal comes before set() builds the set, so a forged one never pays for a quadratic - build. The writer refuses the same sets, so an honest one fails at write, not on every read (unless its elements - only share a hash once read back, as pandas Timestamps that differ only in nanoseconds do).""" + build. The writer refuses the same sets, so an honest one fails at write, not on every read. It also refuses the + elements the round trip could make equal (TestAutoSerializerDatetimeRoundTrip), so no set it writes crosses a + bound only once read back.""" @pytest.fixture def counts(self, monkeypatch: pytest.MonkeyPatch) -> dict[str, int]: @@ -266,15 +269,15 @@ def test_a_same_hash_set_is_refused_before_any_comparison( # Hashed once each, by the sort; set() would have compared each to every earlier one. assert counts == {"hash": len(_SAME_HASH), "eq": 0} - def test_timestamps_that_read_back_equal_are_refused_on_read_past_the_bounds(self) -> None: - """The writer sees distinct nanosecond Timestamps; the reader sees equal datetimes, so equal hashes.""" + def test_timestamps_that_would_read_back_equal_are_refused_at_write(self) -> None: + """Timestamps 1 ns apart are distinct to the writer but would read back as one datetime: 8 of them would + collapse into one element, and 9, or two frozensets of them, would fail every read.""" pd = pytest.importorskip("pandas") stamps = [pd.Timestamp("2026-01-01") + pd.Timedelta(i, "ns") for i in range(9)] serializer = AutoSerializer() - for value in (set(stamps), {frozenset(stamps[:1]), frozenset(stamps[1:2])}): - with pytest.raises(SerializationError, match="for a shared hash, its elements read back equal"): - serializer.deserialize(*serializer.serialize(value)) - assert serializer.deserialize(*serializer.serialize(set(stamps[:8]))) == {datetime(2026, 1, 1)} + for value in (set(stamps[:8]), set(stamps), {frozenset(stamps[:1]), frozenset(stamps[1:2])}): + with pytest.raises(TypeError, match="sub-microsecond precision is not kept"): + serializer.serialize(value) def test_eight_elements_may_share_a_hash_but_not_nine(self) -> None: """hash(-1) == hash(-2), so 3-D offsets over -1 and -2 put 8 honest tuples on one hash; 4-D offsets put 16.""" @@ -433,6 +436,88 @@ def test_the_writer_refuses_a_set_over_the_count(self, kind: type) -> None: AutoSerializer().serialize({"v": [kind(range(_MAX_NON_STRING_SET_ELEMENTS + 1))]}) +_NEW_YORK = ZoneInfo("America/New_York") +# 01:30 happens twice in New York on 2025-11-02: fold=1 is the second, at -05:00, the same instant as _UTC_TWIN. +_FOLDED = datetime(2025, 11, 2, 1, 30, fold=1, tzinfo=_NEW_YORK) +_UTC_TWIN = datetime(2025, 11, 2, 6, 30, tzinfo=UTC) +# The first 01:30 that day, fold=0, at -04:00; and 02:30 on 2025-03-09, a wall time New York skips. +_AMBIGUOUS = datetime(2025, 11, 2, 1, 30, tzinfo=_NEW_YORK) +_SKIPPED = datetime(2025, 3, 9, 2, 30, tzinfo=_NEW_YORK) +_FOLDED_MARKER = b"\x82\xac__datetime__\xc3\xa5value\xb92025-11-02T01:30:00-05:00" +_MICRO_MARKER = b"\x82\xac__datetime__\xc3\xa5value\xba2026-01-01T12:00:00.123456" + + +class TestAutoSerializerDatetimeRoundTrip: + """A datetime is stored as its ISO text, which reads back to the microsecond and without fold. The writer refuses a + value that would read back as another one, and writes every other datetime as it always has.""" + + @pytest.mark.parametrize( + "wrap", + [lambda t: t, lambda t: [t], lambda t: {"k": t}, lambda t: {t}, lambda t: frozenset({t})], + ids=["bare", "list", "dict-value", "set", "frozenset"], + ) + def test_a_timestamp_with_nanoseconds_is_refused_at_write(self, wrap: Callable[[Any], Any]) -> None: + pd = pytest.importorskip("pandas") + with pytest.raises(TypeError, match=r"cannot store this Timestamp: .*sub-microsecond precision is not kept"): + AutoSerializer().serialize(wrap(pd.Timestamp("2026-01-01T12:00:00.123456789"))) + + def test_nat_is_not_refused_as_sub_microsecond(self) -> None: + """pd.NaT is a datetime whose nanosecond is nan, and no Timestamp: the guard leaves it alone.""" + pd = pytest.importorskip("pandas") + serializer = AutoSerializer(enable_integrity_checking=False) + assert serializer.serialize(pd.NaT)[0] == msgpack.packb({"__datetime__": True, "value": "NaT"}) + assert serializer.serialize([pd.NaT])[0] == msgpack.packb([{"__datetime__": True, "value": "NaT"}]) + + def test_a_timestamp_without_nanoseconds_writes_the_bytes_a_datetime_always_has(self) -> None: + pd = pytest.importorskip("pandas") + serializer = AutoSerializer(enable_integrity_checking=False) + stamp = pd.Timestamp("2026-01-01T12:00:00.123456") + assert serializer.serialize(stamp)[0] == _MICRO_MARKER + assert serializer.serialize(datetime(2026, 1, 1, 12, 0, 0, 123456))[0] == _MICRO_MARKER + assert serializer.deserialize(*serializer.serialize(stamp)) == stamp + + @pytest.mark.parametrize( + "value", + [ + {frozenset({_FOLDED}), frozenset({_UTC_TWIN})}, + {_FOLDED, _UTC_TWIN}, + {(_FOLDED,), (_UTC_TWIN,)}, + {(1, ((_FOLDED, "x"),))}, # however deep in its tuples + {_AMBIGUOUS, _AMBIGUOUS.astimezone(UTC)}, # fold=0 in a repeated hour + {(_AMBIGUOUS,), (_AMBIGUOUS.astimezone(UTC),)}, + {_SKIPPED, _SKIPPED.astimezone(UTC)}, # either fold of a skipped hour + {_SKIPPED.replace(fold=1), _SKIPPED.replace(fold=1).astimezone(UTC)}, + ], + ids=["frozensets", "set", "tuples", "deep-tuple", "fold0", "fold0-tuples", "gap", "gap-fold1"], + ) + def test_a_set_holding_a_fold_dependent_datetime_is_refused_at_write(self, value: set) -> None: + """Written, these read back with fewer elements, or as two equal frozensets that every read refuses: PEP 495 + makes a datetime whose offset depends on fold unequal to any datetime in another zone, at either fold, and the + stored text keeps the offset but not fold.""" + assert _FOLDED != _UTC_TWIN and _FOLDED.isoformat() == _UTC_TWIN.astimezone(_NEW_YORK).isoformat() + assert _AMBIGUOUS != _AMBIGUOUS.astimezone(UTC) and _SKIPPED != _SKIPPED.astimezone(UTC) + with pytest.raises(TypeError, match="it holds an aware datetime in a wall-clock hour its zone repeats or skips"): + AutoSerializer().serialize(value) + + def test_a_folded_datetime_outside_a_set_writes_the_bytes_it_always_has(self) -> None: + serializer = AutoSerializer(enable_integrity_checking=False) + assert serializer.serialize(_FOLDED)[0] == _FOLDED_MARKER + assert serializer.serialize([_FOLDED])[0] == b"\x91" + _FOLDED_MARKER + assert serializer.serialize({"k": _FOLDED})[0] == b"\x81\xa1k" + _FOLDED_MARKER + + def test_fold_that_does_not_move_the_offset_is_not_refused(self) -> None: + """Naive, at a fixed offset, or at a wall time its zone neither repeats nor skips, fold leaves the offset alone, + so the datetime reads back equal, at the same hash.""" + serializer = AutoSerializer() + for value in ( + {datetime(2025, 11, 2, 1, 30, fold=1)}, + {datetime(2025, 11, 2, 1, 30, fold=1, tzinfo=UTC)}, + {datetime(2025, 7, 1, 1, 30, fold=1, tzinfo=_NEW_YORK)}, + {datetime(2025, 7, 1, 1, 30, tzinfo=_NEW_YORK), datetime(2025, 7, 1, 5, 30, tzinfo=UTC)}, + ): + assert serializer.deserialize(*serializer.serialize(value)) == value + + class TestAutoSerializerComplexNesting: """Test nested complex objects with new types."""