fix(serializers): refuse AutoSerializer datetimes that read back as another value (LAB-12972)

docs/error-codes.md

@@ -493,7 +493,7 @@ export CACHEKIT_CONNECTION_POOL_SIZE=100
 
 **Exception**: none raised to a `@cache`-decorated caller: the result is returned but not stored in the backend. The exception is `@cache(interop=...)`: there an unsupported return value raises `InteropError` (a `ValueError` subclass, `from cachekit.interop import InteropError`) by design, so a cross-SDK entry is never silently skipped. Calling the serializer directly raises `TypeError`, e.g. `StandardSerializer does not support custom classes (Python-specific types). ...` or `StandardSerializer does not support pandas DataFrames or Series (Python-specific types). ...`
 
-**Cause**: The default serializer handles `None`, `bool`, `int`, `float`, `str`, `bytes`, `list`, `tuple`, `dict`, `datetime`, `date` and `time`. Custom classes, dataclasses and DataFrames are not supported. `serializer='auto'` also refuses a set or frozenset holding more than 1,024 elements that are neither strings nor bytes, or more than 8 of those that share one hash value, or two frozensets that share one, or two tuples that share one in a set where a tuple holds a frozenset (`AutoSerializer cannot store this set: it holds ...`): see [Set Size Bound](serializers/auto.md#set-size-bound).
+**Cause**: The default serializer handles `None`, `bool`, `int`, `float`, `str`, `bytes`, `list`, `tuple`, `dict`, `datetime`, `date` and `time`. Custom classes, dataclasses and DataFrames are not supported. `serializer='auto'` also refuses a set or frozenset holding more than 1,024 elements that are neither strings nor bytes, or more than 8 of those that share one hash value, or two frozensets that share one, or two tuples that share one in a set where a tuple holds a frozenset (`AutoSerializer cannot store this set: it holds ...`): see [Set Size Bound](serializers/auto.md#set-size-bound). It also refuses a pandas `Timestamp` with non-zero nanoseconds anywhere in the value (`AutoSerializer cannot store this Timestamp: it has nanoseconds, ...`), and a set or frozenset holding an aware `datetime` whose UTC offset depends on `fold`, a wall time its zone repeats or skips (`... it holds an aware datetime in a wall-clock hour its zone repeats or skips, ...`), because either would read back as another value.
 
 **When it occurs**:
 ```python
@@ -549,7 +549,7 @@ def get_json_data():
 
 **Message** (logged): `L2 cache decrypt/integrity failure (corruption) for ...: ...`, or `(envelope_shape)` in place of `(corruption)` for the envelope-shape refusal described below
 
-With `serializer='auto'`, a stored set or frozenset over the [Set Size Bound](serializers/auto.md#set-size-bound) is handled the same way. Under `@cache` the log line ends in `SerializationError` and names no bound, because the log carries only the exception's class; the full text, `Invalid set format: ... a hash-flood bound, not a framing error. ...`, appears only when you call `deserialize()` directly. A forged entry, an entry written by a cachekit version without the bound, or a set whose elements read back equal (pandas `Timestamp`s that differ only in nanoseconds, or an aware `datetime` at `fold=1`) causes it; the function recomputes as for any corrupt entry.
+With `serializer='auto'`, a stored set or frozenset over the [Set Size Bound](serializers/auto.md#set-size-bound) is handled the same way. Under `@cache` the log line ends in `SerializationError` and names no bound, because the log carries only the exception's class; the full text, `Invalid set format: ... a hash-flood bound, not a framing error. ...`, appears only when you call `deserialize()` directly. A forged entry or an entry an earlier cachekit version wrote causes it: a version without the bound, or one that accepted a set whose elements read back equal (pandas `Timestamp`s that differ only in nanoseconds, or an aware `datetime` in a wall-clock hour its zone repeats or skips), which the writer now refuses. The function recomputes as for any corrupt entry.
 
 **Exception**: none under `@cache` — cachekit attempts to evict the entry (best effort; a failed delete is logged and the entry stays) and the function recomputes. Calling a serializer's `deserialize()` directly raises `SerializationError` (there is no separate `DeserializationError` class).
 

docs/serializers/auto.md

@@ -30,13 +30,15 @@ How the default `StandardSerializer` (MessagePack) and `AutoSerializer` handle P
 | `tuple` | → `list` | Preserved |
 | `set` | Raises `TypeError` | Preserved |
 | `frozenset` | Raises `TypeError` | Preserved |
-| `datetime` | Preserved | Preserved (ISO-8601 roundtrip) |
+| `datetime` | Preserved | Preserved (ISO-8601 roundtrip, to the microsecond) |
 | `date` / `time` | Preserved | Preserved |
 | `UUID` | Raises `TypeError` | Preserved |
 | `numpy.ndarray` | Not supported | Preserved (zero-copy binary) |
 | `pandas.DataFrame` | Not supported | Preserved (columnar format), top-level value only |
 | `pandas.Series` | Not supported | Preserved, top-level value only |
 
+A pandas `Timestamp` is a `datetime` and is stored as one: it reads back as a `datetime`, exact to the microsecond. A `Timestamp` with non-zero nanoseconds raises `TypeError` at write, bare or inside any container, because the read would drop them and return another value. Round it first (`ts.floor("us")`) or cache its int nanoseconds (`ts.value`). `pd.NaT` is written without an error, but no read can parse it, so every read of it raises `SerializationError`, a miss under `@cache`.
+
 A DataFrame or Series is encoded only as the whole cached value. Nested inside a dict, list or other container it raises `TypeError`; return it on its own, or convert it first (for example with `df.to_dict("list")`). A NumPy array has no such limit and can sit at any depth.
 
 `TypeError` is what the serializer raises when you call it directly. Under `@cache` with a backend, the decorated function's result is returned but not cached, and the failure is logged: see [Serialization unsupported type](../error-codes.md#serialization-unsupported-type).
@@ -71,7 +73,7 @@ Without pyarrow, DataFrames use the columnar MessagePack format. If pyarrow is i
 
 A set or frozenset holds at most 1,024 elements that are neither strings nor bytes, at most 8 of those that share one hash value, and the empty string `""` and empty bytes `b""` at most once each. No two of its frozensets may share a hash, and no two of its tuples may once a frozenset sits anywhere inside one of its tuples. Writing a set over any of these bounds raises `TypeError`; under `@cache` the result is returned but not cached, and the failure is logged (see [Serialization unsupported type](../error-codes.md#serialization-unsupported-type)). Reading one, which a forged entry or an earlier cachekit version can hold, raises `SerializationError`, which a cached read treats as a miss.
 
-A set whose elements change their hash on the round trip can cross these bounds only once read back, so the writer accepts it and every read refuses it. pandas `Timestamp`s that differ only in nanoseconds read back as one `datetime`, so more than 8 of them, or two frozensets of them, are refused, while up to 8 bare ones collapse into one element without an error. An aware `datetime` at `fold=1` reads back with its offset but without `fold`, so it can come back equal to another element of the set.
+The writer also refuses elements that the round trip could make equal to another element, because such a set could cross these bounds only once read back, or read back with fewer elements than it held. A pandas `Timestamp` with nanoseconds raises `TypeError` wherever it sits (see [What It Preserves](#what-it-preserves)): `Timestamp`s that differ only in nanoseconds would read back as one `datetime`. A set or frozenset that holds an aware `datetime` whose UTC offset depends on `fold`, as an element or anywhere inside one of its tuples, raises `TypeError` too. That is a wall time its zone repeats or skips, such as 01:30 on the night New York leaves daylight saving time, at either `fold`. Python never finds such a `datetime` equal to one in another zone, but it reads back with its offset and without `fold`, so it can come back equal to another element. Convert it to UTC first (`dt.astimezone(timezone.utc)`). The same `datetime` outside a set is stored as before. An earlier cachekit version wrote such sets without an error: reading one either refuses it as above or returns fewer elements than were written.
 
 The count takes every element that is not a string or bytes, whatever its type: an int, a float, a bool, `None`, a UUID, a tuple, a frozenset or a datetime. Ints, floats, UUIDs and tuples hash without a random seed, so a forged entry could choose colliding element hashes that make one read quadratic in the set's size. A stored set is a list, so it can also repeat an element, and every copy counts toward both bounds. Strings and bytes hash under a per-process secret, so a set of them is bounded only by the size bounds every read applies, except `""` and `b""`, which hash to 0 in every process. The secret holds while Python's hash randomization is on, its default: a process run with `PYTHONHASHSEED` set to a fixed value hashes strings and bytes predictably, so a forged entry could then flood a set of them too.
 

src/cachekit/serializers/auto_serializer.py

@@ -30,7 +30,7 @@
 import operator
 import sys
 from collections.abc import Iterable
-from datetime import date, datetime, time
+from datetime import date, datetime, time, timezone
 from functools import cached_property
 from importlib.util import find_spec
 from itertools import chain, compress, repeat
@@ -82,6 +82,9 @@
 _MAX_NON_STRING_SET_ELEMENTS = 1024
 _MAX_SAME_HASH_SET_ELEMENTS = 8
 _STR_TYPES = frozenset({str, bytes})
+_TZINFO = operator.attrgetter("tzinfo")
+# Naive, or a fixed offset: no tzinfo of these types makes a datetime's UTC offset depend on fold.
+_FIXED_ZONES = frozenset({type(None), timezone})
 _EMPTY_TWICE = 'the empty str or bytes ("" or b"") more than once'
 
 # Error message constants for unsupported types
@@ -316,6 +319,16 @@ def _auto_default(obj: Any) -> Any:
 
     # Existing: datetime/date/time support (KEEP)
     if isinstance(obj, datetime):
+        # A pandas Timestamp can hold nanoseconds: its ISO text keeps them and fromisoformat drops them, so it would
+        # read back as another value, and set elements 1 ns apart would read back equal. pd.NaT is a datetime but no
+        # Timestamp, so it never reaches the check (its nanosecond is nan).
+        pd_loaded = sys.modules.get("pandas")
+        if pd_loaded is not None and isinstance(obj, pd_loaded.Timestamp) and cast("pd.Timestamp", obj).nanosecond:
+            raise TypeError(
+                f"AutoSerializer cannot store this {type(obj).__name__}: it has nanoseconds, and a datetime is stored to "
+                "the microsecond, so sub-microsecond precision is not kept. Round it first (ts.floor('us')) or cache its "
+                "int nanoseconds (ts.value)."
+            )
         return {"__datetime__": True, "value": obj.isoformat()}
     if isinstance(obj, date):
         return {"__date__": True, "value": obj.isoformat()}
@@ -341,6 +354,14 @@ def _auto_default(obj: Any) -> Any:
                 f"AutoSerializer cannot store this {type(obj).__name__}: it holds {reason}, which its reader refuses "
                 "as a hash-flood bound. Cache the elements as a list instead."
             )
+        # exposed is 0 only when every element is an exact str or bytes, which holds no datetime.
+        if exposed and _fold_shifts_offset(value, types):
+            raise TypeError(
+                f"AutoSerializer cannot store this {type(obj).__name__}: it holds an aware datetime in a wall-clock "
+                "hour its zone repeats or skips, which is stored with its UTC offset but not fold, so it can read back "
+                "equal to another element. Convert it to UTC first (dt.astimezone(timezone.utc)) or cache the elements "
+                "as a list."
+            )
         return {"__set__": True, "value": value, "frozen": isinstance(obj, frozenset)}
 
     # NumPy array support (nested in dicts/lists via msgpack custom encoder)
@@ -378,6 +399,30 @@ def _frozenset_in_tuples(values: list[Any], types: list[type]) -> bool:
             return True
 
 
+def _fold_shifts_offset(values: list[Any], types: list[type]) -> bool:
+    """Whether an aware datetime whose UTC offset depends on fold (a wall time its zone repeats or skips) sits among
+    ``values`` or at any depth inside a tuple among them. ``types`` holds each value's type.
+
+    At either fold, such a datetime never equals one in another zone (PEP 495), but it reads back at a fixed offset,
+    without fold, and can then equal one. A frozenset among ``values`` is checked when its own encode runs. Walked a
+    level at a time in C passes, as _frozenset_in_tuples is: only the datetimes outside a fixed-offset zone (UTC
+    included) reach a Python check.
+    """
+    while values:
+        kinds = set(types)
+        if any(issubclass(t, datetime) for t in kinds):
+            stamps = list(compress(values, map(issubclass, types, repeat(datetime))))
+            zones = list(map(type, map(_TZINFO, stamps)))
+            zoned = compress(stamps, map(operator.not_, map(_FIXED_ZONES.__contains__, zones)))
+            if any(d.utcoffset() != d.replace(fold=1 - d.fold).utcoffset() for d in zoned):
+                return True
+        if not any(issubclass(t, tuple) for t in kinds):
+            return False
+        values = list(chain.from_iterable(compress(values, map(issubclass, types, repeat(tuple)))))
+        types = list(map(type, values))
+    return False
+
+
 def _hash_run_over(values: Iterable[Any], limit: int) -> bool:
     """Whether more than ``limit`` of ``values`` share one hash.