fix(core): sort dict keys before msgpack encoding in DictTransformer

[KR-Ravindra]

Tracking issue

Closes flyteorg/flyte#6776

Problem

DictTransformer serializes untyped dicts (and typed dicts with non-str keys) to a msgpack binary scalar. msgpack preserves insertion order, so two dicts that are equal in Python but built in a different key order produce different bytes. flytepropeller hashes the raw literal bytes for the cache key, so logically identical inputs cause spurious cache misses. It shows up in practice with large generated nested config dicts.

dict_to_binary_literal encodes v as-is, so the byte layout depends on the caller's insertion order.

Fix

Option 2 from the issue discussion — sort before encoding, no new dependency. _sort_dict_keys returns a copy with dict keys sorted recursively through nested dicts, lists and tuples, ordered by (type name, key) so mixed int/str keys can still be ordered; if keys cannot be compared at all the original order is kept, so nothing that encodes today starts failing. It is called right before encoder.encode(...).

Only dicts, lists and tuples are rewritten. Because the sorted copy is what gets encoded, to_python_value returns keys in sorted rather than insertion order. Dataclasses, FlyteFile, FlyteDirectory and anything else nested inside are passed through untouched, so mashumaro's SerializableType handling is unchanged, and mashumaro preserves iteration order, so sorting the input is enough to make the output canonical.

Scope: DictTransformer only. A Dict[...] field inside a dataclass goes through DataclassTransformer and is not changed.

test_guess_dict3 compared output bytes against msgpack.dumps() of the insertion-ordered dict, which no longer holds by design; it now decodes and compares the value, still checking the msgpack tag.

test_dict_to_binary_literal_is_independent_of_key_order asserts two permutations of a nested dict produce byte-identical literals and round-trip correctly, plus Dict[int, str] and mixed str/int/None keys.

On the large-dict cost concern from the issue: for a ~200k-leaf dict (2.1 MB msgpack) the sort takes ~78 ms against ~15 ms for the encode — a few times the encode, and well below the network round trips in a task launch.

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

flyteorg/flyte#7075 (closed) explored fixing this in propeller instead.

Links

• [BUG] DictTransformer has non-deterministic serialization causing unexpected cache misses flyte#6776
• [BUG] Spurious cache misses on dicts that are nested (_calculate_cache_key is nondeterministic) flyte#2924 (earlier report of the same behaviour)

Yes, AI-assisted; reviewed by me.

[KR-Ravindra]

Following up after three weeks. DictTransformer msgpack-encodes untyped dicts without sorting keys, so two equal Python dicts built in different insertion orders serialise to different bytes; flytepropeller hashes those bytes for cache lookups, so a cache hit turns into a miss depending on how the dict happened to be constructed. Sorting keys before encoding makes it deterministic. Tests cover the ordering. Would one of @wild-endeavor / @pingsutw have a moment to look?