Repository navigation
Honor SOURCE_DATE_EPOCH as the clock source for generated timestamps #9
Description
Problem
sil-lift currently generates no timestamps of its own. dateCreated/dateModified are read from the source and written back verbatim; the only other root attributes are version (fixed at 0.13) and producer (caller-supplied). Serialization is therefore fully deterministic today — the same input plus the same in-memory edits always produce the same bytes, which is what makes the byte-exact fidelity contract testable.
Any feature that generates a timestamp from the wall clock breaks that. A dateModified stamped with datetime.now(...) means two runs of the same script over the same input produce different bytes, which:
- makes byte-exact round-trip tests of the stamping feature itself unwritable without monkeypatching the clock;
- defeats diff-based CI gates that assert an export is unchanged;
- makes container/Action runs non-reproducible, which matters because sil-lift ships both a
Dockerfileand a GitHub Action (action.yml) explicitly aimed at pipeline use.
Proposed solution
Honor SOURCE_DATE_EPOCH as the default clock source anywhere the library generates a timestamp, with explicit arguments always taking precedence.
Precedence: an explicit when= argument > SOURCE_DATE_EPOCH > wall clock.
def _default_now() -> datetime:
"""The clock for generated timestamps: SOURCE_DATE_EPOCH if usable, else now."""
raw = os.environ.get("SOURCE_DATE_EPOCH")
if raw is not None:
try:
return datetime.fromtimestamp(int(raw.strip()), tz=timezone.utc)
except (ValueError, OverflowError, OSError):
pass # a malformed value is not honored; fall through to the wall clock
return datetime.now(timezone.utc).replace(microsecond=0)Two properties make this a clean fit:
SOURCE_DATE_EPOCHis integer POSIX seconds, so the result is naturally seconds-precision — no truncation needed, and no fractional-second output to worry about._fmt_dateinsrc/sil_lift/_writer.pyalready rewrites+00:00toZ, so an aware UTC datetime serializes asYYYY-MM-DDTHH:MM:SSZ. That is the shape real FieldWorks exports use: across the seven FLEx 8.3–9.0 exports in The Combine'sBackend.Tests/Assets, all 70,636dateCreated/dateModifiedliterals are exactly that 20-character form, with no bare dates, numeric offsets, or fractional seconds.
The variable is already set by Debian, Nix, Buildah, setuptools, and Sphinx among others, so pipelines that set it for other tools get sil-lift determinism for free rather than needing sil-lift-specific configuration.
Dependency
This is only observable once something in sil-lift generates a timestamp; nothing does today. It is a prerequisite or companion to any timestamp-stamping API rather than an independently visible change — and arguably a prerequisite rather than a follow-up, since it is what makes byte-exact tests of such a feature possible without clock monkeypatching.
Open questions
- Malformed values: the spec permits a consumer to either error or ignore. The sketch above ignores and falls through, which keeps a library from raising on an environment variable the caller may not control — but it means a typo silently reintroduces nondeterminism. An alternative is to ignore it but surface a
Problem-style warning, which fits the existing validation vocabulary. - Scope of "generated timestamp":
dateModifiedonly, ordateCreatedtoo? Both are generated in a from-scratch export, so both presumably. - Forcing the wall clock: with the env var set, is passing
when=datetime.now(timezone.utc)explicitly the sanctioned escape hatch, or is a dedicated override wanted? - CLI surface: whether the
validate/exportcommands should also accept an explicit--nowvalue, which would document the behavior in--helpwhere an environment variable is invisible. - Documentation home: the reproducibility guarantee belongs next to the fidelity contract in
docs/en/fidelity.md, since that is where readers go for byte-exactness claims.
Alternatives considered
- A sil-lift-specific variable (e.g.
SIL_LIFT_NOW). Rejected: it duplicates an established cross-ecosystem standard, and a pipeline already pinningSOURCE_DATE_EPOCHwould have to learn a second name for the same concept. - An injectable clock parameter and nothing else. A
when=parameter should exist regardless, but it does not help when the timestamp is generated inside a call the pipeline does not control — the CLI subcommands and the Docker/Action entrypoint being the concrete cases here. - Making generated timestamps opt-in only, and declaring reproducibility the caller's problem. Viable, but it pushes every pipeline that wants both stamping and determinism into monkeypatching or post-processing, and it leaves the library's own test suite without a supported way to exercise stamping byte-exactly.
Activity
imnasnainaec commentedon Sep 3, 2026
#43 makes the library generate timestamps, so the dependency here is satisfied — this is now independently observable rather than waiting on a stamping API.
It also narrows the change. The clock is read in exactly one place,
default_now()in_writer.py, with aresolve_when()sibling handling an explicitwhen=(normalized to UTC whole seconds; a naive value is refused, since UTC and local would put it hours apart). So the precedence this issue proposes — explicitwhen=>SOURCE_DATE_EPOCH> wall clock — is a change todefault_now()alone, and the escape-hatch question above answers itself: an awarewhen=already overrides everything.One question to add. A reviewer on #43 asked for generated stamps to advance past an entry's previous
dateModified, so that two edits saved inside the same second stay distinguishable. I deferred it: it would have to order naive and bare-datevalues that currently can't be compared, and a date ahead of the clock would ratchet further out on every edit. If it is ever taken up, it belongs with the clock, and it needs an answer here first —SOURCE_DATE_EPOCHis a default source rather than an explicit argument, so "guard the wall clock only" is ambiguous until this issue decides whether a pinned clock counts as explicit. A reproducible build presumably wants no adjustment at all.