@@ -14,6 +14,12 @@ A Python package for versioning and monitoring dataset changes throughout the ma
+
+
+
+Exemplary PDF report (1st page only)
+
+
## Overview
`dstrack` helps data scientists and ML engineers track how datasets evolve over time, catching schema drift, distribution shifts, and unexpected mutations before they silently break pipelines or degrade model performance.
@@ -84,15 +90,37 @@ Each snapshot captures the file's schema, a content fingerprint, and per-column
statistics. See the [Getting Started guide](https://leoyala.github.io/dstrack/getting_started/)
for options such as `--name`, `--reader`, and `--dataset-id`.
+**4. See what changed** - `dstrack log` shows *that* a dataset changed;
+`dstrack diff` shows *what* changed:
+
+```bash
+dstrack diff data.csv
+```
+
+The two snapshots are shown side by side, with the schema changes, the dataset's own
+numbers, and every column whose statistics moved, each colored by how much it should
+worry you. Add `--json` for a pipeline to read, or `--pdf report.pdf` for a formatted
+report to share (needs the `report` extra).
+
+It is a library too, so a comparison can be done entirely from Python:
+
+```python
+from dstrack.diff import diff_snapshots
+
+result = diff_snapshots(old_payload, new_payload)
+print(result.severity, result.summary)
+```
+
## Features
- **Dataset versioning** - snapshot a dataset and track its lineage across pipeline stages
- **Rich snapshots** - schema hash, content fingerprint, and per-column statistics (ranges, null rates, and more)
+- **Snapshot comparison** - schema, statistics and distribution-shift diffs, in the terminal, as JSON, or as a PDF report
- **CSV out of the box** - pure standard-library reader, no heavy dependencies
- **Lightweight CLI** - a small, git-like local store you can commit alongside your code
## Roadmap
-Change detection and drift monitoring are on the way - comparing snapshots, surfacing
-schema and distribution shifts, and failing CI on drift. See the
-[roadmap](docs/roadmap.md) for what's planned.
+Drift monitoring in CI is next - comparing a working file against its latest snapshot
+and failing the build when data drifts. See the [roadmap](docs/roadmap.md) for what's
+planned.
diff --git a/docs/API/diff.md b/docs/API/diff.md
new file mode 100644
index 0000000..63df517
--- /dev/null
+++ b/docs/API/diff.md
@@ -0,0 +1,5 @@
+---
+icon: lucide/git-compare
+---
+
+::: dstrack.diff
diff --git a/docs/assets/diff-report.png b/docs/assets/diff-report.png
new file mode 100644
index 0000000..8458f1a
Binary files /dev/null and b/docs/assets/diff-report.png differ
diff --git a/docs/decisions/0004-diff-semantics-and-severity.md b/docs/decisions/0004-diff-semantics-and-severity.md
new file mode 100644
index 0000000..acb86bf
--- /dev/null
+++ b/docs/decisions/0004-diff-semantics-and-severity.md
@@ -0,0 +1,185 @@
+---
+icon: lucide/git-compare
+---
+
+# ADR-0004: Diff semantics and severity classification
+
+## Status
+:lucide-circle-check: Accepted
+
+## Context
+[ADR-0001](0001-dataset-snapshot-content.md) defines what a snapshot contains and
+[ADR-0003](0003-local-snapshot-store-layout.md) where snapshots live. Neither says what
+it means for two of them to *differ*, which is the whole of the roadmap's "Diff Engine"
+milestone and the question `dstrack diff` exists to answer.
+
+Comparing two snapshots is not one problem but three, and they cannot be answered the
+same way:
+
+1. **Structural.** Is a column still there, and does it still hold the same type? This
+ is exact: nothing is inferred and nothing is thresholded.
+2. **Scalar.** A number moved - is that worth telling anyone? Every answer here is a
+ judgement against a threshold, and the right threshold depends on what kind of
+ number it is.
+3. **Distributional.** Do the values still fall where they used to? No single scalar
+ answers this, and what *can* be asked depends on which statistics the snapshot
+ happened to record.
+
+These choices are expensive to revisit once users depend on them. A threshold that
+changes silently makes yesterday's clean run today's alarm, and a JSON key that changes
+meaning breaks every pipeline reading it.
+
+## Decision
+
+### A single severity vocabulary
+Every comparison, at every level, is classified as one of `unchanged`, `info`,
+`warning`, or `critical`. Groups fold upward: a column is as serious as its most serious
+measurement, and the diff is as serious as its most serious column. One vocabulary means
+the terminal, the JSON and the PDF all rank things identically, and a caller can filter
+on severity without knowing what produced it.
+
+Structural severities are fixed rather than thresholded, because their consequences are:
+
+| Change | Severity | Why |
+|---|---|---|
+| Column removed | `critical` | Anything reading it by name breaks outright. |
+| Column type changed | `critical` | Readers break, and the old statistics stop being comparable. |
+| Column added | `warning` | Notable, but a consumer that ignores it keeps working. |
+| Nullability changed | `info` | A contract change with no immediate breakage. |
+| Joined `constant_columns` / `high_null_columns` | `warning` | A feature just went dead or mostly-null. |
+| Left either set | `info` | Good news, reported without raising the alarm. |
+
+### Relative thresholds for counts, absolute for fractions
+A count is judged by its relative change; a quantity already expressed as a fraction of
+one is judged by how far it moved in absolute terms.
+
+This is not a stylistic split. A null fraction going from 0.001 to 0.002 has doubled and
+means nothing, and grading it relatively would fire an alarm on two rows in a million.
+Conversely a row count going from 1,000 to 1,200 is a 20% change whatever the absolute
+figure, and grading it absolutely would need a different threshold for every dataset.
+
+A value moving from exactly zero to something has no defined relative scale, and is
+classified `warning`: structurally notable, ungradeable.
+
+Defaults: 10% warns and 50% is critical for relative change; 0.02 and 0.10 absolute for
+fractions. Three named presets - `low`, `normal`, `high` - move all thresholds together,
+and are the only sensitivity control the CLI exposes. Individual thresholds are available
+to the Python API through `DiffOptions`; a dozen float flags would be an unusable
+command line.
+
+### Distribution shift: PSI, with a percentile fallback
+For numeric columns the measure is the **population stability index** over the recorded
+histograms, using the industry-standard bands (0.10 warns, 0.25 is critical).
+
+The two histograms almost never share bin edges, because each snapshot derives its edges
+from its own minimum and maximum. Counts cannot be re-binned exactly, so each source
+bin's count is spread across a shared grid **in proportion to overlap length** - assuming
+uniform density within a bin. This is an approximation, and it can distort the score for
+a heavy-tailed column summarized in twenty equal-width bins. It is accepted because it
+needs no dependency, because the alternative of refusing to compare unaligned histograms
+would refuse almost every real comparison, and because the method is always reported
+alongside the score.
+
+Zero counts are smoothed by crediting every bucket **half an observation** (the Jeffreys
+prior) rather than by a fixed floor. A fixed floor treats an empty bucket in a four-row
+snapshot as being as certainly empty as one in a four-million-row snapshot, and reports
+the first as near-total drift; half an observation is a large share of a tiny sample and
+a negligible share of a large one, which is exactly how confident the emptiness is.
+
+When a histogram cannot describe a shape at all - too few bins, or the single degenerate
+bin a constant column produces - the fallback is the largest percentile move, scaled by
+the baseline's p5-p95 spread. The result carries `ShiftMethod.PERCENTILE_SHIFT` so the
+score is never read on the PSI scale.
+
+### Distribution shift for text: total variation distance
+Text columns record only their top values, so the distance is computed over the union of
+both sides' top values **plus a synthetic bucket carrying the untracked tail**
+(`1 - top_values_coverage`). Without that bucket, a column whose top values held steady
+while everything beneath them churned would score zero.
+
+### No rename inference
+A renamed column is reported as one removal and one addition.
+
+Snapshots carry no row-level identity, so the only available heuristic is "same dtype and
+similar statistics" - which cannot distinguish a rename from a genuine drop-and-add of
+two similar features. The cost of the two errors is not symmetric: reporting a rename as
+a drop-and-add is verbose but true, while reporting a drop-and-add as a rename silently
+corrupts the per-column drift table, which is the part users act on. `renamed` remains
+available as a purely additive key should snapshots ever carry column identity.
+
+### Reference resolution: id, then path, then prefix
+`dstrack diff` reads each argument as a full id first, a tracked path second, and a short
+id prefix last.
+
+The id-before-path precedence is inherited from `dstrack log` and exists for the same
+reason: reading a path first would let any file in the current directory shadow a real
+id. The prefix comes *after* the path so a file whose name happens to be eight hex
+characters keeps the meaning `log` already gives it, and *before* the final failure so a
+mistyped path is reported as a path rather than as an unknown id.
+
+Naming one side leaves the other implied - a snapshot alone is compared against the one
+it succeeded - because that is the question most often asked.
+
+### Comparing two datasets is legal
+Two snapshots from different datasets compare normally. The result carries
+`same_dataset: false` and every renderer says so, but nothing is refused: comparing a
+staging table against a production one is a real use, and the engine cannot know which
+comparisons are meaningful.
+
+### Snapshot format versions: refuse only what cannot be read
+Each side's `format_version` is checked against the set of versions this dstrack can
+read (`SUPPORTED_FORMAT_VERSIONS`), not against the one version it writes. Only a
+version outside that set is refused, which in practice means a snapshot written by a
+newer dstrack: guessing at a shape the engine has never seen would produce a diff that
+looks authoritative and is not. A payload recording no version is compared as far as its
+fields allow, like any other hand-built dict.
+
+Matching the written version exactly would be simpler and wrong. Snapshots stay in a
+store indefinitely, so the first format bump would make every existing snapshot
+incomparable - with each other and with the new ones - and the baseline-versus-latest
+comparison users most want after an upgrade would be the one refused. A format bump
+therefore **adds** to the supported set rather than replacing its member, and comes with
+whatever lets the engine read the older shape, whether that is the extractors reading
+both or an upgrade step applied to old payloads before comparison. Two snapshots in
+different supported versions compare normally.
+
+This is a different version from the one below. `format_version` describes the
+snapshot the engine reads; `diff_format_version` describes the result it writes. The
+two move independently, and only the first ever causes a refusal.
+
+### `diff_format_version` and an additive-only promise
+The JSON output carries `diff_format_version`, currently `"1"`. Within a version, keys
+may be **added** but are never renamed, removed, or given a new meaning. Consumers are
+expected to ignore keys they do not recognize.
+
+### Tolerating what a payload does not carry
+Every field is read defensively; a missing or malformed one degrades to "not recorded"
+rather than raising. A metric neither side recorded is omitted from the result entirely.
+
+This is what lets the engine accept dictionaries a caller built by hand, and it is also
+why the diff does **not** require `near_duplicate_estimate`, `row_minhash` or
+`row_hyperloglog`. Those are listed as required in `snapshot_v1.json` but no current
+builder produces them; the gap is pre-existing and out of scope here. Nobody should
+"fix" the diff engine to demand them.
+
+Which statistics variant a column holds is inferred from **the keys actually present**,
+falling back to the declared dtype. Where the two disagree the keys win: the statistics
+are what is about to be compared arithmetically, and trusting a dtype that says
+`float64` over an entry carrying `top_values` would read a dictionary as a number.
+
+## Consequences
+
+- The severity vocabulary and the default thresholds are now a compatibility surface.
+ Changing a default changes what an unchanged pipeline reports, and is a breaking
+ change in everything but name.
+- PSI scores are comparable across dstrack versions only as long as the re-binning and
+ smoothing are unchanged. Both are recorded here for that reason.
+- Every snapshot format bump has to keep the engine able to read the previous format.
+ Dropping a version from `SUPPORTED_FORMAT_VERSIONS` strands every stored snapshot
+ written in it, so a version can only leave the set once migration tooling can rewrite
+ those snapshots.
+- A rename shows up as two changes and will look noisier than users expect. This is a
+ deliberate trade and should be documented wherever the diff output is explained.
+- The diff engine depends on nothing beyond the standard library, so it stays importable
+ anywhere `dstrack` is. Only the PDF report needs the optional `report` extra, and it is
+ imported lazily so its absence cannot break the command.
diff --git a/docs/getting_started.md b/docs/getting_started.md
index 5f39ffc..893c23a 100644
--- a/docs/getting_started.md
+++ b/docs/getting_started.md
@@ -161,6 +161,121 @@ dstrack log
Useful flags: `-n 5` to show only the latest few, `--oneline` to condense each snapshot
to a single line, and `--reverse` to read oldest-first.
+## 5. Compare two snapshots
+
+`dstrack log` says *that* a dataset changed. `dstrack diff` says *what* changed. Point it
+at the dataset and it compares the latest snapshot against the one before it:
+
+```bash
+dstrack diff data.csv
+```
+
+The output puts the two snapshots side by side and colors each difference by how much it
+should worry you: the schema changes first, then the dataset's own numbers, then every
+column whose statistics moved, with a distribution-shift score for each.
+
+Any of these name a side, so you can compare whichever two you mean:
+
+```bash
+dstrack diff a587afd9 0c82297f # two snapshots, by the short ids `log` shows
+dstrack diff --from 0c82297f # that snapshot against where the dataset stands now
+dstrack diff customers.csv orders.csv # the latest snapshot of two datasets
+```
+
+Useful flags: `--oneline` for a single line, `--all` to include what did not change, and
+`--sensitivity low|normal|high` to move the thresholds that decide what counts as a
+significant shift.
+
+For a pipeline, `--json` prints the whole comparison as structured data instead of a
+table:
+
+```bash
+dstrack diff data.csv --json --yes > diff.json
+```
+
+### A shareable report
+
+With the optional `report` extra installed, `--pdf` also writes a formatted report of the
+same comparison - the kind of thing to attach to a review:
+
+```bash
+pip install 'dstrack[report]'
+dstrack diff data.csv --pdf diff-report.pdf
+```
+
+### From Python
+
+The comparison is a library function, not just a command. It takes two snapshot payloads
+as plain dictionaries, so it works on snapshots read out of a store and on ones you built
+yourself:
+
+```python
+from dstrack.diff import diff_snapshots
+from dstrack.paths import resolve_store_root
+from dstrack.store import read_snapshot
+
+store_root = resolve_store_root()
+old = read_snapshot("", "", store_root=store_root)
+new = read_snapshot("", "", store_root=store_root)
+
+result = diff_snapshots(old, new)
+print(result.severity, result.summary)
+for column in result.columns:
+ if column.distribution_shift is not None:
+ print(column.name, column.distribution_shift.score)
+```
+
+??? example "Exemplary bash diff output"
+ ```bash title="Command lind output"
+ ✔ Wrote the comparison report to report.pdf
+ ╭────────────────── dstrack diff ───────────────────╮
+ │ before after │
+ │ snapshot 0d86445c 6f6577b3 │
+ │ name sample sample │
+ │ path data/sample.csv data/sample.csv │
+ │ taken 1 minute ago 55 seconds ago │
+ │ by menjivar menjivar │
+ │ rows 50 54 │
+ │ columns 7 8 │
+ │ source 2dfa4d9c36ae… 5887d2bc5a31… changed │
+ ╰─────────────────── CRITICAL ────────────────────╯
+
+ Schema
+ column before after
+ ────────────────────────────────────────────────
+ + country — string not null
+ ~ age int64 not null int64 null
+
+ Dataset
+ • num rows 50 54 +4 (+8.0%)
+ ⚠ num columns 7 8 +1 (+14.3%)
+
+ Columns
+ Numeric columns
+ column nulls mean std p50 min max uniq shift
+ ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
+ ⚠ id 0 25.5 → 28.11 14.43 → 16.31 25.5 → 27.5 1 50 → 56 50 → 54 PSI 0.23
+ ⚠ age 0 → 0.09259 44.14 → 45.37 18.31 → 18.94 37.5 → 39 18 75 33 → 31 PSI 0.15
+ ❗ balance 0 1,235 → 2,213 684.5 → 1,180 1,226 → 2,192 97.28 → 170.2 2,403 → 4,202 50 → 54 PSI 1.09
+ • is_active 0 0.5 0.5 0.5 0 1 2 PSI 0.00
+ Text columns
+ column nulls uniq coverage avg len shift
+ ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
+ • name 0 28 1 13.88 → 13.93 TVD 0.10 +Margaret Hopper -Katherine Torvalds
+ ⚠ plan 0 3 1 5.9 → 7 TVD 0.18
+ • country (new) — → 0 — → 5 — → 1 — → 2 —
+
+ +4 rows +1 columns 4 drifted
+ ```
+
+??? example "Exemplary PDF diff report"
+
+ 
+ Exemplary PDF report (1st page only)
+
+
+See the [diff API reference](API/diff.md) for the full result type.
+
## Where snapshots live
Each snapshot is a JSON file under its dataset's directory:
@@ -194,6 +309,6 @@ This command creates a synthetic file in a temporary location to measure perform
## What's next
-Change detection and drift monitoring - comparing snapshots to surface schema and
-distribution shifts, and failing CI when data drifts - are planned. See the
-[roadmap](roadmap.md) for the full picture.
+Comparing a working file against its latest snapshot without taking a new one
+(`dstrack status`), and failing CI when data drifts (`--fail-on-drift`), are planned. See
+the [roadmap](roadmap.md) for the full picture.
diff --git a/docs/index.md b/docs/index.md
index bbe6e21..ac04ef5 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -43,6 +43,12 @@ DVC and use `dstrack` alongside it.
+!!! example "Exemplary PDF diff report"
+
+ 
+ Exemplary PDF report (1st page only)
+
+
## Installation
```bash
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 129b768..b6bf184 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -31,11 +31,12 @@ Foundation for all future capabilities: reading a dataset and capturing an immut
Compare two snapshots and surface what changed between them.
-- [ ] Schema diff (added, removed, and type-changed columns)
-- [ ] Statistics diff (value range and distribution shifts)
-- [ ] Basic drift indicators
-- [ ] `dstrack diff` command
-- [ ] Structured JSON output for machine consumption
+- [x] Schema diff (added, removed, and type-changed columns)
+- [x] Statistics diff (value range and distribution shifts)
+- [x] Basic drift indicators
+- [x] `dstrack diff` command
+- [x] Structured JSON output for machine consumption
+- [x] PDF comparison report (optional `report` extra)
## v0.3 - CLI Polish
diff --git a/pyproject.toml b/pyproject.toml
index 36c6d07..baea3a9 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -28,6 +28,11 @@ dependencies = [
"typer>=0.26.8",
]
+[project.optional-dependencies]
+# PDF reporting only. Kept out of the base install so the core stays as
+# dependency-light as it is: most runs never write a report.
+report = ["reportlab>=4.2"]
+
[project.urls]
GitHub = "https://github.com/leoyala/dstrack"
@@ -93,10 +98,18 @@ enable_error_code = ["ignore-without-code", "redundant-expr", "truthy-bool"]
module = [
"numpy.*",
"pandas.*",
+ "reportlab.*",
"sklearn.*",
]
ignore_missing_imports = true
+# -----------------------------------------------------------------------------
+# coverage
+# -----------------------------------------------------------------------------
+[tool.coverage.run]
+# Manual helper for trying out `dstrack diff`; deliberately untested.
+omit = ["sample_data.py"]
+
# -----------------------------------------------------------------------------
# codespell: spell-checking for code, docstrings, and docs
# -----------------------------------------------------------------------------
diff --git a/sample_data.py b/sample_data.py
new file mode 100644
index 0000000..77fd764
--- /dev/null
+++ b/sample_data.py
@@ -0,0 +1,155 @@
+"""Generate and mutate a small CSV for trying out `dstrack diff` by hand.
+
+A development helper, not part of the package: it is not tested and is left
+out of coverage.
+
+ uv run sample_data.py create # write data/sample.csv
+ dstrack track data/sample.csv
+ uv run sample_data.py modify # apply a handful of changes
+ dstrack track data/sample.csv
+ dstrack diff data/sample.csv
+"""
+
+import csv
+import random
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+app = typer.Typer(add_completion=False, no_args_is_help=True)
+
+DATA_DIR = Path("data")
+DEFAULT_FILE = "sample.csv"
+
+FIRST_NAMES = ["Ada", "Alan", "Grace", "Katherine", "Linus", "Margaret", "Dennis"]
+LAST_NAMES = ["Lovelace", "Turing", "Hopper", "Johnson", "Torvalds", "Hamilton"]
+PLANS = ["free", "pro", "enterprise"]
+COUNTRIES = ["NL", "DE", "US", "SV", "JP"]
+
+FileOption = Annotated[
+ str, typer.Option("--file", "-f", help="File name inside the data/ folder.")
+]
+SeedOption = Annotated[
+ int | None, typer.Option(help="Random seed, for a reproducible result.")
+]
+
+
+def _path(file: str) -> Path:
+ DATA_DIR.mkdir(exist_ok=True)
+ return DATA_DIR / file
+
+
+def _row(rng: random.Random, row_id: int) -> dict[str, str]:
+ return {
+ "id": str(row_id),
+ "name": f"{rng.choice(FIRST_NAMES)} {rng.choice(LAST_NAMES)}",
+ "age": str(rng.randint(18, 75)),
+ "balance": f"{rng.uniform(0, 2500):.2f}",
+ "signup_date": f"2023-{rng.randint(1, 12):02d}-{rng.randint(1, 28):02d}",
+ "is_active": rng.choice(["true", "false"]),
+ "plan": rng.choice(PLANS),
+ }
+
+
+def _read(path: Path) -> tuple[list[str], list[dict[str, str]]]:
+ with path.open(newline="") as fh:
+ reader = csv.DictReader(fh)
+ return list(reader.fieldnames or []), list(reader)
+
+
+def _write(path: Path, fields: list[str], rows: list[dict[str, str]]) -> None:
+ with path.open("w", newline="") as fh:
+ writer = csv.DictWriter(fh, fieldnames=fields)
+ writer.writeheader()
+ writer.writerows(rows)
+
+
+@app.command()
+def create(
+ file: FileOption = DEFAULT_FILE,
+ rows: Annotated[int, typer.Option(help="Number of rows to generate.")] = 50,
+ seed: SeedOption = 0,
+) -> None:
+ """Write a fresh sample CSV, overwriting any previous one."""
+ rng = random.Random(seed)
+ data = [_row(rng, i) for i in range(1, rows + 1)]
+ path = _path(file)
+ _write(path, list(data[0]), data)
+ typer.echo(f"Wrote {len(data)} rows to {path}")
+
+
+@app.command()
+def modify(
+ file: FileOption = DEFAULT_FILE,
+ seed: SeedOption = None,
+ schema: Annotated[
+ bool, typer.Option(help="Also change the schema (add/drop a column).")
+ ] = True,
+) -> None:
+ """Apply a few simple changes to the sample CSV, in place."""
+ path = _path(file)
+ if not path.exists():
+ typer.echo(f"{path} does not exist; run `create` first.", err=True)
+ raise typer.Exit(1)
+
+ rng = random.Random(seed)
+ fields, data = _read(path)
+ changes: list[str] = []
+
+ # Drop a few rows.
+ dropped = rng.sample(range(len(data)), k=min(3, len(data)))
+ data = [row for i, row in enumerate(data) if i not in dropped]
+ changes.append(f"dropped {len(dropped)} rows")
+
+ # Append new rows, continuing the id sequence.
+ next_id = max((int(row["id"]) for row in data if row.get("id")), default=0) + 1
+ added = rng.randint(3, 8)
+ for i in range(added):
+ new = _row(rng, next_id + i)
+ data.append({field: new.get(field, "") for field in fields})
+ changes.append(f"added {added} rows")
+
+ # Shift the balance distribution upwards.
+ if "balance" in fields:
+ factor = rng.uniform(1.2, 1.8)
+ for row in data:
+ if row["balance"]:
+ row["balance"] = f"{float(row['balance']) * factor:.2f}"
+ changes.append(f"scaled balance by {factor:.2f}")
+
+ # Blank out some ages to raise the null rate.
+ if "age" in fields:
+ nulled = rng.sample(range(len(data)), k=max(1, len(data) // 10))
+ for i in nulled:
+ data[i]["age"] = ""
+ changes.append(f"nulled {len(nulled)} ages")
+
+ # Move part of the user base onto a single plan.
+ if "plan" in fields:
+ for row in data:
+ if rng.random() < 0.3:
+ row["plan"] = "enterprise"
+ changes.append("moved ~30% of rows to the enterprise plan")
+
+ # Schema: add a country column the first time, drop it the next.
+ if schema:
+ if "country" in fields:
+ fields.remove("country")
+ for row in data:
+ row.pop("country", None)
+ changes.append("dropped column country")
+ else:
+ fields.append("country")
+ for row in data:
+ row["country"] = rng.choice(COUNTRIES)
+ changes.append("added column country")
+
+ _write(path, fields, data)
+ typer.echo(f"Modified {path}:")
+ for change in changes:
+ typer.echo(f" - {change}")
+
+
+if __name__ == "__main__":
+ app()
diff --git a/src/dstrack/_cli.py b/src/dstrack/_cli.py
index 02af29b..500256d 100644
--- a/src/dstrack/_cli.py
+++ b/src/dstrack/_cli.py
@@ -6,6 +6,7 @@
import typer
from dstrack import __version__, console
+from dstrack._diff import diff
from dstrack._log import log
from dstrack._store_template import STORE_TEMPLATE, materialize
from dstrack._track import track
@@ -95,6 +96,13 @@ def init(
"or its path."
)(log)
+app.command(
+ help="Compare two snapshots and show what changed between them, side by "
+ "side. Each argument may be a snapshot id, a dataset id, or the path to a "
+ "tracked dataset; given one, it is compared against the snapshot it "
+ "succeeded."
+)(diff)
+
@app.command(help="Print package version.")
def version() -> None:
diff --git a/src/dstrack/_diff.py b/src/dstrack/_diff.py
new file mode 100644
index 0000000..9208288
--- /dev/null
+++ b/src/dstrack/_diff.py
@@ -0,0 +1,234 @@
+"""The ``dstrack diff`` command: compare two snapshots of a dataset."""
+
+import json
+import sqlite3
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+from dstrack import cache, console
+from dstrack._diff_render import render_diff
+from dstrack._resolve import ensure_index, resolve_pair
+from dstrack.diff import PRESETS, DiffOptions, SnapshotDiff, diff_snapshots, preset
+from dstrack.errors import (
+ AmbiguousSnapshotError,
+ DatasetNotFoundError,
+ IncompatibleSnapshotError,
+ MissingDependencyError,
+ SnapshotNotFoundError,
+ StoreCorruptionError,
+ StoreNotFoundError,
+)
+from dstrack.paths import resolve_store_root
+
+# Failures that mean "what you asked for is not in the store, or is not
+# readable", all of which are reported the same way: a message, and exit 1.
+_USER_ERRORS = (
+ AmbiguousSnapshotError,
+ DatasetNotFoundError,
+ IncompatibleSnapshotError,
+ SnapshotNotFoundError,
+ StoreCorruptionError,
+ ValueError,
+ OSError,
+ sqlite3.DatabaseError,
+)
+
+
+def diff(
+ refs: Annotated[
+ list[str] | None,
+ typer.Argument(
+ help="Snapshots to compare: a snapshot id (in full or by its "
+ "leading characters), a dataset id, or the path to a tracked "
+ "dataset file. Given one, it is compared against the snapshot it "
+ "succeeded; given two, the first is the baseline.",
+ ),
+ ] = None,
+ from_ref: Annotated[
+ str | None,
+ typer.Option(
+ "--from",
+ help="Baseline to compare against. Alone, the dataset's latest "
+ "snapshot is compared to it.",
+ ),
+ ] = None,
+ to_ref: Annotated[
+ str | None,
+ typer.Option(
+ "--to",
+ help="Snapshot to compare. Alone, it is compared against the one "
+ "it succeeded.",
+ ),
+ ] = None,
+ sensitivity: Annotated[
+ str,
+ typer.Option(
+ "--sensitivity",
+ help="How much movement counts as significant: low, normal, or high.",
+ ),
+ ] = "normal",
+ show_all: Annotated[
+ bool,
+ typer.Option("--all", help="Include columns and metrics that did not change."),
+ ] = False,
+ oneline: Annotated[
+ bool,
+ typer.Option("--oneline", help="Condense the comparison to a single line."),
+ ] = False,
+ as_json: Annotated[
+ bool,
+ typer.Option(
+ "--json",
+ help="Print the comparison as JSON instead of a table. Implies a "
+ "non-interactive run, so pass `--yes` in scripts.",
+ ),
+ ] = False,
+ pdf: Annotated[
+ Path | None,
+ typer.Option(
+ "--pdf",
+ dir_okay=False,
+ writable=True,
+ help="Also write a PDF report of the comparison to this path. "
+ "Needs the `report` extra: pip install 'dstrack[report]'.",
+ ),
+ ] = None,
+ yes: Annotated[
+ bool,
+ typer.Option(
+ "-y",
+ "--yes",
+ help="Build the snapshot index without asking, if it does not exist yet.",
+ ),
+ ] = False,
+ rebuild: Annotated[
+ bool,
+ typer.Option(
+ "--rebuild",
+ help="Discard the snapshot index and rebuild it from the dataset logs.",
+ ),
+ ] = False,
+ root: Annotated[
+ Path | None,
+ typer.Option(
+ "--root",
+ help="Path root a REF path is made relative to, matching the one "
+ "`dstrack track` recorded it with. Defaults to the store root (the "
+ "directory containing .dstrack/). Ignored for a REF that is an id.",
+ ),
+ ] = None,
+) -> None:
+ """Compare two snapshots and report what changed between them."""
+ options = _options(sensitivity)
+
+ # Locate the .dstrack/ store; without one there is nothing to compare.
+ try:
+ store_root = resolve_store_root()
+ except StoreNotFoundError as e:
+ console.error(str(e))
+ raise typer.Exit(code=1) from e
+
+ # Under --json the only thing on stdout may be the document itself, so the
+ # index cannot be offered interactively: consent has to come up front.
+ ensure_index(store_root, yes=yes or as_json, rebuild=rebuild, quiet=as_json)
+
+ try:
+ # Sync before resolving: the index does not travel with the repository,
+ # so a snapshot taken elsewhere is only visible once its log is read.
+ reason = cache.sync(store_root)
+ if reason is not None and not as_json:
+ console.warning(f"Rebuilt the snapshot index: {reason}.")
+
+ old, new = resolve_pair(
+ refs or [],
+ from_ref=from_ref,
+ to_ref=to_ref,
+ store_root=store_root,
+ root=root,
+ )
+ result = diff_snapshots(
+ old.payload,
+ new.payload,
+ options=options,
+ old_dataset_id=old.dataset_id,
+ new_dataset_id=new.dataset_id,
+ )
+ except _USER_ERRORS as e:
+ console.error(str(e))
+ raise typer.Exit(code=1) from e
+
+ if pdf is not None and not _write_pdf(result, pdf, quiet=as_json):
+ raise typer.Exit(code=1)
+
+ if as_json:
+ # Deliberately not `console.display`: the document is data, and rich
+ # would wrap and highlight it into something no parser accepts.
+ print(json.dumps(result.to_dict(), indent=2))
+ return
+
+ if not result.same_dataset:
+ console.warning("Comparing snapshots from two different datasets.")
+
+ console.display(
+ render_diff(
+ result,
+ width=console.width(),
+ now=datetime.now(UTC),
+ oneline=oneline,
+ show_unchanged=show_all,
+ )
+ )
+
+
+def _options(sensitivity: str) -> DiffOptions:
+ """Return the thresholds a named sensitivity stands for.
+
+ Args:
+ sensitivity: The preset name, as the user gave it.
+
+ Returns:
+ Its thresholds.
+
+ Raises:
+ typer.BadParameter: If no preset goes by that name. Raised rather than
+ reported, so the failure reads like any other bad option.
+ """
+ try:
+ return preset(sensitivity)
+ except KeyError as e:
+ known = ", ".join(sorted(PRESETS))
+ raise typer.BadParameter(
+ f"Unknown sensitivity {sensitivity!r}. Choose one of: {known}."
+ ) from e
+
+
+def _write_pdf(result: SnapshotDiff, path: Path, *, quiet: bool) -> bool:
+ """Write the PDF report, reporting a failure rather than raising it.
+
+ The reporting module is imported here rather than at the top of the file
+ because it needs an optional dependency: importing it eagerly would make
+ every `dstrack diff` pay for a feature most runs do not use, and would turn
+ a missing extra into an import error on an unrelated command.
+
+ Args:
+ result: The comparison to report.
+ path: Where to write the PDF.
+ quiet: Suppress the success message, for a run whose stdout is data.
+
+ Returns:
+ Whether the report was written.
+ """
+ from dstrack._diff_report import write_pdf_report
+
+ try:
+ write_pdf_report(result, path, now=datetime.now(UTC))
+ except (MissingDependencyError, OSError) as e:
+ console.error(str(e))
+ return False
+
+ if not quiet:
+ console.success(f"Wrote the comparison report to {path}")
+ return True
diff --git a/src/dstrack/_diff_render.py b/src/dstrack/_diff_render.py
new file mode 100644
index 0000000..374c765
--- /dev/null
+++ b/src/dstrack/_diff_render.py
@@ -0,0 +1,808 @@
+"""Rendering of a snapshot comparison for the terminal.
+
+This is the only layer that knows what a diff looks like on screen. It builds
+renderables and returns them; printing is the caller's job.
+
+Everything is laid out as before-and-after pairs, because that is the question
+being asked: not "what does this snapshot say" but "what does it say that the
+other one did not". A value that did not move is shown once and dimmed, so the
+eye lands on the ones that did, and every changed value is colored by how much
+it should worry the reader rather than by which direction it went -- a rising
+null fraction is not good news because the number went up.
+
+The layout adapts to the terminal in three bands. Wide terminals get the full
+per-column tables side by side; medium ones drop the least informative columns;
+narrow ones abandon the tabular form entirely and stack each column as its own
+small card, since a side-by-side table squeezed into forty cells is not a
+side-by-side table any more.
+
+Every value that reaches a cell is user-derived -- a dataset name, a path, a
+column name, a category from the data itself. Cells are therefore assembled as
+`Text`, never interpolated into rich's markup: a column named ``price[/]usd``
+is not markup to be parsed, and treating it as such raises rather than prints.
+"""
+
+from collections.abc import Sequence
+from datetime import datetime
+from typing import Final
+
+from rich import box
+from rich.console import Group, RenderableType
+from rich.panel import Panel
+from rich.table import Table
+from rich.text import Text
+
+from dstrack._log_render import _humanize
+from dstrack.diff import (
+ ColumnKind,
+ ColumnSchemaChange,
+ ColumnStatDiff,
+ DistributionShift,
+ MetricDiff,
+ Severity,
+ ShiftMethod,
+ SnapshotDiff,
+)
+
+_MISSING: Final = "\N{EM DASH}"
+_ARROW: Final = "\N{RIGHTWARDS ARROW}"
+_ELLIPSIS: Final = "\N{HORIZONTAL ELLIPSIS}"
+_SHORT_ID_LEN: Final = 8
+_HASH_LEN: Final = 12
+
+# Layout bands, in cells.
+_WIDE_MIN_WIDTH: Final = 140
+_SIDE_BY_SIDE_MIN_WIDTH: Final = 100
+
+_MAX_LISTED_COLUMNS: Final = 20
+_MAX_LISTED_CATEGORIES: Final = 5
+
+_SEVERITY_STYLES: Final[dict[Severity, str]] = {
+ Severity.UNCHANGED: "dim",
+ Severity.INFO: "cyan",
+ Severity.WARNING: "yellow",
+ Severity.CRITICAL: "bold red",
+}
+_SEVERITY_GLYPHS: Final[dict[Severity, str]] = {
+ Severity.UNCHANGED: " ",
+ Severity.INFO: "\N{BULLET}",
+ Severity.WARNING: "\N{WARNING SIGN}",
+ Severity.CRITICAL: "\N{HEAVY EXCLAMATION MARK SYMBOL}",
+}
+
+# Metrics where a rise is bad news, so the delta takes its color from the
+# severity rather than from its sign.
+_QUALITY_METRICS: Final[frozenset[str]] = frozenset(
+ {"null_fraction", "null_count", "duplicate_row_fraction"}
+)
+
+# Numeric detail columns, most informative first. Trimmed from the end as the
+# terminal narrows.
+_NUMERIC_PRIORITY: Final[tuple[tuple[str, str], ...]] = (
+ ("null_fraction", "nulls"),
+ ("mean", "mean"),
+ ("std", "std"),
+ ("percentiles.p50", "p50"),
+ ("min", "min"),
+ ("max", "max"),
+ ("num_unique", "uniq"),
+)
+_STRING_COLUMNS: Final[tuple[tuple[str, str], ...]] = (
+ ("null_fraction", "nulls"),
+ ("num_unique", "uniq"),
+ ("top_values_coverage", "coverage"),
+ ("avg_char_length", "avg len"),
+)
+_DATETIME_COLUMNS: Final[tuple[tuple[str, str], ...]] = (
+ ("null_fraction", "nulls"),
+ ("range_days", "range (d)"),
+)
+_OTHER_COLUMNS: Final[tuple[tuple[str, str], ...]] = (
+ ("null_fraction", "nulls"),
+ ("null_count", "null count"),
+)
+
+_KIND_TITLES: Final[dict[ColumnKind, str]] = {
+ ColumnKind.NUMERIC: "Numeric columns",
+ ColumnKind.STRING: "Text columns",
+ ColumnKind.DATETIME: "Datetime columns",
+ ColumnKind.OTHER: "Other columns",
+ ColumnKind.UNKNOWN: "Unrecognized columns",
+}
+_CHANGE_SIGNS: Final[dict[str, tuple[str, str]]] = {
+ "added": ("+", "green"),
+ "removed": ("-", "red"),
+ "type_changed": ("~", "yellow"),
+ "nullability_changed": ("~", "dim"),
+}
+_SHIFT_LABELS: Final[dict[ShiftMethod, str]] = {
+ ShiftMethod.PSI: "PSI",
+ ShiftMethod.PERCENTILE_SHIFT: "pctl",
+ ShiftMethod.TOP_VALUES_TVD: "TVD",
+ ShiftMethod.NONE: "",
+}
+
+
+def render_diff(
+ diff: SnapshotDiff,
+ *,
+ width: int,
+ now: datetime,
+ oneline: bool = False,
+ show_unchanged: bool = False,
+) -> RenderableType:
+ """Render a snapshot comparison.
+
+ Args:
+ diff: The comparison to show.
+ width: The terminal width in cells, which decides the layout band.
+ now: The moment each snapshot's age is measured against.
+ oneline: Condense the whole comparison to a single line.
+ show_unchanged: Include columns whose statistics did not move.
+
+ Returns:
+ A renderable comparison.
+ """
+ if oneline:
+ return _oneline(diff, now=now)
+
+ sections: list[RenderableType] = [_identity(diff, now=now)]
+
+ if diff.identical:
+ sections.append(
+ Text("The two snapshots are identical in everything compared.", "green")
+ )
+ return Group(*sections)
+
+ sections.append(_schema_section(diff))
+
+ metrics = _metrics_table(diff.metrics, show_unchanged=show_unchanged)
+ if metrics is not None:
+ sections.extend([Text(), Text("Dataset", style="bold"), metrics])
+
+ sections.extend(_column_sections(diff, width=width, show_unchanged=show_unchanged))
+ sections.extend([Text(), _summary(diff)])
+ return Group(*sections)
+
+
+# ---------------------------------------------------------------------------
+# Identity
+# ---------------------------------------------------------------------------
+
+
+def _identity(diff: SnapshotDiff, *, now: datetime) -> Panel:
+ """Render the two snapshots' identities, side by side.
+
+ Args:
+ diff: The comparison.
+ now: The moment each snapshot's age is measured against.
+
+ Returns:
+ A panel bordered in the diff's overall severity.
+ """
+ grid = Table.grid(padding=(0, 2))
+ grid.add_column(style="dim", no_wrap=True)
+ # No overflow settings on the value columns: rich's defaults ellipsize
+ # them, where a column told not to wrap is dropped whole when the panel has
+ # to shrink, taking the comparison with it.
+ grid.add_column()
+ grid.add_column()
+
+ grid.add_row("", Text("before", style="dim"), Text("after", style="dim"))
+ grid.add_row(
+ "snapshot",
+ Text(_short(diff.old.snapshot_id), style="bold yellow"),
+ Text(_short(diff.new.snapshot_id), style="bold yellow"),
+ )
+ if not diff.same_dataset:
+ grid.add_row(
+ "dataset",
+ Text(diff.old.dataset_id or _MISSING),
+ Text(diff.new.dataset_id or _MISSING),
+ )
+ grid.add_row(
+ "name",
+ Text(diff.old.dataset_name or _MISSING, style="bold"),
+ Text(diff.new.dataset_name or _MISSING, style="bold"),
+ )
+ grid.add_row(
+ "path",
+ Text(diff.old.dataset_path or _MISSING, style="dim"),
+ Text(diff.new.dataset_path or _MISSING, style="dim"),
+ )
+ grid.add_row(
+ "taken",
+ Text(_humanize(diff.old.created_at, now)),
+ Text(_humanize(diff.new.created_at, now)),
+ )
+ grid.add_row(
+ "by",
+ Text(diff.old.created_by or _MISSING),
+ Text(diff.new.created_by or _MISSING),
+ )
+ grid.add_row(
+ "rows",
+ Text(_count(diff.old.num_rows), style="bold"),
+ Text(_count(diff.new.num_rows), style="bold"),
+ )
+ grid.add_row(
+ "columns",
+ Text(_count(diff.old.num_columns), style="bold"),
+ Text(_count(diff.new.num_columns), style="bold"),
+ )
+ grid.add_row(
+ "source",
+ Text(_hash(diff.old.source_hash), style="dim"),
+ _source_verdict(diff),
+ )
+
+ body: list[RenderableType] = [grid]
+ if not diff.same_dataset:
+ body.extend(
+ [
+ Text(),
+ Text(
+ "These snapshots come from two different datasets.",
+ style="yellow",
+ ),
+ ]
+ )
+
+ return Panel(
+ Group(*body),
+ title=Text("dstrack diff"),
+ subtitle=_badge(diff.severity),
+ border_style=_SEVERITY_STYLES[diff.severity],
+ box=box.ROUNDED,
+ expand=False,
+ )
+
+
+def _source_verdict(diff: SnapshotDiff) -> Text:
+ """Render the later snapshot's source hash and what it says.
+
+ Args:
+ diff: The comparison.
+
+ Returns:
+ The hash, followed by whether the underlying bytes are known to match.
+ """
+ text = Text(_hash(diff.new.source_hash), style="dim")
+ if diff.source_hash_equal:
+ text.append(" identical bytes", style="dim green")
+ elif diff.old.source_hash and diff.new.source_hash:
+ text.append(" changed", style="bold red")
+ return text
+
+
+def _badge(severity: Severity) -> Text:
+ """Render a severity as a short badge.
+
+ Args:
+ severity: The severity to show.
+
+ Returns:
+ The severity's name, styled and spaced to read as a label.
+ """
+ return Text(f" {severity.value.upper()} ", style=_SEVERITY_STYLES[severity])
+
+
+# ---------------------------------------------------------------------------
+# Schema
+# ---------------------------------------------------------------------------
+
+
+def _schema_section(diff: SnapshotDiff) -> RenderableType:
+ """Render the schema differences.
+
+ Args:
+ diff: The comparison.
+
+ Returns:
+ A table of column-level changes, or a note that the schema held still.
+ """
+ schema = diff.schema
+ if not schema.changed:
+ lines: list[RenderableType] = [
+ Text(),
+ Text("Schema", style="bold"),
+ Text(" unchanged", style="dim green"),
+ ]
+ if schema.reordered:
+ lines.append(
+ Text(
+ f" {len(schema.reordered)} column(s) reordered "
+ "(schema hash unchanged)",
+ style="dim",
+ )
+ )
+ return Group(*lines)
+
+ table = Table(box=box.SIMPLE_HEAD, padding=(0, 1), show_edge=False)
+ table.add_column("", no_wrap=True)
+ table.add_column("column")
+ table.add_column("before")
+ table.add_column("after")
+
+ changes = [
+ (kind, change)
+ for kind, group in (
+ ("added", schema.added),
+ ("removed", schema.removed),
+ ("type_changed", schema.type_changed),
+ ("nullability_changed", schema.nullability_changed),
+ )
+ for change in group
+ ]
+ for kind, change in changes[:_MAX_LISTED_COLUMNS]:
+ sign, style = _CHANGE_SIGNS[kind]
+ table.add_row(
+ Text(sign, style=style),
+ Text(change.name, style=_SEVERITY_STYLES[change.severity]),
+ _dtype(change, old=True),
+ _dtype(change, old=False),
+ )
+
+ body: list[RenderableType] = [Text(), Text("Schema", style="bold"), table]
+ if len(changes) > _MAX_LISTED_COLUMNS:
+ hidden = len(changes) - _MAX_LISTED_COLUMNS
+ body.append(Text(f" {_ELLIPSIS} and {hidden} more", style="dim"))
+ return Group(*body)
+
+
+def _dtype(change: ColumnSchemaChange, *, old: bool) -> Text:
+ """Render one side of a column's type, with its nullability.
+
+ Args:
+ change: The column-level change.
+ old: Whether to render the baseline side.
+
+ Returns:
+ The dtype, or an em-dash for the side the column was absent from.
+ """
+ dtype = change.old_dtype if old else change.new_dtype
+ nullable = change.old_nullable if old else change.new_nullable
+ present = (
+ (change.old_dtype is not None or change.old_nullable is not None)
+ if old
+ else (change.new_dtype is not None or change.new_nullable is not None)
+ )
+ if dtype is None and not present:
+ return Text(_MISSING, style="dim")
+
+ text = Text(dtype or _MISSING)
+ if nullable is not None:
+ text.append(" null" if nullable else " not null", style="dim")
+ return text
+
+
+# ---------------------------------------------------------------------------
+# Metrics
+# ---------------------------------------------------------------------------
+
+
+def _metrics_table(
+ metrics: Sequence[MetricDiff], *, show_unchanged: bool
+) -> Table | None:
+ """Render a group of scalar comparisons.
+
+ Args:
+ metrics: The comparisons to show.
+ show_unchanged: Include the ones that did not move.
+
+ Returns:
+ A grid of metric, before, after and delta, or ``None`` if nothing is
+ worth showing.
+ """
+ shown = [
+ m for m in metrics if show_unchanged or m.severity is not Severity.UNCHANGED
+ ]
+ if not shown:
+ return None
+
+ grid = Table.grid(padding=(0, 2))
+ grid.add_column(no_wrap=True)
+ grid.add_column(style="dim", no_wrap=True)
+ grid.add_column(justify="right")
+ grid.add_column(justify="right")
+ grid.add_column(justify="right")
+
+ for metric in shown:
+ grid.add_row(
+ Text(
+ _SEVERITY_GLYPHS[metric.severity],
+ style=_SEVERITY_STYLES[metric.severity],
+ ),
+ _label(metric.key),
+ Text(_value(metric.old), style="dim"),
+ Text(_value(metric.new), style=_SEVERITY_STYLES[metric.severity]),
+ _delta(metric),
+ )
+ return grid
+
+
+def _delta(metric: MetricDiff) -> Text:
+ """Render how far a metric moved.
+
+ Args:
+ metric: The comparison.
+
+ Returns:
+ The signed change and its relative size. A metric where a rise is bad
+ news takes its color from the severity, so that a growing null count is
+ never green; every other metric is colored by direction, as
+ `dstrack log` colors a row-count delta.
+ """
+ if metric.delta is None or metric.delta == 0:
+ return Text("")
+
+ text = Text(f"{metric.delta:+,.4g}", style=_delta_style(metric))
+ if metric.pct_change:
+ text.append(f" ({metric.pct_change:+.1%})", style="dim")
+ return text
+
+
+def _delta_style(metric: MetricDiff) -> str:
+ """Return the style a metric's delta is shown in.
+
+ Args:
+ metric: The comparison.
+
+ Returns:
+ A severity style for quality metrics, else green or red by direction.
+ """
+ if metric.key in _QUALITY_METRICS:
+ return _SEVERITY_STYLES[metric.severity]
+ return "green" if (metric.delta or 0) > 0 else "red"
+
+
+# ---------------------------------------------------------------------------
+# Columns
+# ---------------------------------------------------------------------------
+
+
+def _column_sections(
+ diff: SnapshotDiff, *, width: int, show_unchanged: bool
+) -> list[RenderableType]:
+ """Render the per-column comparisons, grouped by statistics variant.
+
+ One table per variant: the meaningful measurements differ between a numeric
+ column and a text one, and a single merged table would be mostly em-dashes.
+
+ Args:
+ diff: The comparison.
+ width: The terminal width, which decides the layout band.
+ show_unchanged: Include columns whose statistics did not move.
+
+ Returns:
+ The renderables to append, empty if no column is worth showing.
+ """
+ shown = [
+ column
+ for column in diff.columns
+ if show_unchanged or column.severity is not Severity.UNCHANGED
+ ]
+ if not shown:
+ return []
+
+ sections: list[RenderableType] = [Text(), Text("Columns", style="bold")]
+ if width < _SIDE_BY_SIDE_MIN_WIDTH:
+ sections.append(_stacked(shown))
+ return sections
+
+ for kind in _KIND_TITLES:
+ group = [column for column in shown if column.kind is kind]
+ if group:
+ sections.extend(
+ [
+ Text(f" {_KIND_TITLES[kind]}", style="dim"),
+ _column_table(group, kind=kind, width=width),
+ ]
+ )
+ return sections
+
+
+def _column_table(
+ columns: Sequence[ColumnStatDiff], *, kind: ColumnKind, width: int
+) -> Table:
+ """Render one variant's columns as a before-and-after table.
+
+ Args:
+ columns: The columns to show, all of one variant.
+ kind: The variant they share.
+ width: The terminal width, which decides how many measurements fit.
+
+ Returns:
+ A table with one row per column and one cell per measurement, each cell
+ holding both sides.
+ """
+ fields = _fields_for(kind, width=width)
+
+ table = Table(box=box.SIMPLE_HEAD, padding=(0, 1), show_edge=False, expand=False)
+ table.add_column("", no_wrap=True)
+ table.add_column("column")
+ for _, label in fields:
+ table.add_column(label, justify="right")
+ if kind in (ColumnKind.NUMERIC, ColumnKind.STRING):
+ table.add_column("shift")
+
+ for column in columns:
+ cells: list[RenderableType] = [
+ Text(
+ _SEVERITY_GLYPHS[column.severity],
+ style=_SEVERITY_STYLES[column.severity],
+ ),
+ _column_name(column),
+ ]
+ metrics = {metric.key: metric for metric in column.metrics}
+ cells.extend(_pair(metrics.get(key)) for key, _ in fields)
+ if kind in (ColumnKind.NUMERIC, ColumnKind.STRING):
+ cells.append(_shift(column.distribution_shift))
+ table.add_row(*cells)
+ return table
+
+
+def _fields_for(kind: ColumnKind, *, width: int) -> tuple[tuple[str, str], ...]:
+ """Return the measurements to show for a variant at a given width.
+
+ Args:
+ kind: The statistics variant.
+ width: The terminal width in cells.
+
+ Returns:
+ Metric key and column label pairs. Numeric columns carry the most
+ measurements and are the only ones trimmed, dropping the least
+ informative first.
+ """
+ if kind is ColumnKind.STRING:
+ return _STRING_COLUMNS
+ if kind is ColumnKind.DATETIME:
+ return _DATETIME_COLUMNS
+ if kind is not ColumnKind.NUMERIC:
+ return _OTHER_COLUMNS
+ if width >= _WIDE_MIN_WIDTH:
+ return _NUMERIC_PRIORITY
+ return _NUMERIC_PRIORITY[:4]
+
+
+def _column_name(column: ColumnStatDiff) -> Text:
+ """Render a column's name, noting a variant change.
+
+ Args:
+ column: The column's comparison.
+
+ Returns:
+ The name, followed by the variants it moved between when it changed
+ kind -- the one case where the statistics are not comparable at all.
+ """
+ text = Text(column.name, style="bold")
+ if column.old_kind is not None:
+ text.append(
+ f" ({column.old_kind.value} {_ARROW} {column.kind.value})", style="bold red"
+ )
+ elif not column.present_in_new:
+ text.append(" (dropped)", style="red")
+ elif not column.present_in_old:
+ text.append(" (new)", style="green")
+ return text
+
+
+def _pair(metric: MetricDiff | None) -> Text:
+ """Render one measurement as its before-and-after pair.
+
+ Args:
+ metric: The comparison, or ``None`` if this column never recorded it.
+
+ Returns:
+ Both values with an arrow between them when they differ, or the single
+ value dimmed when they do not -- an unchanged number is context, not
+ news, and showing it twice would bury the ones that moved.
+ """
+ if metric is None:
+ return Text(_MISSING, style="dim")
+ if metric.severity is Severity.UNCHANGED:
+ return Text(_value(metric.new), style="dim")
+
+ text = Text(_value(metric.old), style="dim")
+ text.append(f" {_ARROW} ", style="dim")
+ text.append(_value(metric.new), style=_SEVERITY_STYLES[metric.severity])
+ return text
+
+
+def _shift(shift: DistributionShift | None) -> Text:
+ """Render a column's distribution shift.
+
+ Args:
+ shift: The measured shift, or ``None`` if the variant records none.
+
+ Returns:
+ The measure's name and score, followed by the categories that came and
+ went for a text column.
+ """
+ if shift is None or shift.score is None:
+ return Text(_MISSING, style="dim")
+
+ text = Text(
+ f"{_SHIFT_LABELS[shift.method]} {shift.score:.2f}",
+ style=_SEVERITY_STYLES[shift.severity],
+ )
+ for sign, style, categories in (
+ ("+", "green", shift.added_categories),
+ ("-", "red", shift.removed_categories),
+ ):
+ for category in categories[:_MAX_LISTED_CATEGORIES]:
+ text.append(" ")
+ text.append(f"{sign}{category}", style=style)
+ if len(categories) > _MAX_LISTED_CATEGORIES:
+ extra = len(categories) - _MAX_LISTED_CATEGORIES
+ text.append(f" {sign}{extra} more", style="dim")
+ return text
+
+
+def _stacked(columns: Sequence[ColumnStatDiff]) -> RenderableType:
+ """Render each column as its own small card, for a narrow terminal.
+
+ A side-by-side table squeezed into forty cells stops being one, so below
+ the threshold the comparison is turned on its side: one heading per column,
+ then one line per measurement that moved.
+
+ Args:
+ columns: The columns to show.
+
+ Returns:
+ The stacked cards.
+ """
+ grid = Table.grid(padding=(0, 1))
+ grid.add_column(no_wrap=True)
+ grid.add_column(style="dim", no_wrap=True)
+ grid.add_column()
+
+ for position, column in enumerate(columns):
+ if position:
+ grid.add_row("", "", "")
+ grid.add_row(
+ Text(
+ _SEVERITY_GLYPHS[column.severity],
+ style=_SEVERITY_STYLES[column.severity],
+ ),
+ _column_name(column),
+ Text(column.kind.value, style="dim"),
+ )
+ for metric in column.metrics:
+ if metric.severity is Severity.UNCHANGED:
+ continue
+ grid.add_row("", _label(metric.key), _pair(metric))
+ if column.distribution_shift is not None and (
+ column.distribution_shift.score is not None
+ ):
+ grid.add_row("", Text("shift"), _shift(column.distribution_shift))
+ return grid
+
+
+# ---------------------------------------------------------------------------
+# Summary and one-line form
+# ---------------------------------------------------------------------------
+
+
+def _summary(diff: SnapshotDiff) -> Text:
+ """Render the headline counts.
+
+ Args:
+ diff: The comparison.
+
+ Returns:
+ A single line naming what changed, or a note that nothing did.
+ """
+ summary = diff.summary
+ text = Text()
+ parts: list[tuple[str, str]] = []
+ if summary.row_delta:
+ parts.append((f"{summary.row_delta:+,} rows", "bold"))
+ if summary.columns_added:
+ parts.append((f"+{summary.columns_added} columns", "green"))
+ if summary.columns_removed:
+ parts.append((f"-{summary.columns_removed} columns", "red"))
+ if summary.columns_type_changed:
+ parts.append((f"{summary.columns_type_changed} columns retyped", "yellow"))
+ if summary.columns_drifted:
+ parts.append((f"{summary.columns_drifted} columns drifted", "yellow"))
+
+ if not parts:
+ return Text("No column changed materially.", style="dim")
+
+ for position, (label, style) in enumerate(parts):
+ if position:
+ text.append(" ")
+ text.append(label, style=style)
+ return text
+
+
+def _oneline(diff: SnapshotDiff, *, now: datetime) -> Text:
+ """Render the whole comparison as one line.
+
+ Args:
+ diff: The comparison.
+ now: Unused, kept so the two render paths take the same arguments.
+
+ Returns:
+ The two ids, the dataset, what changed, and the overall severity.
+ """
+ del now
+ line = Text()
+ line.append(_short(diff.old.snapshot_id), style="bold yellow")
+ line.append(f" {_ARROW} ", style="dim")
+ line.append(_short(diff.new.snapshot_id), style="bold yellow")
+ line.append(" ")
+ line.append(diff.new.dataset_name or _MISSING, style="bold")
+ line.append(" ")
+ line.append_text(_summary(diff))
+ line.append(" ")
+ line.append_text(_badge(diff.severity))
+ return line
+
+
+# ---------------------------------------------------------------------------
+# Formatting
+# ---------------------------------------------------------------------------
+
+
+def _label(key: str) -> Text:
+ """Render a metric key as a human-readable label.
+
+ Args:
+ key: The metric's stable name, possibly dotted.
+
+ Returns:
+ The label to show, with the nesting dropped and underscores spaced.
+ """
+ _, _, leaf = key.rpartition(".")
+ return Text(leaf.replace("_", " "))
+
+
+def _value(value: float | None) -> str:
+ """Format a metric value for display.
+
+ Args:
+ value: The value, or ``None`` if it was not recorded.
+
+ Returns:
+ An em-dash for a missing value; an integer without a decimal point when
+ the number is whole, since a row count reading ``1200.0`` is noise.
+ """
+ if value is None:
+ return _MISSING
+ if float(value).is_integer() and abs(value) < 1e15:
+ return f"{int(value):,}"
+ return f"{value:,.4g}"
+
+
+def _count(value: int | None) -> str:
+ """Format a recorded count.
+
+ Args:
+ value: The count, or ``None`` if it was not recorded.
+
+ Returns:
+ The count with thousands separators, or an em-dash.
+ """
+ return _MISSING if value is None else f"{value:,}"
+
+
+def _short(snapshot_id: str | None) -> str:
+ """Return the leading characters of a snapshot id, as git shows a commit."""
+ return _MISSING if snapshot_id is None else snapshot_id[:_SHORT_ID_LEN]
+
+
+def _hash(value: str | None) -> str:
+ """Return a truncated content hash.
+
+ Args:
+ value: The recorded hash, which is empty for a non-file source.
+
+ Returns:
+ Its leading characters followed by an ellipsis, or an em-dash when
+ there is no hash to show.
+ """
+ if not value:
+ return _MISSING
+ return f"{value[:_HASH_LEN]}{_ELLIPSIS}"
diff --git a/src/dstrack/_diff_report.py b/src/dstrack/_diff_report.py
new file mode 100644
index 0000000..73786a9
--- /dev/null
+++ b/src/dstrack/_diff_report.py
@@ -0,0 +1,1016 @@
+"""A printable report of a snapshot comparison.
+
+The terminal renderer and this one answer different questions. On screen a diff
+is read once, by the person who ran the command, who already knows what they
+compared. A report is read later and elsewhere, often by someone who did not
+run anything, so it carries the context the terminal leaves implicit: what was
+compared, when, by whom, and what the thresholds were that called a change
+significant.
+
+`reportlab` is imported inside
+[write_pdf_report][dstrack._diff_report.write_pdf_report] rather than at module
+scope. It is an optional dependency, and importing it eagerly would make the
+whole module unimportable wherever the ``report`` extra is not installed --
+which is most places, since most runs never write a report.
+
+The counterpart of the terminal renderer's `Text` rule applies here.
+`Paragraph` parses a small HTML dialect, so a column named ``a`` would end
+the build with a parse error rather than printing. Every user-derived string
+therefore reaches a paragraph through [_p][dstrack._diff_report._p], which
+escapes unconditionally.
+"""
+
+import io
+from collections.abc import Sequence
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Final
+from xml.sax.saxutils import escape
+
+from dstrack import __version__
+from dstrack.diff import (
+ ColumnKind,
+ ColumnStatDiff,
+ MetricDiff,
+ SetChange,
+ Severity,
+ ShiftMethod,
+ SnapshotDiff,
+)
+from dstrack.errors import MissingDependencyError
+
+_LOGO_RESOURCE: Final = "assets/dstrack-logo.png"
+
+# The brand palette, read off docs/assets/dstrack-logo.svg.
+_PURPLE_LIGHT: Final = "#a99cff"
+_PURPLE: Final = "#7060e0"
+_PURPLE_DARK: Final = "#4a3cc0"
+_INK: Final = "#1d1568"
+_MUTED: Final = "#6b7280"
+_ROW_TINT: Final = "#f5f4ff"
+_RULE: Final = "#dedcf5"
+_OK: Final = "#2e7d32"
+_WARN: Final = "#b26a00"
+_BAD: Final = "#c62828"
+_PAPER: Final = "#ffffff"
+
+_SEVERITY_COLORS: Final[dict[Severity, str]] = {
+ Severity.UNCHANGED: _MUTED,
+ Severity.INFO: _PURPLE,
+ Severity.WARNING: _WARN,
+ Severity.CRITICAL: _BAD,
+}
+
+_MISSING: Final = "—"
+_ARROW: Final = "→"
+_SHORT_ID_LEN: Final = 8
+_HASH_LEN: Final = 16
+_MAX_LISTED_CATEGORIES: Final = 5
+
+_KIND_TITLES: Final[dict[ColumnKind, str]] = {
+ ColumnKind.NUMERIC: "Numeric columns",
+ ColumnKind.STRING: "Text columns",
+ ColumnKind.DATETIME: "Datetime columns",
+ ColumnKind.OTHER: "Other columns",
+ ColumnKind.UNKNOWN: "Unrecognized columns",
+}
+_CHANGE_LABELS: Final[tuple[tuple[str, str, str], ...]] = (
+ ("added", "Added", _OK),
+ ("removed", "Removed", _BAD),
+ ("type_changed", "Type changed", _WARN),
+ ("nullability_changed", "Nullability", _MUTED),
+)
+_SHIFT_LABELS: Final[dict[ShiftMethod, str]] = {
+ ShiftMethod.PSI: "PSI",
+ ShiftMethod.PERCENTILE_SHIFT: "Percentile shift",
+ ShiftMethod.TOP_VALUES_TVD: "Total variation",
+ ShiftMethod.NONE: "",
+}
+
+
+def write_pdf_report(diff: SnapshotDiff, path: Path, *, now: datetime) -> None:
+ """Write a comparison report to `path` as a PDF.
+
+ Args:
+ diff: The comparison to report.
+ path: Where to write the report. Its parent must exist.
+ now: The moment to stamp the report with.
+
+ Raises:
+ MissingDependencyError: If `reportlab` is not installed.
+ OSError: If `path` cannot be written.
+ """
+ kit = _import_reportlab()
+
+ document = kit.SimpleDocTemplate(
+ str(path),
+ pagesize=kit.A4,
+ leftMargin=18 * kit.mm,
+ rightMargin=18 * kit.mm,
+ topMargin=16 * kit.mm,
+ bottomMargin=20 * kit.mm,
+ title="dstrack dataset diff report",
+ author=f"dstrack {__version__}",
+ subject=_plain_subject(diff),
+ )
+
+ story: list[Any] = []
+ story += _banner(kit, diff)
+ story += _identity(kit, diff)
+ story += _summary(kit, diff)
+ story += _schema(kit, diff)
+ story += _metrics(kit, diff)
+ story += _columns(kit, diff)
+
+ document.build(story, canvasmaker=_numbered_canvas(kit, now=now))
+
+
+# ---------------------------------------------------------------------------
+# The optional dependency
+# ---------------------------------------------------------------------------
+
+
+class _Kit:
+ """The reportlab names this module uses, gathered by a single import.
+
+ Passing one object around keeps every builder below free of its own import
+ block, and keeps the import itself in one place where its failure can be
+ explained rather than raised as an `ImportError` from somewhere deep.
+
+ Attributes:
+ colors: The `reportlab.lib.colors` module.
+ A4: The A4 page size, in points.
+ mm: One millimetre, in points.
+ canvas: The `reportlab.pdfgen.canvas` module.
+ Image: Flowable for an embedded image.
+ KeepTogether: Flowable wrapper that keeps its content on one page.
+ Paragraph: Flowable for styled text.
+ SimpleDocTemplate: The single-template document builder.
+ Spacer: Flowable for vertical whitespace.
+ Table: Flowable for a grid of cells.
+ TableStyle: Styling commands for a `Table`.
+ HRFlowable: Flowable for a horizontal rule.
+ title: Style for the report title.
+ subtitle: Style for the line under the title.
+ heading: Style for section headings.
+ subheading: Style for headings within a section.
+ body: Style for running text.
+ cell: Style for text inside table cells.
+ mono: Style for code-like values inside table cells.
+ note: Style for muted explanatory text.
+ """
+
+ def __init__(self) -> None:
+ from reportlab.lib import colors
+ from reportlab.lib.enums import TA_LEFT
+ from reportlab.lib.pagesizes import A4
+ from reportlab.lib.styles import ParagraphStyle
+ from reportlab.lib.units import mm
+ from reportlab.pdfgen import canvas
+ from reportlab.platypus import (
+ Image,
+ KeepTogether,
+ Paragraph,
+ SimpleDocTemplate,
+ Spacer,
+ Table,
+ TableStyle,
+ )
+ from reportlab.platypus.flowables import HRFlowable
+
+ self.colors = colors
+ self.A4 = A4
+ self.mm = mm
+ self.canvas = canvas
+ self.Image = Image
+ self.KeepTogether = KeepTogether
+ self.Paragraph = Paragraph
+ self.SimpleDocTemplate = SimpleDocTemplate
+ self.Spacer = Spacer
+ self.Table = Table
+ self.TableStyle = TableStyle
+ self.HRFlowable = HRFlowable
+
+ self.title = ParagraphStyle(
+ "dsTitle",
+ fontName="Helvetica-Bold",
+ fontSize=20,
+ leading=24,
+ textColor=colors.HexColor(_PURPLE_DARK),
+ alignment=TA_LEFT,
+ )
+ self.subtitle = ParagraphStyle(
+ "dsSubtitle",
+ fontName="Helvetica",
+ fontSize=10,
+ leading=14,
+ textColor=colors.HexColor(_MUTED),
+ )
+ self.heading = ParagraphStyle(
+ "dsHeading",
+ fontName="Helvetica-Bold",
+ fontSize=13,
+ leading=16,
+ spaceBefore=10,
+ spaceAfter=4,
+ textColor=colors.HexColor(_INK),
+ )
+ self.subheading = ParagraphStyle(
+ "dsSubheading",
+ fontName="Helvetica-Bold",
+ fontSize=9.5,
+ leading=12,
+ spaceBefore=6,
+ spaceAfter=2,
+ textColor=colors.HexColor(_PURPLE_DARK),
+ )
+ self.body = ParagraphStyle(
+ "dsBody",
+ fontName="Helvetica",
+ fontSize=9,
+ leading=12.5,
+ textColor=colors.HexColor(_INK),
+ )
+ self.cell = ParagraphStyle(
+ "dsCell", parent=self.body, fontSize=8.2, leading=10.5
+ )
+ self.mono = ParagraphStyle(
+ "dsMono", parent=self.cell, fontName="Courier", fontSize=7.8
+ )
+ self.note = ParagraphStyle(
+ "dsNote",
+ parent=self.body,
+ fontSize=8.5,
+ textColor=colors.HexColor(_MUTED),
+ )
+
+
+def _import_reportlab() -> _Kit:
+ """Load reportlab, explaining how to get it if it is not there.
+
+ Returns:
+ The names this module builds a document from.
+
+ Raises:
+ MissingDependencyError: If `reportlab` is not installed.
+ """
+ try:
+ return _Kit()
+ except ImportError as e:
+ raise MissingDependencyError(
+ "Writing a PDF report needs `reportlab`, which is not installed.\n"
+ "Install it with: pip install 'dstrack[report]'\n"
+ " or: uv add 'dstrack[report]'"
+ ) from e
+
+
+# ---------------------------------------------------------------------------
+# Sections
+# ---------------------------------------------------------------------------
+
+
+def _banner(kit: _Kit, diff: SnapshotDiff) -> list[Any]:
+ """Build the report's masthead.
+
+ Args:
+ kit: The loaded reportlab names.
+ diff: The comparison.
+
+ Returns:
+ The logo, title, subtitle and rule.
+ """
+ heading = [
+ kit.Paragraph("Dataset Diff Report", kit.title),
+ kit.Paragraph(escape(_plain_subject(diff)), kit.subtitle),
+ ]
+
+ logo = _logo(kit)
+ masthead: Any
+ if logo is None:
+ masthead = kit.Table([[heading]], colWidths=[None])
+ else:
+ masthead = kit.Table(
+ [[logo, heading]], colWidths=[20 * kit.mm, None], hAlign="LEFT"
+ )
+ masthead.setStyle(
+ kit.TableStyle(
+ [
+ ("VALIGN", (0, 0), (-1, -1), "TOP"),
+ ("LEFTPADDING", (0, 0), (0, 0), 0),
+ ("TOPPADDING", (0, 0), (-1, -1), 0),
+ ("BOTTOMPADDING", (0, 0), (-1, -1), 0),
+ ]
+ )
+ )
+ return [
+ masthead,
+ kit.Spacer(1, 5 * kit.mm),
+ kit.HRFlowable(
+ width="100%", thickness=1.5, color=kit.colors.HexColor(_PURPLE_LIGHT)
+ ),
+ kit.Spacer(1, 3 * kit.mm),
+ _chip(kit, diff.severity.value.upper(), _SEVERITY_COLORS[diff.severity]),
+ ]
+
+
+def _logo(kit: _Kit) -> Any | None:
+ """Return the brand logo as a flowable.
+
+ The image is read as bytes rather than as a path. `importlib.resources`
+ only guarantees a real file inside an `as_file` block, and the file is gone
+ once that block exits -- which would be long before `doc.build` gets round
+ to drawing it.
+
+ Args:
+ kit: The loaded reportlab names.
+
+ Returns:
+ The logo, or ``None`` if it could not be read. A broken install should
+ still produce a report: a missing logo is a cosmetic loss, not a reason
+ to fail the command.
+ """
+ from importlib import resources
+
+ try:
+ data = resources.files("dstrack").joinpath(_LOGO_RESOURCE).read_bytes()
+ except (FileNotFoundError, ModuleNotFoundError, OSError):
+ return None
+ return kit.Image(io.BytesIO(data), width=16 * kit.mm, height=16 * kit.mm)
+
+
+def _identity(kit: _Kit, diff: SnapshotDiff) -> list[Any]:
+ """Build the table naming what was compared.
+
+ Args:
+ kit: The loaded reportlab names.
+ diff: The comparison.
+
+ Returns:
+ The section's flowables.
+ """
+ rows: list[list[Any]] = [
+ [
+ _p(kit, "Field", kit.cell, bold=True),
+ _p(kit, "Before", kit.cell, bold=True),
+ _p(kit, "After", kit.cell, bold=True),
+ ]
+ ]
+ rows.append(
+ [
+ _p(kit, "Snapshot", kit.cell),
+ _p(kit, _short(diff.old.snapshot_id), kit.mono),
+ _p(kit, _short(diff.new.snapshot_id), kit.mono),
+ ]
+ )
+ if not diff.same_dataset:
+ rows.append(
+ [
+ _p(kit, "Dataset", kit.cell),
+ _p(kit, diff.old.dataset_id or _MISSING, kit.mono),
+ _p(kit, diff.new.dataset_id or _MISSING, kit.mono),
+ ]
+ )
+ for label, before, after in (
+ ("Name", diff.old.dataset_name, diff.new.dataset_name),
+ ("Path", diff.old.dataset_path, diff.new.dataset_path),
+ ("Taken", _timestamp(diff.old.created_at), _timestamp(diff.new.created_at)),
+ ("By", diff.old.created_by, diff.new.created_by),
+ ):
+ rows.append(
+ [
+ _p(kit, label, kit.cell),
+ _p(kit, before or _MISSING, kit.cell),
+ _p(kit, after or _MISSING, kit.cell),
+ ]
+ )
+ rows.append(
+ [
+ _p(kit, "Rows", kit.cell),
+ _p(kit, _count(diff.old.num_rows), kit.cell),
+ _p(kit, _count(diff.new.num_rows), kit.cell),
+ ]
+ )
+ rows.append(
+ [
+ _p(kit, "Columns", kit.cell),
+ _p(kit, _count(diff.old.num_columns), kit.cell),
+ _p(kit, _count(diff.new.num_columns), kit.cell),
+ ]
+ )
+ rows.append(
+ [
+ _p(kit, "Source hash", kit.cell),
+ _p(kit, _hash(diff.old.source_hash), kit.mono),
+ _p(kit, _hash(diff.new.source_hash), kit.mono),
+ ]
+ )
+
+ section: list[Any] = [
+ kit.Paragraph("What was compared", kit.heading),
+ ]
+ if not diff.same_dataset:
+ section.append(
+ _callout(
+ kit,
+ "These snapshots belong to two different datasets. Column-level "
+ "comparisons still apply, but a difference here is not "
+ "necessarily drift.",
+ )
+ )
+ section.append(_table(kit, rows, widths=[32 * kit.mm, None, None]))
+ return section
+
+
+def _summary(kit: _Kit, diff: SnapshotDiff) -> list[Any]:
+ """Build the plain-language summary of what changed.
+
+ Args:
+ kit: The loaded reportlab names.
+ diff: The comparison.
+
+ Returns:
+ The section's flowables.
+ """
+ summary = diff.summary
+ points: list[str] = []
+
+ if diff.identical:
+ points.append("Nothing that is compared differs between the two snapshots.")
+ if summary.row_delta:
+ points.append(
+ f"Row count moved by {summary.row_delta:+,}, "
+ f"from {_count(diff.old.num_rows)} to {_count(diff.new.num_rows)}."
+ )
+ if summary.columns_added:
+ points.append(f"{summary.columns_added} column(s) added.")
+ if summary.columns_removed:
+ points.append(
+ f"{summary.columns_removed} column(s) removed, which breaks anything "
+ "reading them by name."
+ )
+ if summary.columns_type_changed:
+ points.append(f"{summary.columns_type_changed} column(s) changed type.")
+ if summary.columns_drifted:
+ points.append(
+ f"{summary.columns_drifted} column(s) moved enough statistically to "
+ "be worth a look."
+ )
+ if diff.old.source_hash and diff.new.source_hash:
+ points.append(
+ "The underlying bytes are identical."
+ if diff.source_hash_equal
+ else "The underlying file changed."
+ )
+ if not points:
+ points.append("No change reached the configured thresholds.")
+
+ section: list[Any] = [kit.Paragraph("Summary", kit.heading)]
+ section += [_p(kit, f"• {point}", kit.body) for point in points]
+ section.append(kit.Spacer(1, 2 * kit.mm))
+ section.append(
+ _p(
+ kit,
+ f"Thresholds: a relative change of {diff.options.warn_pct_change:.0%} "
+ f"warns and {diff.options.critical_pct_change:.0%} is critical; for "
+ "values already expressed as a fraction, "
+ f"{diff.options.warn_fraction_shift:.1%} and "
+ f"{diff.options.critical_fraction_shift:.1%} absolute. Population "
+ f"stability index warns at {diff.options.warn_psi:.2f} and is "
+ f"critical at {diff.options.critical_psi:.2f}.",
+ kit.note,
+ )
+ )
+ return section
+
+
+def _schema(kit: _Kit, diff: SnapshotDiff) -> list[Any]:
+ """Build the schema-change table.
+
+ Args:
+ kit: The loaded reportlab names.
+ diff: The comparison.
+
+ Returns:
+ The section's flowables.
+ """
+ section: list[Any] = [kit.Paragraph("Schema changes", kit.heading)]
+ schema = diff.schema
+
+ if not schema.changed:
+ section.append(_p(kit, "The schema is unchanged.", kit.body))
+ if schema.reordered:
+ section.append(
+ _p(
+ kit,
+ f"{len(schema.reordered)} column(s) appear in a different "
+ "order. The schema hash is order-independent, so this is "
+ "the only place a reorder shows.",
+ kit.note,
+ )
+ )
+ return section
+
+ rows: list[list[Any]] = [
+ [
+ _p(kit, "Change", kit.cell, bold=True),
+ _p(kit, "Column", kit.cell, bold=True),
+ _p(kit, "Before", kit.cell, bold=True),
+ _p(kit, "After", kit.cell, bold=True),
+ ]
+ ]
+ tints: list[tuple[int, str]] = []
+ for attribute, label, color in _CHANGE_LABELS:
+ for change in getattr(schema, attribute):
+ tints.append((len(rows), color))
+ rows.append(
+ [
+ _p(kit, label, kit.cell, color=color, bold=True),
+ _p(kit, change.name, kit.cell),
+ _p(kit, _dtype(change.old_dtype, change.old_nullable), kit.cell),
+ _p(kit, _dtype(change.new_dtype, change.new_nullable), kit.cell),
+ ]
+ )
+
+ section.append(
+ _table(kit, rows, widths=[26 * kit.mm, None, 38 * kit.mm, 38 * kit.mm])
+ )
+ return section
+
+
+def _metrics(kit: _Kit, diff: SnapshotDiff) -> list[Any]:
+ """Build the dataset-level metric table.
+
+ Args:
+ kit: The loaded reportlab names.
+ diff: The comparison.
+
+ Returns:
+ The section's flowables, empty if no dataset metric was recorded.
+ """
+ if not diff.metrics and not diff.sets:
+ return []
+
+ rows: list[list[Any]] = [
+ [
+ _p(kit, "Metric", kit.cell, bold=True),
+ _p(kit, "Before", kit.cell, bold=True),
+ _p(kit, "After", kit.cell, bold=True),
+ _p(kit, "Change", kit.cell, bold=True),
+ _p(kit, "Relative", kit.cell, bold=True),
+ ]
+ ]
+ for metric in diff.metrics:
+ color = _SEVERITY_COLORS[metric.severity]
+ rows.append(
+ [
+ _p(kit, _label(metric.key), kit.cell),
+ _p(kit, _value(metric.old), kit.cell),
+ _p(kit, _value(metric.new), kit.cell, color=color),
+ _p(kit, _delta(metric), kit.cell, color=color),
+ _p(
+ kit,
+ f"{metric.pct_change:+.1%}" if metric.pct_change else _MISSING,
+ kit.cell,
+ color=color,
+ ),
+ ]
+ )
+
+ section: list[Any] = [
+ kit.Paragraph("Dataset metrics", kit.heading),
+ _table(kit, rows, widths=[46 * kit.mm, None, None, None, None]),
+ ]
+ section.extend(_p(kit, _set_change(change), kit.note) for change in diff.sets)
+ return section
+
+
+def _set_change(change: SetChange) -> str:
+ """Describe how a set-valued field's membership moved.
+
+ Args:
+ change: The membership change.
+
+ Returns:
+ A sentence naming what joined the set and what left it.
+ """
+ parts = []
+ if change.added:
+ parts.append(f"joined by {', '.join(change.added)}")
+ if change.removed:
+ parts.append(f"left by {', '.join(change.removed)}")
+ return f"{_label(change.key)}: {', '.join(parts)}"
+
+
+def _columns(kit: _Kit, diff: SnapshotDiff) -> list[Any]:
+ """Build the per-column detail, grouped by statistics variant.
+
+ Args:
+ kit: The loaded reportlab names.
+ diff: The comparison.
+
+ Returns:
+ The section's flowables, empty if no column changed.
+ """
+ changed = [
+ column for column in diff.columns if column.severity is not Severity.UNCHANGED
+ ]
+ if not changed:
+ return []
+
+ # Each heading is bound to the first block beneath it, so a page never ends
+ # on a title whose content starts overleaf.
+ section: list[Any] = []
+ pending: list[Any] = [kit.Paragraph("Column detail", kit.heading)]
+ for kind, title in _KIND_TITLES.items():
+ group = [column for column in changed if column.kind is kind]
+ if not group:
+ continue
+ pending.append(kit.Paragraph(escape(title), kit.subheading))
+ blocks = [_column_block(kit, column) for column in group]
+ section.append(kit.KeepTogether([*pending, blocks[0]]))
+ section.extend(blocks[1:])
+ pending = []
+ return section
+
+
+def _column_block(kit: _Kit, column: ColumnStatDiff) -> Any:
+ """Build one column's heading and metric table as an unbreakable unit.
+
+ Args:
+ kit: The loaded reportlab names.
+ column: The column's comparison.
+
+ Returns:
+ A flowable that will not be split across a page boundary, so a
+ column's numbers are never separated from the name they belong to.
+ """
+ heading = _p(
+ kit,
+ _column_title(column),
+ kit.body,
+ bold=True,
+ color=_SEVERITY_COLORS[column.severity],
+ )
+
+ rows: list[list[Any]] = [
+ [
+ _p(kit, "Statistic", kit.cell, bold=True),
+ _p(kit, "Before", kit.cell, bold=True),
+ _p(kit, "After", kit.cell, bold=True),
+ _p(kit, "Change", kit.cell, bold=True),
+ ]
+ ]
+ for metric in column.metrics:
+ if metric.severity is Severity.UNCHANGED:
+ continue
+ color = _SEVERITY_COLORS[metric.severity]
+ rows.append(
+ [
+ _p(kit, _label(metric.key), kit.cell),
+ _p(kit, _value(metric.old), kit.cell),
+ _p(kit, _value(metric.new), kit.cell, color=color),
+ _p(kit, _delta(metric), kit.cell, color=color),
+ ]
+ )
+
+ block: list[Any] = [heading]
+ if len(rows) > 1:
+ block.append(_table(kit, rows, widths=[46 * kit.mm, None, None, None]))
+ else:
+ block.append(_p(kit, "Only the column's presence changed.", kit.note))
+
+ # The shift is a property of the pair, not a value either side recorded, so
+ # it is stated rather than tabulated: a row of it under Before and After
+ # would read as a statistic that went from nothing to 0.23.
+ shift = column.distribution_shift
+ if shift is not None and shift.score is not None:
+ block.append(
+ _p(
+ kit,
+ f"{_SHIFT_LABELS[shift.method]}: {shift.score:.3f}"
+ + (f" {categories}" if (categories := _categories(shift)) else ""),
+ kit.note,
+ color=_SEVERITY_COLORS[shift.severity],
+ )
+ )
+ block.append(kit.Spacer(1, 2 * kit.mm))
+ return kit.KeepTogether(block)
+
+
+# ---------------------------------------------------------------------------
+# Page furniture
+# ---------------------------------------------------------------------------
+
+
+def _numbered_canvas(kit: _Kit, *, now: datetime) -> Any:
+ """Build the canvas class that stamps a footer on every page.
+
+ "Page 3 of 7" cannot be drawn while page 3 is being laid out, because the
+ total is not known until the last page is done. The pages are therefore
+ buffered and their footers drawn at the end, when the count is settled.
+
+ Args:
+ kit: The loaded reportlab names.
+ now: The moment to stamp the report with.
+
+ Returns:
+ A canvas class for `doc.build` to instantiate.
+ """
+ stamp = now.strftime("%Y-%m-%d %H:%M UTC")
+
+ class _NumberedCanvas(kit.canvas.Canvas): # type: ignore[misc,name-defined]
+ def __init__(self, *args: Any, **kwargs: Any) -> None:
+ super().__init__(*args, **kwargs)
+ self._pages: list[dict[str, Any]] = []
+
+ def showPage(self) -> None:
+ self._pages.append(dict(self.__dict__))
+ self._startPage()
+
+ def save(self) -> None:
+ total = len(self._pages)
+ for state in self._pages:
+ self.__dict__.update(state)
+ self._footer(total)
+ super().showPage()
+ super().save()
+
+ def _footer(self, total: int) -> None:
+ width, _ = self._pagesize
+ self.setStrokeColor(kit.colors.HexColor(_RULE))
+ self.setLineWidth(0.5)
+ self.line(18 * kit.mm, 14 * kit.mm, width - 18 * kit.mm, 14 * kit.mm)
+
+ self.setFont("Helvetica", 7.5)
+ self.setFillColor(kit.colors.HexColor(_MUTED))
+ self.drawString(
+ 18 * kit.mm,
+ 10 * kit.mm,
+ f"dstrack {__version__} · generated {stamp}",
+ )
+ self.drawRightString(
+ width - 18 * kit.mm,
+ 10 * kit.mm,
+ f"Page {self._pageNumber} of {total}",
+ )
+
+ return _NumberedCanvas
+
+
+# ---------------------------------------------------------------------------
+# Building blocks
+# ---------------------------------------------------------------------------
+
+
+def _p(
+ kit: _Kit,
+ content: str,
+ style: Any,
+ *,
+ bold: bool = False,
+ color: str | None = None,
+) -> Any:
+ """Build a paragraph from text that may contain anything.
+
+ The single place user-derived text becomes a flowable, and so the single
+ place the escaping has to be right. `Paragraph` reads its input as markup,
+ so a column named ``a`` would abort the build.
+
+ Args:
+ kit: The loaded reportlab names.
+ content: The text, escaped here rather than by the caller.
+ style: The paragraph style to use.
+ bold: Render in the bold face.
+ color: Override the style's text color.
+
+ Returns:
+ The paragraph.
+ """
+ markup = escape(content)
+ if bold:
+ markup = f"{markup}"
+ if color is not None:
+ markup = f'{markup}'
+ return kit.Paragraph(markup, style)
+
+
+def _table(kit: _Kit, rows: Sequence[Sequence[Any]], *, widths: Sequence[Any]) -> Any:
+ """Build a table in the report's house style.
+
+ Args:
+ kit: The loaded reportlab names.
+ rows: The cells, with the header as the first row.
+ widths: Column widths, ``None`` for a column that takes what is left.
+
+ Returns:
+ The styled table, with its header repeated on every page it spans.
+ """
+ table = kit.Table(list(rows), colWidths=list(widths), repeatRows=1, hAlign="LEFT")
+ style = [
+ ("BACKGROUND", (0, 0), (-1, 0), kit.colors.HexColor(_PURPLE_DARK)),
+ ("TEXTCOLOR", (0, 0), (-1, 0), kit.colors.HexColor(_PAPER)),
+ ("GRID", (0, 0), (-1, -1), 0.25, kit.colors.HexColor(_RULE)),
+ ("VALIGN", (0, 0), (-1, -1), "MIDDLE"),
+ ("TOPPADDING", (0, 0), (-1, -1), 3),
+ ("BOTTOMPADDING", (0, 0), (-1, -1), 3),
+ ("LEFTPADDING", (0, 0), (-1, -1), 5),
+ ("RIGHTPADDING", (0, 0), (-1, -1), 5),
+ ]
+ style += [
+ ("BACKGROUND", (0, row), (-1, row), kit.colors.HexColor(_ROW_TINT))
+ for row in range(2, len(rows), 2)
+ ]
+ table.setStyle(kit.TableStyle(style))
+ return table
+
+
+def _chip(kit: _Kit, label: str, color: str) -> Any:
+ """Build a small filled badge.
+
+ Args:
+ kit: The loaded reportlab names.
+ label: The badge's text.
+ color: Its fill color.
+
+ Returns:
+ A one-cell table standing in for a rounded label.
+ """
+ chip = kit.Table(
+ [[_p(kit, label, kit.cell, bold=True, color=_PAPER)]],
+ colWidths=[26 * kit.mm],
+ hAlign="LEFT",
+ )
+ chip.setStyle(
+ kit.TableStyle(
+ [
+ ("BACKGROUND", (0, 0), (-1, -1), kit.colors.HexColor(color)),
+ ("ALIGN", (0, 0), (-1, -1), "CENTER"),
+ ("TOPPADDING", (0, 0), (-1, -1), 3),
+ ("BOTTOMPADDING", (0, 0), (-1, -1), 3),
+ ]
+ )
+ )
+ return chip
+
+
+def _callout(kit: _Kit, message: str) -> Any:
+ """Build a tinted note that should not be read as part of a table.
+
+ Args:
+ kit: The loaded reportlab names.
+ message: The text to show.
+
+ Returns:
+ The callout.
+ """
+ box = kit.Table([[_p(kit, message, kit.note)]], colWidths=["100%"], hAlign="LEFT")
+ box.setStyle(
+ kit.TableStyle(
+ [
+ ("BACKGROUND", (0, 0), (-1, -1), kit.colors.HexColor(_ROW_TINT)),
+ ("LINEBEFORE", (0, 0), (0, -1), 2, kit.colors.HexColor(_WARN)),
+ ("TOPPADDING", (0, 0), (-1, -1), 5),
+ ("BOTTOMPADDING", (0, 0), (-1, -1), 5),
+ ("LEFTPADDING", (0, 0), (-1, -1), 7),
+ ]
+ )
+ )
+ return box
+
+
+# ---------------------------------------------------------------------------
+# Formatting
+# ---------------------------------------------------------------------------
+
+
+def _plain_subject(diff: SnapshotDiff) -> str:
+ """Describe what the report is about, in one line.
+
+ Args:
+ diff: The comparison.
+
+ Returns:
+ The dataset name or names, and the two short snapshot ids.
+ """
+ old_name = diff.old.dataset_name or _MISSING
+ new_name = diff.new.dataset_name or _MISSING
+ subject = old_name if old_name == new_name else f"{old_name} {_ARROW} {new_name}"
+ return (
+ f"{subject} · {_short(diff.old.snapshot_id)} {_ARROW} "
+ f"{_short(diff.new.snapshot_id)}"
+ )
+
+
+def _column_title(column: ColumnStatDiff) -> str:
+ """Name a column, saying what happened to it.
+
+ Args:
+ column: The column's comparison.
+
+ Returns:
+ The name, annotated when the column appeared, disappeared, or changed
+ the kind of value it holds.
+ """
+ if column.old_kind is not None:
+ return f"{column.name} ({column.old_kind.value} {_ARROW} {column.kind.value})"
+ if not column.present_in_new:
+ return f"{column.name} (dropped)"
+ if not column.present_in_old:
+ return f"{column.name} (new)"
+ return column.name
+
+
+def _categories(shift: Any) -> str:
+ """Summarize the categories a text column gained and lost.
+
+ Args:
+ shift: The column's distribution shift.
+
+ Returns:
+ A short list of each, or an em-dash when nothing came or went.
+ """
+ parts = []
+ for sign, values in (
+ ("+", shift.added_categories),
+ ("-", shift.removed_categories),
+ ):
+ if not values:
+ continue
+ shown = ", ".join(values[:_MAX_LISTED_CATEGORIES])
+ if len(values) > _MAX_LISTED_CATEGORIES:
+ shown += f", +{len(values) - _MAX_LISTED_CATEGORIES} more"
+ label = "gained" if sign == "+" else "lost"
+ parts.append(f"{label} {shown}")
+ return "; ".join(parts)
+
+
+def _label(key: str) -> str:
+ """Render a metric key as a human-readable label."""
+ _, _, leaf = key.rpartition(".")
+ return leaf.replace("_", " ").capitalize()
+
+
+def _delta(metric: MetricDiff) -> str:
+ """Render how far a metric moved.
+
+ Args:
+ metric: The comparison.
+
+ Returns:
+ The signed change, or an em-dash when there is none to state.
+ """
+ if metric.delta is None or metric.delta == 0:
+ return _MISSING
+ return f"{metric.delta:+,.4g}"
+
+
+def _value(value: float | None) -> str:
+ """Format a metric value for a table cell."""
+ if value is None:
+ return _MISSING
+ if float(value).is_integer() and abs(value) < 1e15:
+ return f"{int(value):,}"
+ return f"{value:,.4g}"
+
+
+def _count(value: int | None) -> str:
+ """Format a recorded count."""
+ return _MISSING if value is None else f"{value:,}"
+
+
+def _dtype(dtype: str | None, nullable: bool | None) -> str:
+ """Render a column's type and nullability."""
+ if dtype is None and nullable is None:
+ return _MISSING
+ suffix = "" if nullable is None else (", nullable" if nullable else ", not null")
+ return f"{dtype or _MISSING}{suffix}"
+
+
+def _timestamp(value: str | None) -> str:
+ """Render a recorded timestamp for a reader rather than for a machine.
+
+ Args:
+ value: The ISO-8601 timestamp, as the snapshot recorded it.
+
+ Returns:
+ The moment to the minute, or the raw string if it cannot be parsed --
+ an unreadable timestamp is still worth showing, and is not a reason to
+ fail a report.
+ """
+ if not value:
+ return _MISSING
+ try:
+ parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+ except ValueError:
+ return value
+ return parsed.strftime("%Y-%m-%d %H:%M %Z").strip()
+
+
+def _short(snapshot_id: str | None) -> str:
+ """Return the leading characters of a snapshot id."""
+ return _MISSING if snapshot_id is None else snapshot_id[:_SHORT_ID_LEN]
+
+
+def _hash(value: str | None) -> str:
+ """Return a truncated content hash, or an em-dash if there is none."""
+ return f"{value[:_HASH_LEN]}…" if value else _MISSING
diff --git a/src/dstrack/_log.py b/src/dstrack/_log.py
index 68efe15..c17f70d 100644
--- a/src/dstrack/_log.py
+++ b/src/dstrack/_log.py
@@ -1,9 +1,6 @@
"""The ``dstrack log`` command: show a tracked dataset's history."""
-import os
import sqlite3
-import sys
-import uuid
from datetime import UTC, datetime
from itertools import pairwise
from pathlib import Path
@@ -11,8 +8,9 @@
import typer
-from dstrack import cache, console, store
+from dstrack import cache, console
from dstrack._log_render import render_history
+from dstrack._resolve import ensure_index, resolve_target
from dstrack.errors import (
DatasetNotFoundError,
StoreCorruptionError,
@@ -80,7 +78,7 @@ def log(
console.error(str(e))
raise typer.Exit(code=1) from e
- _ensure_index(store_root, yes=yes, rebuild=rebuild)
+ ensure_index(store_root, yes=yes, rebuild=rebuild)
try:
# Sync before resolving: the index does not travel with the repository,
@@ -89,7 +87,7 @@ def log(
if reason is not None:
console.warning(f"Rebuilt the snapshot index: {reason}.")
- dataset_id = _resolve_target(target, store_root=store_root, root=root)
+ dataset_id = resolve_target(target, store_root=store_root, root=root)
history = cache.query_history(dataset_id, store_root=store_root)
except (
DatasetNotFoundError,
@@ -125,144 +123,3 @@ def log(
oneline=oneline,
)
)
-
-
-def _ensure_index(store_root: Path, *, yes: bool, rebuild: bool) -> None:
- """Make sure the snapshot index may be built, asking first if need be.
-
- Building parses every dataset's log, so it is offered rather than done
- silently. Consent is only about the first build: an index that exists but
- cannot be used is rebuilt without asking, since it holds nothing that the
- logs do not.
-
- Args:
- store_root: Path to the ``.dstrack/`` directory.
- yes: Consent given up front, so nothing is asked.
- rebuild: Discard any existing index, which implies consent.
-
- Raises:
- typer.Exit: Code 0 if the user declines to build it; code 1 if there
- is no index and no terminal to ask at.
- """
- path = cache.index_path(store_root)
- if rebuild:
- path.unlink(missing_ok=True)
- console.info("Rebuilding the snapshot index from the dataset logs...")
- return
- if cache.index_exists(store_root):
- return
-
- if not yes:
- if not sys.stdin.isatty():
- console.error(
- f"No snapshot index at `{path}`, and there is no terminal to "
- "ask whether to build one. Re-run with `--yes` to build it."
- )
- raise typer.Exit(code=1)
- console.info(f"No snapshot index at `{path}` yet.")
- if not typer.confirm("Build it now from the dataset logs?"):
- console.warning(
- "Not building the index; there is nothing to show without it."
- )
- raise typer.Exit(code=0)
-
- console.info("Building the snapshot index from the dataset logs...")
-
-
-def _resolve_target(target: str, *, store_root: Path, root: Path | None) -> str:
- """Resolve a user-supplied dataset id or path to a dataset id.
-
- A ``dataset_id`` is a random UUID4, so nothing about it can be
- derived from a path: a path is resolved by matching it against what each
- dataset last recorded. `target` is read as an id first and as a path
- second, but only when the id names a dataset that exists. Reading it as a
- path first would let any file in the current directory shadow a real id,
- and would send a mistyped path into the id lookup, which can only report
- that no dataset goes by that name.
-
- Args:
- target: A dataset id, or a path to a tracked dataset file.
- store_root: Path to the ``.dstrack/`` directory.
- root: Path root a `target` path is made relative to, or ``None`` to
- use the store root.
-
- Returns:
- The id of the dataset `target` names.
-
- Raises:
- DatasetNotFoundError: If `target` is an id no dataset goes by, or a
- path no dataset last recorded.
- ValueError: If `target` and the path root are on different drives, so
- no relative path exists between them.
- """
- canonical = _canonical_uuid(target)
- if canonical is not None and store.dataset_exists(canonical, store_root=store_root):
- return canonical
-
- path_root = root if root is not None else store_root.parent
- dataset_path = Path(
- os.path.relpath(Path(target).resolve(), Path(path_root).resolve())
- ).as_posix()
-
- matched = cache.find_dataset_by_path(dataset_path, store_root=store_root)
- if matched is not None:
- return matched
-
- if canonical is not None:
- raise DatasetNotFoundError(
- f"No dataset {target!r} in the store.\n{_known_datasets(store_root)}"
- )
- raise DatasetNotFoundError(
- f"No tracked dataset at {target!r} (recorded as {dataset_path!r} "
- f"relative to {path_root}).\n"
- "If the file was renamed or moved, its recorded path no longer "
- "matches; pass the dataset id instead. Use `--root` to change the "
- "path root the recorded path is computed against.\n"
- f"{_known_datasets(store_root)}"
- )
-
-
-def _canonical_uuid(value: str) -> str | None:
- """Return `value` as a canonical UUID string, or ``None`` if it is not one.
-
- Datasets are named by ``str(uuid.uuid4())``, but a user may paste an id
- uppercased, braced, or without its dashes, all of which name the same
- dataset. Canonicalizing accepts those spellings, and makes the result
- provably free of path separators before it is joined onto the store.
-
- Args:
- value: The string to interpret.
-
- Returns:
- The canonical form of `value` if it is a UUID in any spelling, else
- ``None``.
- """
- try:
- return str(uuid.UUID(value))
- except (ValueError, AttributeError, TypeError):
- return None
-
-
-def _known_datasets(store_root: Path) -> str:
- """List the store's datasets, for an error message to end on.
-
- A dataset id says nothing on its own, so each is shown with the name and
- path its latest snapshot recorded: that is what lets a reader recognize
- the dataset they meant and copy its id.
-
- Args:
- store_root: Path to the ``.dstrack/`` directory.
-
- Returns:
- An indented listing of every dataset, or a note that there are none.
- """
- datasets = cache.list_datasets(store_root=store_root)
- lines = [
- f" {d.dataset_id} {(d.head.dataset_name if d.head else None) or '-'} "
- f"{(d.head.dataset_path if d.head else None) or '-'}"
- for d in datasets
- ]
- if not lines:
- return "Known datasets:\n (none)\nRun `dstrack track ` to record one."
- listing = "\n".join(lines)
- return f"Known datasets:\n{listing}"
diff --git a/src/dstrack/_resolve.py b/src/dstrack/_resolve.py
new file mode 100644
index 0000000..a236757
--- /dev/null
+++ b/src/dstrack/_resolve.py
@@ -0,0 +1,555 @@
+"""Turning what a user typed into something the store can be asked about.
+
+Commands take a dataset or a snapshot the way a person refers to it -- a path,
+an id, the first few characters of one -- and the store only answers to exact
+ids. This module holds that translation, and the consent flow that must run
+before the index it relies on is built.
+
+It lives here rather than in `dstrack.store` or `dstrack.cache` because
+resolution is a *command's* concern: it prompts, prints, and exits, none of
+which a data-layer module may do. And it lives in one place rather than in each
+command because the precedence between an id and a path is a promise made to
+the user, and two copies of it would drift until `dstrack log` and
+`dstrack diff` answered the same argument differently.
+"""
+
+import os
+import sys
+import uuid
+from collections.abc import Sequence
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Final
+
+import typer
+
+from dstrack import cache, console, store
+from dstrack.errors import (
+ AmbiguousSnapshotError,
+ DatasetNotFoundError,
+ SnapshotNotFoundError,
+ StoreCorruptionError,
+)
+
+# Shortest prefix accepted as a snapshot id. Below this almost any id matches,
+# and the argument is far more likely to be a path.
+_MIN_PREFIX_LEN: Final = 4
+# Longest, i.e. a full canonical UUID.
+_MAX_PREFIX_LEN: Final = 36
+_HEX_CHARS: Final = frozenset("0123456789abcdefABCDEF-")
+_SHORT_ID_LEN: Final = 8
+# Candidates listed back when a prefix is ambiguous. Enough to recognize the
+# one that was meant, not a dump of the store.
+_MAX_CANDIDATES: Final = 10
+# A comparison has exactly two sides.
+_PAIR: Final = 2
+
+
+@dataclass(frozen=True)
+class ResolvedSnapshot:
+ """One command-line argument, resolved to a snapshot and its payload.
+
+ Attributes:
+ dataset_id: Dataset the snapshot belongs to.
+ snapshot_id: The snapshot itself.
+ payload: Its full stored payload.
+ spec: What the user typed, kept for error messages that follow.
+ """
+
+ dataset_id: str
+ snapshot_id: str
+ payload: dict[str, Any]
+ spec: str
+
+
+def ensure_index(
+ store_root: Path, *, yes: bool, rebuild: bool, quiet: bool = False
+) -> None:
+ """Make sure the snapshot index may be built, asking first if need be.
+
+ Building parses every dataset's log, so it is offered rather than done
+ silently. Consent is only about the first build: an index that exists but
+ cannot be used is rebuilt without asking, since it holds nothing that the
+ logs do not.
+
+ Args:
+ store_root: Path to the ``.dstrack/`` directory.
+ yes: Consent given up front, so nothing is asked.
+ rebuild: Discard any existing index, which implies consent.
+ quiet: Suppress the progress notes, for a command whose standard
+ output is a document rather than a report to a person. Errors are
+ still printed: a failure the caller has to act on is not noise.
+
+ Raises:
+ typer.Exit: Code 0 if the user declines to build it; code 1 if there
+ is no index and no terminal to ask at.
+ """
+ path = cache.index_path(store_root)
+ if rebuild:
+ path.unlink(missing_ok=True)
+ if not quiet:
+ console.info("Rebuilding the snapshot index from the dataset logs...")
+ return
+ if cache.index_exists(store_root):
+ return
+
+ if not yes:
+ if not sys.stdin.isatty():
+ console.error(
+ f"No snapshot index at `{path}`, and there is no terminal to "
+ "ask whether to build one. Re-run with `--yes` to build it."
+ )
+ raise typer.Exit(code=1)
+ console.info(f"No snapshot index at `{path}` yet.")
+ if not typer.confirm("Build it now from the dataset logs?"):
+ console.warning(
+ "Not building the index; there is nothing to show without it."
+ )
+ raise typer.Exit(code=0)
+
+ if not quiet:
+ console.info("Building the snapshot index from the dataset logs...")
+
+
+def resolve_target(target: str, *, store_root: Path, root: Path | None) -> str:
+ """Resolve a user-supplied dataset id or path to a dataset id.
+
+ A ``dataset_id`` is a random UUID4, so nothing about it can be
+ derived from a path: a path is resolved by matching it against what each
+ dataset last recorded. `target` is read as an id first and as a path
+ second, but only when the id names a dataset that exists. Reading it as a
+ path first would let any file in the current directory shadow a real id,
+ and would send a mistyped path into the id lookup, which can only report
+ that no dataset goes by that name.
+
+ Args:
+ target: A dataset id, or a path to a tracked dataset file.
+ store_root: Path to the ``.dstrack/`` directory.
+ root: Path root a `target` path is made relative to, or ``None`` to
+ use the store root.
+
+ Returns:
+ The id of the dataset `target` names.
+
+ Raises:
+ DatasetNotFoundError: If `target` is an id no dataset goes by, or a
+ path no dataset last recorded.
+ ValueError: If `target` and the path root are on different drives, so
+ no relative path exists between them.
+ """
+ canonical = canonical_uuid(target)
+ if canonical is not None and store.dataset_exists(canonical, store_root=store_root):
+ return canonical
+
+ dataset_path = recorded_path(target, store_root=store_root, root=root)
+
+ matched = cache.find_dataset_by_path(dataset_path, store_root=store_root)
+ if matched is not None:
+ return matched
+
+ raise dataset_not_found(
+ target,
+ canonical=canonical,
+ dataset_path=dataset_path,
+ store_root=store_root,
+ root=root,
+ )
+
+
+def recorded_path(target: str, *, store_root: Path, root: Path | None) -> str:
+ """Return the relative POSIX path a dataset at `target` would be recorded as.
+
+ Mirrors what `dstrack track` stores, so that a path typed on the command
+ line and a path read out of a snapshot can be compared as strings.
+
+ Args:
+ target: A path to a dataset file, as the user typed it.
+ store_root: Path to the ``.dstrack/`` directory.
+ root: Path root the result is made relative to, or ``None`` to use the
+ store root, i.e. the directory containing ``.dstrack/``.
+
+ Returns:
+ The path relative to the path root, in POSIX form.
+
+ Raises:
+ ValueError: If `target` and the path root are on different drives, so
+ no relative path exists between them.
+ """
+ path_root = root if root is not None else store_root.parent
+ return Path(
+ os.path.relpath(Path(target).resolve(), Path(path_root).resolve())
+ ).as_posix()
+
+
+def dataset_not_found(
+ target: str,
+ *,
+ canonical: str | None,
+ dataset_path: str,
+ store_root: Path,
+ root: Path | None,
+) -> Exception:
+ """Build the error for a `target` that names no dataset.
+
+ An id-shaped argument and a path-shaped one fail for different reasons and
+ are fixed in different ways, so they are reported differently. Both end on
+ the store's contents, which is what lets a reader recognize the dataset
+ they meant.
+
+ Args:
+ target: The argument as the user typed it.
+ canonical: `target` as a canonical UUID, or ``None`` if it is not one.
+ dataset_path: What `target` was matched as, when read as a path.
+ store_root: Path to the ``.dstrack/`` directory.
+ root: Path root `dataset_path` was computed against, or ``None``.
+
+ Returns:
+ The exception to raise. Returned rather than raised so the caller's
+ traceback starts at the failure, not inside this helper.
+ """
+ if canonical is not None:
+ return DatasetNotFoundError(
+ f"No dataset {target!r} in the store.\n{known_datasets(store_root)}"
+ )
+ path_root = root if root is not None else store_root.parent
+ return DatasetNotFoundError(
+ f"No tracked dataset at {target!r} (recorded as {dataset_path!r} "
+ f"relative to {path_root}).\n"
+ "If the file was renamed or moved, its recorded path no longer "
+ "matches; pass the dataset id instead. Use `--root` to change the "
+ "path root the recorded path is computed against.\n"
+ f"{known_datasets(store_root)}"
+ )
+
+
+def canonical_uuid(value: str) -> str | None:
+ """Return `value` as a canonical UUID string, or ``None`` if it is not one.
+
+ Datasets are named by ``str(uuid.uuid4())``, but a user may paste an id
+ uppercased, braced, or without its dashes, all of which name the same
+ dataset. Canonicalizing accepts those spellings, and makes the result
+ provably free of path separators before it is joined onto the store.
+
+ Args:
+ value: The string to interpret.
+
+ Returns:
+ The canonical form of `value` if it is a UUID in any spelling, else
+ ``None``.
+ """
+ try:
+ return str(uuid.UUID(value))
+ except (ValueError, AttributeError, TypeError):
+ return None
+
+
+def known_datasets(store_root: Path) -> str:
+ """List the store's datasets, for an error message to end on.
+
+ A dataset id says nothing on its own, so each is shown with the name and
+ path its latest snapshot recorded: that is what lets a reader recognize
+ the dataset they meant and copy its id.
+
+ Args:
+ store_root: Path to the ``.dstrack/`` directory.
+
+ Returns:
+ An indented listing of every dataset, or a note that there are none.
+ """
+ datasets = cache.list_datasets(store_root=store_root)
+ lines = [
+ f" {d.dataset_id} {(d.head.dataset_name if d.head else None) or '-'} "
+ f"{(d.head.dataset_path if d.head else None) or '-'}"
+ for d in datasets
+ ]
+ if not lines:
+ return "Known datasets:\n (none)\nRun `dstrack track ` to record one."
+ listing = "\n".join(lines)
+ return f"Known datasets:\n{listing}"
+
+
+def resolve_snapshot_ref(
+ spec: str, *, store_root: Path, root: Path | None
+) -> ResolvedSnapshot:
+ """Resolve one user-supplied reference to a snapshot.
+
+ A reference is read in the order that keeps each reading from shadowing
+ the one before it:
+
+ 1. a full id, which names either a snapshot outright or a dataset whose
+ latest snapshot is meant;
+ 2. the path of a tracked file, matched as `dstrack log` matches it;
+ 3. the leading characters of a snapshot id, the way git takes a short
+ commit hash.
+
+ The prefix is tried *after* the path so that a file whose name happens to
+ be eight hex characters keeps the meaning `dstrack log` already gives it,
+ and *before* the failure so that a mistyped path is reported as a path
+ rather than as an unknown id.
+
+ Args:
+ spec: The reference, as the user typed it.
+ store_root: Path to the ``.dstrack/`` directory.
+ root: Path root a `spec` path is made relative to, or ``None`` to use
+ the store root.
+
+ Returns:
+ The snapshot it names, with its payload already read.
+
+ Raises:
+ AmbiguousSnapshotError: If a prefix matches more than one snapshot.
+ DatasetNotFoundError: If `spec` names nothing in the store.
+ SnapshotNotFoundError: If it names a dataset that has no snapshots.
+ StoreCorruptionError: If the payload cannot be read back.
+ ValueError: If `spec` and the path root are on different drives.
+ """
+ canonical = canonical_uuid(spec)
+ if canonical is not None:
+ located = _match_prefix(canonical, spec, store_root=store_root)
+ if located is not None:
+ return _load(located, spec, store_root=store_root)
+ if store.dataset_exists(canonical, store_root=store_root):
+ return _head_of(canonical, spec, store_root=store_root)
+
+ dataset_path = recorded_path(spec, store_root=store_root, root=root)
+ matched = cache.find_dataset_by_path(dataset_path, store_root=store_root)
+ if matched is not None:
+ return _head_of(matched, spec, store_root=store_root)
+
+ if canonical is None and _looks_like_prefix(spec):
+ located = _match_prefix(spec, spec, store_root=store_root)
+ if located is not None:
+ return _load(located, spec, store_root=store_root)
+
+ raise dataset_not_found(
+ spec,
+ canonical=canonical,
+ dataset_path=dataset_path,
+ store_root=store_root,
+ root=root,
+ )
+
+
+def resolve_pair(
+ specs: Sequence[str],
+ *,
+ from_ref: str | None,
+ to_ref: str | None,
+ store_root: Path,
+ root: Path | None,
+) -> tuple[ResolvedSnapshot, ResolvedSnapshot]:
+ """Resolve the two sides of a comparison from what the command was given.
+
+ Two spellings are accepted and cannot be mixed: positional arguments, or
+ the explicit `--from` / `--to` options. Naming one side leaves the other
+ implied -- a single snapshot is compared against the one it succeeded,
+ which is the question `dstrack diff` is usually asked.
+
+ Args:
+ specs: Positional references, in the order they were given.
+ from_ref: The baseline, named explicitly.
+ to_ref: The later snapshot, named explicitly.
+ store_root: Path to the ``.dstrack/`` directory.
+ root: Path root a path reference is made relative to.
+
+ Returns:
+ The baseline and the later snapshot, in that order.
+
+ Raises:
+ typer.BadParameter: If the two spellings are mixed, if more than two
+ positional references are given, or if none is given at all.
+ SnapshotNotFoundError: If an implied side does not exist, e.g. the
+ parent of a dataset's first snapshot.
+ """
+ explicit = from_ref is not None or to_ref is not None
+ if explicit and specs:
+ raise typer.BadParameter(
+ "Name the snapshots either positionally or with --from/--to, not both."
+ )
+ if len(specs) > _PAIR:
+ raise typer.BadParameter(
+ f"Expected at most two snapshots to compare, got {len(specs)}."
+ )
+ if not explicit and not specs:
+ raise typer.BadParameter("Name at least one snapshot or dataset to compare.")
+
+ def resolve(spec: str) -> ResolvedSnapshot:
+ return resolve_snapshot_ref(spec, store_root=store_root, root=root)
+
+ if from_ref is not None and to_ref is not None:
+ return resolve(from_ref), resolve(to_ref)
+ if to_ref is not None:
+ new = resolve(to_ref)
+ return _parent_of(new, store_root=store_root), new
+ if from_ref is not None:
+ # Only the baseline was named, so the question is "what has happened
+ # since?", which the dataset's latest snapshot answers.
+ old = resolve(from_ref)
+ return old, _head_of(old.dataset_id, from_ref, store_root=store_root)
+
+ if len(specs) == _PAIR:
+ return resolve(specs[0]), resolve(specs[1])
+
+ new = resolve(specs[0])
+ return _parent_of(new, store_root=store_root), new
+
+
+def _parent_of(snapshot: ResolvedSnapshot, *, store_root: Path) -> ResolvedSnapshot:
+ """Return the snapshot a resolved one succeeded.
+
+ The parent is read from the payload already in hand rather than looked up
+ again: a snapshot records the one it followed, which makes it the
+ authoritative answer and a free one.
+
+ Args:
+ snapshot: The snapshot whose predecessor is wanted.
+ store_root: Path to the ``.dstrack/`` directory.
+
+ Returns:
+ The predecessor, with its payload read.
+
+ Raises:
+ SnapshotNotFoundError: If the snapshot has no parent, i.e. it is the
+ first its dataset recorded.
+ """
+ parent = snapshot.payload.get("parent_snapshot_id")
+ if not isinstance(parent, str) or not parent:
+ raise SnapshotNotFoundError(
+ f"Snapshot {snapshot.snapshot_id[:_SHORT_ID_LEN]} is the first "
+ f"snapshot of dataset {snapshot.dataset_id}; there is nothing to "
+ "compare it against. Name a second snapshot explicitly."
+ )
+ return _load(
+ store.SnapshotLocation(dataset_id=snapshot.dataset_id, snapshot_id=parent),
+ snapshot.spec,
+ store_root=store_root,
+ )
+
+
+def _head_of(dataset_id: str, spec: str, *, store_root: Path) -> ResolvedSnapshot:
+ """Return a dataset's latest snapshot.
+
+ Args:
+ dataset_id: The dataset.
+ spec: What the user typed to name it, for error messages.
+ store_root: Path to the ``.dstrack/`` directory.
+
+ Returns:
+ The snapshot the dataset's ``HEAD`` names.
+
+ Raises:
+ SnapshotNotFoundError: If the dataset has never been snapshotted.
+ """
+ head = store.read_head(dataset_id, store_root=store_root)
+ if head is None:
+ raise SnapshotNotFoundError(
+ f"Dataset {dataset_id} has no snapshots yet, so there is nothing "
+ "to compare. Run `dstrack track ` first."
+ )
+ return _load(
+ store.SnapshotLocation(dataset_id=dataset_id, snapshot_id=head),
+ spec,
+ store_root=store_root,
+ )
+
+
+def _load(
+ location: store.SnapshotLocation, spec: str, *, store_root: Path
+) -> ResolvedSnapshot:
+ """Read a located snapshot's payload.
+
+ Args:
+ location: Which dataset and snapshot to read.
+ spec: What the user typed, carried through for error messages.
+ store_root: Path to the ``.dstrack/`` directory.
+
+ Returns:
+ The snapshot with its payload.
+
+ Raises:
+ SnapshotNotFoundError: If the payload is not on disk.
+ StoreCorruptionError: If it cannot be read back as a JSON object.
+ """
+ return ResolvedSnapshot(
+ dataset_id=location.dataset_id,
+ snapshot_id=location.snapshot_id,
+ payload=store.read_snapshot(
+ location.dataset_id, location.snapshot_id, store_root=store_root
+ ),
+ spec=spec,
+ )
+
+
+def _looks_like_prefix(spec: str) -> bool:
+ """Return whether `spec` could be the leading characters of a snapshot id.
+
+ Args:
+ spec: The reference, as the user typed it.
+
+ Returns:
+ ``True`` if it is hex-and-dashes and of a plausible length. A shorter
+ string would match almost anything, and a longer one is not a UUID at
+ all.
+ """
+ return _MIN_PREFIX_LEN <= len(spec) <= _MAX_PREFIX_LEN and set(spec) <= _HEX_CHARS
+
+
+def _match_prefix(
+ prefix: str, spec: str, *, store_root: Path
+) -> store.SnapshotLocation | None:
+ """Find the one snapshot whose id starts with `prefix`.
+
+ Args:
+ prefix: The canonical id or leading characters to match.
+ spec: What the user typed, for the error message.
+ store_root: Path to the ``.dstrack/`` directory.
+
+ Returns:
+ The single match, or ``None`` if nothing matched.
+
+ Raises:
+ AmbiguousSnapshotError: If more than one snapshot matched.
+ """
+ matches = cache.find_snapshots_by_prefix(
+ prefix, store_root=store_root, limit=_MAX_CANDIDATES
+ )
+ if not matches:
+ return None
+ if len(matches) == 1:
+ return matches[0]
+ raise AmbiguousSnapshotError(
+ f"Snapshot id {spec!r} is ambiguous; it matches {len(matches)} "
+ f"snapshots:\n{_candidates(matches, store_root=store_root)}\n"
+ "Give more characters of the id."
+ )
+
+
+def _candidates(matches: Sequence[store.SnapshotLocation], *, store_root: Path) -> str:
+ """Describe ambiguous matches well enough to tell them apart.
+
+ A bare list of ids is no help in choosing between them, so each is shown
+ with what its snapshot recorded. A payload that cannot be read is still
+ listed by id: it is a candidate whether or not it describes itself.
+
+ Args:
+ matches: The snapshots a prefix matched.
+ store_root: Path to the ``.dstrack/`` directory.
+
+ Returns:
+ One indented line per candidate.
+ """
+ lines = []
+ for match in matches:
+ try:
+ payload = store.read_snapshot(
+ match.dataset_id, match.snapshot_id, store_root=store_root
+ )
+ except (OSError, SnapshotNotFoundError, StoreCorruptionError):
+ payload = {}
+ name = payload.get("dataset_name") or "-"
+ created = payload.get("created_at") or "-"
+ lines.append(
+ f" {match.snapshot_id[:_SHORT_ID_LEN]} {name} {created} "
+ f"dataset {match.dataset_id}"
+ )
+ return "\n".join(lines)
diff --git a/src/dstrack/assets/diff-report.png b/src/dstrack/assets/diff-report.png
new file mode 100644
index 0000000..8458f1a
Binary files /dev/null and b/src/dstrack/assets/diff-report.png differ
diff --git a/src/dstrack/assets/dstrack-logo.png b/src/dstrack/assets/dstrack-logo.png
new file mode 100644
index 0000000..ae0d18b
Binary files /dev/null and b/src/dstrack/assets/dstrack-logo.png differ
diff --git a/src/dstrack/cache.py b/src/dstrack/cache.py
index 547cda7..a9f6f3b 100644
--- a/src/dstrack/cache.py
+++ b/src/dstrack/cache.py
@@ -28,13 +28,13 @@
import contextlib
import logging
import sqlite3
-from collections.abc import Iterator
+from collections.abc import Generator
from pathlib import Path
from typing import Final
from dstrack import store
from dstrack.errors import IndexUnusable
-from dstrack.store import DatasetSummary, LogEntry
+from dstrack.store import DatasetSummary, LogEntry, SnapshotLocation
_log = logging.getLogger(__name__)
@@ -45,6 +45,9 @@
# than querying it with the wrong shape.
_SCHEMA_VERSION: Final = "1"
+# Escape character for SQL ``LIKE`` patterns built from user-supplied text.
+_LIKE_ESCAPE: Final = "\\"
+
_SCHEMA: Final = """
CREATE TABLE meta (
key TEXT PRIMARY KEY,
@@ -222,6 +225,62 @@ def find_dataset_by_path(dataset_path: str, *, store_root: Path) -> str | None:
return dataset_id
+def find_snapshots_by_prefix(
+ prefix: str, *, store_root: Path, limit: int = 10
+) -> list[SnapshotLocation]:
+ """Return the snapshots whose id starts with `prefix`, sorted by id.
+
+ Snapshot ids are UUID4s, which nobody types in full, so a user names one by
+ its leading characters the way git takes a short commit hash. The lookup
+ spans every dataset: a snapshot id says nothing about which dataset holds
+ it, so which one does is part of the answer rather than part of the
+ question.
+
+ Args:
+ prefix: Leading characters of a snapshot id. Matched literally --
+ SQL's ``LIKE`` wildcards are escaped, so a prefix containing ``_``
+ or ``%`` matches only ids that really contain them.
+ store_root: Path to the ``.dstrack/`` directory.
+ limit: Cap on how many matches are returned. Reporting an ambiguous
+ prefix needs enough candidates to list back to the user, not all
+ of them.
+
+ Returns:
+ Every matching snapshot, sorted by id and capped at `limit`. Empty if
+ the prefix matches nothing, or the index holds no snapshots yet.
+ """
+ with _connect(store_root) as conn:
+ rows = conn.execute(
+ "SELECT dataset_id, snapshot_id FROM snapshots "
+ f"WHERE snapshot_id LIKE ? ESCAPE '{_LIKE_ESCAPE}' "
+ "ORDER BY snapshot_id LIMIT ?",
+ (f"{_escape_like(prefix)}%", limit),
+ ).fetchall()
+ return [
+ SnapshotLocation(dataset_id=row["dataset_id"], snapshot_id=row["snapshot_id"])
+ for row in rows
+ ]
+
+
+def _escape_like(value: str) -> str:
+ """Escape a string for use as a literal prefix in a SQL ``LIKE`` pattern.
+
+ ``_`` matches any single character and ``%`` any run of them, so a prefix
+ carrying either would silently match ids the user did not name. The escape
+ character itself has to go first, or escaping the others would re-escape
+ the backslashes just introduced.
+
+ Args:
+ value: The literal text to match.
+
+ Returns:
+ `value` with every ``LIKE`` meta character escaped.
+ """
+ for char in (_LIKE_ESCAPE, "%", "_"):
+ value = value.replace(char, f"{_LIKE_ESCAPE}{char}")
+ return value
+
+
def list_datasets(*, store_root: Path) -> list[DatasetSummary]:
"""Return every indexed dataset, described by its latest snapshot.
@@ -248,7 +307,7 @@ def list_datasets(*, store_root: Path) -> list[DatasetSummary]:
@contextlib.contextmanager
-def _connect(store_root: Path) -> Iterator[sqlite3.Connection]:
+def _connect(store_root: Path) -> Generator[sqlite3.Connection]:
"""Open the index, creating its parent directory if needed.
Args:
diff --git a/src/dstrack/console.py b/src/dstrack/console.py
index c19eec2..98ee323 100644
--- a/src/dstrack/console.py
+++ b/src/dstrack/console.py
@@ -27,6 +27,16 @@ def display(renderable: RenderableType) -> None:
console.print(renderable)
+def width() -> int:
+ """Return the console's current width in cells.
+
+ For output whose layout has to adapt to the terminal. Reading it here
+ rather than inside a renderer keeps the renderer a pure function of its
+ arguments, and so testable at any width.
+ """
+ return console.width
+
+
def success(message: str) -> None:
"""Print a success message prefixed with a check mark."""
console.print(f"[bold green]\N{HEAVY CHECK MARK}[/bold green] {escape(message)}")
diff --git a/src/dstrack/diff/__init__.py b/src/dstrack/diff/__init__.py
new file mode 100644
index 0000000..65e7eaa
--- /dev/null
+++ b/src/dstrack/diff/__init__.py
@@ -0,0 +1,103 @@
+"""Comparison of two dataset snapshots.
+
+Takes two snapshot payloads -- the plain dicts
+[SnapshotBuilder][dstrack.snapshot.SnapshotBuilder] produces and
+[write_snapshot][dstrack.store.write_snapshot] stores -- and reports what
+changed between them: schema, dataset-level metrics, and per-column
+statistics, each classified by how much it should worry the reader.
+
+Neither input has to come from a store. Every field is read defensively and
+anything missing degrades to "not recorded", so a dict assembled by hand, or
+one written by a newer dstrack, is compared as far as it can be rather than
+refused.
+
+Layers
+------
+- [diff_snapshots][dstrack.diff.diff_snapshots] is the whole public entry
+ point.
+- [SnapshotDiff.to_dict][dstrack.diff.SnapshotDiff.to_dict] renders that result
+ as plain JSON-serializable data, for a pipeline to consume.
+- [DiffOptions][dstrack.diff.DiffOptions] carries the thresholds that decide
+ unchanged from changed from significant; [preset][dstrack.diff.preset]
+ returns the named sensitivities the CLI offers.
+
+Examples:
+ ```python
+ >>> from dstrack.diff import diff_snapshots
+ >>> old = {
+ ... "dataset_name": "customers",
+ ... "num_rows": 100,
+ ... "columns": [{"name": "age", "dtype": "int64", "nullable": True}],
+ ... "column_stats": {
+ ... "age": {"null_count": 0, "null_fraction": 0.0, "mean": 30.0, "std": 5.0}
+ ... },
+ ... }
+ >>> new = {
+ ... "dataset_name": "customers",
+ ... "num_rows": 150,
+ ... "columns": [{"name": "age", "dtype": "float64", "nullable": True}],
+ ... "column_stats": {
+ ... "age": {"null_count": 15, "null_fraction": 0.1, "mean": 42.0, "std": 5.0}
+ ... },
+ ... }
+ >>> result = diff_snapshots(old, new)
+ >>> result.severity.value
+ 'critical'
+ >>> [change.name for change in result.schema.type_changed]
+ ['age']
+ >>> result.summary.row_delta
+ 50
+ >>> sorted(result.to_dict())[:3]
+ ['columns', 'diff_format_version', 'identical']
+
+ ```
+
+ Comparing two snapshots already in a store means reading their payloads
+ first, with [read_snapshot][dstrack.store.read_snapshot].
+"""
+
+from dstrack.diff._engine import diff_snapshots
+from dstrack.diff._model import (
+ DEFAULT_OPTIONS,
+ DIFF_FORMAT_VERSION,
+ PRESETS,
+ ColumnChangeKind,
+ ColumnKind,
+ ColumnSchemaChange,
+ ColumnStatDiff,
+ DiffOptions,
+ DiffSummary,
+ DistributionShift,
+ MetricDiff,
+ SchemaDiff,
+ SetChange,
+ Severity,
+ ShiftMethod,
+ SnapshotDiff,
+ SnapshotRef,
+ preset,
+ worst,
+)
+
+__all__ = [
+ "DEFAULT_OPTIONS",
+ "DIFF_FORMAT_VERSION",
+ "PRESETS",
+ "ColumnChangeKind",
+ "ColumnKind",
+ "ColumnSchemaChange",
+ "ColumnStatDiff",
+ "DiffOptions",
+ "DiffSummary",
+ "DistributionShift",
+ "MetricDiff",
+ "SchemaDiff",
+ "SetChange",
+ "Severity",
+ "ShiftMethod",
+ "SnapshotDiff",
+ "SnapshotRef",
+ "diff_snapshots",
+ "preset",
+ "worst",
+]
diff --git a/src/dstrack/diff/_engine.py b/src/dstrack/diff/_engine.py
new file mode 100644
index 0000000..632e294
--- /dev/null
+++ b/src/dstrack/diff/_engine.py
@@ -0,0 +1,1080 @@
+"""Comparing two snapshot payloads and classifying what changed.
+
+Three kinds of comparison happen here, and they are kept apart on purpose.
+
+*Structural* comparison is exact: a column is in both column lists or it is
+not, and its dtype either matches or it does not. Nothing is inferred and
+nothing is thresholded.
+
+*Scalar* comparison asks how far a number moved, and needs to know what kind of
+number it is. A count is judged by its relative change; a quantity already
+expressed as a fraction of one is judged by its absolute shift, because
+relative change is meaningless near zero -- a null fraction going from 0.001 to
+0.002 has doubled and means nothing.
+
+*Distributional* comparison asks whether a column's values still fall where
+they used to, which no single scalar answers. Each column kind gets the measure
+its recorded statistics can support, and every score is reported alongside the
+[ShiftMethod][dstrack.diff._model.ShiftMethod] that produced it, since the
+scales are not comparable to each other.
+"""
+
+import math
+from collections.abc import Mapping, Sequence
+from datetime import datetime
+from typing import Any, Final
+
+from dstrack.diff import _extract
+from dstrack.diff._model import (
+ DEFAULT_OPTIONS,
+ ColumnChangeKind,
+ ColumnKind,
+ ColumnSchemaChange,
+ ColumnStatDiff,
+ DiffOptions,
+ DiffSummary,
+ DistributionShift,
+ MetricDiff,
+ SchemaDiff,
+ SetChange,
+ Severity,
+ ShiftMethod,
+ SnapshotDiff,
+ SnapshotRef,
+ worst,
+)
+from dstrack.errors import IncompatibleSnapshotError
+from dstrack.snapshot._version import SUPPORTED_FORMAT_VERSIONS
+
+# Metrics that are already fractions of one, and so are judged by how far they
+# moved in absolute terms rather than by their relative change.
+_FRACTION_METRICS: Final[frozenset[str]] = frozenset(
+ {
+ "null_fraction",
+ "duplicate_row_fraction",
+ "top_values_coverage",
+ "near_duplicate_estimate",
+ }
+)
+
+# Dataset-level scalars, in the order they are rendered.
+_DATASET_METRICS: Final[tuple[str, ...]] = (
+ "num_rows",
+ "num_columns",
+ "duplicate_row_fraction",
+ "near_duplicate_estimate",
+)
+
+# Set-valued dataset fields. Joining one is a quality signal worth acting on;
+# leaving it is good news, and is reported without raising the alarm.
+_SET_FIELDS: Final[tuple[str, ...]] = ("constant_columns", "high_null_columns")
+
+_NULL_METRICS: Final[tuple[str, ...]] = ("null_fraction", "null_count")
+_PERCENTILES: Final[tuple[str, ...]] = ("p5", "p25", "p50", "p75", "p95", "p99")
+_NUMERIC_METRICS: Final[tuple[str, ...]] = (
+ *_NULL_METRICS,
+ "num_unique",
+ "mean",
+ "std",
+ "min",
+ "max",
+)
+_STRING_METRICS: Final[tuple[str, ...]] = (
+ *_NULL_METRICS,
+ "num_unique",
+ "top_values_coverage",
+ "avg_char_length",
+ "min_char_length",
+ "max_char_length",
+ "avg_token_count",
+ "min_token_count",
+ "max_token_count",
+)
+_DATETIME_METRICS: Final[tuple[str, ...]] = (*_NULL_METRICS, "range_days")
+
+# Smoothing for PSI. A bucket empty on one side makes the log term infinite, so
+# every bucket is credited half an observation before the counts are turned into
+# proportions -- the Jeffreys prior. A fixed floor would do as well at keeping
+# the arithmetic finite, but not at keeping the answer meaningful: it treats an
+# empty bucket in a four-row snapshot as being as certainly empty as one in a
+# four-million-row snapshot, and reports the first as near-total drift. Half an
+# observation is instead a large share of a tiny sample and a negligible one of
+# a large sample, which is exactly how confident the emptiness is.
+_PSI_PRIOR: Final = 0.5
+# The bucket standing for everything outside a string column's top values, so
+# that a shift confined to the untracked tail is still visible.
+_OTHER_BUCKET: Final = "\x00other"
+_MIN_HISTOGRAM_EDGES: Final = 2
+# Below this a measured shift is floating-point noise, not movement. Two
+# identical histograms can score a few multiples of epsilon rather than a
+# clean zero, and that must still read as "nothing changed".
+_NEGLIGIBLE: Final = 1e-9
+
+
+def diff_snapshots(
+ old: Mapping[str, Any],
+ new: Mapping[str, Any],
+ *,
+ options: DiffOptions | None = None,
+ old_dataset_id: str | None = None,
+ new_dataset_id: str | None = None,
+) -> SnapshotDiff:
+ """Compare two snapshot payloads.
+
+ Args:
+ old: The earlier snapshot's payload, i.e. the baseline.
+ new: The later snapshot's payload.
+ options: Thresholds deciding what counts as a significant change.
+ Defaults to [DEFAULT_OPTIONS][dstrack.diff._model.DEFAULT_OPTIONS].
+ old_dataset_id: Dataset the baseline belongs to, if known. Never
+ present in a payload, so it can only come from whoever read it.
+ new_dataset_id: Dataset the later snapshot belongs to, if known.
+
+ Returns:
+ Everything that differs between them, each difference classified by
+ severity.
+
+ Raises:
+ IncompatibleSnapshotError: If either payload records a snapshot format
+ version this dstrack cannot read. Any supported version is accepted,
+ including two different ones, and a payload that records none is
+ compared as far as its fields allow.
+ """
+ active = options if options is not None else DEFAULT_OPTIONS
+
+ old_ref = _extract.snapshot_ref(old, dataset_id=old_dataset_id)
+ new_ref = _extract.snapshot_ref(new, dataset_id=new_dataset_id)
+ _check_format_version(old_ref)
+ _check_format_version(new_ref)
+
+ schema = _diff_schema(old, new, old_ref=old_ref, new_ref=new_ref)
+ metrics = _diff_dataset_metrics(old, new, options=active)
+ sets = _diff_sets(old, new)
+ columns = _diff_columns(old, new, options=active)
+
+ severity = worst(
+ [
+ schema.severity,
+ worst(m.severity for m in metrics),
+ worst(s.severity for s in sets),
+ worst(c.severity for c in columns),
+ ]
+ )
+ return SnapshotDiff(
+ old=old_ref,
+ new=new_ref,
+ identical=severity is Severity.UNCHANGED and not schema.changed,
+ same_dataset=_same_dataset(old_ref, new_ref),
+ source_hash_equal=_hashes_equal(old_ref.source_hash, new_ref.source_hash),
+ schema=schema,
+ metrics=metrics,
+ sets=sets,
+ columns=columns,
+ summary=_summarize(schema, columns, metrics, severity=severity),
+ severity=severity,
+ options=active,
+ )
+
+
+def _check_format_version(ref: SnapshotRef) -> None:
+ """Raise if a snapshot records a format version that cannot be compared.
+
+ Only a version that is *recorded and unknown* is refused. Each side is
+ checked against every version this dstrack reads, not against the one it
+ writes, so snapshots taken before an upgrade stay comparable -- with each
+ other and with snapshots taken after it. A payload recording none is a
+ hand-built dict, which is a supported input; refusing it would make the
+ library unusable for anything but stored snapshots.
+
+ Args:
+ ref: The identity fields read from one side.
+
+ Raises:
+ IncompatibleSnapshotError: If the recorded version is not one this
+ dstrack can read, typically because a newer dstrack wrote it.
+ """
+ if (
+ ref.format_version is not None
+ and ref.format_version not in SUPPORTED_FORMAT_VERSIONS
+ ):
+ supported = ", ".join(repr(v) for v in sorted(SUPPORTED_FORMAT_VERSIONS))
+ raise IncompatibleSnapshotError(
+ f"Snapshot {ref.snapshot_id or ''} is in format version "
+ f"{ref.format_version!r}, which this dstrack cannot read "
+ f"(supported: {supported}). It may have been written by a newer "
+ "dstrack."
+ )
+
+
+def _same_dataset(old: SnapshotRef, new: SnapshotRef) -> bool:
+ """Return whether both sides are known to belong to the same dataset.
+
+ Args:
+ old: Identity of the baseline.
+ new: Identity of the later snapshot.
+
+ Returns:
+ ``True`` when both carry the same dataset id, or when neither carries
+ one at all -- two hand-built payloads are taken at face value as two
+ versions of one thing, which is what a caller comparing them means.
+ """
+ if old.dataset_id is None and new.dataset_id is None:
+ return True
+ return old.dataset_id == new.dataset_id
+
+
+def _hashes_equal(old: str | None, new: str | None) -> bool:
+ """Return whether two recorded source hashes prove identical bytes.
+
+ Args:
+ old: The baseline's ``source_hash``.
+ new: The later snapshot's ``source_hash``.
+
+ Returns:
+ ``True`` only if both are present, non-empty, and equal. An empty hash
+ is what a non-file source records, and two of them say nothing about
+ whether the sources matched.
+ """
+ return bool(old) and bool(new) and old == new
+
+
+# ---------------------------------------------------------------------------
+# Schema
+# ---------------------------------------------------------------------------
+
+
+def _diff_schema(
+ old: Mapping[str, Any],
+ new: Mapping[str, Any],
+ *,
+ old_ref: SnapshotRef,
+ new_ref: SnapshotRef,
+) -> SchemaDiff:
+ """Compare the two snapshots' column lists.
+
+ A renamed column reads as one removal and one addition. Snapshots carry no
+ row-level identity, so the only way to guess otherwise is "same dtype,
+ similar statistics", which cannot tell a rename from a genuine drop-and-add
+ of two similar features -- and a wrong guess silently corrupts the
+ per-column drift table, which is the part a reader acts on.
+
+ Args:
+ old: The baseline payload.
+ new: The later payload.
+ old_ref: Identity fields already read from `old`.
+ new_ref: Identity fields already read from `new`.
+
+ Returns:
+ Every schema-level difference, classified.
+ """
+ old_columns = _extract.column_schema(old)
+ new_columns = _extract.column_schema(new)
+
+ added = tuple(
+ ColumnSchemaChange(
+ name=name,
+ kind=ColumnChangeKind.ADDED,
+ old_dtype=None,
+ new_dtype=entry.dtype,
+ old_nullable=None,
+ new_nullable=entry.nullable,
+ severity=Severity.WARNING,
+ )
+ for name, entry in new_columns.items()
+ if name not in old_columns
+ )
+ removed = tuple(
+ ColumnSchemaChange(
+ name=name,
+ kind=ColumnChangeKind.REMOVED,
+ old_dtype=entry.dtype,
+ new_dtype=None,
+ old_nullable=entry.nullable,
+ new_nullable=None,
+ # Critical rather than merely notable: anything reading this
+ # dataset by name breaks outright, where an addition it ignores.
+ severity=Severity.CRITICAL,
+ )
+ for name, entry in old_columns.items()
+ if name not in new_columns
+ )
+
+ type_changed: list[ColumnSchemaChange] = []
+ nullability_changed: list[ColumnSchemaChange] = []
+ reordered: list[str] = []
+ for name, new_entry in new_columns.items():
+ old_entry = old_columns.get(name)
+ if old_entry is None:
+ continue
+ if old_entry.dtype != new_entry.dtype:
+ type_changed.append(
+ ColumnSchemaChange(
+ name=name,
+ kind=ColumnChangeKind.TYPE_CHANGED,
+ old_dtype=old_entry.dtype,
+ new_dtype=new_entry.dtype,
+ old_nullable=old_entry.nullable,
+ new_nullable=new_entry.nullable,
+ severity=Severity.CRITICAL,
+ )
+ )
+ elif old_entry.nullable != new_entry.nullable:
+ nullability_changed.append(
+ ColumnSchemaChange(
+ name=name,
+ kind=ColumnChangeKind.NULLABILITY_CHANGED,
+ old_dtype=old_entry.dtype,
+ new_dtype=new_entry.dtype,
+ old_nullable=old_entry.nullable,
+ new_nullable=new_entry.nullable,
+ severity=Severity.INFO,
+ )
+ )
+ if old_entry.position != new_entry.position:
+ reordered.append(name)
+
+ changes = (*added, *removed, *type_changed, *nullability_changed)
+ return SchemaDiff(
+ added=added,
+ removed=removed,
+ type_changed=tuple(type_changed),
+ nullability_changed=tuple(nullability_changed),
+ reordered=tuple(reordered),
+ schema_hash_equal=_schema_hashes_equal(old_ref, new_ref, changes),
+ severity=worst(c.severity for c in changes),
+ )
+
+
+def _schema_hashes_equal(
+ old: SnapshotRef, new: SnapshotRef, changes: Sequence[ColumnSchemaChange]
+) -> bool:
+ """Return whether both sides describe the same schema.
+
+ Args:
+ old: Identity of the baseline.
+ new: Identity of the later snapshot.
+ changes: The column-level differences already found.
+
+ Returns:
+ The recorded hashes' equality when both sides recorded one, else
+ whether any column-level difference was found. The hash is only a
+ shortcut; the column lists are what the rest of the diff is built from.
+ """
+ if old.schema_hash and new.schema_hash:
+ return old.schema_hash == new.schema_hash
+ return not changes
+
+
+# ---------------------------------------------------------------------------
+# Scalars
+# ---------------------------------------------------------------------------
+
+
+def _diff_dataset_metrics(
+ old: Mapping[str, Any], new: Mapping[str, Any], *, options: DiffOptions
+) -> tuple[MetricDiff, ...]:
+ """Compare the dataset-level scalars both payloads carry.
+
+ Args:
+ old: The baseline payload.
+ new: The later payload.
+ options: The active thresholds.
+
+ Returns:
+ One entry per metric at least one side recorded, in render order. A
+ metric neither side recorded is omitted entirely, which is how fields
+ the current builders never produce -- ``near_duplicate_estimate`` and
+ the row sketches -- are tolerated without being special-cased.
+ """
+ return tuple(
+ diff
+ for key in _DATASET_METRICS
+ if (
+ diff := _compare(
+ key,
+ _extract.number(old.get(key)),
+ _extract.number(new.get(key)),
+ options=options,
+ )
+ )
+ is not None
+ )
+
+
+def _compare(
+ key: str, old: float | None, new: float | None, *, options: DiffOptions
+) -> MetricDiff | None:
+ """Compare one scalar and classify how far it moved.
+
+ Args:
+ key: The metric's stable name, which decides how it is judged.
+ old: The baseline value, or ``None`` if not recorded.
+ new: The later value, or ``None`` if not recorded.
+ options: The active thresholds.
+
+ Returns:
+ The comparison, or ``None`` if neither side recorded the metric and
+ there is therefore nothing to say about it.
+ """
+ if old is None and new is None:
+ return None
+ if old is None or new is None:
+ # A metric appearing or disappearing is worth showing, but it is a
+ # change in what was recorded, not a change in the data.
+ return MetricDiff(
+ key=key,
+ old=old,
+ new=new,
+ delta=None,
+ pct_change=None,
+ severity=Severity.INFO,
+ )
+
+ if math.isclose(
+ old, new, rel_tol=options.rel_tolerance, abs_tol=options.abs_tolerance
+ ):
+ return MetricDiff(key, old, new, 0.0, 0.0, Severity.UNCHANGED)
+
+ delta = new - old
+ pct_change = delta / abs(old) if old != 0 else None
+ return MetricDiff(
+ key=key,
+ old=old,
+ new=new,
+ delta=delta,
+ pct_change=pct_change,
+ severity=_classify(key, delta=delta, pct_change=pct_change, options=options),
+ )
+
+
+def _classify(
+ key: str, *, delta: float, pct_change: float | None, options: DiffOptions
+) -> Severity:
+ """Grade a change that is already known to be real.
+
+ Args:
+ key: The metric's stable name.
+ delta: How far the value moved.
+ pct_change: The move as a fraction of the baseline, or ``None`` when
+ the baseline was zero.
+ options: The active thresholds.
+
+ Returns:
+ The severity to report.
+ """
+ if key in _FRACTION_METRICS:
+ return _band(
+ abs(delta), options.warn_fraction_shift, options.critical_fraction_shift
+ )
+ if pct_change is None:
+ # Zero to something. Structurally notable -- no rows to some rows, no
+ # nulls to some nulls -- and on no relative scale that could grade it.
+ return Severity.WARNING
+ return _band(abs(pct_change), options.warn_pct_change, options.critical_pct_change)
+
+
+def _band(magnitude: float, warn: float, critical: float) -> Severity:
+ """Place a magnitude in a severity band.
+
+ Args:
+ magnitude: The absolute size of the change.
+ warn: Where ``WARNING`` starts.
+ critical: Where ``CRITICAL`` starts.
+
+ Returns:
+ The band it falls in. Anything real but below `warn` is ``INFO``: it
+ happened, and the reader decides whether it matters. A magnitude at
+ the floor is ``UNCHANGED``, which is what keeps a snapshot compared
+ against itself reporting as identical -- the distribution measures are
+ graded here directly, and for them zero means zero rather than "too
+ small to have been filtered out already".
+ """
+ if magnitude <= _NEGLIGIBLE:
+ return Severity.UNCHANGED
+ if magnitude >= critical:
+ return Severity.CRITICAL
+ if magnitude >= warn:
+ return Severity.WARNING
+ return Severity.INFO
+
+
+def _diff_sets(old: Mapping[str, Any], new: Mapping[str, Any]) -> tuple[SetChange, ...]:
+ """Compare the set-valued dataset fields.
+
+ Args:
+ old: The baseline payload.
+ new: The later payload.
+
+ Returns:
+ One entry per field whose membership changed. A column *joining*
+ ``constant_columns`` or ``high_null_columns`` is a warning -- a feature
+ just went dead or mostly-null -- while one leaving is reported as
+ information, since a column that started varying again is good news.
+ """
+ changes: list[SetChange] = []
+ for key in _SET_FIELDS:
+ before = set(_extract.string_tuple(old.get(key)))
+ after = set(_extract.string_tuple(new.get(key)))
+ entered = tuple(sorted(after - before))
+ left = tuple(sorted(before - after))
+ if not entered and not left:
+ continue
+ changes.append(
+ SetChange(
+ key=key,
+ added=entered,
+ removed=left,
+ severity=Severity.WARNING if entered else Severity.INFO,
+ )
+ )
+ return tuple(changes)
+
+
+# ---------------------------------------------------------------------------
+# Columns
+# ---------------------------------------------------------------------------
+
+
+def _diff_columns(
+ old: Mapping[str, Any], new: Mapping[str, Any], *, options: DiffOptions
+) -> tuple[ColumnStatDiff, ...]:
+ """Compare every column's statistics.
+
+ Args:
+ old: The baseline payload.
+ new: The later payload.
+ options: The active thresholds.
+
+ Returns:
+ One entry per column either side recorded statistics for, ordered by
+ the new schema and then by name, so the output reads in the order the
+ dataset's columns actually appear.
+ """
+ old_stats = _extract.column_stats(old)
+ new_stats = _extract.column_stats(new)
+ old_schema = _extract.column_schema(old)
+ new_schema = _extract.column_schema(new)
+
+ ordered = list(new_schema)
+ ordered += sorted(set(new_stats) - set(ordered))
+ ordered += sorted((set(old_stats) | set(old_schema)) - set(ordered))
+
+ return tuple(
+ _diff_column(
+ name,
+ old_stats.get(name),
+ new_stats.get(name),
+ old_dtype=old_schema[name].dtype if name in old_schema else None,
+ new_dtype=new_schema[name].dtype if name in new_schema else None,
+ old_rows=_extract.integer(old.get("num_rows")),
+ new_rows=_extract.integer(new.get("num_rows")),
+ options=options,
+ )
+ for name in ordered
+ if name in old_stats or name in new_stats
+ )
+
+
+def _diff_column(
+ name: str,
+ old: Mapping[str, Any] | None,
+ new: Mapping[str, Any] | None,
+ *,
+ old_dtype: str | None,
+ new_dtype: str | None,
+ old_rows: int | None,
+ new_rows: int | None,
+ options: DiffOptions,
+) -> ColumnStatDiff:
+ """Compare one column's statistics.
+
+ Args:
+ name: The column's name.
+ old: Its baseline statistics, or ``None`` if the column is new.
+ new: Its later statistics, or ``None`` if the column was dropped.
+ old_dtype: Its baseline dtype, used only as a hint about which
+ statistics variant to expect.
+ new_dtype: Its later dtype.
+ old_rows: The baseline's row count, for normalizing category counts.
+ new_rows: The later row count.
+ options: The active thresholds.
+
+ Returns:
+ The column's comparison. When the two sides hold different statistics
+ variants, only the null counts are comparable, and the change is
+ reported as critical: the column now holds a different kind of value.
+ """
+ old_kind = _extract.column_kind(old, old_dtype) if old is not None else None
+ new_kind = _extract.column_kind(new, new_dtype) if new is not None else None
+ kind = new_kind if new_kind is not None else (old_kind or ColumnKind.UNKNOWN)
+
+ kind_changed = (
+ old_kind is not None and new_kind is not None and old_kind != new_kind
+ )
+ keys = _NULL_METRICS if kind_changed else _metric_keys(kind)
+
+ metrics = tuple(
+ diff
+ for key in keys
+ if (
+ diff := _compare(
+ key,
+ _stat(old, key),
+ _stat(new, key),
+ options=options,
+ )
+ )
+ is not None
+ )
+ shift = (
+ None
+ if kind_changed or old is None or new is None
+ else _distribution_shift(
+ kind,
+ old,
+ new,
+ old_rows=old_rows,
+ new_rows=new_rows,
+ options=options,
+ )
+ )
+
+ severities = [m.severity for m in metrics]
+ if shift is not None:
+ severities.append(shift.severity)
+ if kind_changed:
+ severities.append(Severity.CRITICAL)
+ elif old is None or new is None:
+ # The schema diff already reports the add or drop with its own
+ # severity; repeating it here would double-count it in the roll-up.
+ severities.append(Severity.UNCHANGED)
+
+ return ColumnStatDiff(
+ name=name,
+ kind=kind,
+ old_kind=old_kind if kind_changed else None,
+ present_in_old=old is not None,
+ present_in_new=new is not None,
+ metrics=metrics,
+ distribution_shift=shift,
+ severity=worst(severities),
+ )
+
+
+def _metric_keys(kind: ColumnKind) -> tuple[str, ...]:
+ """Return the scalars worth comparing for a statistics variant.
+
+ Args:
+ kind: The variant the column's entry holds.
+
+ Returns:
+ The metric keys, in render order.
+ """
+ if kind is ColumnKind.NUMERIC:
+ return (*_NUMERIC_METRICS, *(f"percentiles.{p}" for p in _PERCENTILES))
+ if kind is ColumnKind.STRING:
+ return _STRING_METRICS
+ if kind is ColumnKind.DATETIME:
+ return _DATETIME_METRICS
+ return _NULL_METRICS
+
+
+def _stat(stats: Mapping[str, Any] | None, key: str) -> float | None:
+ """Read one statistic out of a column's entry.
+
+ Args:
+ stats: The column's statistics, or ``None``.
+ key: The metric's name, dotted for a nested field such as
+ ``"percentiles.p50"``.
+
+ Returns:
+ The value as a float, or ``None`` if it is absent or unusable.
+ Datetime bounds are ISO strings rather than numbers, and are converted
+ to epoch seconds so they compare like any other scalar.
+ """
+ if stats is None:
+ return None
+
+ head, _, tail = key.partition(".")
+ if tail:
+ nested = stats.get(head)
+ return (
+ _extract.number(nested.get(tail)) if isinstance(nested, Mapping) else None
+ )
+
+ value = stats.get(key)
+ if key in ("min", "max") and isinstance(value, str):
+ return _epoch_seconds(value)
+ return _extract.number(value)
+
+
+def _epoch_seconds(value: str) -> float | None:
+ """Parse an ISO-8601 timestamp into epoch seconds.
+
+ Args:
+ value: The recorded timestamp, as a datetime column's bound records it.
+
+ Returns:
+ The moment as epoch seconds, or ``None`` if it cannot be parsed. A
+ trailing ``Z`` is translated first, matching how the stats computer
+ reads these values back.
+ """
+ try:
+ return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
+ except ValueError:
+ return None
+
+
+# ---------------------------------------------------------------------------
+# Distributions
+# ---------------------------------------------------------------------------
+
+
+def _distribution_shift(
+ kind: ColumnKind,
+ old: Mapping[str, Any],
+ new: Mapping[str, Any],
+ *,
+ old_rows: int | None,
+ new_rows: int | None,
+ options: DiffOptions,
+) -> DistributionShift | None:
+ """Measure how far a column's value distribution moved.
+
+ Args:
+ kind: The statistics variant, which decides the measure.
+ old: The column's baseline statistics.
+ new: The column's later statistics.
+ old_rows: The baseline's row count.
+ new_rows: The later row count.
+ options: The active thresholds.
+
+ Returns:
+ The shift, or ``None`` for a variant that records no distribution at
+ all -- a datetime or unrecognized column has bounds, not a shape.
+ """
+ if kind is ColumnKind.NUMERIC:
+ return _numeric_shift(old, new, options=options)
+ if kind is ColumnKind.STRING:
+ return _string_shift(
+ old, new, old_rows=old_rows, new_rows=new_rows, options=options
+ )
+ return None
+
+
+def _numeric_shift(
+ old: Mapping[str, Any], new: Mapping[str, Any], *, options: DiffOptions
+) -> DistributionShift:
+ """Measure a numeric column's shift, preferring PSI over its histogram.
+
+ Args:
+ old: The column's baseline statistics.
+ new: The column's later statistics.
+ options: The active thresholds.
+
+ Returns:
+ A PSI score when both histograms are usable, else a percentile-based
+ fallback, else a ``NONE`` shift saying nothing could be measured.
+ """
+ psi = _population_stability_index(old.get("histogram"), new.get("histogram"))
+ if psi is not None:
+ return DistributionShift(
+ method=ShiftMethod.PSI,
+ score=psi,
+ severity=_band(psi, options.warn_psi, options.critical_psi),
+ )
+
+ fallback = _percentile_shift(old.get("percentiles"), new.get("percentiles"))
+ if fallback is not None:
+ return DistributionShift(
+ method=ShiftMethod.PERCENTILE_SHIFT,
+ score=fallback,
+ severity=_band(
+ fallback,
+ options.warn_percentile_shift,
+ options.critical_percentile_shift,
+ ),
+ detail="histogram unusable; compared on percentiles",
+ )
+
+ return DistributionShift(
+ method=ShiftMethod.NONE,
+ score=None,
+ severity=Severity.UNCHANGED,
+ detail="no comparable distribution recorded",
+ )
+
+
+def _population_stability_index(old: Any, new: Any) -> float | None:
+ """Return the PSI between two histograms, or ``None`` if it cannot be computed.
+
+ The two histograms rarely share bin edges, since each snapshot's edges are
+ derived from its own minimum and maximum. Counts cannot be re-binned
+ exactly, so each source bin's count is spread across the bins of a shared
+ grid in proportion to how much of it they overlap -- the piecewise-uniform
+ density assumption, which is the standard choice and needs no dependency.
+
+ Args:
+ old: The baseline ``histogram`` mapping.
+ new: The later ``histogram`` mapping.
+
+ Returns:
+ The PSI, or ``None`` when either histogram is unusable: too few edges,
+ a count list that does not match its edges, no observations at all, or
+ the single degenerate bin a constant column produces, none of which
+ describe a shape that can be compared.
+ """
+ old_grid = _histogram(old)
+ new_grid = _histogram(new)
+ if old_grid is None or new_grid is None:
+ return None
+
+ old_edges, old_counts = old_grid
+ new_edges, new_counts = new_grid
+
+ lower = min(old_edges[0], new_edges[0])
+ upper = max(old_edges[-1], new_edges[-1])
+ union = sorted({e for e in (*old_edges, *new_edges) if lower <= e <= upper})
+ union = [lower, *[e for e in union if lower < e < upper], upper]
+ if len(union) < _MIN_HISTOGRAM_EDGES:
+ return None
+
+ old_p = _proportions(_redistribute(old_edges, old_counts, union))
+ new_p = _proportions(_redistribute(new_edges, new_counts, union))
+
+ return sum(
+ (after - before) * math.log(after / before)
+ for before, after in zip(old_p, new_p, strict=True)
+ )
+
+
+def _proportions(counts: Sequence[float]) -> list[float]:
+ """Turn bin counts into smoothed proportions summing to one.
+
+ Args:
+ counts: One count per bin of the shared grid.
+
+ Returns:
+ The proportions, every one of them strictly positive so the PSI's
+ logarithms are defined. See
+ [_PSI_PRIOR][dstrack.diff._engine._PSI_PRIOR] for why the smoothing is
+ a share of an observation rather than a fixed floor.
+ """
+ total = sum(counts) + _PSI_PRIOR * len(counts)
+ return [(count + _PSI_PRIOR) / total for count in counts]
+
+
+def _histogram(raw: Any) -> tuple[list[float], list[float]] | None:
+ """Read a histogram, rejecting one that cannot describe a distribution.
+
+ Args:
+ raw: The recorded ``histogram`` mapping.
+
+ Returns:
+ Its edges and counts, or ``None`` if it is unusable. A single bin is
+ rejected along with the malformed cases: `StatsComputer` emits one for
+ a constant column, and it carries no shape to compare.
+ """
+ if not isinstance(raw, Mapping):
+ return None
+ edges = _extract.float_list(raw.get("bin_edges"))
+ counts = _extract.float_list(raw.get("counts"))
+ if edges is None or counts is None:
+ return None
+ if len(edges) < _MIN_HISTOGRAM_EDGES or len(counts) != len(edges) - 1:
+ return None
+ if len(counts) < _MIN_HISTOGRAM_EDGES or sum(counts) <= 0:
+ return None
+ if edges[-1] <= edges[0]:
+ return None
+ return edges, counts
+
+
+def _redistribute(
+ edges: Sequence[float], counts: Sequence[float], union: Sequence[float]
+) -> list[float]:
+ """Spread a histogram's counts onto a shared grid.
+
+ Args:
+ edges: The histogram's own bin edges, ascending.
+ counts: Its per-bin counts.
+ union: The shared grid's edges, ascending and spanning both histograms.
+
+ Returns:
+ One count per bin of `union`, together holding the same total as
+ `counts` (up to floating-point error).
+ """
+ spread = [0.0] * (len(union) - 1)
+
+ for index, count in enumerate(counts):
+ low, high = edges[index], edges[index + 1]
+ span = high - low
+ for target in range(len(spread)):
+ overlap = min(high, union[target + 1]) - max(low, union[target])
+ if overlap <= 0:
+ continue
+ # A zero-width source bin cannot be subdivided; its whole count
+ # belongs to whichever target bin contains it.
+ spread[target] += count * (overlap / span) if span > 0 else count
+
+ return spread
+
+
+def _percentile_shift(old: Any, new: Any) -> float | None:
+ """Return the largest percentile move, relative to the baseline's spread.
+
+ Args:
+ old: The baseline ``percentiles`` mapping.
+ new: The later ``percentiles`` mapping.
+
+ Returns:
+ The move as a fraction of the baseline's p5-p95 spread, or ``None`` if
+ the two share no comparable percentile. Falling back to ``|p50|`` and
+ then to 1.0 keeps the measure defined for a column whose middle 90% is
+ a single value.
+ """
+ if not isinstance(old, Mapping) or not isinstance(new, Mapping):
+ return None
+
+ moves = [
+ abs(after - before)
+ for key in _PERCENTILES
+ if (before := _extract.number(old.get(key))) is not None
+ and (after := _extract.number(new.get(key))) is not None
+ ]
+ if not moves:
+ return None
+
+ p5 = _extract.number(old.get("p5"))
+ p95 = _extract.number(old.get("p95"))
+ p50 = _extract.number(old.get("p50"))
+ spread = (p95 - p5) if p5 is not None and p95 is not None else 0.0
+ scale = spread or abs(p50 or 0.0) or 1.0
+ return max(moves) / scale
+
+
+def _string_shift(
+ old: Mapping[str, Any],
+ new: Mapping[str, Any],
+ *,
+ old_rows: int | None,
+ new_rows: int | None,
+ options: DiffOptions,
+) -> DistributionShift:
+ """Measure a string column's shift as total variation distance.
+
+ Only the top values are recorded, so the tail they do not cover is carried
+ as one synthetic bucket. Without it, a column whose top values held steady
+ while everything else churned would score zero.
+
+ Args:
+ old: The column's baseline statistics.
+ new: The column's later statistics.
+ old_rows: The baseline's row count, for turning counts into shares.
+ new_rows: The later row count.
+ options: The active thresholds.
+
+ Returns:
+ The distance, in ``[0, 1]``, with the categories that came and went.
+ """
+ old_values = _extract.counts(old.get("top_values"))
+ new_values = _extract.counts(new.get("top_values"))
+ if not old_values and not new_values:
+ return DistributionShift(
+ method=ShiftMethod.NONE,
+ score=None,
+ severity=Severity.UNCHANGED,
+ detail="no top values recorded",
+ )
+
+ old_share, old_note = _category_shares(old, old_values, rows=old_rows)
+ new_share, new_note = _category_shares(new, new_values, rows=new_rows)
+
+ keys = set(old_share) | set(new_share)
+ distance = 0.5 * sum(
+ abs(new_share.get(key, 0.0) - old_share.get(key, 0.0)) for key in keys
+ )
+ return DistributionShift(
+ method=ShiftMethod.TOP_VALUES_TVD,
+ score=distance,
+ severity=_band(distance, options.warn_tvd, options.critical_tvd),
+ added_categories=tuple(sorted(set(new_values) - set(old_values))),
+ removed_categories=tuple(sorted(set(old_values) - set(new_values))),
+ detail=old_note or new_note,
+ )
+
+
+def _category_shares(
+ stats: Mapping[str, Any], values: Mapping[str, float], *, rows: int | None
+) -> tuple[dict[str, float], str | None]:
+ """Turn recorded category counts into shares of the column's values.
+
+ Args:
+ stats: The column's statistics, for its null fraction and coverage.
+ values: Its ``top_values`` counts.
+ rows: The snapshot's row count, or ``None`` if not recorded.
+
+ Returns:
+ The shares -- including the bucket standing for everything outside the
+ top values -- and a note when the normalization had to be weakened
+ because the row count was missing.
+ """
+ counted = sum(values.values())
+ null_fraction = _extract.number(stats.get("null_fraction")) or 0.0
+ coverage = _extract.number(stats.get("top_values_coverage"))
+
+ note: str | None = None
+ non_null = float(rows) * (1.0 - null_fraction) if rows is not None else 0.0
+ if non_null <= 0:
+ non_null = counted
+ note = "row count unavailable; shares normalized over the top values only"
+
+ shares = (
+ {key: count / non_null for key, count in values.items()} if non_null else {}
+ )
+ covered = (
+ coverage if coverage is not None else (counted / non_null if non_null else 1.0)
+ )
+ shares[_OTHER_BUCKET] = max(0.0, 1.0 - covered)
+ return shares, note
+
+
+# ---------------------------------------------------------------------------
+# Summary
+# ---------------------------------------------------------------------------
+
+
+def _summarize(
+ schema: SchemaDiff,
+ columns: Sequence[ColumnStatDiff],
+ metrics: Sequence[MetricDiff],
+ *,
+ severity: Severity,
+) -> DiffSummary:
+ """Reduce the whole comparison to the handful of numbers a reader scans first.
+
+ Args:
+ schema: The schema-level differences.
+ columns: The per-column comparisons.
+ metrics: The dataset-level comparisons.
+ severity: The diff's overall severity.
+
+ Returns:
+ The headline counts.
+ """
+ rows = next((m for m in metrics if m.key == "num_rows"), None)
+ drifted = sum(
+ 1
+ for column in columns
+ if column.severity in (Severity.WARNING, Severity.CRITICAL)
+ )
+ return DiffSummary(
+ columns_added=len(schema.added),
+ columns_removed=len(schema.removed),
+ columns_type_changed=len(schema.type_changed),
+ columns_drifted=drifted,
+ row_delta=int(rows.delta)
+ if rows is not None and rows.delta is not None
+ else None,
+ severity=severity,
+ )
diff --git a/src/dstrack/diff/_extract.py b/src/dstrack/diff/_extract.py
new file mode 100644
index 0000000..b262b21
--- /dev/null
+++ b/src/dstrack/diff/_extract.py
@@ -0,0 +1,305 @@
+"""Reading a raw snapshot payload without trusting it.
+
+The engine works on two plain mappings, and this is the only layer that touches
+them directly. Everything is read with ``.get()`` and checked at the boundary:
+a field of the wrong type, or one a payload never carried, degrades to "not
+recorded" rather than raising. That is the same stance
+[store][dstrack.store] takes when it reads a log line -- a command that
+compares two snapshots must not die on a field it was only going to print.
+
+The other job here is deciding which statistics variant a column holds. A
+payload is a dict, so the frozen dataclasses
+[StatsComputer][dstrack.snapshot.StatsComputer] produced are long gone by the
+time the engine sees them, and which arithmetic is valid has to be recovered.
+"""
+
+import math
+from collections.abc import Mapping
+from dataclasses import dataclass
+from typing import Any, Final
+
+from dstrack.diff._model import ColumnKind, SnapshotRef
+
+# The dtype vocabulary `StatsComputer` dispatches on when it picks an
+# accumulator. Kept in step with `dstrack.snapshot._stats._NUMERIC_DTYPES`.
+_NUMERIC_DTYPES: Final[frozenset[str]] = frozenset(
+ {"int8", "int16", "int32", "int64", "float16", "float32", "float64", "bool"}
+)
+_STRING_DTYPES: Final[frozenset[str]] = frozenset({"string", "str", "object"})
+_DATETIME_DTYPES: Final[frozenset[str]] = frozenset({"datetime64", "datetime"})
+
+# Keys that only one variant of the stats dataclasses ever carries. Checked in
+# this order, so a payload carrying several is read as the most specific.
+_KIND_MARKERS: Final[tuple[tuple[ColumnKind, frozenset[str]], ...]] = (
+ (ColumnKind.STRING, frozenset({"top_values", "avg_char_length"})),
+ (ColumnKind.DATETIME, frozenset({"range_days"})),
+ (ColumnKind.NUMERIC, frozenset({"histogram", "percentiles", "mean", "std"})),
+)
+
+
+@dataclass(frozen=True, slots=True)
+class SchemaEntry:
+ """One column as a snapshot's ``columns`` list described it.
+
+ Attributes:
+ name: The column's name.
+ dtype: Its recorded dtype, or ``None`` if the payload did not say.
+ nullable: Whether it was recorded as nullable, or ``None``.
+ position: Its index in the ``columns`` list, which is what makes a
+ reorder visible.
+ """
+
+ name: str
+ dtype: str | None
+ nullable: bool | None
+ position: int
+
+
+def snapshot_ref(
+ payload: Mapping[str, Any], *, dataset_id: str | None = None
+) -> SnapshotRef:
+ """Read a payload's identity fields.
+
+ Args:
+ payload: The snapshot payload.
+ dataset_id: Which dataset holds the snapshot. Never present in the
+ payload itself, so it is supplied by whoever read it out of a store.
+
+ Returns:
+ The identity of that side of the comparison, with anything the payload
+ did not record left as ``None``.
+ """
+ return SnapshotRef(
+ snapshot_id=text(payload.get("snapshot_id")),
+ dataset_id=dataset_id,
+ parent_snapshot_id=text(payload.get("parent_snapshot_id")),
+ dataset_name=text(payload.get("dataset_name")),
+ dataset_path=text(payload.get("dataset_path")),
+ created_at=text(payload.get("created_at")),
+ created_by=text(payload.get("created_by")),
+ source_type=text(payload.get("source_type")),
+ source_hash=text(payload.get("source_hash")),
+ schema_hash=text(payload.get("schema_hash")),
+ format_version=text(payload.get("format_version")),
+ num_rows=integer(payload.get("num_rows")),
+ num_columns=integer(payload.get("num_columns")),
+ )
+
+
+def column_schema(payload: Mapping[str, Any]) -> dict[str, SchemaEntry]:
+ """Read a payload's ``columns`` list, keyed by column name.
+
+ Args:
+ payload: The snapshot payload.
+
+ Returns:
+ One entry per named column, in the order the payload listed them. A
+ malformed entry, or one with no usable name, is skipped; a name listed
+ twice keeps its first position, since that is the one a reorder is
+ measured against.
+ """
+ entries: dict[str, SchemaEntry] = {}
+ raw = payload.get("columns")
+ if not isinstance(raw, list):
+ return entries
+
+ for position, item in enumerate(raw):
+ if not isinstance(item, Mapping):
+ continue
+ name = text(item.get("name"))
+ if name is None or name in entries:
+ continue
+ nullable = item.get("nullable")
+ entries[name] = SchemaEntry(
+ name=name,
+ dtype=text(item.get("dtype")),
+ nullable=nullable if isinstance(nullable, bool) else None,
+ position=position,
+ )
+ return entries
+
+
+def column_stats(payload: Mapping[str, Any]) -> dict[str, Mapping[str, Any]]:
+ """Read a payload's ``column_stats`` mapping.
+
+ Args:
+ payload: The snapshot payload.
+
+ Returns:
+ Each column's statistics, keyed by column name. Empty if the payload
+ recorded none, or recorded something that is not a mapping.
+ """
+ raw = payload.get("column_stats")
+ if not isinstance(raw, Mapping):
+ return {}
+ return {
+ str(name): stats for name, stats in raw.items() if isinstance(stats, Mapping)
+ }
+
+
+def column_kind(stats: Mapping[str, Any] | None, dtype: str | None) -> ColumnKind:
+ """Decide which statistics variant a column's entry holds.
+
+ Two sources are consulted, and they are not equally trusted. The declared
+ dtype is a hint; the keys actually present are the evidence. When they
+ disagree the keys win, because the stats are what is about to be compared
+ arithmetically, and a column declared ``float64`` whose entry carries
+ ``top_values`` would otherwise have a dict read as a number.
+
+ Args:
+ stats: The column's ``column_stats`` entry, or ``None`` if it has none.
+ dtype: The dtype the ``columns`` list declared, or ``None``.
+
+ Returns:
+ The variant to compare the column as. ``OTHER`` when only null counts
+ are present, and ``UNKNOWN`` when neither source says anything, in
+ which case only null counts are compared either way.
+ """
+ if stats is not None:
+ for kind, markers in _KIND_MARKERS:
+ if markers & stats.keys():
+ return kind
+
+ declared = _kind_from_dtype(dtype)
+ if declared is not None:
+ return declared
+
+ if stats is not None:
+ # An entry carrying nothing but null counts is what `OtherColumnStats`
+ # is: a column whose dtype the reader did not recognize.
+ return ColumnKind.OTHER
+ return ColumnKind.UNKNOWN
+
+
+def _kind_from_dtype(dtype: str | None) -> ColumnKind | None:
+ """Map a declared dtype to a statistics variant.
+
+ Args:
+ dtype: The recorded dtype, or ``None``.
+
+ Returns:
+ The variant that dtype implies, or ``None`` if there is no dtype to go
+ on. An unrecognized dtype maps to ``OTHER``, matching what
+ `StatsComputer` does with it.
+ """
+ if dtype is None:
+ return None
+ lowered = dtype.lower()
+ if lowered in _NUMERIC_DTYPES:
+ return ColumnKind.NUMERIC
+ if lowered in _STRING_DTYPES:
+ return ColumnKind.STRING
+ if lowered.startswith("datetime") or lowered in _DATETIME_DTYPES:
+ return ColumnKind.DATETIME
+ return ColumnKind.OTHER
+
+
+def number(value: Any) -> float | None:
+ """Read a value as a float, or ``None`` if it is not one.
+
+ Args:
+ value: The recorded value.
+
+ Returns:
+ The value as a float. Booleans are rejected: `bool` is a subclass of
+ `int`, and a flag that slipped into a numeric field would otherwise be
+ averaged as 0 or 1. Non-finite floats are rejected too, since every
+ comparison downstream would propagate the NaN rather than report it.
+ """
+ if isinstance(value, bool) or not isinstance(value, int | float):
+ return None
+ result = float(value)
+ return result if math.isfinite(result) else None
+
+
+def integer(value: Any) -> int | None:
+ """Read a value as an int, or ``None`` if it is not one.
+
+ Args:
+ value: The recorded value.
+
+ Returns:
+ The value as an int, or ``None``. Booleans are rejected for the same
+ reason [number][dstrack.diff._extract.number] rejects them; a float
+ carrying an exact integer is accepted, since JSON does not distinguish
+ ``3`` from ``3.0``.
+ """
+ if isinstance(value, bool):
+ return None
+ if isinstance(value, int):
+ return value
+ if isinstance(value, float) and value.is_integer():
+ return int(value)
+ return None
+
+
+def text(value: Any) -> str | None:
+ """Read a value as a string, or ``None`` if it is not one.
+
+ Args:
+ value: The recorded value.
+
+ Returns:
+ The string, or ``None``. A value of another type is *not* coerced: a
+ `dataset_name` that came back as a list is a payload that lied, and
+ showing ``"[1, 2]"`` as the name would hide that.
+ """
+ return value if isinstance(value, str) else None
+
+
+def string_tuple(value: Any) -> tuple[str, ...]:
+ """Read a value as a tuple of strings.
+
+ Args:
+ value: The recorded value, expected to be a list of names.
+
+ Returns:
+ Every string it holds, in order. Empty if it is not a list, so a
+ malformed ``constant_columns`` reads as "no constant columns" rather
+ than stopping the comparison.
+ """
+ if not isinstance(value, list):
+ return ()
+ return tuple(item for item in value if isinstance(item, str))
+
+
+def counts(value: Any) -> dict[str, float]:
+ """Read a value as a mapping of category name to count.
+
+ Args:
+ value: The recorded value, expected to be a ``top_values`` mapping.
+
+ Returns:
+ Each entry whose key is a string and whose count is a usable number.
+ Empty if the value is not a mapping.
+ """
+ if not isinstance(value, Mapping):
+ return {}
+ result: dict[str, float] = {}
+ for key, raw in value.items():
+ parsed = number(raw)
+ if isinstance(key, str) and parsed is not None:
+ result[key] = parsed
+ return result
+
+
+def float_list(value: Any) -> list[float] | None:
+ """Read a value as a list of floats.
+
+ Args:
+ value: The recorded value, e.g. a histogram's ``bin_edges``.
+
+ Returns:
+ The floats it holds, or ``None`` if it is not a list or any element is
+ unusable. Unlike the other readers this is all-or-nothing: a histogram
+ missing one edge is not a shorter histogram, it is an unusable one.
+ """
+ if not isinstance(value, list):
+ return None
+ parsed: list[float] = []
+ for item in value:
+ as_float = number(item)
+ if as_float is None:
+ return None
+ parsed.append(as_float)
+ return parsed
diff --git a/src/dstrack/diff/_model.py b/src/dstrack/diff/_model.py
new file mode 100644
index 0000000..d08d530
--- /dev/null
+++ b/src/dstrack/diff/_model.py
@@ -0,0 +1,504 @@
+"""Result types for a snapshot comparison, and the knobs that shape it.
+
+Pure data: nothing here computes a difference, and nothing here knows how one
+is displayed. The engine fills these in, the renderers read them, and
+[SnapshotDiff.to_dict][dstrack.diff._model.SnapshotDiff.to_dict] turns them
+into the JSON a pipeline consumes.
+
+Every quantity is compared through one type,
+[MetricDiff][dstrack.diff._model.MetricDiff], whether it is a dataset's row
+count or a single column's mean. One type means renderers and serializers have
+one code path rather than one per metric family, and a metric added later needs
+no new plumbing.
+"""
+
+import dataclasses
+from collections.abc import Iterable
+from dataclasses import dataclass, field
+from enum import StrEnum
+from typing import Any, Final
+
+# Bumped only when a key already emitted changes meaning or disappears. Adding
+# a key is not a version change: consumers are expected to ignore what they do
+# not recognize.
+DIFF_FORMAT_VERSION: Final = "1"
+# Relative thresholds, for counts and other unbounded magnitudes.
+DEFAULT_REL_TOLERANCE: Final = 1e-3
+DEFAULT_ABS_TOLERANCE: Final = 1e-12
+DEFAULT_WARN_PCT_CHANGE: Final = 0.10
+DEFAULT_CRITICAL_PCT_CHANGE: Final = 0.50
+# Absolute thresholds, for quantities that are already fractions in [0, 1].
+DEFAULT_WARN_FRACTION_SHIFT: Final = 0.02
+DEFAULT_CRITICAL_FRACTION_SHIFT: Final = 0.10
+# The industry-standard PSI bands: below 0.1 no real shift, above 0.25 a major one.
+DEFAULT_WARN_PSI: Final = 0.10
+DEFAULT_CRITICAL_PSI: Final = 0.25
+DEFAULT_WARN_TVD: Final = 0.10
+DEFAULT_CRITICAL_TVD: Final = 0.25
+DEFAULT_WARN_PERCENTILE_SHIFT: Final = 0.10
+DEFAULT_CRITICAL_PERCENTILE_SHIFT: Final = 0.25
+
+
+class Severity(StrEnum):
+ """How much a single comparison should worry the reader."""
+
+ UNCHANGED = "unchanged"
+ INFO = "info"
+ WARNING = "warning"
+ CRITICAL = "critical"
+
+
+# Ascending, so `worst()` can fold a group of severities into the highest one.
+_SEVERITY_ORDER: Final[tuple[Severity, ...]] = (
+ Severity.UNCHANGED,
+ Severity.INFO,
+ Severity.WARNING,
+ Severity.CRITICAL,
+)
+
+
+def worst(severities: Iterable[Severity]) -> Severity:
+ """Return the highest severity in `severities`, or ``UNCHANGED`` if empty.
+
+ How a whole diff is classified: a group is exactly as serious as its most
+ serious member, applied at every level from a column's metrics up to the
+ diff itself.
+
+ Args:
+ severities: The severities to fold.
+
+ Returns:
+ The most serious of them.
+ """
+ return max(severities, key=_SEVERITY_ORDER.index, default=Severity.UNCHANGED)
+
+
+class ColumnChangeKind(StrEnum):
+ """What happened to a column between the two schemas."""
+
+ ADDED = "added"
+ REMOVED = "removed"
+ TYPE_CHANGED = "type_changed"
+ NULLABILITY_CHANGED = "nullability_changed"
+
+
+class ColumnKind(StrEnum):
+ """Which statistics variant a column's ``column_stats`` entry holds."""
+
+ NUMERIC = "numeric"
+ STRING = "string"
+ DATETIME = "datetime"
+ OTHER = "other"
+ UNKNOWN = "unknown"
+
+
+class ShiftMethod(StrEnum):
+ """How a column's distribution shift was measured.
+
+ Each measure has its own scale and its own thresholds, so a score is only
+ meaningful alongside the method that produced it.
+ """
+
+ PSI = "psi"
+ PERCENTILE_SHIFT = "percentile_shift"
+ TOP_VALUES_TVD = "top_values_tvd"
+ NONE = "none"
+
+
+@dataclass(frozen=True, slots=True)
+class SnapshotRef:
+ """Identity of one side of the comparison, as its payload recorded it.
+
+ Every field is optional. `diff_snapshots` accepts dicts that never touched
+ a store, and a payload written by another version of dstrack may omit a
+ field entirely; neither is a reason to refuse to compare.
+
+ Attributes:
+ snapshot_id: Unique identifier of the snapshot.
+ dataset_id: Which dataset holds the snapshot. Never present in a
+ payload -- it is a store concept -- so it is supplied by the caller
+ or left unset.
+ parent_snapshot_id: Identifier of the snapshot this one was derived
+ from, if any.
+ dataset_name: Human-readable name of the dataset.
+ dataset_path: Path of the tracked data, relative to the root it was
+ tracked against.
+ created_at: Timestamp at which the snapshot was created.
+ created_by: Who or what created the snapshot.
+ source_type: Kind of source the snapshot was taken from (e.g. `file`).
+ source_hash: Content hash of the source data.
+ schema_hash: Hash of the dataset's schema.
+ format_version: Version of the snapshot payload format.
+ num_rows: Number of rows in the dataset.
+ num_columns: Number of columns in the dataset.
+ """
+
+ snapshot_id: str | None = None
+ dataset_id: str | None = None
+ parent_snapshot_id: str | None = None
+ dataset_name: str | None = None
+ dataset_path: str | None = None
+ created_at: str | None = None
+ created_by: str | None = None
+ source_type: str | None = None
+ source_hash: str | None = None
+ schema_hash: str | None = None
+ format_version: str | None = None
+ num_rows: int | None = None
+ num_columns: int | None = None
+
+
+@dataclass(frozen=True, slots=True)
+class MetricDiff:
+ """One scalar quantity as it stood on each side.
+
+ Attributes:
+ key: Stable machine name, e.g. ``"num_rows"`` or ``"percentiles.p50"``.
+ old: Value on the left side, or ``None`` if it was not recorded.
+ new: Value on the right side, or ``None`` if it was not recorded.
+ delta: ``new - old``, or ``None`` when either side is missing.
+ pct_change: ``delta / abs(old)``, or ``None`` when `old` is zero or
+ either side is missing. A fraction, not a percentage.
+ severity: How the change classifies under the active options.
+ """
+
+ key: str
+ old: float | None
+ new: float | None
+ delta: float | None
+ pct_change: float | None
+ severity: Severity
+
+
+@dataclass(frozen=True, slots=True)
+class ColumnSchemaChange:
+ """One column's schema entry, compared between snapshots.
+
+ Attributes:
+ name: Column name.
+ kind: How the column changed (added, removed, retyped, ...).
+ old_dtype: Data type on the old side, or ``None`` if absent or not
+ recorded.
+ new_dtype: Data type on the new side, or ``None`` if absent or not
+ recorded.
+ old_nullable: Nullability on the old side, or ``None`` if absent or not
+ recorded.
+ new_nullable: Nullability on the new side, or ``None`` if absent or not
+ recorded.
+ severity: How the change classifies under the active options.
+ """
+
+ name: str
+ kind: ColumnChangeKind
+ old_dtype: str | None
+ new_dtype: str | None
+ old_nullable: bool | None
+ new_nullable: bool | None
+ severity: Severity
+
+
+@dataclass(frozen=True, slots=True)
+class SchemaDiff:
+ """Every difference between the two snapshots' column lists.
+
+ Attributes:
+ added: Columns present only on the new side.
+ removed: Columns present only on the old side.
+ type_changed: Columns on both sides whose data type differs.
+ nullability_changed: Columns on both sides whose nullability differs.
+ reordered: Names present on both sides at a different position.
+ ``schema_hash`` is computed over *sorted* pairs, so a pure reorder
+ leaves the two hashes equal; this is the only field that shows it.
+ schema_hash_equal: Whether both sides recorded the same
+ ``schema_hash``. Display-only: the explicit column lists are what
+ the other fields are derived from.
+ severity: The worst severity among the schema changes.
+ """
+
+ added: tuple[ColumnSchemaChange, ...] = ()
+ removed: tuple[ColumnSchemaChange, ...] = ()
+ type_changed: tuple[ColumnSchemaChange, ...] = ()
+ nullability_changed: tuple[ColumnSchemaChange, ...] = ()
+ reordered: tuple[str, ...] = ()
+ schema_hash_equal: bool = True
+ severity: Severity = Severity.UNCHANGED
+
+ @property
+ def changed(self) -> bool:
+ """Whether any column was added, removed, or redefined."""
+ return bool(
+ self.added or self.removed or self.type_changed or self.nullability_changed
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class DistributionShift:
+ """How far a column's value distribution moved.
+
+ Attributes:
+ method: Which measure `score` is on. Always read the two together.
+ score: The measure's value, or ``None`` when `method` is ``NONE``.
+ severity: How the shift classifies under the active options.
+ added_categories: Values the new top-values list carries that the old
+ one did not. User-derived: renderers must build these with
+ `rich.text.Text`, and the PDF must escape them.
+ removed_categories: The mirror image.
+ detail: A short note on how the score was arrived at, e.g. why a
+ fallback measure was used.
+ """
+
+ method: ShiftMethod
+ score: float | None
+ severity: Severity
+ added_categories: tuple[str, ...] = ()
+ removed_categories: tuple[str, ...] = ()
+ detail: str | None = None
+
+
+@dataclass(frozen=True, slots=True)
+class ColumnStatDiff:
+ """One column's statistics, compared between snapshots.
+
+ Attributes:
+ name: Column name.
+ kind: The statistics variant on the new side, or on the old side when
+ the column was removed.
+ old_kind: The variant on the old side, when it differs from `kind`. A
+ column that changed variant has no comparable statistics beyond its
+ null counts, which is why such a change is always ``CRITICAL``.
+ present_in_old: Whether the column has statistics on the old side.
+ present_in_new: Whether the column has statistics on the new side.
+ metrics: Every scalar at least one side recorded, in a stable order.
+ distribution_shift: ``None`` for kinds that carry no distribution.
+ severity: The worst severity among the column's metrics and shift.
+ """
+
+ name: str
+ kind: ColumnKind
+ old_kind: ColumnKind | None
+ present_in_old: bool
+ present_in_new: bool
+ metrics: tuple[MetricDiff, ...]
+ distribution_shift: DistributionShift | None
+ severity: Severity
+
+
+@dataclass(frozen=True, slots=True)
+class SetChange:
+ """A set-valued dataset field, e.g. ``constant_columns``.
+
+ Attributes:
+ key: Name of the field that changed.
+ added: Members present only on the new side.
+ removed: Members present only on the old side.
+ severity: How the change classifies under the active options.
+ """
+
+ key: str
+ added: tuple[str, ...]
+ removed: tuple[str, ...]
+ severity: Severity
+
+
+@dataclass(frozen=True, slots=True)
+class DiffSummary:
+ """Headline counts, for a one-line render and a report's summary block.
+
+ Attributes:
+ columns_added: Number of columns present only on the new side.
+ columns_removed: Number of columns present only on the old side.
+ columns_type_changed: Number of columns whose data type changed.
+ columns_drifted: Columns whose own severity reached ``WARNING`` or
+ above, counting statistical movement rather than schema edits.
+ row_delta: Change in row count (new minus old), or ``None`` when either
+ side did not record one.
+ severity: The overall severity of the comparison.
+ """
+
+ columns_added: int
+ columns_removed: int
+ columns_type_changed: int
+ columns_drifted: int
+ row_delta: int | None
+ severity: Severity
+
+
+@dataclass(frozen=True, slots=True)
+class DiffOptions:
+ """Thresholds deciding unchanged vs. changed vs. significant.
+
+ Every threshold is a fraction, never a percentage.
+
+ Relative thresholds apply to counts and unbounded magnitudes. Absolute
+ *shift* thresholds apply to quantities that are already fractions in
+ ``[0, 1]``, where relative change is meaningless near zero: a null fraction
+ going from 0.001 to 0.002 is a 100% relative change and nothing at all.
+
+ Attributes:
+ rel_tolerance: Relative difference below which two numbers count as
+ equal.
+ abs_tolerance: Absolute difference below which two numbers count as
+ equal; covers comparisons against zero, where a relative
+ tolerance is meaningless.
+ warn_pct_change: Relative change in a count or magnitude at which it
+ is flagged as a warning.
+ critical_pct_change: Relative change in a count or magnitude at which
+ it is flagged as critical.
+ warn_fraction_shift: Absolute shift in a ``[0, 1]`` fraction (e.g. a
+ null fraction) at which it is flagged as a warning.
+ critical_fraction_shift: Absolute shift in a ``[0, 1]`` fraction at
+ which it is flagged as critical.
+ warn_psi: Population stability index (PSI) at which a numeric
+ distribution shift is a warning.
+ critical_psi: Population stability index at which a numeric
+ distribution shift is critical.
+ warn_tvd: Total variation distance at which a categorical
+ distribution shift is a warning.
+ critical_tvd: Total variation distance at which a categorical
+ distribution shift is critical.
+ warn_percentile_shift: Largest percentile move, as a fraction of the
+ baseline's p5-p95 spread, at which a shift is a warning. Used
+ when histograms are unusable for PSI.
+ critical_percentile_shift: Same measure, at which a shift is
+ critical.
+ """
+
+ rel_tolerance: float = DEFAULT_REL_TOLERANCE
+ abs_tolerance: float = DEFAULT_ABS_TOLERANCE
+ warn_pct_change: float = DEFAULT_WARN_PCT_CHANGE
+ critical_pct_change: float = DEFAULT_CRITICAL_PCT_CHANGE
+ warn_fraction_shift: float = DEFAULT_WARN_FRACTION_SHIFT
+ critical_fraction_shift: float = DEFAULT_CRITICAL_FRACTION_SHIFT
+ warn_psi: float = DEFAULT_WARN_PSI
+ critical_psi: float = DEFAULT_CRITICAL_PSI
+ warn_tvd: float = DEFAULT_WARN_TVD
+ critical_tvd: float = DEFAULT_CRITICAL_TVD
+ warn_percentile_shift: float = DEFAULT_WARN_PERCENTILE_SHIFT
+ critical_percentile_shift: float = DEFAULT_CRITICAL_PERCENTILE_SHIFT
+
+
+DEFAULT_OPTIONS: Final = DiffOptions()
+
+# Named sensitivities, so the CLI can offer a choice without exposing twelve
+# float flags. `low` is for datasets that move constantly and where only a
+# structural break matters; `high` is for a curated table that should not move
+# at all.
+PRESETS: Final[dict[str, DiffOptions]] = {
+ "low": DiffOptions(
+ warn_pct_change=0.25,
+ critical_pct_change=1.00,
+ warn_fraction_shift=0.05,
+ critical_fraction_shift=0.20,
+ warn_psi=0.25,
+ critical_psi=0.50,
+ warn_tvd=0.25,
+ critical_tvd=0.50,
+ warn_percentile_shift=0.25,
+ critical_percentile_shift=0.50,
+ ),
+ "normal": DEFAULT_OPTIONS,
+ "high": DiffOptions(
+ warn_pct_change=0.02,
+ critical_pct_change=0.10,
+ warn_fraction_shift=0.005,
+ critical_fraction_shift=0.02,
+ warn_psi=0.05,
+ critical_psi=0.10,
+ warn_tvd=0.05,
+ critical_tvd=0.10,
+ warn_percentile_shift=0.05,
+ critical_percentile_shift=0.10,
+ ),
+}
+
+
+def preset(name: str) -> DiffOptions:
+ """Return a named sensitivity preset.
+
+ Args:
+ name: One of the keys of [PRESETS][dstrack.diff._model.PRESETS].
+
+ Returns:
+ The thresholds that preset stands for.
+
+ Raises:
+ KeyError: If no preset goes by that name.
+ """
+ try:
+ return PRESETS[name]
+ except KeyError:
+ known = ", ".join(sorted(PRESETS))
+ raise KeyError(
+ f"Unknown sensitivity {name!r}. Known presets: {known}."
+ ) from None
+
+
+@dataclass(frozen=True, slots=True)
+class SnapshotDiff:
+ """Everything that changed between two snapshots.
+
+ Attributes:
+ identical: Whether every comparison came back ``UNCHANGED`` and no
+ column was added, removed, or redefined. Not the same as
+ `source_hash_equal`: two snapshots of one unchanged file still
+ differ in ``created_at`` and ``snapshot_id``, which are identity
+ rather than content and are never compared.
+ same_dataset: Whether both sides belong to the same dataset. ``False``
+ is legitimate -- comparing two datasets is a supported use -- and
+ is surfaced rather than raised on.
+ source_hash_equal: Whether both sides recorded the same non-empty
+ source hash, i.e. the bytes behind them are known to be identical.
+ old: The baseline snapshot.
+ new: The snapshot compared against the baseline.
+ schema: Column additions, removals and changes between the schemas.
+ metrics: Dataset-level scalar quantities (e.g. row count), compared.
+ sets: Changes to set-valued dataset fields, e.g. ``constant_columns``.
+ columns: Per-column statistics, compared.
+ summary: Counts of changes by kind and severity.
+ severity: The highest severity across every comparison.
+ options: The thresholds the diff was classified with.
+ """
+
+ old: SnapshotRef
+ new: SnapshotRef
+ identical: bool
+ same_dataset: bool
+ source_hash_equal: bool
+ schema: SchemaDiff
+ metrics: tuple[MetricDiff, ...]
+ sets: tuple[SetChange, ...]
+ columns: tuple[ColumnStatDiff, ...]
+ summary: DiffSummary
+ severity: Severity
+ options: DiffOptions = field(default=DEFAULT_OPTIONS)
+
+ def to_dict(self) -> dict[str, Any]:
+ """Return the diff as plain JSON-serializable data.
+
+ Enum members are flattened to their string values and tuples become
+ lists, so the result survives `json.dumps` with no custom encoder and
+ compares equal to a literal in a test.
+
+ Returns:
+ A dict carrying ``diff_format_version`` plus every field of this
+ object. Keys are stable within a
+ [DIFF_FORMAT_VERSION][dstrack.diff._model.DIFF_FORMAT_VERSION]:
+ new ones may appear, existing ones are not renamed or removed.
+ """
+ return {
+ "diff_format_version": DIFF_FORMAT_VERSION,
+ **dataclasses.asdict(self, dict_factory=_plain),
+ }
+
+
+def _plain(items: list[tuple[str, Any]]) -> dict[str, Any]:
+ """`dict_factory` that unwraps enum members to their string values.
+
+ Args:
+ items: The field name/value pairs `dataclasses.asdict` collected.
+
+ Returns:
+ The same pairs as a dict, with every `StrEnum` replaced by its value so
+ the result is JSON-serializable without a custom encoder.
+ """
+ return {key: (v.value if isinstance(v, StrEnum) else v) for key, v in items}
diff --git a/src/dstrack/errors.py b/src/dstrack/errors.py
index c6e493b..9a187a8 100644
--- a/src/dstrack/errors.py
+++ b/src/dstrack/errors.py
@@ -14,9 +14,25 @@ class DatasetNotFoundError(Exception):
"""Raised when a named dataset does not exist in the local store."""
+class SnapshotNotFoundError(Exception):
+ """Raised when a snapshot cannot be located in the local store."""
+
+
+class AmbiguousSnapshotError(Exception):
+ """Raised when a short snapshot id prefix matches more than one snapshot."""
+
+
+class IncompatibleSnapshotError(Exception):
+ """Raised when a snapshot's format version cannot be compared."""
+
+
class InputTooLargeError(Exception):
"""Raised when a dataset exceeds the configured snapshot row limit."""
class IndexUnusable(Exception):
"""The index exists but cannot be queried, and must be rebuilt."""
+
+
+class MissingDependencyError(Exception):
+ """Raised when an optional feature's dependency is not installed."""
diff --git a/src/dstrack/snapshot/_version.py b/src/dstrack/snapshot/_version.py
index 31f65b1..7287d66 100644
--- a/src/dstrack/snapshot/_version.py
+++ b/src/dstrack/snapshot/_version.py
@@ -3,11 +3,23 @@
Bump [FORMAT_VERSION][dstrack.snapshot._version.FORMAT_VERSION] and
[SCHEMA_PATH][dstrack.snapshot._version.SCHEMA_PATH] together whenever
[SnapshotMetadata][dstrack.snapshot._metadata.SnapshotMetadata]'s shape changes.
+
+A bump must not drop the old version from
+[SUPPORTED_FORMAT_VERSIONS][dstrack.snapshot._version.SUPPORTED_FORMAT_VERSIONS]:
+snapshots already in a store keep the version they were written in, and
+removing it would make every one of them unreadable after an upgrade. Add the
+new version alongside it, and teach the readers to handle both.
"""
from pathlib import Path
+from typing import Final
FORMAT_VERSION = "1"
SCHEMA_PATH = (
Path(__file__).parent.parent / "schemas" / f"snapshot_v{FORMAT_VERSION}.json"
)
+
+# Every format version this dstrack can read, which is the one it writes plus
+# each older one it still understands. A version outside this set was written by
+# a newer dstrack, or is not a dstrack snapshot at all.
+SUPPORTED_FORMAT_VERSIONS: Final[frozenset[str]] = frozenset({FORMAT_VERSION})
diff --git a/src/dstrack/store.py b/src/dstrack/store.py
index 42f5cad..8081179 100644
--- a/src/dstrack/store.py
+++ b/src/dstrack/store.py
@@ -33,12 +33,16 @@
import sys
import tempfile
import uuid
-from collections.abc import Iterator, Mapping
+from collections.abc import Generator, Iterator, Mapping
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Final
-from dstrack.errors import DatasetNotFoundError, StoreCorruptionError
+from dstrack.errors import (
+ DatasetNotFoundError,
+ SnapshotNotFoundError,
+ StoreCorruptionError,
+)
# Lightweight identity fields copied from a snapshot into its `log.jsonl` line.
# A subset of ADR-0001's fields: only those the current builders produce.
@@ -113,6 +117,23 @@ class DatasetSummary:
head: LogEntry | None
+@dataclass(frozen=True)
+class SnapshotLocation:
+ """Where a snapshot lives: which dataset owns it, and its own id.
+
+ A ``snapshot_id`` is a random UUID4 that says nothing about which dataset
+ holds it, but every path into the store starts at ``datasets//``.
+ Pairing the two is what lets a snapshot found by id be read back.
+
+ Attributes:
+ dataset_id: Dataset the snapshot belongs to.
+ snapshot_id: Identifier of the snapshot itself.
+ """
+
+ dataset_id: str
+ snapshot_id: str
+
+
@dataclass(frozen=True)
class SnapshotWriteResult:
"""Outcome of writing one snapshot into the store.
@@ -287,6 +308,74 @@ def read_head(dataset_id: str, *, store_root: Path) -> str | None:
return _read_head(_resolve_dataset_dir(dataset_id, store_root))
+def snapshot_path(dataset_id: str, snapshot_id: str, *, store_root: Path) -> Path:
+ """Return the path of a snapshot's full JSON payload. Need not exist.
+
+ Args:
+ dataset_id: Dataset the snapshot belongs to.
+ snapshot_id: Snapshot to locate.
+ store_root: Path to the ``.dstrack/`` directory.
+
+ Returns:
+ The ``datasets//snapshots/.json`` path.
+
+ Raises:
+ ValueError: If either id resolves to a path outside the store.
+ """
+ snapshots_dir = _resolve_dataset_dir(dataset_id, store_root) / _SNAPSHOTS_DIRNAME
+ path = snapshots_dir / f"{snapshot_id}.json"
+ _ensure_direct_child(
+ base=snapshots_dir,
+ candidate=path,
+ kind="snapshot_id",
+ value=snapshot_id,
+ )
+ return path
+
+
+def read_snapshot(
+ dataset_id: str, snapshot_id: str, *, store_root: Path
+) -> dict[str, Any]:
+ """Return a snapshot's full payload, as [write_snapshot][dstrack.store.write_snapshot] wrote it.
+
+ The payload is returned as the plain mapping it was stored as, with no
+ schema validation: what a reader may do with a field it does not recognize
+ is the reader's concern, and a payload written by a different version of
+ dstrack is still worth returning.
+
+ Args:
+ dataset_id: Dataset the snapshot belongs to.
+ snapshot_id: Snapshot to read.
+ store_root: Path to the ``.dstrack/`` directory.
+
+ Returns:
+ The snapshot payload.
+
+ Raises:
+ ValueError: If either id resolves to a path outside the store.
+ SnapshotNotFoundError: If the dataset holds no such snapshot.
+ StoreCorruptionError: If the file is not a JSON object.
+ OSError: If the file cannot be read.
+ """
+ path = snapshot_path(dataset_id, snapshot_id, store_root=store_root)
+ if not path.is_file():
+ raise SnapshotNotFoundError(
+ f"No snapshot {snapshot_id!r} in dataset {dataset_id!r} (expected "
+ f"it at {path})."
+ )
+
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except json.JSONDecodeError as e:
+ raise StoreCorruptionError(f"{path} is not valid JSON: {e}") from e
+
+ if not isinstance(payload, dict):
+ raise StoreCorruptionError(
+ f"{path} does not hold a JSON object, but a {type(payload).__name__}."
+ )
+ return payload
+
+
def dataset_log_path(dataset_id: str, *, store_root: Path) -> Path:
"""Return the path of a dataset's ``log.jsonl``. Need not exist.
@@ -713,7 +802,7 @@ def _atomic_write(path: Path, text: str) -> None:
@contextlib.contextmanager
-def _store_lock(store_root: Path) -> Iterator[None]:
+def _store_lock(store_root: Path) -> Generator[None]:
"""Hold an exclusive, cross-process lock on the store for the block's body.
Serializes writers across processes so the resolve-parent-then-write
diff --git a/tests/test_diff_cli.py b/tests/test_diff_cli.py
new file mode 100644
index 0000000..7309af0
--- /dev/null
+++ b/tests/test_diff_cli.py
@@ -0,0 +1,346 @@
+"""Tests for the `dstrack diff` command in src/dstrack/_diff.py."""
+
+import json
+from pathlib import Path
+
+import pytest
+from typer.testing import CliRunner
+
+from dstrack import cache, store
+from dstrack._cli import app
+
+runner = CliRunner()
+
+
+# ---------------------------------------------------------------------------
+# Fixtures / helpers
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture
+def store_root(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
+ """Create an empty local store under a fresh cwd and return its root."""
+ monkeypatch.delenv("DSTRACK_ROOT_PATH", raising=False)
+ monkeypatch.chdir(tmp_path)
+ root = tmp_path / ".dstrack"
+ (root / "datasets").mkdir(parents=True)
+ return root
+
+
+def _write_csv(path: Path, rows: int = 3, extra_column: bool = False) -> Path:
+ """Write a small, valid CSV file and return its path."""
+ header = "a,b,c" if extra_column else "a,b"
+ lines = [header] + [
+ f"{i},{i * 2}" + (f",{i * 3}" if extra_column else "") for i in range(rows)
+ ]
+ path.write_text("\n".join(lines) + "\n", encoding="utf-8")
+ return path
+
+
+def _track(path: Path) -> None:
+ """Track `path` through the CLI."""
+ result = runner.invoke(app, ["track", str(path)])
+ assert result.exit_code == 0, result.output
+
+
+def _dataset_id_of(root: Path, path: Path) -> str:
+ """Return the id of the dataset whose snapshots recorded `path`."""
+ for snapshot in root.glob("datasets/*/snapshots/*.json"):
+ payload = json.loads(snapshot.read_text(encoding="utf-8"))
+ if Path(payload["dataset_path"]).name == path.name:
+ return str(snapshot.parent.parent.name)
+ raise AssertionError(f"no dataset recorded for {path}")
+
+
+@pytest.fixture
+def tracked(tmp_path: Path, store_root: Path) -> Path:
+ """Track one CSV twice, so it has a snapshot and a parent.
+
+ The index is built here rather than by the first command under test, so
+ that no test's output depends on whether it happened to run first.
+ """
+ csv = _write_csv(tmp_path / "data.csv")
+ _track(csv)
+ _write_csv(csv, rows=9, extra_column=True)
+ _track(csv)
+ cache.sync(store_root)
+ return csv
+
+
+def _diff(*args: str, width: str = "160") -> object:
+ """Invoke `dstrack diff` with `--yes`, at a fixed terminal width."""
+ return runner.invoke(app, ["diff", *args, "--yes"], env={"COLUMNS": width})
+
+
+def _snapshots(root: Path, csv: Path) -> list[str]:
+ """Return a dataset's snapshot ids, oldest first."""
+ dataset_id = _dataset_id_of(root, csv)
+ entries = store.read_log_entries(dataset_id, store_root=root)
+ return [entry.snapshot_id for entry in entries]
+
+
+# ---------------------------------------------------------------------------
+# Happy path
+# ---------------------------------------------------------------------------
+
+
+def test_a_path_compares_the_latest_snapshot_to_its_parent(
+ store_root: Path, tracked: Path
+) -> None:
+ """The default question: what changed in the last snapshot of this file?"""
+ result = _diff(str(tracked))
+
+ first, second = _snapshots(store_root, tracked)
+ assert result.exit_code == 0, result.output
+ assert first[:8] in result.output
+ assert second[:8] in result.output
+ assert "+6" in result.output, "6 rows should have been added"
+
+
+def test_two_snapshot_ids_compare_in_the_order_given(
+ store_root: Path, tracked: Path
+) -> None:
+ """Naming both sides explicitly is the unambiguous spelling."""
+ first, second = _snapshots(store_root, tracked)
+
+ forward = _diff(first, second)
+ backward = _diff(second, first)
+
+ assert forward.exit_code == 0, forward.output
+ assert backward.exit_code == 0, backward.output
+ assert "+6" in forward.output, "first -> second adds 6 rows"
+ assert "-6" in backward.output, "second -> first removes 6 rows"
+
+
+def test_short_prefixes_work_like_short_commit_hashes(
+ store_root: Path, tracked: Path
+) -> None:
+ """Nobody types a UUID in full so it should be allowed to pass the beginning of the UUID."""
+ first, second = _snapshots(store_root, tracked)
+
+ result_short_uid = _diff(first[:8], second[:8])
+ result_long_uid = _diff(first, second)
+
+ assert result_short_uid.exit_code == 0, result_short_uid.output
+ assert result_short_uid.output == result_long_uid.output, (
+ "comparison should be identical"
+ )
+
+
+def test_from_and_to_name_the_two_sides(store_root: Path, tracked: Path) -> None:
+ """The explicit spelling, for when the order matters and should be obvious."""
+ first, second = _snapshots(store_root, tracked)
+
+ forward = _diff("--from", first, "--to", second)
+ backward = _diff("--from", second, "--to", first)
+
+ assert forward.exit_code == 0, forward.output
+ assert backward.exit_code == 0, backward.output
+ assert "+6" in forward.output, "--from first --to second adds 6 rows"
+ assert "-6" in backward.output, "--from second --to first removes 6 rows"
+
+
+def test_to_alone_compares_against_the_snapshot_it_succeeded(
+ store_root: Path, tracked: Path
+) -> None:
+ """Naming only the later side asks what that snapshot changed."""
+ first, second = _snapshots(store_root, tracked)
+
+ result = _diff("--to", second)
+
+ assert result.exit_code == 0, result.output
+ assert first[:8] in result.output
+ assert second[:8] in result.output
+ assert result.output.index(first[:8]) < result.output.index(second[:8]), (
+ "the parent should be the 'before' side"
+ )
+ assert "+6 rows" in result.output
+
+
+def test_from_alone_compares_against_where_the_dataset_stands_now(
+ store_root: Path, tracked: Path
+) -> None:
+ """Naming only the baseline asks what has happened since."""
+ first, second = _snapshots(store_root, tracked)
+
+ result = _diff("--from", first)
+
+ assert result.exit_code == 0, result.output
+ assert first[:8] in result.output
+ assert second[:8] in result.output
+ assert result.output.index(first[:8]) < result.output.index(second[:8]), (
+ "the baseline should be the 'before' side and the latest the 'after'"
+ )
+ assert "+6 rows" in result.output
+
+
+def test_oneline_condenses_the_comparison(store_root: Path, tracked: Path) -> None:
+ """The form that fits a CI log line."""
+ result = _diff(str(tracked), "--oneline")
+
+ assert result.exit_code == 0, result.output
+ assert len(result.output.strip().splitlines()) == 1
+
+
+def test_a_narrow_terminal_stacks_the_columns(store_root: Path, tracked: Path) -> None:
+ """The layout follows the terminal it is printed to."""
+ result = _diff(str(tracked), width="60")
+
+ assert result.exit_code == 0, result.output
+ assert max(len(line) for line in result.output.splitlines()) <= 60
+
+
+def test_all_includes_what_did_not_change(store_root: Path, tracked: Path) -> None:
+ """`--all` is for reading the whole picture."""
+ default = _diff(str(tracked))
+ everything = _diff(str(tracked), "--all")
+
+ assert everything.exit_code == 0, everything.output
+ assert len(everything.output.splitlines()) > len(default.output.splitlines())
+
+
+# ---------------------------------------------------------------------------
+# JSON output
+# ---------------------------------------------------------------------------
+
+
+def test_json_output_parses_and_carries_its_version(
+ store_root: Path, tracked: Path
+) -> None:
+ """A pipeline has to be able to read stdout with `json.loads`."""
+ result = _diff(str(tracked), "--json")
+
+ assert result.exit_code == 0, result.output
+ document = json.loads(result.output)
+ assert "diff_format_version" in document
+ assert "schema" in document
+ assert document["summary"]["row_delta"] == 6
+
+
+def test_json_output_carries_no_decoration(store_root: Path, tracked: Path) -> None:
+ """Panels, box drawing and progress notes would all break a parser."""
+ result = _diff(str(tracked), "--json")
+
+ assert "\N{BOX DRAWINGS LIGHT VERTICAL}" not in result.output
+ assert "\x1b[" not in result.output
+
+
+def test_sensitivity_changes_what_is_reported(store_root: Path, tracked: Path) -> None:
+ """The same movement is graded differently at different sensitivities."""
+ low = json.loads(_diff(str(tracked), "--json", "--sensitivity", "low").output)
+ high = json.loads(_diff(str(tracked), "--json", "--sensitivity", "high").output)
+
+ assert low["options"]["warn_psi"] > high["options"]["warn_psi"]
+
+
+def test_an_unknown_sensitivity_lists_the_ones_that_exist(
+ store_root: Path, tracked: Path
+) -> None:
+ """A typo should say what to type instead."""
+ result = _diff(str(tracked), "--sensitivity", "aggressive")
+
+ assert result.exit_code == 2
+ assert "normal" in result.output
+
+
+# ---------------------------------------------------------------------------
+# Failures
+# ---------------------------------------------------------------------------
+
+
+def test_an_unknown_target_lists_the_known_datasets(
+ tmp_path: Path, store_root: Path, tracked: Path
+) -> None:
+ """An error a reader can act on names what does exist."""
+ result = _diff(str(tmp_path / "nope.csv"))
+
+ assert result.exit_code == 1
+ assert "Known datasets" in result.output
+
+
+def test_a_first_snapshot_has_nothing_to_compare_against(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """One snapshot and no parent is a dead end worth explaining."""
+ csv = _write_csv(tmp_path / "only.csv")
+ _track(csv)
+
+ result = _diff(str(csv))
+
+ assert result.exit_code == 1
+ assert "first snapshot" in result.output
+
+
+def test_mixing_positionals_with_from_and_to_is_refused(
+ store_root: Path, tracked: Path
+) -> None:
+ """Two spellings of the same thing must not be combined."""
+ first, second = _snapshots(store_root, tracked)
+
+ result = runner.invoke(app, ["diff", first, "--to", second, "--yes"])
+
+ assert result.exit_code == 2
+ assert "not both" in result.output
+
+
+def test_more_than_two_references_is_refused(store_root: Path, tracked: Path) -> None:
+ """A comparison has exactly two sides."""
+ first, second = _snapshots(store_root, tracked)
+
+ result = _diff(first, second, first)
+
+ assert result.exit_code == 2
+ assert "at most two" in result.output
+
+
+def test_no_reference_at_all_is_refused(store_root: Path, tracked: Path) -> None:
+ """There is no sensible default for what to compare."""
+ result = _diff()
+
+ assert result.exit_code == 2
+ assert "at least one" in result.output
+
+
+def test_no_index_and_no_terminal_says_how_to_proceed(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """A script with no tty cannot be asked, so it is told what to pass."""
+ csv = _write_csv(tmp_path / "data.csv")
+ _track(csv)
+
+ result = runner.invoke(app, ["diff", str(csv)])
+
+ assert result.exit_code == 1
+ assert "--yes" in result.output
+
+
+# ---------------------------------------------------------------------------
+# Cross-dataset
+# ---------------------------------------------------------------------------
+
+
+def test_comparing_two_datasets_warns_but_succeeds(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """Comparing across datasets is supported, and the reader is told."""
+ first = _write_csv(tmp_path / "a.csv")
+ second = _write_csv(tmp_path / "b.csv", rows=9)
+ _track(first)
+ _track(second)
+
+ result = _diff(str(first), str(second))
+
+ assert result.exit_code == 0, result.output
+ assert "two different datasets" in result.output
+
+
+# ---------------------------------------------------------------------------
+# Registration
+# ---------------------------------------------------------------------------
+
+
+def test_diff_is_listed_in_the_top_level_help() -> None:
+ """A command nobody can discover is a command nobody uses."""
+ result = runner.invoke(app, [])
+
+ assert "diff" in result.output
diff --git a/tests/test_diff_engine.py b/tests/test_diff_engine.py
new file mode 100644
index 0000000..d808f67
--- /dev/null
+++ b/tests/test_diff_engine.py
@@ -0,0 +1,564 @@
+"""Tests for the diff engine in src/dstrack/diff/."""
+
+import json
+from typing import Any
+
+import pytest
+
+from dstrack.diff import (
+ ColumnKind,
+ Severity,
+ ShiftMethod,
+ diff_snapshots,
+ preset,
+)
+from dstrack.diff._engine import _redistribute
+from dstrack.diff._model import DiffOptions
+from dstrack.errors import IncompatibleSnapshotError
+
+# ---------------------------------------------------------------------------
+# Fixtures / helpers
+# ---------------------------------------------------------------------------
+
+
+def _numeric(**overrides: Any) -> dict[str, Any]:
+ """Build a numeric column's statistics entry."""
+ stats: dict[str, Any] = {
+ "null_count": 0,
+ "null_fraction": 0.0,
+ "mean": 10.0,
+ "std": 2.0,
+ "min": 0.0,
+ "max": 20.0,
+ "num_unique": 50,
+ "percentiles": {
+ "p5": 2.0,
+ "p25": 6.0,
+ "p50": 10.0,
+ "p75": 14.0,
+ "p95": 18.0,
+ "p99": 19.0,
+ },
+ "histogram": {
+ "bin_edges": [0.0, 5.0, 10.0, 15.0, 20.0],
+ "counts": [100, 400, 400, 100],
+ },
+ }
+ stats.update(overrides)
+ return stats
+
+
+def _string(**overrides: Any) -> dict[str, Any]:
+ """Build a text column's statistics entry."""
+ stats: dict[str, Any] = {
+ "null_count": 0,
+ "null_fraction": 0.0,
+ "num_unique": 3,
+ "top_values": {"NY": 500, "LA": 300, "SF": 200},
+ "top_values_coverage": 1.0,
+ "avg_char_length": 2.0,
+ "min_char_length": 2.0,
+ "max_char_length": 2.0,
+ "avg_token_count": 1.0,
+ "min_token_count": 1.0,
+ "max_token_count": 1.0,
+ }
+ stats.update(overrides)
+ return stats
+
+
+def _payload(**overrides: Any) -> dict[str, Any]:
+ """Build a minimal but complete snapshot payload."""
+ payload: dict[str, Any] = {
+ "format_version": "1",
+ "snapshot_id": "11111111-1111-4111-8111-111111111111",
+ "created_at": "2026-01-01T00:00:00+00:00",
+ "created_by": "tester",
+ "dataset_name": "sales",
+ "dataset_path": "data/sales.csv",
+ "source_type": "file",
+ "source_hash": "a" * 64,
+ "num_rows": 1000,
+ "num_columns": 2,
+ "columns": [
+ {"name": "price", "dtype": "float64", "nullable": True},
+ {"name": "city", "dtype": "string", "nullable": True},
+ ],
+ "schema_hash": "b" * 64,
+ "column_stats": {"price": _numeric(), "city": _string()},
+ "duplicate_row_fraction": 0.0,
+ "constant_columns": [],
+ "high_null_columns": [],
+ }
+ payload.update(overrides)
+ return payload
+
+
+def _mutate(**changes: Any) -> dict[str, Any]:
+ """Build a payload that differs from `_payload()` in the given fields."""
+ payload = json.loads(json.dumps(_payload()))
+ payload.update(changes)
+ return payload
+
+
+# ---------------------------------------------------------------------------
+# Identity and the no-change case
+# ---------------------------------------------------------------------------
+
+
+def test_a_snapshot_compared_to_itself_is_identical() -> None:
+ """Nothing is reported when both sides hold the same payload."""
+ result = diff_snapshots(_payload(), _payload())
+
+ assert result.identical is True
+ assert result.severity is Severity.UNCHANGED
+ assert not result.schema.changed
+ assert result.source_hash_equal is True
+
+
+def test_empty_source_hashes_do_not_prove_identical_bytes() -> None:
+ """Two non-file sources both record "", which says nothing about content."""
+ result = diff_snapshots(
+ _mutate(source_hash=""), _mutate(source_hash="", num_rows=1200)
+ )
+
+ assert result.source_hash_equal is False
+
+
+def test_dataset_ids_decide_whether_it_is_one_dataset() -> None:
+ """Comparing two datasets is surfaced rather than refused."""
+ across = diff_snapshots(
+ _payload(), _payload(), old_dataset_id="ds-a", new_dataset_id="ds-b"
+ )
+ within = diff_snapshots(
+ _payload(), _payload(), old_dataset_id="ds-a", new_dataset_id="ds-a"
+ )
+
+ assert across.same_dataset is False
+ assert within.same_dataset is True
+
+
+# ---------------------------------------------------------------------------
+# Schema
+# ---------------------------------------------------------------------------
+
+
+def test_added_column_is_a_warning() -> None:
+ """A new column is worth knowing about, but nothing downstream breaks."""
+ new = _mutate(
+ columns=[
+ {"name": "price", "dtype": "float64", "nullable": True},
+ {"name": "city", "dtype": "string", "nullable": True},
+ {"name": "tier", "dtype": "string", "nullable": True},
+ ],
+ )
+ new["column_stats"]["tier"] = _string()
+
+ result = diff_snapshots(_payload(), new)
+
+ assert [c.name for c in result.schema.added] == ["tier"]
+ assert result.schema.added[0].severity is Severity.WARNING
+ assert result.summary.columns_added == 1
+
+
+def test_removed_column_is_critical() -> None:
+ """Dropping a column breaks anything that reads it by name."""
+ new = _mutate(columns=[{"name": "price", "dtype": "float64", "nullable": True}])
+ del new["column_stats"]["city"]
+
+ result = diff_snapshots(_payload(), new)
+
+ assert [c.name for c in result.schema.removed] == ["city"]
+ assert result.severity is Severity.CRITICAL
+ assert result.summary.columns_removed == 1
+
+
+def test_type_change_is_critical() -> None:
+ """A column that changed dtype is reported as a retype, not an add and drop."""
+ new = _mutate(
+ columns=[
+ {"name": "price", "dtype": "int64", "nullable": True},
+ {"name": "city", "dtype": "string", "nullable": True},
+ ]
+ )
+
+ result = diff_snapshots(_payload(), new)
+
+ assert [c.name for c in result.schema.type_changed] == ["price"]
+ assert result.schema.type_changed[0].severity is Severity.CRITICAL
+ assert not result.schema.added
+ assert not result.schema.removed
+
+
+def test_nullability_change_is_information_only() -> None:
+ """A column that became nullable is noted without raising the alarm."""
+ new = _mutate(
+ columns=[
+ {"name": "price", "dtype": "float64", "nullable": False},
+ {"name": "city", "dtype": "string", "nullable": True},
+ ]
+ )
+
+ result = diff_snapshots(_payload(), new)
+
+ assert [c.name for c in result.schema.nullability_changed] == ["price"]
+ assert result.schema.severity is Severity.INFO
+
+
+def test_reordered_columns_are_the_only_signal_of_a_reorder() -> None:
+ """A pure reorder leaves the schema hash equal, so it is reported separately."""
+ new = _mutate(
+ columns=[
+ {"name": "city", "dtype": "string", "nullable": True},
+ {"name": "price", "dtype": "float64", "nullable": True},
+ ]
+ )
+
+ result = diff_snapshots(_payload(), new)
+
+ assert set(result.schema.reordered) == {"price", "city"}
+ assert result.schema.schema_hash_equal is True
+ assert result.identical is True
+
+
+def test_a_rename_reads_as_a_removal_and_an_addition() -> None:
+ """No rename is inferred: the engine reports only what it can know."""
+ new = _mutate(
+ columns=[
+ {"name": "cost", "dtype": "float64", "nullable": True},
+ {"name": "city", "dtype": "string", "nullable": True},
+ ]
+ )
+ new["column_stats"]["cost"] = new["column_stats"].pop("price")
+
+ result = diff_snapshots(_payload(), new)
+
+ assert [c.name for c in result.schema.added] == ["cost"]
+ assert [c.name for c in result.schema.removed] == ["price"]
+
+
+# ---------------------------------------------------------------------------
+# Scalar classification
+# ---------------------------------------------------------------------------
+
+
+def test_a_count_is_judged_by_its_relative_change() -> None:
+ """A 50% jump in rows is critical; the delta and ratio are both reported."""
+ result = diff_snapshots(_payload(), _mutate(num_rows=1500))
+
+ rows = next(m for m in result.metrics if m.key == "num_rows")
+ assert rows.delta == 500
+ assert rows.pct_change == pytest.approx(0.5)
+ assert rows.severity is Severity.CRITICAL
+ assert result.summary.row_delta == 500
+
+
+def test_growth_from_zero_has_no_relative_scale() -> None:
+ """Zero to something is notable and cannot be graded as a ratio."""
+ result = diff_snapshots(_mutate(num_rows=0), _mutate(num_rows=10))
+
+ rows = next(m for m in result.metrics if m.key == "num_rows")
+ assert rows.pct_change is None
+ assert rows.severity is Severity.WARNING
+
+
+@pytest.mark.parametrize(
+ ("null_fraction", "expected"),
+ [
+ (0.0, Severity.UNCHANGED),
+ (0.01, Severity.INFO),
+ (0.05, Severity.WARNING),
+ (0.20, Severity.CRITICAL),
+ ],
+)
+def test_a_fraction_is_judged_by_its_absolute_shift(
+ null_fraction: float, expected: Severity
+) -> None:
+ """A null fraction moving 0.001 to 0.002 has doubled and means nothing."""
+ new = _mutate()
+ new["column_stats"]["price"]["null_fraction"] = null_fraction
+
+ result = diff_snapshots(_payload(), new)
+
+ column = next(c for c in result.columns if c.name == "price")
+ metric = next(m for m in column.metrics if m.key == "null_fraction")
+ assert metric.severity is expected
+
+
+def test_a_metric_neither_side_recorded_is_not_reported() -> None:
+ """Fields in the schema that no builder produces are simply absent."""
+ result = diff_snapshots(_payload(), _payload())
+
+ assert not [m for m in result.metrics if m.key == "near_duplicate_estimate"]
+
+
+def test_a_metric_only_one_side_recorded_is_information() -> None:
+ """A field appearing is a change in what was recorded, not in the data."""
+ result = diff_snapshots(_payload(), _mutate(near_duplicate_estimate=0.3))
+
+ metric = next(m for m in result.metrics if m.key == "near_duplicate_estimate")
+ assert metric.old is None
+ assert metric.new == pytest.approx(0.3)
+ assert metric.severity is Severity.INFO
+
+
+# ---------------------------------------------------------------------------
+# Set-valued fields
+# ---------------------------------------------------------------------------
+
+
+def test_joining_the_constant_columns_warns_and_leaving_informs() -> None:
+ """A feature going dead is actionable; one coming back to life is not."""
+ joined = diff_snapshots(_payload(), _mutate(constant_columns=["price"]))
+ left = diff_snapshots(_mutate(constant_columns=["price"]), _payload())
+
+ assert joined.sets[0].added == ("price",)
+ assert joined.sets[0].severity is Severity.WARNING
+ assert left.sets[0].removed == ("price",)
+ assert left.sets[0].severity is Severity.INFO
+
+
+# ---------------------------------------------------------------------------
+# Distribution shift
+# ---------------------------------------------------------------------------
+
+
+def test_identical_histograms_score_zero() -> None:
+ """A distribution that did not move must not read as movement."""
+ result = diff_snapshots(_payload(), _payload())
+
+ column = next(c for c in result.columns if c.name == "price")
+ assert column.distribution_shift is not None
+ assert column.distribution_shift.method is ShiftMethod.PSI
+ assert column.distribution_shift.score == pytest.approx(0.0)
+ assert column.distribution_shift.severity is Severity.UNCHANGED
+
+
+def test_disjoint_histograms_are_critical() -> None:
+ """Values that moved to a range they never occupied is the clearest drift."""
+ new = _mutate()
+ new["column_stats"]["price"]["histogram"] = {
+ "bin_edges": [100.0, 105.0, 110.0, 115.0, 120.0],
+ "counts": [100, 400, 400, 100],
+ }
+
+ result = diff_snapshots(_payload(), new)
+
+ column = next(c for c in result.columns if c.name == "price")
+ assert column.distribution_shift is not None
+ assert column.distribution_shift.score is not None
+ assert column.distribution_shift.score > DiffOptions().critical_psi
+ assert column.distribution_shift.severity is Severity.CRITICAL
+
+
+def test_counts_are_redistributed_onto_a_shared_grid_without_loss() -> None:
+ """Re-binning conserves the total, or every proportion after it is wrong."""
+ spread = _redistribute(
+ [0.0, 5.0, 10.0, 15.0, 20.0],
+ [100, 400, 400, 100],
+ [0.0, 3.0, 5.0, 10.0, 12.0, 15.0, 20.0],
+ )
+
+ assert sum(spread) == pytest.approx(1000.0, abs=1e-9)
+ assert len(spread) == 6
+
+
+def test_histograms_with_different_edges_still_compare() -> None:
+ """Each snapshot derives its edges from its own range, so they rarely align."""
+ new = _mutate()
+ new["column_stats"]["price"]["histogram"] = {
+ "bin_edges": [0.0, 6.0, 12.0, 18.0, 24.0],
+ "counts": [50, 150, 400, 600],
+ }
+
+ result = diff_snapshots(_payload(), new)
+
+ column = next(c for c in result.columns if c.name == "price")
+ assert column.distribution_shift is not None
+ assert column.distribution_shift.method is ShiftMethod.PSI
+ assert column.distribution_shift.score is not None
+
+
+def test_a_degenerate_histogram_falls_back_to_percentiles() -> None:
+ """A constant column's single bin carries no shape to compare."""
+ old = _mutate()
+ new = _mutate()
+ for payload in (old, new):
+ payload["column_stats"]["price"]["histogram"] = {
+ "bin_edges": [3.0, 4.0],
+ "counts": [1000],
+ }
+ new["column_stats"]["price"]["percentiles"]["p50"] = 14.0
+
+ result = diff_snapshots(old, new)
+
+ column = next(c for c in result.columns if c.name == "price")
+ assert column.distribution_shift is not None
+ assert column.distribution_shift.method is ShiftMethod.PERCENTILE_SHIFT
+ assert column.distribution_shift.detail is not None
+
+
+def test_a_new_dominant_category_shifts_a_text_column() -> None:
+ """Top-value churn is what drift looks like for a text column."""
+ new = _mutate()
+ new["column_stats"]["city"]["top_values"] = {"MIA": 700, "LA": 300}
+
+ result = diff_snapshots(_payload(), new)
+
+ column = next(c for c in result.columns if c.name == "city")
+ assert column.distribution_shift is not None
+ assert column.distribution_shift.method is ShiftMethod.TOP_VALUES_TVD
+ assert column.distribution_shift.added_categories == ("MIA",)
+ assert column.distribution_shift.removed_categories == ("NY", "SF")
+ assert column.distribution_shift.severity is Severity.CRITICAL
+
+
+def test_the_untracked_tail_is_carried_as_its_own_bucket() -> None:
+ """Coverage falling means the tail grew, even if the top values held."""
+ new = _mutate()
+ new["column_stats"]["city"]["top_values_coverage"] = 0.4
+
+ result = diff_snapshots(_payload(), new)
+
+ column = next(c for c in result.columns if c.name == "city")
+ assert column.distribution_shift is not None
+ assert column.distribution_shift.score is not None
+ assert column.distribution_shift.score > 0
+
+
+# ---------------------------------------------------------------------------
+# Column kinds
+# ---------------------------------------------------------------------------
+
+
+def test_a_column_that_changed_kind_is_critical_and_barely_comparable() -> None:
+ """Numbers and text share nothing but their null counts."""
+ new = _mutate(
+ columns=[
+ {"name": "price", "dtype": "string", "nullable": True},
+ {"name": "city", "dtype": "string", "nullable": True},
+ ]
+ )
+ new["column_stats"]["price"] = _string()
+
+ result = diff_snapshots(_payload(), new)
+
+ column = next(c for c in result.columns if c.name == "price")
+ assert column.old_kind is ColumnKind.NUMERIC
+ assert column.kind is ColumnKind.STRING
+ assert column.severity is Severity.CRITICAL
+ assert {m.key for m in column.metrics} == {"null_fraction", "null_count"}
+ assert column.distribution_shift is None
+
+
+def test_the_recorded_keys_outrank_a_declared_dtype() -> None:
+ """Arithmetic follows the statistics present, not the dtype claimed."""
+ payload = _mutate(columns=[{"name": "price", "dtype": "float64", "nullable": True}])
+ payload["column_stats"] = {"price": _string()}
+
+ result = diff_snapshots(payload, payload)
+
+ column = next(c for c in result.columns if c.name == "price")
+ assert column.kind is ColumnKind.STRING
+
+
+# ---------------------------------------------------------------------------
+# Options
+# ---------------------------------------------------------------------------
+
+
+def test_presets_change_what_counts_as_significant() -> None:
+ """The same movement is graded differently at different sensitivities."""
+ new = _mutate(num_rows=1150)
+
+ assert diff_snapshots(_payload(), new, options=preset("low")).severity is (
+ Severity.INFO
+ )
+ assert diff_snapshots(_payload(), new, options=preset("high")).severity is (
+ Severity.CRITICAL
+ )
+
+
+def test_an_unknown_preset_names_the_ones_that_exist() -> None:
+ """A typo'd sensitivity should say what to type instead."""
+ with pytest.raises(KeyError, match="normal"):
+ preset("aggressive")
+
+
+# ---------------------------------------------------------------------------
+# Format version and malformed input
+# ---------------------------------------------------------------------------
+
+
+def test_a_foreign_format_version_is_refused() -> None:
+ """A payload this dstrack cannot read must not be compared as if it could."""
+ with pytest.raises(IncompatibleSnapshotError, match="cannot read"):
+ diff_snapshots(_mutate(format_version="99"), _payload())
+
+
+def test_older_supported_format_versions_stay_comparable(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """An upgrade must not strand snapshots written in an older format."""
+ monkeypatch.setattr(
+ "dstrack.diff._engine.SUPPORTED_FORMAT_VERSIONS", frozenset({"1", "2"})
+ )
+
+ result = diff_snapshots(_payload(), _mutate(format_version="2"))
+
+ assert result.old.format_version == "1"
+ assert result.new.format_version == "2"
+
+
+def test_a_payload_recording_no_format_version_is_accepted() -> None:
+ """Hand-built dicts are a supported input and record no version."""
+ result = diff_snapshots({"num_rows": 1}, {"num_rows": 2})
+
+ assert result.summary.row_delta == 1
+
+
+@pytest.mark.parametrize(
+ "payload",
+ [
+ {"num_rows": "many"},
+ {"column_stats": []},
+ {"columns": "not a list"},
+ {"constant_columns": {"not": "a list"}},
+ {"column_stats": {"a": {"mean": None, "histogram": {"bin_edges": "x"}}}},
+ ],
+)
+def test_a_malformed_payload_degrades_rather_than_raises(
+ payload: dict[str, Any],
+) -> None:
+ """A display command must not die on a field it was only going to print."""
+ result = diff_snapshots(payload, _payload())
+
+ assert result.severity in tuple(Severity)
+
+
+# ---------------------------------------------------------------------------
+# Serialization
+# ---------------------------------------------------------------------------
+
+
+def test_to_dict_is_json_serializable_and_carries_its_version() -> None:
+ """The JSON output must survive `json.dumps` with no custom encoder."""
+ result = diff_snapshots(_payload(), _mutate(num_rows=1500))
+
+ document = result.to_dict()
+ encoded = json.dumps(document)
+
+ assert "diff_format_version" in document
+ assert json.loads(encoded)["summary"]["row_delta"] == 500
+ assert "severity" in document
+
+
+def test_to_dict_holds_no_enum_objects() -> None:
+ """Enums are flattened to their values, so the result compares to literals."""
+ result = diff_snapshots(_payload(), _mutate(num_rows=1500))
+
+ document = result.to_dict()
+
+ assert document["severity"] == "critical"
+ assert isinstance(document["severity"], str)
+ assert document["columns"][0]["kind"] == "numeric"
diff --git a/tests/test_diff_render.py b/tests/test_diff_render.py
new file mode 100644
index 0000000..46cb905
--- /dev/null
+++ b/tests/test_diff_render.py
@@ -0,0 +1,224 @@
+"""Tests for the terminal renderer in src/dstrack/_diff_render.py."""
+
+from datetime import UTC, datetime
+from typing import Any
+
+import pytest
+from rich.console import Console
+
+from dstrack._diff_render import _MAX_LISTED_COLUMNS, render_diff
+from dstrack.diff import SnapshotDiff, diff_snapshots
+
+NOW = datetime(2026, 1, 2, tzinfo=UTC)
+
+
+# ---------------------------------------------------------------------------
+# Fixtures / helpers
+# ---------------------------------------------------------------------------
+
+
+def _payload(**overrides: Any) -> dict[str, Any]:
+ """Build a small snapshot payload with one numeric and one text column."""
+ payload: dict[str, Any] = {
+ "snapshot_id": "11111111-1111-4111-8111-111111111111",
+ "created_at": "2026-01-01T00:00:00+00:00",
+ "created_by": "tester",
+ "dataset_name": "sales",
+ "dataset_path": "data/sales.csv",
+ "source_hash": "a" * 64,
+ "num_rows": 1000,
+ "num_columns": 2,
+ "columns": [
+ {"name": "price", "dtype": "float64", "nullable": True},
+ {"name": "city", "dtype": "string", "nullable": True},
+ ],
+ "column_stats": {
+ "price": {
+ "null_count": 0,
+ "null_fraction": 0.0,
+ "mean": 10.0,
+ "std": 2.0,
+ "min": 0.0,
+ "max": 20.0,
+ "num_unique": 50,
+ "percentiles": {"p5": 2.0, "p50": 10.0, "p95": 18.0},
+ "histogram": {
+ "bin_edges": [0.0, 5.0, 10.0, 15.0, 20.0],
+ "counts": [100, 400, 400, 100],
+ },
+ },
+ "city": {
+ "null_count": 0,
+ "null_fraction": 0.0,
+ "num_unique": 3,
+ "top_values": {"NY": 500, "LA": 300, "SF": 200},
+ "top_values_coverage": 1.0,
+ "avg_char_length": 2.0,
+ },
+ },
+ "duplicate_row_fraction": 0.0,
+ "constant_columns": [],
+ "high_null_columns": [],
+ }
+ payload.update(overrides)
+ return payload
+
+
+def _changed_diff() -> SnapshotDiff:
+ """Build a diff with a row change, a null change and a category change."""
+ new = _payload(
+ snapshot_id="22222222-2222-4222-8222-222222222222",
+ num_rows=1500,
+ source_hash="b" * 64,
+ )
+ new["column_stats"]["price"]["null_fraction"] = 0.2
+ new["column_stats"]["price"]["mean"] = 40.0
+ new["column_stats"]["city"]["top_values"] = {"MIA": 900, "LA": 600}
+ return diff_snapshots(_payload(), new, old_dataset_id="ds", new_dataset_id="ds")
+
+
+def _render(diff: SnapshotDiff, *, width: int = 160, **kwargs: Any) -> str:
+ """Render a diff at a given width and return its plain text."""
+ console = Console(
+ width=width, record=True, force_terminal=False, legacy_windows=False
+ )
+ console.print(render_diff(diff, width=width, now=NOW, **kwargs))
+ return console.export_text()
+
+
+# ---------------------------------------------------------------------------
+# Layout bands
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize("width", [160, 120, 60])
+def test_the_comparison_renders_at_every_width(width: int) -> None:
+ """Wide, medium and narrow terminals must all produce output."""
+ output = _render(_changed_diff(), width=width)
+
+ assert "sales" in output
+ assert "price" in output
+ assert "city" in output
+
+
+def test_a_narrow_terminal_stacks_each_column_instead_of_tabulating() -> None:
+ """A side-by-side table squeezed into forty cells is not one any more."""
+ wide = _render(_changed_diff(), width=160)
+ narrow = _render(_changed_diff(), width=60)
+
+ # The stacked form spells each statistic out on its own line, so the
+ # per-column header row that only the tabular form has is gone.
+ assert "null fraction" in narrow
+ assert "uniq" in wide
+ assert max(len(line) for line in narrow.splitlines()) <= 60
+
+
+def _numeric_header(output: str) -> str:
+ """Return the header row of the numeric column table."""
+ lines = output.splitlines()
+ start = next(i for i, line in enumerate(lines) if "Numeric columns" in line)
+ return lines[start + 1]
+
+
+def test_a_wide_terminal_shows_more_statistics_than_a_medium_one() -> None:
+ """The least informative measurements are dropped first as space runs out."""
+ wide = _numeric_header(_render(_changed_diff(), width=160))
+ medium = _numeric_header(_render(_changed_diff(), width=120))
+
+ assert "min" in wide
+ assert "max" in wide
+ assert "uniq" in wide
+ assert "min" not in medium
+ assert "p50" in medium
+
+
+# ---------------------------------------------------------------------------
+# Markup safety
+# ---------------------------------------------------------------------------
+
+
+def test_names_that_look_like_markup_are_printed_verbatim() -> None:
+ """A dataset called `sales[/]report` is not markup, and must not be parsed."""
+ old = _payload(dataset_name="sales[/]report")
+ new = _payload(
+ dataset_name="sales[/]report",
+ snapshot_id="22222222-2222-4222-8222-222222222222",
+ num_rows=1500,
+ )
+ for payload in (old, new):
+ payload["columns"][0]["name"] = "price[/]usd"
+ payload["column_stats"]["price[/]usd"] = payload["column_stats"].pop("price")
+ new["column_stats"]["price[/]usd"]["mean"] = 40.0
+
+ output = _render(diff_snapshots(old, new))
+
+ assert "sales[/]report" in output
+ assert "price[/]usd" in output
+
+
+def test_a_category_that_looks_like_markup_is_printed_verbatim() -> None:
+ """Categories come straight from the data and are the least trusted of all."""
+ new = _payload(snapshot_id="22222222-2222-4222-8222-222222222222")
+ new["column_stats"]["city"]["top_values"] = {"[bold]NY": 900, "LA": 600}
+
+ output = _render(diff_snapshots(_payload(), new))
+
+ assert "[bold]NY" in output
+
+
+# ---------------------------------------------------------------------------
+# Content
+# ---------------------------------------------------------------------------
+
+
+def test_an_unchanged_comparison_says_so_and_shows_no_tables() -> None:
+ """Two identical snapshots need one sentence, not an empty table."""
+ output = _render(diff_snapshots(_payload(), _payload()))
+
+ assert "identical" in output
+ assert "Schema" not in output
+
+
+def test_show_unchanged_includes_a_column_that_did_not_move() -> None:
+ """`--all` is for reading the whole picture, not only what changed."""
+ diff = _changed_diff()
+
+ default = _render(diff)
+ everything = _render(diff, show_unchanged=True)
+
+ assert len(everything.splitlines()) > len(default.splitlines())
+
+
+def test_oneline_is_a_single_line() -> None:
+ """The condensed form has to fit a shell prompt or a CI log line."""
+ output = _render(_changed_diff(), oneline=True).strip()
+
+ assert len(output.splitlines()) == 1
+ assert "CRITICAL" in output
+
+
+def test_a_long_list_of_schema_changes_is_capped() -> None:
+ """A cross-dataset diff can add hundreds of columns; the table stays readable."""
+ total = _MAX_LISTED_COLUMNS + 5
+ new = _payload(snapshot_id="22222222-2222-4222-8222-222222222222")
+ new["columns"] = [
+ {"name": f"col{i}", "dtype": "float64", "nullable": True} for i in range(total)
+ ]
+ new["column_stats"] = {
+ f"col{i}": {"null_count": 0, "null_fraction": 0.0} for i in range(total)
+ }
+
+ output = _render(diff_snapshots(_payload(), new))
+
+ assert "and 7 more" in output
+
+
+def test_comparing_two_datasets_is_called_out(tmp_path: object) -> None:
+ """A cross-dataset comparison is legitimate, but the reader should know."""
+ diff = diff_snapshots(
+ _payload(), _payload(num_rows=1500), old_dataset_id="a", new_dataset_id="b"
+ )
+
+ output = _render(diff)
+
+ assert "two different datasets" in output
diff --git a/tests/test_diff_report.py b/tests/test_diff_report.py
new file mode 100644
index 0000000..04d4a46
--- /dev/null
+++ b/tests/test_diff_report.py
@@ -0,0 +1,254 @@
+"""Tests for the PDF report in src/dstrack/_diff_report.py."""
+
+import base64
+import hashlib
+import re
+import sys
+import zlib
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from dstrack.diff import SnapshotDiff, diff_snapshots
+from dstrack.errors import MissingDependencyError
+
+pytest.importorskip("reportlab", reason="the `report` extra is not installed")
+
+from dstrack._diff_report import write_pdf_report
+
+NOW = datetime(2026, 1, 2, 12, 30, tzinfo=UTC)
+
+
+# ---------------------------------------------------------------------------
+# Fixtures / helpers
+# ---------------------------------------------------------------------------
+
+
+def _payload(columns: int = 2, **overrides: Any) -> dict[str, Any]:
+ """Build a snapshot payload with `columns` numeric columns."""
+ payload: dict[str, Any] = {
+ "snapshot_id": "11111111-1111-4111-8111-111111111111",
+ "created_at": "2026-01-01T00:00:00+00:00",
+ "created_by": "tester",
+ "dataset_name": "sales",
+ "dataset_path": "data/sales.csv",
+ "source_hash": "a" * 64,
+ "num_rows": 1000,
+ "num_columns": columns,
+ "columns": [
+ {"name": f"col{i}", "dtype": "float64", "nullable": True}
+ for i in range(columns)
+ ],
+ "column_stats": {
+ f"col{i}": {
+ "null_count": 0,
+ "null_fraction": 0.0,
+ "mean": 10.0,
+ "std": 2.0,
+ "min": 0.0,
+ "max": 20.0,
+ "num_unique": 50,
+ "percentiles": {"p5": 2.0, "p50": 10.0, "p95": 18.0},
+ "histogram": {
+ "bin_edges": [0.0, 5.0, 10.0, 15.0, 20.0],
+ "counts": [100, 400, 400, 100],
+ },
+ }
+ for i in range(columns)
+ },
+ "duplicate_row_fraction": 0.0,
+ "constant_columns": [],
+ "high_null_columns": [],
+ }
+ payload.update(overrides)
+ return payload
+
+
+def _diff(columns: int = 2, **overrides: Any) -> SnapshotDiff:
+ """Build a diff in which every column moved."""
+ old = _payload(columns)
+ new = _payload(
+ columns,
+ snapshot_id="22222222-2222-4222-8222-222222222222",
+ num_rows=1500,
+ source_hash="b" * 64,
+ **overrides,
+ )
+ for stats in new["column_stats"].values():
+ stats["mean"] = 40.0
+ stats["null_fraction"] = 0.2
+ return diff_snapshots(old, new, old_dataset_id="ds", new_dataset_id="ds")
+
+
+def _pages(pdf: bytes) -> int:
+ """Count the pages a PDF declares."""
+ return pdf.count(b"/Type /Page") - pdf.count(b"/Type /Pages")
+
+
+def _text(pdf: bytes) -> str:
+ """Return the text drawn on a reportlab PDF, one line of it per space.
+
+ Reportlab ASCII85-encodes and deflates its page streams, so the words
+ are not visible in the raw bytes. Decoding them here avoids adding a PDF
+ parser to the test dependencies. A line can be drawn as several `Tj`
+ fragments (an escaped `<` gets one of its own), so fragments are joined
+ as they sit and only the `T*` line breaks between them become spaces.
+ """
+ lines: list[str] = []
+ for match in re.finditer(rb"stream\r?\n(.*?)endstream", pdf, re.DOTALL):
+ data = match.group(1).strip()
+ try:
+ data = zlib.decompress(base64.a85decode(data.removesuffix(b"~>")))
+ except (ValueError, zlib.error):
+ continue # an image or font stream, not page content
+ for block in re.findall(rb"BT(.*?)ET", data, re.DOTALL):
+ for line in block.split(b"T*"):
+ fragments = re.findall(rb"\((.*?)(? None:
+ """The file has to be a PDF before anything else about it matters."""
+ path = tmp_path / "diff.pdf"
+
+ write_pdf_report(_diff(), path, now=NOW)
+
+ content = path.read_bytes()
+ assert content.startswith(b"%PDF-")
+ assert content.endswith(b"%%EOF\n")
+ assert len(content) > 1024
+ assert _pages(content) >= 1
+
+
+def test_an_unchanged_comparison_still_produces_a_report(tmp_path: Path) -> None:
+ """ "Nothing changed" is a result worth reporting, not a reason to fail."""
+ path = tmp_path / "same.pdf"
+
+ write_pdf_report(diff_snapshots(_payload(), _payload()), path, now=NOW)
+
+ assert path.read_bytes().startswith(b"%PDF-")
+
+
+def test_many_columns_produce_a_multi_page_report(tmp_path: Path) -> None:
+ """A wide dataset must paginate rather than overflow one page."""
+ path = tmp_path / "wide.pdf"
+
+ write_pdf_report(_diff(columns=30), path, now=NOW)
+
+ assert _pages(path.read_bytes()) > 1
+
+
+def test_a_cross_dataset_comparison_is_reported(tmp_path: Path) -> None:
+ """The report carries the caveat the terminal prints as a warning."""
+ across = tmp_path / "across.pdf"
+ within = tmp_path / "within.pdf"
+ old, new = _payload(), _payload(num_rows=1500)
+
+ write_pdf_report(
+ diff_snapshots(old, new, old_dataset_id="a", new_dataset_id="b"),
+ across,
+ now=NOW,
+ )
+ write_pdf_report(
+ diff_snapshots(old, new, old_dataset_id="a", new_dataset_id="a"),
+ within,
+ now=NOW,
+ )
+
+ assert "two different datasets" in _text(across.read_bytes())
+ assert "two different datasets" not in _text(within.read_bytes())
+
+
+# ---------------------------------------------------------------------------
+# Escaping
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+ "markup", ["a", "tom & jerry", 'say "what"', "bold", "x"]
+)
+def test_text_that_looks_like_markup_is_drawn_verbatim(
+ tmp_path: Path, markup: str
+) -> None:
+ """`Paragraph` parses a markup dialect, so every string from the data must
+ be escaped: dataset and column names, and the categories a column gained
+ and lost, which come straight from the data and are the least trusted.
+
+ Each place gets its own prefix, so one that is drawn correctly cannot
+ hide another that is not.
+ """
+ path = tmp_path / "escaped.pdf"
+ dataset, column = f"dataset {markup}", f"column {markup}"
+ removed, added = f"removed {markup}", f"added {markup}"
+ old = _payload(columns=1, dataset_name=dataset)
+ new = _payload(
+ columns=1,
+ dataset_name=dataset,
+ snapshot_id="22222222-2222-4222-8222-222222222222",
+ )
+ for payload, category, count in ((old, removed, 600), (new, added, 900)):
+ payload["columns"][0]["name"] = column
+ payload["column_stats"] = {
+ column: {
+ "null_count": 0,
+ "null_fraction": 0.0,
+ "num_unique": 2,
+ "top_values": {category: count, "plain": 1000 - count},
+ "top_values_coverage": 1.0,
+ }
+ }
+
+ write_pdf_report(diff_snapshots(old, new), path, now=NOW)
+
+ text = _text(path.read_bytes())
+ for drawn in (dataset, column, removed, added):
+ assert drawn in text, f"{drawn!r} should be drawn verbatim"
+
+
+# ---------------------------------------------------------------------------
+# The optional dependency and the packaged logo
+# ---------------------------------------------------------------------------
+
+
+def test_a_missing_reportlab_explains_how_to_install_it(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """An optional feature must name its extra, not raise an ImportError."""
+ monkeypatch.setitem(sys.modules, "reportlab", None)
+ monkeypatch.setitem(sys.modules, "reportlab.lib", None)
+
+ with pytest.raises(MissingDependencyError, match=r"dstrack\[report\]"):
+ write_pdf_report(_diff(), tmp_path / "nope.pdf", now=NOW)
+
+
+def test_the_packaged_logo_matches_the_one_in_the_docs() -> None:
+ """Two copies of an asset drift; this is the only thing that stops it."""
+ packaged = Path("src/dstrack/assets/dstrack-logo.png").read_bytes()
+ documented = Path("docs/assets/dstrack-logo.png").read_bytes()
+
+ assert hashlib.sha256(packaged).hexdigest() == (
+ hashlib.sha256(documented).hexdigest()
+ )
+
+
+def test_a_report_still_builds_without_its_logo(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """A missing asset is a cosmetic loss, not a reason to fail the command."""
+ from dstrack import _diff_report
+
+ monkeypatch.setattr(_diff_report, "_LOGO_RESOURCE", "assets/not-there.png")
+ path = tmp_path / "nologo.pdf"
+
+ write_pdf_report(_diff(), path, now=NOW)
+
+ assert path.read_bytes().startswith(b"%PDF-")
diff --git a/tests/test_diff_resolve.py b/tests/test_diff_resolve.py
new file mode 100644
index 0000000..cdcbc8e
--- /dev/null
+++ b/tests/test_diff_resolve.py
@@ -0,0 +1,427 @@
+"""Tests for reference resolution in src/dstrack/_resolve.py and the store reads it uses."""
+
+import json
+import uuid
+from pathlib import Path
+
+import pytest
+from typer.testing import CliRunner
+
+from dstrack import cache, store
+from dstrack._cli import app
+from dstrack._resolve import resolve_pair, resolve_snapshot_ref
+from dstrack.errors import (
+ AmbiguousSnapshotError,
+ DatasetNotFoundError,
+ SnapshotNotFoundError,
+ StoreCorruptionError,
+)
+
+runner = CliRunner()
+
+
+# ---------------------------------------------------------------------------
+# Fixtures / helpers
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture
+def store_root(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
+ """Create an empty local store under a fresh cwd and return its root."""
+ monkeypatch.delenv("DSTRACK_ROOT_PATH", raising=False)
+ monkeypatch.chdir(tmp_path)
+ root = tmp_path / ".dstrack"
+ (root / "datasets").mkdir(parents=True)
+ return root
+
+
+def _write_csv(path: Path, rows: int = 3) -> Path:
+ """Write a small, valid CSV file and return its path."""
+ lines = ["a,b"] + [f"{i},{i * 2}" for i in range(rows)]
+ path.write_text("\n".join(lines) + "\n", encoding="utf-8")
+ return path
+
+
+def _track(root: Path, path: Path, *args: str) -> str:
+ """Track `path` through the CLI and return its dataset id."""
+ result = runner.invoke(app, ["track", str(path), *args])
+ assert result.exit_code == 0, result.output
+ return _dataset_id_of(root, path)
+
+
+def _dataset_id_of(root: Path, path: Path) -> str:
+ """Return the id of the dataset whose snapshots recorded `path`."""
+ for snapshot in root.glob("datasets/*/snapshots/*.json"):
+ payload = json.loads(snapshot.read_text(encoding="utf-8"))
+ if Path(payload["dataset_path"]).name == path.name:
+ return str(snapshot.parent.parent.name)
+ raise AssertionError(f"no dataset recorded for {path}")
+
+
+def _synced(root: Path) -> Path:
+ """Bring the index up to date, so prefix and path lookups can see the store."""
+ cache.sync(root)
+ return root
+
+
+def _resolve(spec: str, root: Path) -> object:
+ """Resolve one reference against a synced store."""
+ return resolve_snapshot_ref(spec, store_root=_synced(root), root=None)
+
+
+def _head(root: Path, dataset_id: str) -> str:
+ """Return a dataset's latest snapshot id."""
+ head = store.read_head(dataset_id, store_root=root)
+ assert head is not None
+ return head
+
+
+# ---------------------------------------------------------------------------
+# store.snapshot_path and store.read_snapshot
+# ---------------------------------------------------------------------------
+
+
+def test_read_snapshot_returns_what_was_written(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """A stored payload round-trips through the read API unchanged."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ snapshot_id = _head(store_root, dataset_id)
+
+ payload = store.read_snapshot(dataset_id, snapshot_id, store_root=store_root)
+
+ assert payload["snapshot_id"] == snapshot_id
+ assert payload["dataset_name"] == "data"
+ assert payload["num_rows"] == 3
+
+
+@pytest.mark.parametrize(
+ "snapshot_id", ["../../escape", "nested/id", "/absolute", "../sibling"]
+)
+def test_a_traversing_snapshot_id_is_refused(
+ store_root: Path, snapshot_id: str
+) -> None:
+ """An id must name one file directly inside the dataset's snapshots/.
+
+ A bare ``..`` is deliberately not in this list: the ``.json`` suffix makes
+ it the ordinary filename ``...json``, which stays inside the directory and
+ is therefore nothing to refuse.
+ """
+ with pytest.raises(ValueError, match="snapshot_id"):
+ store.snapshot_path(str(uuid.uuid4()), snapshot_id, store_root=store_root)
+
+
+def test_reading_a_snapshot_that_is_not_there_says_so(store_root: Path) -> None:
+ """A missing payload is reported as missing, not as corruption."""
+ dataset_id = str(uuid.uuid4())
+ (store_root / "datasets" / dataset_id / "snapshots").mkdir(parents=True)
+
+ with pytest.raises(SnapshotNotFoundError, match="No snapshot"):
+ store.read_snapshot(dataset_id, str(uuid.uuid4()), store_root=store_root)
+
+
+def test_a_payload_that_is_not_an_object_is_corruption(store_root: Path) -> None:
+ """The store wrote an object; anything else means the file was damaged."""
+ dataset_id = str(uuid.uuid4())
+ snapshot_id = str(uuid.uuid4())
+ snapshots = store_root / "datasets" / dataset_id / "snapshots"
+ snapshots.mkdir(parents=True)
+ (snapshots / f"{snapshot_id}.json").write_text("[1, 2, 3]", encoding="utf-8")
+
+ with pytest.raises(StoreCorruptionError, match="JSON object"):
+ store.read_snapshot(dataset_id, snapshot_id, store_root=store_root)
+
+
+# ---------------------------------------------------------------------------
+# cache.find_snapshots_by_prefix
+# ---------------------------------------------------------------------------
+
+
+def test_a_prefix_finds_the_snapshot_and_the_dataset_holding_it(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """A snapshot id says nothing about its dataset, so the lookup returns both."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ snapshot_id = _head(store_root, dataset_id)
+
+ matches = cache.find_snapshots_by_prefix(
+ snapshot_id[:8], store_root=_synced(store_root)
+ )
+
+ assert len(matches) == 1
+ assert matches[0].dataset_id == dataset_id
+ assert matches[0].snapshot_id == snapshot_id
+
+
+@pytest.mark.parametrize("prefix", ["____", "%", "a_b%"])
+def test_like_metacharacters_in_a_prefix_match_nothing(
+ tmp_path: Path, store_root: Path, prefix: str
+) -> None:
+ """An unescaped `_` is a wildcard and would match ids the user never named."""
+ _track(store_root, _write_csv(tmp_path / "data.csv"))
+
+ matches = cache.find_snapshots_by_prefix(prefix, store_root=_synced(store_root))
+
+ assert matches == []
+
+
+# ---------------------------------------------------------------------------
+# resolve_snapshot_ref
+# ---------------------------------------------------------------------------
+
+
+def test_a_full_snapshot_id_resolves_to_that_snapshot(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """The most explicit reference there is."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ snapshot_id = _head(store_root, dataset_id)
+
+ resolved = _resolve(snapshot_id, store_root)
+
+ assert resolved.snapshot_id == snapshot_id
+ assert resolved.dataset_id == dataset_id
+ assert resolved.payload["snapshot_id"] == snapshot_id
+
+
+def test_a_dataset_id_resolves_to_its_latest_snapshot(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """Naming a dataset means naming where it stands now."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ _write_csv(csv, rows=5)
+ _track(store_root, csv)
+
+ resolved = _resolve(dataset_id, store_root)
+
+ assert resolved.snapshot_id == _head(store_root, dataset_id)
+ assert resolved.payload["num_rows"] == 5
+
+
+@pytest.mark.parametrize("spelling", ["upper", "nodashes", "braced"])
+def test_an_id_is_recognized_however_it_was_pasted(
+ tmp_path: Path, store_root: Path, spelling: str
+) -> None:
+ """Uppercase, brace-wrapped and dash-free spellings all name the same thing."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ snapshot_id = _head(store_root, dataset_id)
+ variants = {
+ "upper": snapshot_id.upper(),
+ "nodashes": snapshot_id.replace("-", ""),
+ "braced": f"{{{snapshot_id}}}",
+ }
+
+ resolved = _resolve(variants[spelling], store_root)
+
+ assert resolved.snapshot_id == snapshot_id
+
+
+def test_a_short_prefix_resolves_like_a_short_commit_hash(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """Nobody types a UUID in full."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ snapshot_id = _head(store_root, dataset_id)
+
+ resolved = _resolve(snapshot_id[:8], store_root)
+
+ assert resolved.snapshot_id == snapshot_id
+
+
+def test_an_ambiguous_prefix_lists_what_it_matched(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """A prefix matching several ids must show which, so one can be chosen."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ snapshot_id = _head(store_root, dataset_id)
+
+ # Plant a second snapshot sharing the first four characters of its id.
+ twin_id = snapshot_id[:4] + str(uuid.uuid4())[4:]
+ payload = store.read_snapshot(dataset_id, snapshot_id, store_root=store_root)
+ payload["snapshot_id"] = twin_id
+ payload["parent_snapshot_id"] = snapshot_id
+ snapshots = store_root / "datasets" / dataset_id / "snapshots"
+ (snapshots / f"{twin_id}.json").write_text(json.dumps(payload), encoding="utf-8")
+ log = store_root / "datasets" / dataset_id / "log.jsonl"
+ with log.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps({**payload, "snapshot_id": twin_id}) + "\n")
+ (store_root / "datasets" / dataset_id / "HEAD").write_text(
+ f"{twin_id}\n", encoding="utf-8"
+ )
+
+ with pytest.raises(AmbiguousSnapshotError, match="ambiguous"):
+ _resolve(snapshot_id[:4], store_root)
+
+
+def test_a_prefix_shorter_than_the_minimum_is_not_treated_as_one(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """Three characters match almost anything and are far more likely a path."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ prefix = _head(store_root, dataset_id)[:3]
+
+ with pytest.raises(DatasetNotFoundError):
+ _resolve(prefix, store_root)
+
+
+def test_a_tracked_path_resolves_to_its_latest_snapshot(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """The reference a user reaches for first."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+
+ resolved = _resolve(str(csv), store_root)
+
+ assert resolved.dataset_id == dataset_id
+
+
+def test_a_tracked_file_that_was_deleted_still_resolves(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """History outlives the file, exactly as it does for `dstrack log`."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ csv.unlink()
+
+ resolved = _resolve(str(csv), store_root)
+
+ assert resolved.dataset_id == dataset_id
+
+
+def test_an_unknown_reference_lists_the_store_s_datasets(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """An error a reader can act on names what does exist."""
+ _track(store_root, _write_csv(tmp_path / "data.csv"))
+
+ with pytest.raises(DatasetNotFoundError, match="Known datasets"):
+ _resolve(str(tmp_path / "nope.csv"), store_root)
+
+
+def test_a_dataset_with_no_snapshots_says_so(store_root: Path) -> None:
+ """An empty dataset directory is not an error, but it cannot be compared."""
+ dataset_id = str(uuid.uuid4())
+ (store_root / "datasets" / dataset_id).mkdir(parents=True)
+
+ with pytest.raises(SnapshotNotFoundError, match="no snapshots"):
+ _resolve(dataset_id, store_root)
+
+
+# ---------------------------------------------------------------------------
+# resolve_pair
+# ---------------------------------------------------------------------------
+
+
+def _pair(
+ root: Path,
+ specs: list[str],
+ *,
+ from_ref: str | None = None,
+ to_ref: str | None = None,
+) -> tuple[object, object]:
+ """Resolve both sides against a synced store."""
+ return resolve_pair(
+ specs,
+ from_ref=from_ref,
+ to_ref=to_ref,
+ store_root=_synced(root),
+ root=None,
+ )
+
+
+def test_one_reference_compares_a_snapshot_against_its_parent(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """The question `dstrack diff` is usually asked."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ first = _head(store_root, dataset_id)
+ _write_csv(csv, rows=7)
+ _track(store_root, csv)
+ second = _head(store_root, dataset_id)
+
+ old, new = _pair(store_root, [str(csv)])
+
+ assert old.snapshot_id == first
+ assert new.snapshot_id == second
+
+
+def test_the_first_snapshot_of_a_dataset_has_nothing_to_compare_against(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """There is no parent, and saying so beats comparing against nothing."""
+ csv = _write_csv(tmp_path / "data.csv")
+ _track(store_root, csv)
+
+ with pytest.raises(SnapshotNotFoundError, match="first snapshot"):
+ _pair(store_root, [str(csv)])
+
+
+def test_two_references_are_taken_in_the_order_given(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """The first argument is the baseline."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ first = _head(store_root, dataset_id)
+ _write_csv(csv, rows=7)
+ _track(store_root, csv)
+ second = _head(store_root, dataset_id)
+
+ old, new = _pair(store_root, [first, second])
+
+ assert (old.snapshot_id, new.snapshot_id) == (first, second)
+
+
+def test_from_alone_compares_against_where_the_dataset_stands_now(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """Naming only a baseline asks what has happened since."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ first = _head(store_root, dataset_id)
+ _write_csv(csv, rows=7)
+ _track(store_root, csv)
+
+ old, new = _pair(store_root, [], from_ref=first)
+
+ assert old.snapshot_id == first
+ assert new.snapshot_id == _head(store_root, dataset_id)
+
+
+def test_to_alone_compares_against_the_snapshot_it_succeeded(
+ tmp_path: Path, store_root: Path
+) -> None:
+ """Naming only a later snapshot asks what that snapshot changed."""
+ csv = _write_csv(tmp_path / "data.csv")
+ dataset_id = _track(store_root, csv)
+ first = _head(store_root, dataset_id)
+ _write_csv(csv, rows=7)
+ _track(store_root, csv)
+ second = _head(store_root, dataset_id)
+
+ old, new = _pair(store_root, [], to_ref=second)
+
+ assert (old.snapshot_id, new.snapshot_id) == (first, second)
+
+
+def test_two_datasets_may_be_compared(tmp_path: Path, store_root: Path) -> None:
+ """Comparing across datasets is a supported use, not an error."""
+ a = _write_csv(tmp_path / "a.csv")
+ b = _write_csv(tmp_path / "b.csv", rows=9)
+ dataset_a = _track(store_root, a)
+ dataset_b = _track(store_root, b)
+
+ old, new = _pair(store_root, [str(a), str(b)])
+
+ assert old.dataset_id == dataset_a
+ assert new.dataset_id == dataset_b
diff --git a/tests/test_log.py b/tests/test_log.py
index af3726a..cb86b47 100644
--- a/tests/test_log.py
+++ b/tests/test_log.py
@@ -12,15 +12,15 @@
import typer
from typer.testing import CliRunner
-from dstrack import _log, cache
+from dstrack import _resolve, cache
from dstrack._cli import app
-from dstrack._log import (
- _canonical_uuid,
- _ensure_index,
- _known_datasets,
- _resolve_target,
-)
from dstrack._log_render import _humanize
+from dstrack._resolve import (
+ canonical_uuid,
+ ensure_index,
+ known_datasets,
+ resolve_target,
+)
from dstrack.errors import DatasetNotFoundError, StoreCorruptionError
runner = CliRunner()
@@ -70,14 +70,14 @@ def _dataset_id_of(root: Path, path: Path) -> str:
def _pretend_terminal(monkeypatch: pytest.MonkeyPatch) -> None:
- """Make `_log` see a terminal on stdin, so it offers to build the index.
+ """Make `_resolve` see a terminal on stdin, so it offers to build the index.
`CliRunner` installs its own `sys.stdin` for the duration of an invocation,
which would undo a patch of the real one; the module's own reference to
`sys` is replaced instead. Only `stdin.isatty` is ever read through it.
"""
monkeypatch.setattr(
- _log, "sys", SimpleNamespace(stdin=SimpleNamespace(isatty=lambda: True))
+ _resolve, "sys", SimpleNamespace(stdin=SimpleNamespace(isatty=lambda: True))
)
@@ -351,14 +351,14 @@ def test_ensure_index_rebuild_discards_the_existing_index(
assert _log_cmd(str(csv)).exit_code == 0
assert cache.index_exists(store_root)
- _ensure_index(store_root, yes=False, rebuild=True)
+ ensure_index(store_root, yes=False, rebuild=True)
assert not cache.index_exists(store_root)
def test_ensure_index_rebuild_on_a_missing_index_is_fine(store_root: Path) -> None:
"""--rebuild does not require an index to already be there."""
- _ensure_index(store_root, yes=False, rebuild=True)
+ ensure_index(store_root, yes=False, rebuild=True)
assert not cache.index_exists(store_root)
@@ -378,7 +378,7 @@ def test_log_rebuild_still_shows_the_history(tmp_path: Path, store_root: Path) -
def test_ensure_index_without_a_terminal_raises_exit_one(store_root: Path) -> None:
"""The no-tty branch exits rather than blocking on a prompt."""
with pytest.raises(typer.Exit) as excinfo:
- _ensure_index(store_root, yes=False, rebuild=False)
+ ensure_index(store_root, yes=False, rebuild=False)
assert excinfo.value.exit_code == 1
@@ -404,13 +404,13 @@ def test_canonical_uuid_accepts_alternate_spellings(
"""Uppercased, braced, dashless and urn ids all name the same dataset."""
value = str(uuid.uuid4())
- assert _canonical_uuid(spell(value)) == value
+ assert canonical_uuid(spell(value)) == value
@pytest.mark.parametrize("value", ["", "data.csv", "not-a-uuid", "../etc/passwd"])
def test_canonical_uuid_rejects_non_uuids(value: str) -> None:
"""Anything that is not a UUID in some spelling is not an id."""
- assert _canonical_uuid(value) is None
+ assert canonical_uuid(value) is None
def test_resolve_target_prefers_an_existing_id_over_a_path(
@@ -423,7 +423,7 @@ def test_resolve_target_prefers_an_existing_id_over_a_path(
# A file in the cwd named exactly like the id must not be picked up.
_write_csv(tmp_path / dataset_id)
- resolved = _resolve_target(dataset_id, store_root=store_root, root=None)
+ resolved = resolve_target(dataset_id, store_root=store_root, root=None)
assert resolved == dataset_id
@@ -434,7 +434,7 @@ def test_resolve_target_falls_back_to_a_path(tmp_path: Path, store_root: Path) -
dataset_id = _track(store_root, csv)
cache.sync(store_root)
- assert _resolve_target(str(csv), store_root=store_root, root=None) == dataset_id
+ assert resolve_target(str(csv), store_root=store_root, root=None) == dataset_id
def test_resolve_target_relative_path_matches(tmp_path: Path, store_root: Path) -> None:
@@ -443,7 +443,7 @@ def test_resolve_target_relative_path_matches(tmp_path: Path, store_root: Path)
dataset_id = _track(store_root, csv)
cache.sync(store_root)
- assert _resolve_target("data.csv", store_root=store_root, root=None) == dataset_id
+ assert resolve_target("data.csv", store_root=store_root, root=None) == dataset_id
def test_resolve_target_unknown_id_reports_the_id(store_root: Path) -> None:
@@ -452,7 +452,7 @@ def test_resolve_target_unknown_id_reports_the_id(store_root: Path) -> None:
missing = str(uuid.uuid4())
with pytest.raises(DatasetNotFoundError, match="No dataset") as excinfo:
- _resolve_target(missing, store_root=store_root, root=None)
+ resolve_target(missing, store_root=store_root, root=None)
assert missing in str(excinfo.value)
@@ -462,7 +462,7 @@ def test_resolve_target_unknown_path_suggests_the_id(store_root: Path) -> None:
cache.sync(store_root)
with pytest.raises(DatasetNotFoundError, match="No tracked dataset") as excinfo:
- _resolve_target("missing.csv", store_root=store_root, root=None)
+ resolve_target("missing.csv", store_root=store_root, root=None)
message = str(excinfo.value)
assert "missing.csv" in message
@@ -478,9 +478,9 @@ def test_resolve_target_honours_an_explicit_root(
dataset_id = _track(store_root, csv, "--root", str(nested))
cache.sync(store_root)
- assert _resolve_target(str(csv), store_root=store_root, root=nested) == dataset_id
+ assert resolve_target(str(csv), store_root=store_root, root=nested) == dataset_id
with pytest.raises(DatasetNotFoundError):
- _resolve_target(str(csv), store_root=store_root, root=None)
+ resolve_target(str(csv), store_root=store_root, root=None)
# ---------------------------------------------------------------------------
@@ -492,7 +492,7 @@ def test_known_datasets_on_an_empty_store(store_root: Path) -> None:
"""An empty store says so, and points at the command that fills it."""
cache.sync(store_root)
- listing = _known_datasets(store_root)
+ listing = known_datasets(store_root)
assert "(none)" in listing
assert "dstrack track" in listing
@@ -506,7 +506,7 @@ def test_known_datasets_lists_id_name_and_path(
dataset_id = _track(store_root, csv, "--name", "clients")
cache.sync(store_root)
- listing = _known_datasets(store_root)
+ listing = known_datasets(store_root)
assert dataset_id in listing
assert "clients" in listing
@@ -519,7 +519,7 @@ def test_known_datasets_marks_missing_fields(store_root: Path) -> None:
(store_root / "datasets" / dataset_id).mkdir()
cache.sync(store_root)
- listing = _known_datasets(store_root)
+ listing = known_datasets(store_root)
assert dataset_id in listing
assert "-" in listing
@@ -536,12 +536,12 @@ def test_known_datasets_drops_a_dataset_deleted_from_the_store(
removed_id = _track(store_root, _write_csv(tmp_path / "gone.csv"))
kept_id = _track(store_root, _write_csv(tmp_path / "kept.csv"))
cache.sync(store_root)
- assert removed_id in _known_datasets(store_root)
+ assert removed_id in known_datasets(store_root)
shutil.rmtree(store_root / "datasets" / removed_id)
cache.sync(store_root)
- listing = _known_datasets(store_root)
+ listing = known_datasets(store_root)
assert removed_id not in listing
assert kept_id in listing
@@ -596,7 +596,7 @@ def test_log_surfaces_a_corrupt_store(
def _boom(*args: object, **kwargs: object) -> bool:
raise StoreCorruptionError("log.jsonl is not readable")
- monkeypatch.setattr(_log.store, "dataset_exists", _boom)
+ monkeypatch.setattr(_resolve.store, "dataset_exists", _boom)
result = _log_cmd(dataset_id)
diff --git a/uv.lock b/uv.lock
index 2636cda..709ee02 100644
--- a/uv.lock
+++ b/uv.lock
@@ -20,6 +20,80 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" },
]
+[[package]]
+name = "charset-normalizer"
+version = "3.4.9"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/bd/2a/23f34ec9d04624958e137efdc394888716353190e75f25dd22c7a2c7a8aa/charset_normalizer-3.4.9.tar.gz", hash = "sha256:673611bbd43f0810bec0b0f028ddeaaa501190339cac411f347ac76917c3ae7b", size = 152439, upload-time = "2026-07-07T14:34:58.454Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0b/e3/85ec501f206fb049259288c1f3506e53876937fb00edb47009348e66756b/charset_normalizer-3.4.9-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:0e94703ec9684807f20cfb5eed95c70f67f2a8f21ad620146d7b5a13677b93e5", size = 317075, upload-time = "2026-07-07T14:32:56.021Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/69/2a5385192e67175f7d8bd5ce4f57c24bc956439adeae5c13a99aa28a53d1/charset_normalizer-3.4.9-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2a441ea71902098ffe78c5abe6c494f44160b4af614ed16c3d9a3b1d17fd8ee2", size = 213837, upload-time = "2026-07-07T14:32:57.78Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/46/03ddc7da576d814fe0a36dd1f0fd3258e95404b4b2e3c026b7923d7e133f/charset_normalizer-3.4.9-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:304b13570067b2547562e308af560b3963857b1fa90bd6afd978130130fe2d6a", size = 235503, upload-time = "2026-07-07T14:32:59.205Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/6e/de0229a7ef40f6f9d28a837eebf4ec47bdca5dab4e900c84f22919af636a/charset_normalizer-3.4.9-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4773092f8019072343a7447203308b176e10199920eb02d6195e81bbb3274c29", size = 229944, upload-time = "2026-07-07T14:33:00.803Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/34/49b9060e8418b14fb5cba9cf6bfb383111e2538a03a1fb18e66a95aeb3d5/charset_normalizer-3.4.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:04ce310cb89c15df659582aee80a0603788732a5e017d5bd5c81158106ce249c", size = 221276, upload-time = "2026-07-07T14:33:02.199Z" },
+ { url = "https://files.pythonhosted.org/packages/44/95/80282cce0fae9c3061203d723ee87da996aed79679e65d8935050ee7ca1f/charset_normalizer-3.4.9-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:c0323c9daef75ef2e5083624b4585018a0c9d5e3b40f607eed81a311270b934b", size = 205260, upload-time = "2026-07-07T14:33:03.698Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/74/2f62c8821b969ea3bd67cc2e6976834f48ca5d12664d2559ebcd9bcfbed7/charset_normalizer-3.4.9-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:871ff67ea1aad4dfd91736464934d56b32dac49f9fbe16cddba36198a7b3a0db", size = 217786, upload-time = "2026-07-07T14:33:05.12Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/8d/feabb82cb49fcad14515b1d7d1ca4787b0da7fc723a212bf89bc9e0fac52/charset_normalizer-3.4.9-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:67830fc78e67501f47bb950471b2dcb9b35b140084429318e862895a8e89c993", size = 216798, upload-time = "2026-07-07T14:33:06.629Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/ff/c946d63bc3786d5b84d960b0f7ab7e25b828486a946b5aa997625bcaf6a6/charset_normalizer-3.4.9-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:3d92613ec25e43b05f042302531ec0f00b8445190e43325880cbd6ab7c2581da", size = 206429, upload-time = "2026-07-07T14:33:08.006Z" },
+ { url = "https://files.pythonhosted.org/packages/af/ba/5e5007c370702f85d2ef75791fac7943ed41e080364a673b20142e430e3e/charset_normalizer-3.4.9-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:280081916dc341820640489a66e4696049401ef1cf6dd672f672e70ad915aca3", size = 223066, upload-time = "2026-07-07T14:33:09.783Z" },
+ { url = "https://files.pythonhosted.org/packages/83/d5/9096aa3cf532dfad237861544eb47a0f20d5adbf1039760fed8eaae935d9/charset_normalizer-3.4.9-cp311-cp311-win32.whl", hash = "sha256:ac351b3b8014eead140e77e9717e2992c6bbe30b63bc3422422eb84865412e3d", size = 150456, upload-time = "2026-07-07T14:33:11.217Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/a1/e29995109e455dc8eff8d0fac6ae509be39561318a7cfeac5d33ad029213/charset_normalizer-3.4.9-cp311-cp311-win_amd64.whl", hash = "sha256:6366a16e1a25018694d6a5d784d09b046edc9eac40ea2b54065c3052672516a1", size = 161410, upload-time = "2026-07-07T14:33:12.743Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/8d/1569f4d0032d6ba2a4fe4591c35bf87868c600c41a71eb5c2e1ffa8464c2/charset_normalizer-3.4.9-cp311-cp311-win_arm64.whl", hash = "sha256:1d22856ffbe153a602df38e4a5464f0b748a54002e0d69ac6d2ad0a197cc99ec", size = 152649, upload-time = "2026-07-07T14:33:14.173Z" },
+ { url = "https://files.pythonhosted.org/packages/70/4a/ecbd131485c07fcdfad54e28946d513e3da22ef3b4bd854dcafae54ec739/charset_normalizer-3.4.9-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:45b0cc4e3556cd875e09102988d1ab8356c998b596c9fced84547c8138b487a0", size = 319300, upload-time = "2026-07-07T14:33:15.666Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/96/5d9364e3342d69f3a045e1777bc47c85c383e6e9466d561b33fdb419d1f9/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9b2aff1c7b3884512b9512c3eaadd9bab39fb45042ffaaa1dd08ff2b9f8109d9", size = 215802, upload-time = "2026-07-07T14:33:17.031Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/4c/5361f9aa7f2cb58d94f2ab831b3d493f69efb1d239654b4744e3c09527cb/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9104ed0bd76a429d46f9ec0dbc9b08ad1d2dcdf2b00a5a0daa1c145329b35b44", size = 237171, upload-time = "2026-07-07T14:33:18.576Z" },
+ { url = "https://files.pythonhosted.org/packages/50/78/ce342ca4ff30b2eb49fe6d9578df85974f90c67d294113e94efdd9664cbd/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7b86a2b16095d250c6f58b3d9b2eee6f4147754344f3dab0922f7c9bf7d226c9", size = 233075, upload-time = "2026-07-07T14:33:20.084Z" },
+ { url = "https://files.pythonhosted.org/packages/01/c4/4fa4c8b3097a11f3c5f09a35b72ed6855fb1d332469504962ab7bafcc702/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5e226f6218febc71f6c1fc2fafb91c226f75bdc1d8fb12d66823716e891608fd", size = 224256, upload-time = "2026-07-07T14:33:21.747Z" },
+ { url = "https://files.pythonhosted.org/packages/87/3a/ad914516df7e358a81aae018caa5e0470ba827fa6d763b1d2e87d920a5f6/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:90c44bc373b7687f6948b693cceaea1348ae0975d7474746559494468e3c1d84", size = 208784, upload-time = "2026-07-07T14:33:23.313Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/74/3c12f9755717dfe5c5c87da63f35d765fa0c00382ec26bf23f7fae34f2ba/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9cdef90ae47919cae358d8ab15797a800ed41da7aba5d72419fb510729e2ed4b", size = 219928, upload-time = "2026-07-07T14:33:24.814Z" },
+ { url = "https://files.pythonhosted.org/packages/33/9a/895095b83e7907abd6d3d99aad3a38ad0d9686cc186cb0c94c24320fe63e/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:60f44ade2cf573dad7a277e6f8ca9a51a21dda572b13bd7d8539bb3cd5dbedde", size = 218489, upload-time = "2026-07-07T14:33:26.42Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/34/ef5c05f412f42520d7709b7d3784d19640839eb7366ded1755511585429f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:a1786910334ed46ab1dd73222f2cd1e05c2c3bb39f6dddb4f8b36fc382058a39", size = 210267, upload-time = "2026-07-07T14:33:27.952Z" },
+ { url = "https://files.pythonhosted.org/packages/83/dc/9b29fa4412b318bf3bfea985c35d67eb55e04b59a7c3f2237168b0e0be6f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:03d07803992c6c7bbc976327f34b18b6160327fc81cb82c9d504720ac0be3b62", size = 226030, upload-time = "2026-07-07T14:33:29.397Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/42/6dbc00b8cd16011691203e33570fa42ed5746599a2e878112d16eab403a3/charset_normalizer-3.4.9-cp312-cp312-win32.whl", hash = "sha256:78841cccf1af7b40f6f716338d50c0902dbe88d9f800b3c973b7a9a0a693a642", size = 151185, upload-time = "2026-07-07T14:33:30.781Z" },
+ { url = "https://files.pythonhosted.org/packages/80/cc/f920afd1a23c58ccd53c1d36085a71893a4737ff5e66e0371efab6809850/charset_normalizer-3.4.9-cp312-cp312-win_amd64.whl", hash = "sha256:4b3dac63058cc36820b0dd072f89898604e2d39686fe05321729d00d8ac185a0", size = 162557, upload-time = "2026-07-07T14:33:32.176Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/e6/0386d43a261ff4e4b30c5857af7df877254b46bec7b9d1b74b6bf969a90b/charset_normalizer-3.4.9-cp312-cp312-win_arm64.whl", hash = "sha256:78fa18e436a1a0e58dbd7e02fc4473f3f32cceb12df9dfca542d075961c307d2", size = 152665, upload-time = "2026-07-07T14:33:33.711Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/06/97ec2aeae780b31d742b6352218b43841a6871e2564578ca522dce4a45c3/charset_normalizer-3.4.9-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:440eede837960000d74978f0eba527be106b5b9aee0daf779d395276ed0b0614", size = 317688, upload-time = "2026-07-07T14:33:35.408Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/39/8ff066c672434225f8d25f8b739f992af250944392173dcc88362681c9bf/charset_normalizer-3.4.9-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:21e764fd1e70b6a3e205a0e46f3051701f98a8cb3fad66eeb80e48bb502f8698", size = 214982, upload-time = "2026-07-07T14:33:36.996Z" },
+ { url = "https://files.pythonhosted.org/packages/92/8f/3a47a3667c83c2df9483d91644c6c107de3bf8874aa1793da9d3012eb986/charset_normalizer-3.4.9-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e4fd89cc178bced6ad29cb3e6dd4aa63fa5017c3524dbd0b25998fb64a87cc8b", size = 236460, upload-time = "2026-07-07T14:33:38.536Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/60/b22cdbee7e4013dab8b0d7647fc6181120fbbbc8f7025c226d15bd5a47fc/charset_normalizer-3.4.9-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bd47ba7fc3ca94896759ea0109775132d3e7ab921fbf54038e1bab2e46c313c9", size = 232003, upload-time = "2026-07-07T14:33:40.059Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/f8/72eb13dcabe7257035cea8aefd922caad2f110d252bf9f67c4c2ca763aee/charset_normalizer-3.4.9-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:84fd18bcc17526fc2b3c1af7d2b9217d32c9c04448c16ec693b9b4f1985c3d33", size = 223149, upload-time = "2026-07-07T14:33:41.631Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/3e/faee8f9de92b14ee1198e9163252bb15efee7301b31256a3b6d9ebfdd0dd/charset_normalizer-3.4.9-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:5b10cd92fc5c498b35a8635df6d5a100207f88b63a4dc1de7ef9a548e1e2cd63", size = 207901, upload-time = "2026-07-07T14:33:43.209Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/25/45f30093ae27dd7b92a793b61882a38685f993700113ca36e0c9c14965e1/charset_normalizer-3.4.9-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a4fbdde9dd4a9ce5fd52c2b3a347bb50cc89483ef783f1cb00d408c13f7a96c0", size = 219176, upload-time = "2026-07-07T14:33:44.725Z" },
+ { url = "https://files.pythonhosted.org/packages/48/18/c8f397329c35e32f6a837e488986f4ae03bd2abebc453b48714991630c2f/charset_normalizer-3.4.9-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:416c229f77e5ea25b3dfd4b582f8d73d7e43c22320302b9ab128a2d3a0b38efe", size = 217356, upload-time = "2026-07-07T14:33:46.192Z" },
+ { url = "https://files.pythonhosted.org/packages/86/7e/5ce0bba863470fd1902d5e5843968951bddf38abe4742fc97116ef4598b3/charset_normalizer-3.4.9-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:75286256590a6320cf106a0d28970d3560aad9ee09aa7b34fb40524792436d35", size = 209614, upload-time = "2026-07-07T14:33:47.705Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/ef/2473d3c4d869155be4af1191111d59c4d5c4e0173026f7e85b176e23bf65/charset_normalizer-3.4.9-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:69b157c5d3292bcd443faca052f3096f637f1e074b98212a933c074ae23dc3b8", size = 224991, upload-time = "2026-07-07T14:33:49.238Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/a3/53ddae3db108a088156aa8ddfafd411ebbc1340f48c5573f697b27f69a39/charset_normalizer-3.4.9-cp313-cp313-win32.whl", hash = "sha256:51307f5c71007673a2bf8232ad973483d281e74cb99c8c5a990af1eefa6277d9", size = 150622, upload-time = "2026-07-07T14:33:50.711Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/ef/6953a77c7cf2c2ff9998e6f575ab3e380119f100223381565a4f94c1f836/charset_normalizer-3.4.9-cp313-cp313-win_amd64.whl", hash = "sha256:fe2c7201c642b7c308f1675355ad7ff7b66acfe3541625efe5a3ad38f29d6115", size = 161947, upload-time = "2026-07-07T14:33:52.197Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/fb/d560d1d1555debbfe7849d9cac6145c1b537709d79576bf22557ed803b82/charset_normalizer-3.4.9-cp313-cp313-win_arm64.whl", hash = "sha256:611057cc5d5c0afc743ba8be6bd828c17e0aaa8643f9d0a9b9bb7dea80eb8012", size = 152594, upload-time = "2026-07-07T14:33:53.486Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/8d/496817fa0944239ecae662dd57ea765cfeaec6a735f9f025d4b7b72e7143/charset_normalizer-3.4.9-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:0327fcd59a935777d83410750c50600ee9571af2846f71ce40f25b13da1ef380", size = 317253, upload-time = "2026-07-07T14:33:54.994Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/f9/ef4a69ea338ad3c0deceea0f5f7d2380ae8b52132b06d652cb0d2cd86706/charset_normalizer-3.4.9-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8a79d9f4d8001473a30c163556b3c3bfebec837495a412dde78b51672f6134f9", size = 215898, upload-time = "2026-07-07T14:33:56.334Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/e7/5ddfd76fc061eb52de219658a4aa431cbacadf0a0219c8854f00da50d289/charset_normalizer-3.4.9-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:33bdcc2a32c0a0e861f60841a512c8acc658c87c2ac59d89e3a46dacf7d866e4", size = 236718, upload-time = "2026-07-07T14:33:57.9Z" },
+ { url = "https://files.pythonhosted.org/packages/49/ba/768fa3f36048d81c477a0ce61f813bc1454d80917ccfe550abd9f44f5e24/charset_normalizer-3.4.9-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f840ed6d8ecba8255df8c42b87fadeda98ddfc6eeec05e2dc66e26d46dd6f58a", size = 232519, upload-time = "2026-07-07T14:33:59.811Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/c4/b3e049d2aa3766180c78507110543d9d50894cc97f57de543f1be521dcdc/charset_normalizer-3.4.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c25fe15c70c59eb7c5ce8c06a1f3fa1da0ecc5ea1e7a5922c40fd2fa9b0d5046", size = 223143, upload-time = "2026-07-07T14:34:01.517Z" },
+ { url = "https://files.pythonhosted.org/packages/19/79/55c32d06d76ae4feafe053f061f3e3ab70bcf19f4007797ce8c3efda7830/charset_normalizer-3.4.9-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:f7fb7d750cfa0a070d2c24e831fd3481019a60dd317ea2b39acbcebc08b6ed81", size = 206742, upload-time = "2026-07-07T14:34:03.04Z" },
+ { url = "https://files.pythonhosted.org/packages/10/e0/47c079dd82d217c807479cd59ffd30af56307ea31c108b75758970459ad3/charset_normalizer-3.4.9-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4d1c96a7a18b9690a4d46df09e3e3382406ae3213727cd1019ebade1c4a81917", size = 219191, upload-time = "2026-07-07T14:34:04.657Z" },
+ { url = "https://files.pythonhosted.org/packages/42/ab/b9bc2e77d6b44a7e46ef62ec5cac1c9a6ba7b9135a5d560f002696ec9995/charset_normalizer-3.4.9-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a4cfde78a9f2880208d16a93b795726a3017d5977e08d1e162a7a31322479c41", size = 218328, upload-time = "2026-07-07T14:34:06.115Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/78/c9c71d599f5aa2d42bcdd35cbbd46d7f535351a57e40ff7d8e5a7e219401/charset_normalizer-3.4.9-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:d4d6fcde76f94f5cb9e43e9e9a61f16dacefd228cbbf6f1a09bd9b219a92f1a1", size = 207406, upload-time = "2026-07-07T14:34:07.554Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/39/c914445c321a845097ce4f6ac7de9a18228a77b766272125a1ce00d851eb/charset_normalizer-3.4.9-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:898f0e9068ca27d37f8e83a5b962821df851532e6c4a7d615c1c033f9da6eedf", size = 225157, upload-time = "2026-07-07T14:34:09.061Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/f2/c0d4b8508565a36bc5c624e88ed297f5b0b1095011034d7f5b83a69908b5/charset_normalizer-3.4.9-cp314-cp314-win32.whl", hash = "sha256:c1c948747b03be832dceed96ca815cef7360de9aa19d37c730f8e3f6101aca48", size = 151095, upload-time = "2026-07-07T14:34:10.901Z" },
+ { url = "https://files.pythonhosted.org/packages/49/fd/a1d26144398c67486422a72bf5812cda22cb4ccfcd95a290fb41ceb4b8e2/charset_normalizer-3.4.9-cp314-cp314-win_amd64.whl", hash = "sha256:16b65ea0f2465b6fb52aa22de5eca612aa964ddfec00a912e26f4656cbef890b", size = 162796, upload-time = "2026-07-07T14:34:12.47Z" },
+ { url = "https://files.pythonhosted.org/packages/20/95/d75e82f8ce9fd323ebf059c16c9aadefb22a1ecde13b7840b35835e4886c/charset_normalizer-3.4.9-cp314-cp314-win_arm64.whl", hash = "sha256:40a126142a56b2dfc0aacbad1de8310cbf60da7656db0e6b16eebd48e3e93519", size = 153334, upload-time = "2026-07-07T14:34:14.044Z" },
+ { url = "https://files.pythonhosted.org/packages/00/5e/17398df3a139985ba9d11ed072531986f408c8fca952835ef1ab1820c02b/charset_normalizer-3.4.9-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:609b3ba8fcc0fb5ab7af00719d0fb6ad0cb518e48e7712d12fd68f1327951198", size = 338848, upload-time = "2026-07-07T14:34:15.688Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/91/7253a32e86b7e1d1239b1b36ba6dd0f021a21107ab33054b53119cc083b9/charset_normalizer-3.4.9-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:51447e9aa2684679af07ca5021c3db526e0284347ebf4ffcec1154c3350cfe32", size = 223022, upload-time = "2026-07-07T14:34:17.248Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/32/2e64bd2be10e89c61e57ebe6a93fd98ae88eb7ebe414b5121f22c96c69eb/charset_normalizer-3.4.9-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:cc1b0fff8ead343dae06305f954eb8468ba0ec1a97881f42489d198e4ce3c632", size = 241590, upload-time = "2026-07-07T14:34:18.813Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/ef/d96ec496cfea0c21db43b0ad03891308b02388d054cc902cf0e5a1ad6a88/charset_normalizer-3.4.9-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fa36ec09ef71d158186bc79e359ff5fdd6e7996fe8ab638f00d6b93139ba4fcf", size = 239584, upload-time = "2026-07-07T14:34:20.52Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/ce/9af95f7876194bd7a14e3dfe4a4de2e0bff02666a3910d72beafd06cc297/charset_normalizer-3.4.9-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:df115d4d83168fdf2cae48ef1ff6d1cb4c466364e30861b37121de0f3bf1b990", size = 230224, upload-time = "2026-07-07T14:34:22.189Z" },
+ { url = "https://files.pythonhosted.org/packages/52/94/af74dde74a3996bd959c350709bfe50e297823d70a8c1cbd54b838880863/charset_normalizer-3.4.9-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f86c6358749bd4fda175388691e3ba8c46e24c5347d0afd20f9b7edfc9faf07d", size = 212667, upload-time = "2026-07-07T14:34:23.857Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/f0/f1c4fe746c395922961b5916ed1d7d6e7d4c84851d19ed43cc89980ec953/charset_normalizer-3.4.9-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:32286a2c8d167e897177b673176c1e3e00d4057caf5d2b64eef9a3666b03018e", size = 227179, upload-time = "2026-07-07T14:34:25.586Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/56/6c745619ac397e8871e2bcd3cea1eec86b877488f33888b3aef5c3ed506e/charset_normalizer-3.4.9-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:83aed2c10721ddd90f68140685391b50811a880af20654c59af6b6c66c40513c", size = 225372, upload-time = "2026-07-07T14:34:27.212Z" },
+ { url = "https://files.pythonhosted.org/packages/78/ad/98aae8630ac71f16711968e38a5acfecce41b778bf2f0312851020f565a8/charset_normalizer-3.4.9-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:cd6c3d4b783c556fa00bf540854e42f135e2f256abd29669fcd0da0f2dec79c2", size = 215222, upload-time = "2026-07-07T14:34:28.774Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/40/9593d54209765207a7f11073c06494c1721e4ca4a0a426c597679bf7f91e/charset_normalizer-3.4.9-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ee2f2a527e3c1a6e6411eb4209642e138b544a2d72fe5d0d76daf77b24063534", size = 231958, upload-time = "2026-07-07T14:34:30.345Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/27/693ee5e8a18191eb38647360c51cd505013e2bd3b366aa43fd5344c21e3c/charset_normalizer-3.4.9-cp314-cp314t-win32.whl", hash = "sha256:0d861473f743244d349b50f850d10eb87aeb22bbdcc8e64f79273c94af5a8226", size = 155580, upload-time = "2026-07-07T14:34:31.884Z" },
+ { url = "https://files.pythonhosted.org/packages/80/3f/bd97d3d9c613013d07cb7733d299385b41df37f0471310f5a73dc359f0b8/charset_normalizer-3.4.9-cp314-cp314t-win_amd64.whl", hash = "sha256:9b8e0f3107e2200b76f6054de99016eac3ee6762713587b36baaa7e4bd2ae177", size = 167620, upload-time = "2026-07-07T14:34:33.438Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/c6/eee9dca4439b1061f76373f06ea855678cc4a64c1c3c90b50e479edbb8eb/charset_normalizer-3.4.9-cp314-cp314t-win_arm64.whl", hash = "sha256:19ac87f93086ce37b86e098888555c4b4bc48102279bae3350098c0ed664b501", size = 158037, upload-time = "2026-07-07T14:34:35.018Z" },
+ { url = "https://files.pythonhosted.org/packages/98/2b/f97f1c193fb855c345d678f5077d6926034db0722df74c8f057020e05a25/charset_normalizer-3.4.9-py3-none-any.whl", hash = "sha256:68e5f26a1ad57ded6d1cfb85331d1c1a195314756471d97758c48498bb4dcdf5", size = 64538, upload-time = "2026-07-07T14:34:56.993Z" },
+]
+
[[package]]
name = "click"
version = "8.4.1"
@@ -148,6 +222,11 @@ dependencies = [
{ name = "typer" },
]
+[package.optional-dependencies]
+report = [
+ { name = "reportlab" },
+]
+
[package.dev-dependencies]
dev = [
{ name = "jsonschema" },
@@ -159,9 +238,11 @@ dev = [
[package.metadata]
requires-dist = [
+ { name = "reportlab", marker = "extra == 'report'", specifier = ">=4.2" },
{ name = "rich", specifier = ">=15.0.0" },
{ name = "typer", specifier = ">=0.26.8" },
]
+provides-extras = ["report"]
[package.metadata.requires-dev]
dev = [
@@ -455,6 +536,91 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl", hash = "sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189", size = 57328, upload-time = "2026-04-27T01:46:07.06Z" },
]
+[[package]]
+name = "pillow"
+version = "12.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1c/3d/bb7fca845737cf9d7dbde16ed1843984665ff2e0a518f5db43e77ec540b9/pillow-12.3.0.tar.gz", hash = "sha256:3b8182a766685eaa002637e28b4ec8d6b18819a0c71f579bf0dbaa5830297cce", size = 47025035, upload-time = "2026-07-01T11:56:38.965Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/c8/0a78b0e02d7ac54bc03e5321c9220da52f0c2ea83b21f7c40e7f3169c502/pillow-12.3.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:00808c5e14ef63ac5161091d242999076604ff74b883423a11e5d7bbb38bf756", size = 5392415, upload-time = "2026-07-01T11:53:47.162Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/5b/a02d30018abd97ced9f5a6c63d28597694a00d066516b9c1c6de45859fc9/pillow-12.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:37d6d0a00072fd2948eb22bce7e1475f34569d90c87c59f7a2ec59541b77f7a6", size = 4785266, upload-time = "2026-07-01T11:53:49.079Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/98/766667a4be768150a202836acd9fad19c06824ca86c4286d3cf6b274964e/pillow-12.3.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bcb46e2f9feff8d06323983bd83ed00c201fdcab3d74973e7072a889b3979fcd", size = 6263814, upload-time = "2026-07-01T11:53:51.32Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/2d/ede717bc1144f63886c21fd349bb95860b0d1a21149ff16f2bb362b612b6/pillow-12.3.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:23d27a3e0307ec2244cc51e7287b919aa68d097504ebe19df4e76a98a3eea5bd", size = 6934408, upload-time = "2026-07-01T11:53:53.487Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/48/9c58b685e69d49c31af6c8eb9012055fab7e665785165c84796e2c73ce72/pillow-12.3.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:4f883547d4b7f0495ebe7056b0cc2aea76094e7a4abc8e933540f3271df27d9c", size = 6337160, upload-time = "2026-07-01T11:53:55.457Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/fa/dc2a5c0ba6df93f67c31d34b808b7ce440b40cdbf96f0b81cde1d1e6fa93/pillow-12.3.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:236ff70b9312fb68943c703aa842ca6a758abfa45ac187a5e7c1452e96ef72b5", size = 7045172, upload-time = "2026-07-01T11:53:57.736Z" },
+ { url = "https://files.pythonhosted.org/packages/86/a5/444817a4d4c4c2417df00513086ca196f388d8f9ef40c2e4ccd1ad1af54b/pillow-12.3.0-cp311-cp311-win32.whl", hash = "sha256:10e41f0fbf1eec8cfd234b8fe17a4caac7c9d0db4c204d3c173a8f9f6ef3232b", size = 6472232, upload-time = "2026-07-01T11:53:59.767Z" },
+ { url = "https://files.pythonhosted.org/packages/63/c6/4bad1b18d132a50b27e1365e1ab163616f7a5bb56d330f66f9d1d9d4f9d4/pillow-12.3.0-cp311-cp311-win_amd64.whl", hash = "sha256:8e95e1385e4998ae9694eeaa4730ba5457ff61185b3a55e2e7bea0880aef452a", size = 7233653, upload-time = "2026-07-01T11:54:02.066Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/16/00f91ab7760dc842f5aad55217e80fc4a7067a0604535249bc8a2d6d9870/pillow-12.3.0-cp311-cp311-win_arm64.whl", hash = "sha256:ebaea975e03d3141d9d3a507df75c9b3ec90fa9d2ffd07567b3a978d9d790b26", size = 2568195, upload-time = "2026-07-01T11:54:04.622Z" },
+ { url = "https://files.pythonhosted.org/packages/37/bf/fb3ebff8ddcb76aac5a01389251bbbb9519922a9b520d8247c1ca864a25d/pillow-12.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ba09209fbe443b4acccebe845d8a138b89a8f4fbaeedd44953490b5315d5e965", size = 5345969, upload-time = "2026-07-01T11:54:06.397Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/66/9a386a92561f402389a4fc70c18838bf6d35eb5eb5c6850b4b2dc64f5048/pillow-12.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ffd0c5368496f41b0944be820fcb7a838aa6e623d250b01acf2643939c3f99d7", size = 4780323, upload-time = "2026-07-01T11:54:09.351Z" },
+ { url = "https://files.pythonhosted.org/packages/25/27/ac8f99618ffd3dde21db0f4d4b1d2ab00c0880595bfd17df103f7f39fd0c/pillow-12.3.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d9c7f76c0673154f044e9d78c8655fb4213f6ca31a836df48b40fe5d187717b9", size = 6266838, upload-time = "2026-07-01T11:54:11.71Z" },
+ { url = "https://files.pythonhosted.org/packages/84/21/a35af28dcc61f37ed850a2d64c65c701321dfbf25085e469d5559360cbbf/pillow-12.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:78cb2c6865a35ab8ff8b75fd122f6033b92a62c82801110e48ddd6c936a45d91", size = 6940830, upload-time = "2026-07-01T11:54:13.732Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/51/8b08617af3ad95e33ce6d7dd2c99ed6c8298f7fb131636303956be022e25/pillow-12.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e491916b378fba47242221bb9ead245211b70d504f495d105d17b14a24b4907c", size = 6344383, upload-time = "2026-07-01T11:54:15.756Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/72/cf78ac9780bb93c28328f408973845a309d4d145041665f734572ced1b52/pillow-12.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0dd2064cbc55aaec028ef5fbb60fa47bb6c3e7918e07ff17935284b227a9d2df", size = 7052934, upload-time = "2026-07-01T11:54:17.721Z" },
+ { url = "https://files.pythonhosted.org/packages/20/20/25e0f4dc178a6bc0696793720055519a0de89e7661dae886992decbd2f81/pillow-12.3.0-cp312-cp312-win32.whl", hash = "sha256:dbce0b29841537a2fa4a214c2bbf14de3587c9680caa9b4e217568472490b28f", size = 6472684, upload-time = "2026-07-01T11:54:19.839Z" },
+ { url = "https://files.pythonhosted.org/packages/45/89/da2f7971a317f83d807fdd4065c0af40208e59e692cc43d315a71a0e96d1/pillow-12.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:a2b55dd6b2a4c4b7d87ffa56bdb33fdc5fdb9a462173861a7bc097f17d91cb09", size = 7227137, upload-time = "2026-07-01T11:54:22.025Z" },
+ { url = "https://files.pythonhosted.org/packages/de/47/4845a0a6c0dbf1db8456bd9fc791f13c5ced7ced20606d08a0aacfd25b49/pillow-12.3.0-cp312-cp312-win_arm64.whl", hash = "sha256:331b624368d4f1d069149002f25f44bc61c8919ce8ddb3c45bdad8f6e2d89510", size = 2568267, upload-time = "2026-07-01T11:54:24.051Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/ac/31fb64e1e7efb5a4b50cd3d92049ba89ac6e4d8d3bb6a74e15048ca3353e/pillow-12.3.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:21900ce7ba264168cd50defae43cd75d25c833ad4ad6e73ffc5596d12e25ac89", size = 4161684, upload-time = "2026-07-01T11:54:25.934Z" },
+ { url = "https://files.pythonhosted.org/packages/87/b4/9805e23d2b4d77842b468513841fda254ee42f0289d25088340e4ff46e2d/pillow-12.3.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:4e8c2a84d977f50b9daed6eeaf3baef67d00d5d74d932288f02cb94518ee3ace", size = 4255487, upload-time = "2026-07-01T11:54:27.935Z" },
+ { url = "https://files.pythonhosted.org/packages/df/39/ecf519435a200c693fe053a6ee4d835b41cf963a4dfc2551c4e637cb2a71/pillow-12.3.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:ae26d61dfa7a47befdc7572b521024e8745f3d809bd95ca9505a7bba9ef849ec", size = 3696433, upload-time = "2026-07-01T11:54:29.813Z" },
+ { url = "https://files.pythonhosted.org/packages/42/92/2fc3ffad878ae8dd5469ec1bc8eb83b71f48e13efdf68f02709003982a32/pillow-12.3.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7a743ff716f746fc19a9557f60dab1600d4613255f8a7aeb3cdde4db7eb15a66", size = 5345889, upload-time = "2026-07-01T11:54:31.97Z" },
+ { url = "https://files.pythonhosted.org/packages/10/76/8803c13605b763d33d156c4678fc77f8443389c0c51c8aef707bb02015f4/pillow-12.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d69141514cc30b774ceea5e3ed3a6635c8d8a96edf664689b890f4089111fb35", size = 4780109, upload-time = "2026-07-01T11:54:34.026Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/01/e18aff37cb0b4aac47ac90f016d347a49aca667ef97f190b06ac2aabc928/pillow-12.3.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f7401aebd7f581d7f83a439d87d474999317ee099218e5ad25d125290990ba65", size = 6263736, upload-time = "2026-07-01T11:54:36.131Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/62/de5bdd77d935331f4f802edc11e4d82950f642caad6cb2f949837b8560e2/pillow-12.3.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3", size = 6937129, upload-time = "2026-07-01T11:54:38.216Z" },
+ { url = "https://files.pythonhosted.org/packages/70/4d/105627a13300c5e0df1d174230b32fd1273062c96f7745fd552b945d1e1d/pillow-12.3.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:571b9fcb07b97ef3a492028fb3d2dc0993ca23a06138b0315286566d29ef718a", size = 6339562, upload-time = "2026-07-01T11:54:40.354Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/1d/f13de01a553988ab895ba1c722e06cf3144d4f57656fd5b81b6d881f1179/pillow-12.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:756c768d0c9c2955feb7a56c37ea24aea2e369f8d36a88da270b6a9f19e62b5e", size = 7049439, upload-time = "2026-07-01T11:54:42.489Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/f9/066794cca041b969964f779ee5fa66a9498bbf34248ac39c5d7954e4198f/pillow-12.3.0-cp313-cp313-win32.whl", hash = "sha256:a876864214e136f0eb367788dbd7df045f4806801518e2cfe9e13229cfe06d8f", size = 6473287, upload-time = "2026-07-01T11:54:44.9Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/9b/7a58e61d62be561da3a356fe2384d4059a6345fc130e23ef1c36a5b81d24/pillow-12.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:1cca606cd25738df4ed873d5ad46bbdb3d83b5cbca291f6b4ff13a4df6b0bbe8", size = 7239691, upload-time = "2026-07-01T11:54:47.141Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/b0/c4ed4f0ef8f8fa5ee8351537db6650bb8189f7e118842978dd6589065692/pillow-12.3.0-cp313-cp313-win_arm64.whl", hash = "sha256:b629de27fda84b42cde7edef0d85f13b958b47f6e9bbcbba9b673c562a89bd8b", size = 2568185, upload-time = "2026-07-01T11:54:49.137Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/01/001f65b68192f0228cc1dbbc8d2530ab5d58b61037ba0587f946fea607cd/pillow-12.3.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:9cf95fe4d0f84c82d282745d9bb08ad9f926efa00be4697e767b814ce40d4330", size = 4161736, upload-time = "2026-07-01T11:54:51.156Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/d2/0219746d0fd16fc8a84498e79452375be3797d3ce4044596ce565164b84f/pillow-12.3.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:8728f216dcdb6e6d555cf971cb34076139ad74b31fc2c14da4fafc741c5f6217", size = 4255435, upload-time = "2026-07-01T11:54:53.414Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/02/8d0bc62ef0302318c46ff2a512822d2610e81c7aa46c9b3abe6cbaca5ad0/pillow-12.3.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:a45650e8ce7fafffd731db8550230db6b0d306d181a90b67d3e6bca2f1990930", size = 3696262, upload-time = "2026-07-01T11:54:55.739Z" },
+ { url = "https://files.pythonhosted.org/packages/85/e2/73c77d218410b14f5f2d565e8a998d5317b7b9c75368d29985139f7a46f0/pillow-12.3.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:ba54cfebe86920a559a7c4d6b9050791c20513650a1952ebe3368c7dc70306f8", size = 5350344, upload-time = "2026-07-01T11:54:57.657Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/da/32c752228ae345f489e3a42499d817b6c3996da7e8a3bc7a04fc806b243b/pillow-12.3.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:e158cb00350dc278f3b91551101aa7d12415a66ebf2c91d8d5ac14e56ddd3ad0", size = 4780131, upload-time = "2026-07-01T11:54:59.713Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/9d/8b2c807dbef61a5197c047afe99823787eb66f63daf9fb2432f91d6f0462/pillow-12.3.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e9aeb04d6aef139de265b29683e119b638208f88cf73cdd1658aa07221165321", size = 6263757, upload-time = "2026-07-01T11:55:01.778Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/44/c85361f65dbe00eea8576ee467c768d25129989efb76e94f205e9ca9bb46/pillow-12.3.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:251bf95b67017e27b13d82f5b326234ca62d70f9cf4c2b9032de2358a3b12c7b", size = 6936962, upload-time = "2026-07-01T11:55:03.93Z" },
+ { url = "https://files.pythonhosted.org/packages/18/7e/e483414b35800b86b6f08dbbc7803fb5cd52c4d6f897f47d53ea2c7e6f65/pillow-12.3.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fe3cca2e4e8a592be0f269a1ca4835c25199d9f3ce815c8491048f785b0a0198", size = 6339171, upload-time = "2026-07-01T11:55:05.989Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/f4/68c491844841ede6bed70189546b3ee9731cf9f2cbad396faff5e1ccba45/pillow-12.3.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:23aceaa007d6172b02c277f0cd359c79492bbb14f7072b4ede9fbcaf20648130", size = 7048116, upload-time = "2026-07-01T11:55:08.131Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/34/77f3f793fed8efc7d243f21b33c5a3f0d1c97ee70346d3db855587e155ff/pillow-12.3.0-cp314-cp314-win32.whl", hash = "sha256:af8d94b0db561cf68b88a267c5c44b49e134f525d0dc2cb7ed413a66bc23559a", size = 6467209, upload-time = "2026-07-01T11:55:10.408Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/e0/492879f69d94f91f60fc8cd05ba03650e9520afebb2fb7aa12777d7c7f38/pillow-12.3.0-cp314-cp314-win_amd64.whl", hash = "sha256:fdafc9cce40277e0f7a0feabce0ee50dd2fa1800f3b38015e51296b5e814048d", size = 7237707, upload-time = "2026-07-01T11:55:12.745Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/ac/6b11f2875f1c2ac040d84e1bbf9cf22a88038f901ca1037898b280b38365/pillow-12.3.0-cp314-cp314-win_arm64.whl", hash = "sha256:e91206ee562682b51b98ef4b26a6ef48fd84e15fd4c4bc5ec768eb641d206838", size = 2565995, upload-time = "2026-07-01T11:55:14.736Z" },
+ { url = "https://files.pythonhosted.org/packages/52/69/c2208e56af9bfc1913afb24020297a691eb1d4ef688474c8a04913f65e04/pillow-12.3.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:164b31cd1a0490ab6efae01aa5df49da7061be0af1b30e035b6e9a1bfe34ee6e", size = 5352503, upload-time = "2026-07-01T11:55:17.076Z" },
+ { url = "https://files.pythonhosted.org/packages/07/70/e5686d753e898a45d778ff1718dba8516ead6ab6b95d85fc8c4b70650cf2/pillow-12.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:5afb51d599ea772b8365ae807ae557f18bccfe46ab261fd1c2a9ed700fc6eb17", size = 4782956, upload-time = "2026-07-01T11:55:19.448Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/37/25c6692f06927ee973ff18c8d9ee98ad0b4d84ee67a09610c2dd1447958e/pillow-12.3.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3edce1d53195db527e0191f84b71d02022de0540bf43a16ed734ed7537b07385", size = 6322855, upload-time = "2026-07-01T11:55:21.613Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/91/420637fcb8f1bc11029e403b4538e6694744428d8246118e45719f944556/pillow-12.3.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bf16ba1b4d0b6b7c8e534936632270cf70eb00dbe09005bc345b2677b726855c", size = 6989642, upload-time = "2026-07-01T11:55:24.006Z" },
+ { url = "https://files.pythonhosted.org/packages/10/08/b94d7811281ccf0d143a1cf768d1c49e1e54af63e7b708ab2ee3eb87face/pillow-12.3.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:24870b09b224f7ae3c39ed07d10e819d06f8720bc551847b1d623832b5b0e28d", size = 6391281, upload-time = "2026-07-01T11:55:26.252Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/87/24233f785f55474dc02ce3e739c5528a77e3a862e9333d1dd7a25cc31f70/pillow-12.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:30f2aa603c41533cc25c05acd0da21636e84a315768feb631c937177db558931", size = 7096716, upload-time = "2026-07-01T11:55:28.318Z" },
+ { url = "https://files.pythonhosted.org/packages/23/26/fcb2f6e37175b04f53570b59937867e2b80ee1685e744023153028fc14f9/pillow-12.3.0-cp314-cp314t-win32.whl", hash = "sha256:4b0a7fe987b14c31ebda6083f74f22b561fd3739bc0ac51e019622e3d72668c7", size = 6474125, upload-time = "2026-07-01T11:55:30.956Z" },
+ { url = "https://files.pythonhosted.org/packages/90/de/3634abee5f1c9e13c56787b7d5517b0ba8d6de51700b95578cf338349c9f/pillow-12.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:962864dc93511324d51ddbb5b9f8731bf71675b93ca612a07441896f4688fb8c", size = 7242939, upload-time = "2026-07-01T11:55:34.044Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/2a/fd13f8eb24de5714a6eb444a3d67e2842c6c576e159a43793adf23051351/pillow-12.3.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0740a512dc522224c77d9aa5a8d70d8b7d73fb91f2c21125d8d025d3b8990e45", size = 2567506, upload-time = "2026-07-01T11:55:35.988Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/dc/8fdce34ec725a33c81c6ba122b904d6b9024e50ea9ac7bede62fab54506c/pillow-12.3.0-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:0feb2e9d6ad6c9e3c06effe9d00f3f1e618a6643273576b016f591e9315a7139", size = 4162063, upload-time = "2026-07-01T11:55:37.941Z" },
+ { url = "https://files.pythonhosted.org/packages/76/66/2044b9a63d3b84ff048228dfcb7cd9bf0df983e8470971bf7d4c57b693de/pillow-12.3.0-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:9e881fca225083806662a5c43d627d215f258ff43c890f831966c7d7ba9c7402", size = 4255549, upload-time = "2026-07-01T11:55:40.022Z" },
+ { url = "https://files.pythonhosted.org/packages/52/7e/1f67e6f4ece6b582ee4b539decbcc9f848dc245a93ed8cd7338bafef72f1/pillow-12.3.0-cp315-cp315-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:4998562bf62a445225f22e07c896bb04b35b1b1f2eb6d760584c9c51d7a5f78c", size = 3696331, upload-time = "2026-07-01T11:55:41.98Z" },
+ { url = "https://files.pythonhosted.org/packages/12/40/d306fc2c8e4d45d7f175c77edca7063be7b86fe7fe6e68f4353bf71d808c/pillow-12.3.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:dc624f6bc473dacdf7ef7eb8678d0d08edf15cd94fad6ae5c7d6cc67a4e4902f", size = 5350370, upload-time = "2026-07-01T11:55:44.028Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/44/668fb1437e8ce420f62d6106eb66e44a5971602a4d794615bdf79315d82d/pillow-12.3.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:71d6097b330eea8fd15097780c8e89cb1a8ce7838669f48c5bacd6f663dd4701", size = 4780147, upload-time = "2026-07-01T11:55:46.073Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/08/93fa2e70e30a2d81547e481b6ee2bb9522117221fb1e0ce4b5df70967677/pillow-12.3.0-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:28ce87c5ab450a9dd970b52e5aca5fe63ed432d18a2eaddd1979a00a1ba24ace", size = 6273659, upload-time = "2026-07-01T11:55:48.264Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/6d/043e96ff814fc31a33077e4cba86082167db520c93632afdf2042febbb0c/pillow-12.3.0-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6b02afb9b97f65fbca5f31db6a2a3ba21aa93030225f150fa3f249717e938fb4", size = 6947439, upload-time = "2026-07-01T11:55:50.503Z" },
+ { url = "https://files.pythonhosted.org/packages/af/92/ba71d2ee2ac0edf3fa33bd9d5ee9ee080da70b1766f3ca3934f9938ddac9/pillow-12.3.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:1182d52bc2d5e5d7d0949503aa7e36d12f42205dc287e4883f407b1988820d39", size = 6353577, upload-time = "2026-07-01T11:55:52.697Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/ce/e63064e2122923ff687c8ad792d0d736a7b3920a56a46982e81a7fdd25d6/pillow-12.3.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e795b7eb908249c4e43c7c99fac7c2c75dab0c43566e37db472a355f63693d71", size = 7060394, upload-time = "2026-07-01T11:55:55.149Z" },
+ { url = "https://files.pythonhosted.org/packages/54/76/a09cc3ccc8d773a7283d34c38bec1708f9e3cc932093cbc4c5e71ac4060b/pillow-12.3.0-cp315-cp315-win32.whl", hash = "sha256:57b3d78c95ba9059768b10e28b813002261d3f3dfc55cc48b0c988f625175827", size = 6467375, upload-time = "2026-07-01T11:55:57.769Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/03/1846c49ba3b1d5550392a4bbd06d6fb4578e1cd91a803198b5c90f5f7d53/pillow-12.3.0-cp315-cp315-win_amd64.whl", hash = "sha256:fa4ecea169a355be7a3ade2c783e2ed12f0e40d2c5621cda8b3297faf7fbb9f5", size = 7237048, upload-time = "2026-07-01T11:55:59.975Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/bb/89f35dcc79610423f9f195504d7def7f0d1416a711541b42867e25fe3412/pillow-12.3.0-cp315-cp315-win_arm64.whl", hash = "sha256:877c3f311ff35410f690861c4409e7ccbf0cd2f878e50628a28e5a0bb689e658", size = 2566006, upload-time = "2026-07-01T11:56:02.143Z" },
+ { url = "https://files.pythonhosted.org/packages/30/88/707027ba09942dfa2c28759b5c222d769290a41c6d20ea60ec250801941f/pillow-12.3.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:e9871b1ffbfa9656b60aeee92ed5136a5742696006fa322b29ea3d8da0ecc9cf", size = 5352509, upload-time = "2026-07-01T11:56:04.2Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/6d/00352fa25332c2569cd387851f568cc5a4b75a9adbfb37ac4fbce4c02eec/pillow-12.3.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:53aa02d20d10c3d814d536aa4e5ac9b84ca0ff5a88377963b085ad6822f93e64", size = 4783167, upload-time = "2026-07-01T11:56:06.631Z" },
+ { url = "https://files.pythonhosted.org/packages/13/4f/9e049dfa21af7c22427275720e2490267ba8138120add5c4c574deb69782/pillow-12.3.0-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:446c34dcc4324b084a53b705127dc15717b22c5e140ae0a3c38349d4efec071e", size = 6329237, upload-time = "2026-07-01T11:56:08.868Z" },
+ { url = "https://files.pythonhosted.org/packages/36/16/cf6eeaae8d0fce8dd390a33437cf68c5d5bd73834a2bc6e2f14efda0ab45/pillow-12.3.0-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cf1845d02ad822a369a49f2bb9345b1614744267682e7a03527dc3bf6eea1777", size = 6997047, upload-time = "2026-07-01T11:56:11.379Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/69/dbf769bdd55f48bf5733cac28edc6364ffaa072ec9ba336266e4fe66be55/pillow-12.3.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:186941b6aef820ad110fb01fb06eb925374dc3a21b17e37ec9a53b250c6fe2d1", size = 6400440, upload-time = "2026-07-01T11:56:13.908Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/e1/ffc9cfc2eea0d178da8018e18e959301ad9d6bc9f3edb7181e748a474b97/pillow-12.3.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:f13c32a3abd6079a66d9526e18dad9b6d280384d49d7c54040cd57b6424041d9", size = 7105895, upload-time = "2026-07-01T11:56:16.575Z" },
+ { url = "https://files.pythonhosted.org/packages/18/f0/a5595c1e8c3ae44b9828cb2f0fa8155e5095ef04d6327b8f61cf44a3df85/pillow-12.3.0-cp315-cp315t-win32.whl", hash = "sha256:1657923d2d45afb66526e5b933e5b3052e6bdea196c90d3abb2424e18c77dae8", size = 6474384, upload-time = "2026-07-01T11:56:18.855Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/04/62bcd9f844984c5938d3b05264a61d797a29d3e0812341a8204af70bbdee/pillow-12.3.0-cp315-cp315t-win_amd64.whl", hash = "sha256:8cd2f7bdda092d99c9fc2fb7391354f306d01443d22785d0cbfafa2e2c8bb418", size = 7243537, upload-time = "2026-07-01T11:56:21.214Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/68/1f3066acedf37673694a7141381d8f811ae97f30d34413d236abe7d489f1/pillow-12.3.0-cp315-cp315t-win_arm64.whl", hash = "sha256:06ff022112bc9cbf83b60f8e028d94ad87b60621706487e65f673de61610ab59", size = 2567491, upload-time = "2026-07-01T11:56:23.506Z" },
+ { url = "https://files.pythonhosted.org/packages/75/18/2e8b40223153ccbc60df07f9e8928dc0c76202aa4e55ae9f53962b6510d6/pillow-12.3.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:b3c777e849237620b022f7f297dd67705f9f5cf1685f09f02e46f93e92725468", size = 5302510, upload-time = "2026-07-01T11:56:25.736Z" },
+ { url = "https://files.pythonhosted.org/packages/46/3e/51fabf59d5ab801ceab709453d3ab6b180083496579549de4c45ced6528a/pillow-12.3.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:b343699e8308bdc51978310e1c959c584e7869cc8c40780058c87da7781a1e94", size = 4736058, upload-time = "2026-07-01T11:56:28.041Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/20/22fe9384b7949e25fb1293bcfc84fb82590ff4ea6b37c95b24d26d793d86/pillow-12.3.0-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fbd139c8447d25dd750ab79ee274cc5e1fe80fc56340ab10b18a195e1b6eca3e", size = 5237776, upload-time = "2026-07-01T11:56:30.263Z" },
+ { url = "https://files.pythonhosted.org/packages/08/14/f6ba68107680ffa74b39985f3f30884e41318fbc4250caa423c79b4788bb/pillow-12.3.0-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e7e480451b9fa137494bccd3a7d69adbe8ac65a87d97be61e11f1b1050a5bac3", size = 5860358, upload-time = "2026-07-01T11:56:32.68Z" },
+ { url = "https://files.pythonhosted.org/packages/36/54/0169bc772ec491108b62f644f8ecf1fe5d8ae5ebafde2ee2142210166903/pillow-12.3.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:04f01d28a6aaff387bf842a13be313df23ba0597a44f1a976c9feb3c6ff4711a", size = 7231786, upload-time = "2026-07-01T11:56:35.046Z" },
+]
+
[[package]]
name = "platformdirs"
version = "4.10.0"
@@ -618,6 +784,19 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" },
]
+[[package]]
+name = "reportlab"
+version = "5.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "charset-normalizer" },
+ { name = "pillow" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/41/d6/4b7b0cf56880eb96533e607967be6a939e344675601e033d113a0bfa1f4e/reportlab-5.0.0.tar.gz", hash = "sha256:e4494a0c6623ae213bb856fba523171b2b54a7bf629fda02d5e525a7b899a784", size = 3701928, upload-time = "2026-06-18T11:34:31.145Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a3/07/70085c17a369605f15e301d10ab902115019b1126c7253d964afc230c7d6/reportlab-5.0.0-py3-none-any.whl", hash = "sha256:9d5a3affa84919e1111ede580031266a570e93b1ce388219621347965ff1d93c", size = 1956710, upload-time = "2026-06-18T11:34:29.07Z" },
+]
+
[[package]]
name = "rich"
version = "15.0.0"
+ 
+
+
+