[Jammy2211]

Summary

Results-library phase 4 (closes #1756; epic ledger PyAutoMind draft/feature/autofit/results_library_epic.md). This phase covers shipping a finished results library and checking a rebuild against its reference, through a new af.export package:

• package: part zips plus a manifest, checksums and a provenance README. A rebuild rewrites only the parts that changed.
• verify: checks a package is whole.
• compare: diffs two results tables.
• scatter_between: run-to-run scatter.

It generalises the DR1 tools merge_catalogue.distribute (part zips, manifest, SHA256SUMS) and compare_catalogues.py (1,138 lines).

API Changes

Additive only. af.export is a new module.

• af.export.package(source, dist, cap_gb=|n_parts=, include_dataset=, title=, repos=, index=, notes=, replan=False, force=False) → PackageReport.
• af.export.verify(dist, allow_empty=...) → VerifyReport (.ok, .problems).
• af.export.compare(a, b, key="id", exact=(), gate=(), threshold=3, floor=0, rel_floor=0, columns=None) → ParityReport (.passed, .columns, .to_markdown()).
• af.export.scatter_between(a, b, paths, key=...) → ScatterReport.
• CLI python -m autofit.export {package,verify,compare}: exit 0 = OK, 1 = problems / tables differ, 2 = error.
  See full details below.

Test Plan

• test_autofit/export: 37 tests, 15 of them review regressions:
  • package into 3 parts, then verify;
  • a one-object change rewrites one part and the others stay byte-identical;
  • the same files give the same zip in any timezone;
  • new objects fill the last part then a new one; the cap is remembered;
  • replan removes stale parts;
  • verify catches an empty member, a corrupt byte, corrupted deflate data, a bad SHA256SUMS line, ragged rows, a row-count mismatch, manifest drift, a missing part and an unlisted zip;
  • compare: reruns report their scatter and PASS; a gated shift FAILs; identity, duplicate and column-set failures; unknown gate names; one-sided blanks; big-integer ids; floors;
  • scatter_between;
  • the CLI.
• Full suite, serial: 4,262 passed, 1 failed. The failure is test_fit_status.py::test_sigusr1_then_sigterm_in_a_fitting_process, a signal-timing flake under load. It passes 5 of 5 alone, and this branch does not touch autofit/non_linear.
• Independent review: 16 findings (4 high, 6 medium, 6 low), all fixed:
  • compare silently skipped missing, exact and gated columns;
  • one-sided blanks passed exact gates;
  • stale parts were left after a replan;
  • verify crashed on corruption (exit 2 instead of 1);
  • duplicate members;
  • columns= dropped the σ companions;
  • float-equal big integer ids;
  • cap_gb was forgotten and n_parts ignored on a rebuild;
  • the scatter key could collapse;
  • non-UTF-8 and very wide CSVs crashed packaging;
  • n_parts was not exact;
  • false "ragged" on a trailing blank line;
  • zips depended on the local timezone;
  • no force option;
  • object dot-files (.completed) were dropped;
  • the CLI report write was unguarded.
• Witness W1d on the DR1 laptop mirror (Workspace/Science/scratch/results_library_p4_w1d/):
  • 113 vis_pix zips packaged into 3 parts, and verify OK;
  • after a one-object change only part_002.zip was rewritten, and the other three are byte-identical (sha256sum -c);
  • an emptied member was caught by both its checksum and the empty-file check (exit 1).
  • Finding: the mirror's 13 "vis_lp reruns" (output/ vs output_sed/) are copies, with z = 0 everywhere. Calibrating real run-to-run scatter (P19's median z of 15) needs genuine reruns, which is phase-5 census work.

Full API Changes (for automation & release notes)

Added

• autofit.export.package:
  • package(...), PackageReport, PackageException;
  • index_hash(index), write_zip(zip_path, members), part_name(i), sha256(path), csv_records(handle).
• autofit.export.verify: verify(dist, allow_empty=(".completed", "*/.completed")) and VerifyReport.
• autofit.export.parity: compare(...), ParityReport, ColumnParity, scatter_between(...), ScatterReport and Scatter.
• autofit.export.readme: write_readme(dist, document, title, notes).
• python -m autofit.export subcommands package, verify and compare.
• autofit/__init__.py: from . import export, so af.export.* works.

Migration

• None required.

Decisions flagged:

• Everything lives under af.export, including scatter_between; the ledger said af.agg, which doesn't exist.
• The packaging unit is a folder of object folders. Producing that folder from a results index is left to callers and phase 5.
• Value columns gate only when named. That is the P19 lesson.
• A rebuild detects changes from name, size and mtime; force=True rewrites every zip.
• Packaging deletes any part_*.zip in dist it did not write.

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code