Repository navigation
feat: results library phase 4 — packaging, distribution and parity #1756
Description
Overview
Results-library phase 4 (epic ledger: PyAutoMind draft/feature/autofit/results_library_epic.md). It ships a finished library and checks a rebuild against a reference. It generalises the DR1 tools: merge_catalogue.distribute (part zips, manifest, SHA256SUMS) and compare_catalogues.py (a 1,138-line parity tool). It also adds a run-to-run scatter helper, because two identical Nautilus runs differed by a median z of 15.4 (report P19), which makes "within 3σ" the wrong parity test.
Plan
af.export.package(...):- size-capped part zips with each object kept whole, plus a tables zip;
parts_manifest.csv,SHA256SUMSandpackage.json;- STORED media / DEFLATED text, atomic zips;
- optional
include_dataset; - a README / KNOWN_ISSUES template with provenance.
- Delta rebuild: objects keep their part, and only changed parts are rewritten. Unchanged parts stay byte-identical.
af.export.verify(dist): checksums, CRC, no empty members, CSVs parse with matching row counts, manifest coverage.af.export.compare(a, b, key=):- identity, exact and tolerance-by-σ rules, with an error floor;
- duplicate-key detection;
- values are informational unless gated;
- a markdown report.
af.export.scatter_between(agg_a, agg_b, paths): the per-parameter z distribution between two runs.- CLI:
python -m autofit.export {package,verify,compare}. - Witness W1d on the DR1 laptop mirror, including the 13 real vis_lp reruns.
Tier: judge — merge mode: human /prm
Detailed implementation plan
Affected Repositories
- PyAutoFit (primary)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./fit/PyAutoFit | main | clean |
Suggested branch: feature/results-library-p4-package
Implementation Steps
autofit/export/package.py:package();- planning: greedy by cap, or N contiguous near-equal parts; previous assignments are reused when the item set allows;
- a deterministic zip writer: sorted members, file mtimes,
compress_type_for; package.json: schema, items→part, per-part content digest, CSV row counts, provenance.
autofit/export/verify.py:verify()andVerifyReport.autofit/export/parity.py:compare(),ParityReport(passed, per-column stats,to_markdown()) andscatter_between().autofit/export/readme.py: the README / KNOWN_ISSUES template. It records git revisions of the given repos and a results-index hash.autofit/export/__main__.py: the CLI.autofit/__init__.pygainsfrom . import export.- Tests:
- package into 3 parts and verify;
- a delta rebuild leaves 2 parts byte-identical;
- verify catches an empty member, a corrupt byte, manifest drift and a row-count mismatch;
- compare: identity, exact and duplicate failures; reruns pass with their scatter reported; error floor;
scatter_between;- the CLI.
Key Files
autofit/export/(new)autofit/__init__.py
Original Prompt
Click to expand starting prompt
PyAutoMind active/results_library_p4_package.md. Human: "start phase 4"; plan approved 2026-11-02 ("I approve").
Activity
Jammy2211 commentedon Nov 2, 2026
Library PR Created: #1757 (pending-release). Full suite serial 4262 passed + 1 unrelated signal-timing flake (passes 5/5 alone); independent review 16 findings fixed with regressions; W1d on the DR1 laptop mirror: 3-part package verifies, one-object rebuild leaves 3 zips byte-identical, injected empty member caught. Finding: the mirror's 13 vis_lp 'reruns' are copies (z=0) — real scatter calibration needs genuine reruns (phase 5). No workspace impact (new additive module; adoption is phase 5). Tier judge → human /prm.
Jammy2211 commentedon Nov 2, 2026
Shipped. PyAutoFit#1757 merged (pending-release) — af.export package / verify / compare / scatter_between + CLI. Witness W1d on the DR1 laptop mirror passed. Next: phase 5 (organism integration + 500-lens DR1 v1.0 census).