Repository navigation
feat: results library phase 2 — completeness over a declared grid + enrich #1752
Description
Overview
Results-library phase 2 (epic ledger: PyAutoMind draft/feature/autofit/results_library_epic.md). Phase 1 gave PyAutoFit an index of every result and its state; phase 2 answers "what is missing?" against a declared expectation (datasets × searches × required products, optionally derived from an hpc-campaign campaign.yaml), and adds Aggregator.enrich to add a new product to thousands of existing zips atomically — replacing the force_pickle_overwrite + SLURM reload pass used for coolest.json / wcs.json in DR1.
Plan
- Declare what you expect:
library.yaml,af.ResultsLibrary(expected=...), orfrom_campaign: campaign.yaml(single source of truth — no second manifest). - Completeness report over the phase-1 index: every (dataset, search, product) is done / missing / incomplete / failed with a reason;
<name>.missing.csvwritten atomically. A zip without a required product is incomplete, not done. Aggregator.enrich(fn, name, overwrite=False, shard=(i, N)): atomic add-or-replace of a zip member, other members byte-identical, stable sharding for Slurm or a laptop pool.- CLI:
python -m autofit.results check library.yamlandenrich module:function --name … --shard i/N. - autofit_workspace guide
scripts/features/results_library.py(declare → index → check → enrich) on the gaussian example. - Witness W1b: read-only completeness check on the DR1 laptop mirror.
Tier: judge — merge mode: human /prm
Detailed implementation plan
Affected Repositories
- PyAutoFit (primary)
- autofit_workspace (after library)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./fit/PyAutoFit | main | clean |
| ./fit/autofit_workspace | main | clean |
Suggested branch: feature/results-library-p2-completeness
Implementation Steps
autofit/results/library.py—ExpectedResult(dataset, search, products, location),ResultsLibrarywithfrom_yaml(datasets:/datasets_file:,searches: {name: {path_prefix, products}}) andfrom_campaign(loadsautofit.campaign.Campaign; manifest index → dataset fields; stagesuccess.zip_completetemplate → expected location;success.members→ required products; mirrorscollector.check_success). Matching: location first, else(unique_tag | object_id, name); superseded records ignored.autofit/results/completeness.py—CompletenessReport(rows, counts,write_missing_csvviaopen_atomic). Named products fromrecord.products; path products viazip_names()(central directory only, no index schema bump). Reasons:no_result,not_completed:<state>,missing_product:<p>,location_mismatch,duplicate.autofit/results/enrich.py— generalisepaths/abstract.py::_replace_zip_memberto a shared add-or-replace helper (same-dir temp +os.replace); product →files/<name>.json(dict/list) or bytes / file path;overwrite=Falseskips results that have it; per-result error capture, never aborts the batch.autofit/results/shard.py— stableint(record.key, 16) % Nselection (phase 3 reuses it).Aggregator.enrichthin method;force_pickle_overwritedocstring points atenrich(no removal this phase).- CLI subcommands
check,enrichinautofit/results/__main__.py. - Tests: holes fixture (missing zip, no
.completed, missingimage/, missing named member, superseded duplicate) → every gap with right reason; enrich over 100 zips keeps sha256 of other members; killed enrich leaves original zip intact; shards disjoint + cover;from_campaignon a small campaign.yaml; CLI round-trip.
File ownership vs phase 3 (ruling F11)
Phase 2 owns results/{library,completeness,enrich,shard}.py, Aggregator.enrich, the zip-member helper. Phase 3 owns aggregator/summary/aggregate_csv/, aggregate_fits, search_output.plain, chunk/merge. Both add subcommands to results/__main__.py — second to merge rebases.
Key Files
autofit/results/— new modules aboveautofit/aggregator/aggregator.py—enrichautofit/non_linear/paths/abstract.py—_replace_zip_member→ shared helperautofit/campaign/spec.py,collector.py— read-only reuse forfrom_campaign
Original Prompt
Click to expand starting prompt
See PyAutoMind active/results_library_p2_completeness.md. Human request (verbatim): "Continbue the HPC kit and results epic where we are making it so PyAutoFit can do auto run campaign things"; plan approved 2026-11-02 ("I approve").
Activity
Jammy2211 commentedon Nov 2, 2026
Library PR Created: #1753 (pending-release). Workspace guide: PyAutoLabs/autofit_workspace#180 (merge after #1753 + release). Full suite serial 4204 passed; independent review 8 findings fixed; W1b witness on the DR1 laptop mirror 226/226 done (wrong-tree and naming-drift both surfaced). Tier judge → human /prm.
Jammy2211 commentedon Nov 2, 2026
Shipped. PyAutoFit#1753 merged (pending-release) — ResultsLibrary + completeness check + Aggregator.enrich + CLI check/enrich; autofit_workspace#180 merged (features/results_library.py guide). Witness W1b on the DR1 laptop mirror: 226/226 done. Next: phase 3 (merged as #1755) → phase 4.