Repository navigation
feat: results library phase 3 — incremental honest exports + chunk/merge #1754
Description
Overview
Results-library phase 3 (epic ledger: PyAutoMind draft/feature/autofit/results_library_epic.md). Exports become incremental and honest. A rebuild only re-reads new or changed results. Every result, and every expected result, gets a row or a reason, never a silent drop. A columns.json sidecar describes each column. A scheduler-agnostic chunk/merge splits big exports across Slurm tasks or a laptop pool. It generalises the DR1 hand tooling (merge_catalogue.merge_csv, build_wcs_package.plain_from).
Plan
- Incremental
AggregateCSV.save(path, cache=True): a row cache keyed by result fingerprint plus column definitions. - Per-row error capture into
status_reason.expected=(phase 2'sCompletenessReport) adds rows for missing or unfinished expected results. Optional prior-edge flag columns. <name>.columns.jsonsidecar: source, kind, unit, bound type (quantile_1sigma), n_blank.SearchOutput.plain(name): a plain-JSON view of typed outputs.- Chunk/merge:
save(shard="i/N"), thenmerge_csv(...)orpython -m autofit.results merge. It checks header agreement, drops duplicate keys and sorts deterministically. - AggregateFITS: atomic writes plus per-result error capture.
- Witness W1c: a re-export of the DR1 laptop mirror.
Tier: judge — merge mode: human /prm
Detailed implementation plan
Affected Repositories
- PyAutoFit (primary)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./fit/PyAutoFit | main | clean |
Suggested branch: feature/results-library-p3-exports, stacked on feature/results-library-p2-completeness (PR #1753). The PR targets that branch and is retargeted to main after #1753 merges.
Implementation Steps
aggregator/summary/aggregate_csv/cache.py:RowCache(<path>.rows.jsonl, atomic), keyed by output path +stat_fingerprint+ a column-spec hash. A computed column's hash covers its__qualname__and code bytes. Labels are always recomputed.Row.dict()error capture writesstatus_reason="error:<Type>: <msg>";missing_latent_summaryis used when not strict.AggregateCSV.save(path, cache=False, shard=None, expected=None, sort_by=None, columns_json=True). The default behaviour is unchanged.aggregate_csv/columns.pywrites the sidecar.add_variable(..., unit=None, prior_edge=False); the edge test is "within 1% of the prior's width from a limit".aggregate_csv/merge.pyprovidesmerge_csv, and themergeCLI subcommand goes inresults/__main__.py.SearchOutput.plain(name)walks autonerves dictable tags (instance / dict / list / tuple / ndarray / path / np.number).- AggregateFITS:
output_to_folderwrites viaopen_atomic, plus per-result error capture. - Tests:
- a 15k synthetic refresh opens only the 50 changed results;
- a killed export leaves the CSV and cache intact;
- a 4-shard merge is byte-identical to the serial export;
- the merge refusals fire;
- expected-reason rows appear;
columns.jsonandplain()are checked.
File ownership vs phase 2 (ruling F11)
This task owns aggregator/summary/, SearchOutput.plain and the merge subcommand. It reuses phase 2's results/shard.py and results/completeness.py read-only.
Key Files
autofit/aggregator/summary/aggregate_csv/{__init__,row,column}.pyand the newcache.py,columns.py,merge.pyautofit/aggregator/summary/aggregate_fits.pyautofit/aggregator/search_output.pyautofit/results/__main__.py
Original Prompt
Click to expand starting prompt
PyAutoMind active/results_library_p3_exports.md. Human: "start phase 3 in parallel too"; plan approved 2026-11-02 ("I approve").
Activity
Jammy2211 commentedon Nov 2, 2026
Library PR Created: #1755 (pending-release), stacked on #1753 — merge #1753 first, then retarget #1755 to main. Full suite serial 4226 passed; independent review 6+3 findings fixed with regressions; W1c on the DR1 laptop mirror: refresh 11 s all-cached, cached==fresh, killed export leaves files intact, 4-shard merge == serial. No workspace impact (API is additive; phase-5 assistant/pipeline adoption). Tier judge → human /prm.
Jammy2211 commentedon Nov 2, 2026
Shipped. PyAutoFit#1755 merged into main (pending-release) — incremental AggregateCSV row cache, honest rows + expected reasons, columns.json, prior-edge flags, merge_csv/merge CLI, SearchOutput.plain, AggregateFITS atomic writes. Witness W1c on the DR1 laptop mirror: refresh 11 s all cached, cached == fresh, kill -9 safe, 4-shard merge == serial. Next: phase 4 (package + parity).