[Jammy2211]

Summary

Results-library phase 3 (closes #1754; epic ledger PyAutoMind draft/feature/autofit/results_library_epic.md). Exports become incremental (a refresh reads only new or changed results) and honest: every result, and every expected result, gets a row or a reason, and a columns.json sidecar describes each column. A scheduler-agnostic chunk/merge splits a big export across Slurm array tasks or a laptop process pool and merges it byte-identical to the serial export. This generalises the DR1 hand tooling (merge_catalogue.merge_csv, build_wcs_package.plain_from).

Stacked on #1753 (phase 2): it reuses autofit/results/shard.py and CompletenessReport. The base is feature/results-library-p2-completeness. Merge #1753 first, then retarget this PR to main.

API Changes

Additive. AggregateCSV.save(path) with no new arguments keeps its columns, column order and no-sort behaviour.

• AggregateCSV.save(path, cache=False, shard=None, expected=None, sort_by=None, on_error="raise", columns_json=False).
• AggregateCSV.add_variable(..., unit=None, prior_edge=False).
• af.merge_csv(parts, path, key="id", sort_by=None) and python -m autofit.results merge OUT PARTS....
• SearchOutput.plain(name): a typed files/<name>.json as plain Python.
• AggregateFITS.output_to_folder(..., skip_errors=False): atomic writes, returns the skipped results.
  See full details below.

Test Plan

• test_exports.py: 21 tests, 10 of them regressions from the independent review. They cover:
  • refreshing after 30 new results computes only those 30, and the output equals a fresh export;
  • an in-place file change, an edited column, or a changed captured value is re-read;
  • a killed export leaves the CSV and cache byte-identical;
  • 4- and 8-shard merges are byte-identical to the serial export, including empty shards, a symlinked root with expected=, and merging a part twice;
  • the merge refusals fire;
  • error rows and expected rows carry reasons;
  • columns.json content;
  • plain(), including compound-key dicts;
  • the prior edge measured in log space;
  • the spec digest is stable across PYTHONHASHSEED values.
• Existing summary tests unchanged and green. A new AggregateFITS skip_errors test.
• Full suite, serial: 4,226 passed, 2 skipped, 1 xfailed.
• Independent review: 6 bugs plus 3 minor findings, all fixed with tests:
  • the header was built from rows, so shards disagreed;
  • expected rows were sharded by an unresolved path;
  • in-place file rewrites were missed by the fingerprint;
  • closures and helper functions were missing from the cache key;
  • a cached CSV could differ from a fresh one;
  • plain() mishandled compound-key dicts;
  • minor: hash-seed-dependent digests, merge dedup of rows with an empty key, the prior edge for log priors.
• Witness W1c on the DR1 laptop mirror, 113 vis_pix zips read in place (Workspace/Science/scratch/results_library_p3_w1c/):
  • first export 76 s; refresh 11–17 s with all 113 rows from the cache;
  • edited column code → all rows recomputed;
  • an export killed with kill -9 left the previous CSV and cache byte-identical;
  • the cached CSV equals the fresh one;
  • 4 parallel shards merged byte-identical to the serial export.
• The "15k fixture" in the ledger is scaled down to 150 results in the unit tests. W1c is the real-data scale check.

Full API Changes (for automation & release notes)

Added

• autofit.aggregator.summary.aggregate_csv.cache: RowCache, cache_path, row_fingerprint, function_digest, spec_digest and csv_text.
• autofit.aggregator.summary.aggregate_csv.merge: merge_csv, sort_rows, write_rows and MergeException. merge_csv and MergeException are re-exported from aggregate_csv, and merge_csv as af.merge_csv.
• AggregateCSV.stats: computed, cached, errors and expected from the last save.
• Column(..., unit=None, prior_edge=False), with Column.fields(), Column.describe(field) and Column.spec(). ComputedColumn and LabelColumn gain spec and describe.
• Row: reasons, latent_summary (the old _latent_summary is kept as an alias), parameter_paths, and dict(columns=None).
• autofit.aggregator.search_output.plain(document) and SearchOutput.plain(name).
• AggregateFITS.output_to_folder(..., skip_errors=False), which returns a list of (search_output, exception).
• CLI subcommand merge in python -m autofit.results.

Changed

• AggregateCSV.save writes the CSV header from the column definitions rather than from the rows. With the default arguments the columns and their order are the same as before.
• Values are written through str() (as csv does), so an ndarray in a computed column no longer breaks the blank-column check.

Migration

• None required.

Decisions flagged:

• Incremental caching is CSV-only, not FITS.
• Byte-identity between a merge and a serial export requires the same sort_by (shards default to id).
• A computed column whose dict keys vary between shards still makes the part headers differ, and the merge refuses them.
• on_error defaults to "raise", as before. Pass "row", or expected=, to get rows with reasons.

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code