[Jammy2211]

Summary

Results-library phase 2 (closes #1752; epic ledger PyAutoMind draft/feature/autofit/results_library_epic.md). Phase 1 indexed every result and its state; this phase answers "what is missing?" against a declared expectation and adds enrich to put a new product into thousands of finished results without rerunning fits. That replaces the force_pickle_overwrite + Slurm reload pass DR1 used for coolest.json / wcs.json.

API Changes

New, additive only — nothing removed or renamed.

• af.ResultsLibrary declares the expected results (datasets × searches × required products) via .grid(...), .from_yaml("library.yaml") or .from_campaign("campaign.yaml"). .check() returns a CompletenessReport with a status and reason for every expected result, and .write_missing_csv() writes the gaps.
• Aggregator.enrich(fn, name, overwrite=False, shard=None) adds files/<name> to every result atomically.
• CLI: python -m autofit.results check library.yaml (exit 0 = complete, 1 = gaps, 2 = error) and python -m autofit.results enrich ROOTS --fn module:function --name NAME [--shard i/N].
• autofit.tools.util.write_zip_member is the shared atomic zip add-or-replace helper.
  See full details below.

Test Plan

• test_autofit/results covers the following (58 passed, 22 new):
  • A holes fixture, with each gap's reason checked: missing, no .completed, missing product, interrupted, superseded sibling, failed before search.json, duplicate, unexpected.
  • library.yaml refusals.
  • from_campaign agreeing with collector.check_success on every task, plus the short-manifest-line case.
  • Enrich over 100 zips: other members keep their SHA256 and compression, and a killed write leaves the zip byte-identical.
  • Folder + zip results get both copies and survive a re-zip.
  • A pass stopped between the two copies is finished by a rerun.
  • Shards are disjoint and cover every result.
  • CLI exit codes.
• Full suite, serial: 4,204 passed, 2 skipped, 1 xfailed. -n 8 collides on shared /tmp paths in grid-search tests, and those pass serially.
• Independent review: 8 findings, all fixed, each with a test:
  • enrich via from_index skipped the folder copy;
  • the write order could leave a split state;
  • campaign matching was looser than check_success;
  • a short manifest line crashed the check;
  • an exact location reported a superseded result as no_result;
  • CLI errors exited 1;
  • a string product was split into characters;
  • the shard key could be uneven.
• Witness W1b, read-only, on the DR1 laptop mirror (Workspace/Science/scratch/results_library_p2_w1b/):
  • consensus-100 manifest: 0 of 200 done, 226 unexpected (wrong tree, surfaced);
  • 113 mirror objects: 226 of 226 done including files/coolest.json;
  • a deliberate key-naming mismatch (13 _v5 objects) showed up as 13 missing paired with 13 unexpected.
• autofit_workspace guide scripts/features/results_library.py runs in test mode (separate PR).

Full API Changes (for automation & release notes)

Added

• autofit.results.library:
  • ResultsLibrary(expected, roots, index_dir=None, name="library", latest_key=...), with:
    • .grid(datasets, searches, roots, **kw)
    • .from_campaign(campaign, stages=None, products=None, roots=None, **kw)
    • .from_yaml(path)
    • .index() and .check(update=True)
  • ExpectedResult(dataset, search, products=(), location=None, exact=False, error=None)
  • LibraryException
  • Exported as af.ResultsLibrary and from autofit.results.
• autofit.results.completeness:
  • check(library, records)
  • CompletenessReport, with .results, .unexpected, .counts(), .product_counts(), .is_complete, .gaps(), .write_missing_csv(directory=".") and .summary()
  • ResultCheck
  • has_product(record, product, zip_only=False)
• autofit.results.enrich: enrich(search_outputs, fn, name, overwrite=False, shard=None), EnrichReport and member_name.
• autofit.results.shard: parse_shard, shard_of and in_shard.
• autofit.aggregator.Aggregator.enrich(fn, name, overwrite=False, shard=None).
• autofit.tools.util.write_zip_member(zip_path, arcname, data=None, file_path=None, compress_type=None).
• CLI subcommands check and enrich in python -m autofit.results.

Changed

• autofit.non_linear.paths.abstract._replace_zip_member now delegates to write_zip_member. Its behaviour is unchanged; the temporary file is now named <zip>.<pid>.tmp.
• The force_pickle_overwrite docstring now points to Aggregator.enrich. The flag itself is unchanged; removing it is a later step.

Migration

• None required.

Decisions flagged:

• The workspace guide lives at scripts/features/results_library.py.
• A library with several roots must name an index_dir when it is declared, not when it is checked.
• Running two enrich passes on the same results at once is unsupported and documented as such (no locking).

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

[Jammy2211]

Workspace PR: PyAutoLabs/autofit_workspace#180