[Jammy2211]

Summary

PR 1a of results-library phase 1 (#1739, epic ledger draft/feature/autofit/results_library_epic.md). PyAutoFit can now list, classify and count every result under any number of output folders, with no hand tooling.

• ResultState joins the shared status vocabulary (non_linear/status/states.py): completed, running, interrupted_valid_ckpt, corrupt_ckpt, corrupt_zip, failed, superseded, never_started.

• Results index (autofit/results/), per ruling F2:

  • one JSON record per result is the truth;
  • a local SQLite cache (ResultsCache) is rebuilt from those records on demand;
  • indexing is incremental on a stat fingerprint, so unchanged results are never reopened;
  • the layout doesn't matter (flat, nested, custom roots);
  • --index-dir keeps read-only trees untouched.

• Classification: the status record is read first; otherwise the zip or folder decides (.completed plus a CRC check), and otherwise the checkpoint probe from phase 0. A running status record that hasn't been written for 10 minutes is not trusted.

• Unreadable is not corrupt: a transient I/O error is retried, then counted as unreadable and the previous record is kept. Witness W1 found this case (see below).

• superseded: within one (path_prefix, unique_tag, name), every completed result except the newest by completed_at.

• dataset_identity.json at fit start (ruling F10), through a new Analysis.dataset_identity() hook. It records:

  • object_id (default: unique_tag) and external ids;
  • each input with its size and SHA256;
  • an optional centre;
  • the versions of the libraries the fit ran with.

  A resumed fit whose inputs changed logs a warning. Config switches: output.dataset_identity and output.dataset_identity_sha256.

• Aggregator.latest_per(key, by="completed_at") and SearchOutput.completed_at: taken from the status record or the zip's .completed member, never the zip's mtime.

• CLI: python -m autofit.results {index,status,count}.

Decisions taken (please check)

1. latest_per default key is ("path_prefix", "unique_tag", "name"), not the ledger's ("unique_tag", "name"). In the Euclid DR1 layout every tile shares unique_tag="initial_lens_model" and the object is in path_prefix, so the ledger key would collapse all tiles into one group. The 3-field key handles both layouts.
2. never_started also covers an output folder with no checkpoint and no status record, since a rerun of it starts from scratch. Phase 2 adds "expected but absent".
3. Library versions go in dataset_identity.json, because fit outputs don't record them anywhere else today.

Test plan

• test_autofit/results: every state class; a CRC failure; a moved result; nested layouts; incremental re-index (only new results opened); removal; superseded and restore; read-only --index-dir; SQLite cache; CLI; I/O error counted as unreadable; identity from real Drawer fits (sha256, default, off switch, missing input); latest_per on real fits.
• Scale (synthetic 15k): first index 37 s, re-index after 10 new 6.7 s (10 opened), load 15,010 records 1.2 s.
• Full PyAutoFit suite (see comment).
• Witness W1 (read-only, DR1 laptop mirror): 338 results, 325 completed, 13 superseded, 0 corrupt, 0 location mismatches. Report: Workspace/Science/scratch/results_library_p1_w1/README.md. The mirror is partial; the full census is the phase-5 witness.

Follow-up (not here): a PyAutoArray/PyAutoLens dataset_identity() that fills real input paths and the cutout centre.

Refs #1739

🤖 Generated with Claude Code

[Jammy2211]

Full PyAutoFit suite (PyAuto313, local) on 1a and 1b: 4,010 / 4,015 passed, the one failure was the 600-line decomposition guard on abstract_search.py (fixed in b2630e5; guard + results tests 41 passed after). The conformance file set now lists files/dataset_identity.json (1934430). Smoke on the 1b head: autofit, autogalaxy, autolens, autolens_test, euclid, howtolens — 167 PASS, 0 FAIL (autocti legs not runnable on this laptop: GSL headers missing, unrelated). Heart STALE (release validation only) acknowledged by the human for this ship.