Affected Repositories

• PyAutoFit (primary)

Branch Survey

Suggested branch: feature/results-library-p1-index (two stacked PRs, like phase 0)

Implementation Steps

PR 1a — index + identity

1. autofit/non_linear/status/states.py: ResultState(str, Enum) = completed, running, interrupted_valid_ckpt, corrupt_ckpt, corrupt_zip, failed, superseded, never_started (never_started needs the phase-2 grid; defined now).
2. autofit/non_linear/status/identity.py: write_dataset_identity(search, analysis) writes files/dataset_identity.json atomically (open_atomic) with object_id (hook value, else unique_tag), external_ids, inputs {name: {path, bytes, sha256}}, optional centre, and a schema stamp. Called from start_resume_fit beside status_notify(self, "start") (abstract_search.py:407). Best effort: a failure only logs. FitStatus._outputs (record.py:351) already looks for it at this path.
3. Analysis.dataset_identity() returns None by default (analysis.py). Config output.dataset_identity: true, output.dataset_identity_sha256: true in config/general.yaml.
4. autofit/results/record.py: ResultRecord dataclass (zip/dir path, size, mtime, path_prefix, unique_tag, name, identifier, state, completed_at from .completed/status.json, products present, library versions, identity summary, status summary, schema).
5. autofit/results/classify.py: reads .status.json first; otherwise from disk: zip with .completed and a clean testzip → completed; BadZipFile → corrupt_zip; directory without .completed → phase-0 checkpoint probe → interrupted_valid_ckpt / corrupt_ckpt; status failed → failed.
6. autofit/results/index.py: ResultsIndex(roots, index_dir=None); walks with os.scandir and recognises search outputs flat or nested; records go to <index_dir>/records/<sha1(abs path)>.json; incremental on (path, size, mtime); prunes deleted records; superseded marked after latest_per.
7. autofit/results/cache.py: SQLite cache in ~/.cache/autofit/results/<hash>.sqlite (or a temp dir), rebuilt from the JSON records; never truth.
8. Aggregator.latest_per(key=("unique_tag","name"), by="completed_at").
9. autofit/results/__main__.py: index <root>… [--index-dir], status (one line per state class), count [--state].

PR 1b — lazy reads
10. ZipSearchOutput built on zipfile.Path; FileOutput subclasses read through path.open(); FITS from bytes.
11. SearchOutput.open_member(path) public; value(name, kind=None) keeps the current order (JSON first) for back-compat, warns once when a name is ambiguous; kind="fits"|"json"|"pickle"|"csv" selects explicitly (report P15).
12. Aggregator.from_index(root, index_dir=None, states=("completed",)) builds zip-backed outputs with zero extraction.

Tests

• 15k synthetic results: index timing, then re-index after 10 new results in seconds
• every state class, including a corrupt zip and a poisoned checkpoint
• from_index aggregation extracts zero bytes
• identity sha256 + off switch
• latest_per ties
• CLI
• autofit + autolens_test smokes

Witness W1

python -m autofit.results index Workspace/Science/active/euclid_dr1/output* --index-dir <scratch> + status. The laptop mirror is partial (~226 zips under output/dr1_sep1); the report says so. Table goes in Workspace/Science/scratch/results_library_p1_w1/. Read-only.

Follow-up (named, not here)

PyAutoArray/PyAutoLens dataset_identity() that fills real input paths and the cutout centre — additive, a separate small PR.

Key Files

• autofit/non_linear/status/states.py, identity.py (new)
• autofit/non_linear/search/abstract_search.py, autofit/non_linear/analysis/analysis.py
• autofit/results/ (new), autofit/aggregator/aggregator.py, search_output.py, file_output.py
• autofit/config/general.yaml

Epic: results-library
Phase: 1
Blocked-by: phase 0 (COMPLETE 2026-10-29)
Concurrent-with: hpc-campaign phase 2 (ruling F11 — two PyAutoFit worktrees; this task owns autofit/results/, aggregator/, non_linear/status/identity.py, the ResultState enum in states.py, the identity hook in abstract_search.py/analysis.py, config/general.yaml identity keys)
Consequence: judge
Autonomy: human-required
Witness: W1 — python -m autofit.results index + status over the laptop DR1 mirror (Workspace/Science/active/euclid_dr1/output*, partial: say so), index written OUTSIDE the tree, state-class table in Workspace/Science/scratch/results_library_p1_w1/. Read-only; DR1 tree never modified.

Results library phase 1 — results index, state classes, identity, lazy reads, CLI

Original request (human, 2026-10-30): "Ctoninue the PyAutoLEns II HPC runs and Results"

Ledger: draft/feature/autofit/results_library_epic.md §Phase 1 (rulings F2 index store, F10 identity).
Lane prompt: Workspace/Science/scratch/pyautolens_ii_campaign_results_scoping_2026-10-28/PROMPT_results_library.md §Phase 1.
Evidence: results_library_epic_report.md (P3, P4, P5, P6, P13, P15, P16, P22, P25; gaps 1–5, 17).

@PyAutoFit scope:

• ResultState (completed, running, interrupted_valid_ckpt, corrupt_ckpt, corrupt_zip, failed,
  superseded, never_started) added to the shared vocabulary autofit/non_linear/status/states.py.
• autofit/results/: per-result JSON records (truth) under <root>/.autofit_index/ or --index-dir,
  local SQLite cache rebuilt on demand (never truth, never on NFS), incremental on (zip path, size, mtime);
  reads status.json first, classifies from disk otherwise; layout-agnostic, many roots.
• dataset_identity.json at fit start (files/), Analysis.dataset_identity() hook, sha256 default on
  with a config off switch (F10).
• Aggregator.latest_per(key=("unique_tag","name"), by="completed_at").
• Lazy zip reads: SearchOutput.open_member, value(name, kind=...), Aggregator.from_index(root).
• CLI python -m autofit.results {index,status,count}.

Done: 15k synthetic fixture indexes in minutes, re-index after 10 new results in seconds; aggregate over
the index with zero bytes extracted; W1 witness.