Skip to content

feat: read zipped results in place — ZipMember, from_zip, value(kind=), Aggregator.from_index (results-library p1b) - #1744

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/results-library-p1b-lazy-reads
Oct 30, 2026
Merged

Jammy2211 merged 1 commit into
mainfrom
feature/results-library-p1b-lazy-reads

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

PR 1b of results-library phase 1 (#1739). Stacked on #1743; merge that first.

Zipped results are now read in place, with nothing extracted to disk:

  • ZipMember (aggregator/zip_member.py): a pathlib-like path into a zip. Unlike zipfile.Path it holds no open file: each open reopens the zip, and zipfile keeps the stream valid after the ZipFile closes. Aggregating thousands of zips therefore never hits the open-file limit. Member names are cached per (size, mtime).
  • FileOutput / SearchOutput read through path.open(), so SearchOutput.from_zip(zip_path) works unchanged; FITS members load from memory.
  • SearchOutput.open_member(path) is public.
  • value(name, kind=...) (report P15) picks between tracer.json and tracer.fits. A bare ambiguous name keeps today's order (json, pickle, csv, fits) and warns once, so existing scripts behave the same.
  • Aggregator.from_index(roots, index_dir=None, states=("completed",)) aggregates over the results index and leaves superseded results out by default.

Test plan

  • ZipMember path semantics; a zipped fixture reads the same as the folder fixture (id, model, search, children, json/fits values); open_member; value(kind=) and the ambiguity warning; from_index on real fits with extractall/extract patched to raise and the tree byte-identical afterwards.
  • Full PyAutoFit suite (see comment).

Refs #1739

🤖 Generated with Claude Code

…, value(kind=), Aggregator.from_index (results-library p1, PR 1b)

- ZipMember: a pathlib-like path into a result zip that holds no open file
  (each open reopens the zip), so thousands of zipped outputs never hit the
  open-file limit; member names cached per (size, mtime).
- FileOutput / SearchOutput read through path.open(), so a SearchOutput can
  be backed by a zip (SearchOutput.from_zip) with nothing extracted; FITS
  members load from memory.
- SearchOutput.open_member(path) public; value(name, kind=...) chooses
  between files sharing a name (tracer.json vs tracer.fits) and warns once
  when a bare name is ambiguous (order unchanged: json, pickle, csv, fits).
- Aggregator.from_index(roots, index_dir, states=('completed',)) aggregates
  over the results index, superseded results left out by default.

Refs #1739

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jammy2211

Copy link
Copy Markdown
Collaborator Author

Full PyAutoFit suite (PyAuto313, local) on 1a and 1b: 4,010 / 4,015 passed, the one failure was the 600-line decomposition guard on abstract_search.py (fixed in b2630e5; guard + results tests 41 passed after). The conformance file set now lists files/dataset_identity.json (1934430). Smoke on the 1b head: autofit, autogalaxy, autolens, autolens_test, euclid, howtolens — 167 PASS, 0 FAIL (autocti legs not runnable on this laptop: GSL headers missing, unrelated). Heart STALE (release validation only) acknowledged by the human for this ship.

Base automatically changed from feature/results-library-p1-index to main October 30, 2026 15:41
@Jammy2211
Jammy2211 merged commit 2404765 into main Oct 30, 2026
5 checks passed
@Jammy2211
Jammy2211 deleted the feature/results-library-p1b-lazy-reads branch October 30, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant