Repository navigation
Description
Overview
Workspace scripts auto-simulate their dataset with an 8-line should_simulate + import subprocess, sys + subprocess.run(...) block. This task adds a one-line helper, al.util.dataset.simulate_if_needed(dataset_path, simulator), to PyAutoArray (phase 1). It also teaches the PyAutoHands dataset-allowlist guard to recognise the helper (phase 2), so that scripts swept onto it stay covered. The workspace sweep (phase 3) waits for the library release.
Plan
- Phase 1 — PyAutoArray: add
simulate_if_needednext toshould_simulate. It runs the simulator in a subprocess (not runpy), so simulator JAX/matplotlib state cannot leak into the caller. Point theshould_simulatedocstring example at it, and test it withsubprocess.runmocked. - Phase 2 — PyAutoHands:
check_dataset_allowlistleg 2 findssimulate_if_neededcall sites as well asshould_simulate, with tests. - Phase 3 (after release, not started): one sweep PR per workspace repo, replacing the 8-line block with the one-liner.
Tier: glance — merge mode: human /prm for phases 1–2. The Witness check only applies after the phase 3 sweep, so the phase 1–2 PRs are not auto-merged.
Detailed implementation plan
Work Classification: Library (phases 1–2); Workspace (phase 3, deferred)
Worktree root: none. This is a cloud session; work is on the branches below.
Affected Repositories
- PyAutoArray (primary)
- PyAutoHands
- Phase 3 (deferred): autolens/autogalaxy/autofit workspaces, *_workspace_test, HowToLens, HowToGalaxy, assistants
Branch Survey
| Repository | Branch | Dirty? |
|---|---|---|
| PyAutoArray | claude/pyautoarray-simulate-helper-yuqd16 | clean, pushed |
| PyAutoHands | claude/pyautoarray-simulate-helper-yuqd16 | clean, pushed |
Implementation Steps
autoarray/util/dataset_util.py:simulate_if_needed(dataset_path, simulator)callsshould_simulate(str(dataset_path)). If that returns True, it callssubprocess.run([sys.executable, str(simulator)], check=True). It returns whether the simulator ran. Its docstring explains why it uses a subprocess rather than runpy. Theshould_simulatedocstring's usage example now points at the helper.test_autoarray/util/test_dataset_util.py: tests mocksubprocess.runand cover the True and False branches with str and Path arguments, plus a missing-dataset end-to-end case.autohands/check_dataset_allowlist.py: addsDELETING_CALLS = ("should_simulate", "simulate_if_needed").should_simulate_sitesmatches both names and takes the first positional argument or thedataset_path=keyword. The guard's messages and docstrings mention both functions.tests/test_dataset_allowlist_capped_deletion.py: tests check that the old block and the one-liner resolve to the same line and path, that the keyword form resolves, that a call with no path is ignored, and that a swept script reaching tracked files still fails the guard.- Phase 3 (after release): mechanical sweep, one PR per workspace repo, regenerating notebooks. Non-standard blocks are left alone and listed in each PR.
Key Files
PyAutoArray/autoarray/util/dataset_util.py: helper and docstringPyAutoHands/autohands/check_dataset_allowlist.py: leg-2 call-site scan
Notes
aa.util.datasetis thedataset_utilmodule, so the helper is reachable there automatically.al./ag.util.datasetare assumed to re-export the same module, since the workspaces already callal.util.dataset.should_simulate; this was not checked in-session.- PyAutoHands test suite: 559 passed, 10 failed. The same 10 failures (notebook-generation tests) occur on
mainwithout this change.
Original Prompt
Click to expand starting prompt
One-line auto-simulate helper: al.util.dataset.simulate_if_needed
Type: feature
Target: PyAutoArray
Repos:
- PyAutoArray
- PyAutoFit
- PyAutoGalaxy
- PyAutoLens
- HowToGalaxy
- HowToLens
- PyAutoHands
- workspaces
Difficulty: too-large
Autonomy: supervised
Priority: normal
Memory: wiki/lensing/sources/dark-matter-substructure.md; reading-queue.md; wiki/galaxies/sources/massive-ellipticals.md
Status: formalised
Consequence: glance
Witness: after the sweep,grep -rn "subprocess.run" --include=*.pynext toshould_simulatereturns zero hits in every workspace, andcheck_dataset_allowlistreports the same number of guarded call sites before and after.
Review-minutes: 3
Unattended: needs-slicing
Phases: (1) PyAutoArray helper + docstring + test; (2) PyAutoHands guard recognises simulate_if_needed; (3) after release, one sweep PR per workspace repo.
Workspace scripts auto-simulate their dataset with an 8-line block that exposes
subprocess/sys to users who only need to know "simulate if missing":
if al.util.dataset.should_simulate(str(dataset_path)):
import subprocess
import sys
subprocess.run(
[sys.executable, "scripts/simulator/lens_x2.py"],
check=True,
)Replace it with one line:
al.util.dataset.simulate_if_needed(dataset_path, simulator="scripts/simulator/lens_x2.py")Library (PyAutoArray, first)
Add to autoarray/util/dataset_util.py, beside should_simulate:
def simulate_if_needed(dataset_path, simulator):
if not should_simulate(str(dataset_path)):
return
import subprocess
import sys
subprocess.run([sys.executable, str(simulator)], check=True)- Keep subprocess isolation (do NOT use runpy — simulator JAX/matplotlib state must not leak into the caller).
- Accept str or Path for both args.
- Update the
should_simulatedocstring's usage example to point at the helper. - Unit test: mock subprocess.run; called when should_simulate is True, not called when False.
- Confirm it is reachable as
al.util.dataset.simulate_if_needed/ag./aa.(check howutil.datasetis re-exported).
PyAutoHands guard
autohands/check_dataset_allowlist.py should_simulate_sites (~line 238) AST-scans
for should_simulate(...) call sites to prove a capped run cannot rmtree committed
files. Extend it to also match simulate_if_needed (first arg = dataset path),
with a test, so swept scripts stay covered by the guard.
Workspace sweep (after the library release)
Replace the pattern across every workspace (autolens/autogalaxy/autofit workspaces,
*_workspace_test, HowToLens/HowToGalaxy, assistants), one PR per repo. Mechanical
rewrite; keep the dataset_path and simulator path exactly as in each script.
Regenerate notebooks where the repo generates them. Smoke-test a sample per repo.
Watch for variants (extra argv, different indentation, aa./ag. prefixes) and
leave any non-standard block untouched, listing it in the PR.
Activity
Jammy2211 commentedon Nov 1, 2026
Shipped (phases 1–2)
aa.util.dataset.simulate_if_needed(dataset_path, simulator)next toshould_simulate. It runs the simulator in a subprocess (sys.executable,check=True, not runpy) only whenshould_simulatereturns True, and returns whether the simulator ran. Theshould_simulatedocstring example points at it. Tests mocksubprocess.run. Labelledpending-release.check_dataset_allowlistleg 2 now matchessimulate_if_neededas well asshould_simulate, taking the first positional argument or thedataset_path=keyword, so swept scripts stay guarded. A follow-up commit removed a repo name from a code comment, which had tripped the tenant-firewall check.Not shipped: phase 3, the workspace sweep (one PR per workspace repo replacing the 8-line block). It waits for the PyAutoArray release that carries #670, and is re-filed in PyAutoMind as its own draft prompt.
Generated by Claude Code