Repository navigation
fix: validate adapt-image cache against dataset mask (stale cache IndexError) #516
Description
Overview
The 2026-07-21 census reported an IndexError: index 177 is out of bounds for axis 0 with size 177
in mapper_util.adaptive_pixel_signals_from, diagnosed as an off-by-one in multi-plane
(double-Einstein-ring) adapt-data indexing. That diagnosis is wrong. adaptive_pixel_signals_from
is correct and the bug is not multi-plane specific.
The real defect is in PyAutoGalaxy: galaxy_name_image_dict_via_result_from caches per-galaxy adapt
images to files/galaxy_images_snr.fits, keyed only by the search identifier (model + search).
The dataset mask is not part of that key, so when dataset shaping changes while the model stays
identical, a stale cache from a different mask is silently loaded — producing an adapt image of the
wrong length, which blows up much later as a raw IndexError deep in the inversion.
Plan
- Validate the cached mask against the expected dataset mask when loading the adapt-image cache; treat
a mismatch as a cache miss so the images are recomputed and the cache overwritten. - Correct the docstring that currently claims cache staleness is "structurally guarded" — that
guarantee only covers model/search changes, not dataset/mask changes. - In PyAutoArray, replace the bare
IndexErrorwith a clear exception naming the stale-cache cause, so
any future mask mismatch is legible at the point it occurs rather than 5 frames deep. - Add numpy-only unit tests on both sides.
- Clear the stale pre-docs: add nufftax + FINUFFT citation guidance #396 caches from the census
output/, confirm the double-Einstein-ring SLaM
script runs green, and correct the NEEDS_FIX marker with the real diagnosis.
Detailed implementation plan
Affected Repositories
- PyAutoGalaxy (primary)
- PyAutoArray (loud guard)
- autolens_workspace (census marker + stale output cleanup)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./PyAutoGalaxy | main | clean |
| ./PyAutoArray | main | clean |
| ./autolens_workspace | main | dirty (pre-existing regenerated dataset binaries, unrelated) |
Suggested branch: feature/adapt-image-cache-mask-validation
Evidence
Reproduction matrix — scripts/imaging/features/advanced/double_einstein_ring/slam.py,
PYAUTO_TEST_MODE=2:
| Data size | output/ state |
Result |
|---|---|---|
| full (100x100) | cleared | PASS (exit 0, masks equal: True, 2828/2828) |
PYAUTO_SMALL_DATASETS=1 (16x16) |
cleared | PASS (208/208) |
| either | census output/ left in place |
IndexError: index 177 out of bounds for size 177 |
The failing expression is the subscript adapt_data[slim_index_for_sub_slim_index], not the
xp.take — the traceback's ~~~~^^^^ marker pins it.
PyAutoArray 656be94b (#396, merged 2026-07-19, current main HEAD) changed the
PYAUTO_SMALL_DATASETS cap from 15x15 to 16x16. The census ran against an output/ holding pre-#396
caches: cached adapt image 15x15 = 177 pixels, current dataset mask 16x16 = 208 pixels.
Identical model => identical search identifier => stale cache silently loaded. Slim indices run to 207
against 177-length data, so the first out-of-range index is 177 — which only looks like an
off-by-one.
Implementation Steps
PyAutoGalaxy/autogalaxy/analysis/adapt_images/adapt_images.py_galaxy_image_dict_from_cache(cache_path, mask=None): after rebuilding the cachedMask2Dfrom
HDU 0, compare against the expectedmask; on mismatch returnNoneso the caller recomputes and
overwrites the cache.galaxy_name_image_dict_via_result_from: passmask=result.mask.Result.maskresolves to
analysis.dataset.mask(autogalaxy/analysis/result.py:65), which is cheap and does not
rebuild the max-log-likelihood fit — the caching win from autolens_profiling#70 is preserved.- Fix the staleness claim in the docstring (currently around line 127).
PyAutoArray/autoarray/inversion/mappers/abstract.pyMapper.pixel_signals_from: raise a clear exception when
len(adapt_data) != over_sampler.mask.pixels_in_mask, naming the stale adapt-image cache as the
likely cause. A loud crash, not a silent guard — no fallback, no clamping.
- Tests (numpy only — no JAX in library unit tests)
- PyAutoGalaxy: write a cache under mask A; loading with expected mask B is a miss, with mask A is a
hit. - PyAutoArray: adapt_data whose length disagrees with the mask raises the clear exception.
- PyAutoGalaxy: write a cache under mask A; loading with expected mask B is a miss, with mask A is a
- Verification
- Plant a deliberately stale cache and re-run the DSPL SLaM script; confirm it recovers (recomputes)
instead of crashing. - Re-run clean at full size and under
PYAUTO_SMALL_DATASETS=1.
- Plant a deliberately stale cache and re-run the DSPL SLaM script; confirm it recovers (recomputes)
- Census follow-up
- Clear pre-docs: add nufftax + FINUFFT citation guidance #396 adapt caches from
output/, re-run to confirm green, correct/remove the NEEDS_FIX
marker with the real diagnosis.
- Clear pre-docs: add nufftax + FINUFFT citation guidance #396 adapt caches from
Key Files
PyAutoGalaxy/autogalaxy/analysis/adapt_images/adapt_images.py— cache read/write + staleness docstringPyAutoGalaxy/autogalaxy/analysis/result.py—Result.mask, the cheap expected-mask accessorPyAutoArray/autoarray/inversion/mappers/abstract.py—pixel_signals_from, where the mismatch surfacesPyAutoArray/autoarray/inversion/mappers/mapper_util.py—adaptive_pixel_signals_from(correct, not changed)
Original Prompt
Click to expand starting prompt
The original census prompt diagnosed this as an off-by-one in PyAutoArray
mapper_util.adaptive_pixel_signals_from, targeting PyAutoArray +
autolens_workspace, and asked to "fix the indexing in mapper_util". That diagnosis was
disproved by reproduction (see Evidence above); the prompt file
PyAutoMind/active/double_einstein_ring_adaptive_pixel_signals_indexerror.md retains the original
report verbatim alongside the corrected analysis.
Activity
Jammy2211 commentedon Jul 22, 2026
Library PRs created
IndexErrorBoth labelled
pending-release.Verification
Reproduced the census failure deterministically by planting a genuine stale cache — the loose
files/copyand the archived copy inside each search's
.zip, sincerestore()re-extracts the zip and defeatspoisoning only the loose file. The planted caches are 15x15 / 177 pixels, matching the census exactly.
main:IndexError: index 177 is out of bounds for axis 0 with size 177— same frame, same numberssource_pix[2]with the likelihood actually evaluatedSuites: PyAutoGalaxy 1006 passed, PyAutoArray 927 passed.
Census follow-up
NEEDS_FIXmarker exists fordouble_einstein_ring— the parallel chat did not re-add one, so therewas nothing to remove.
autolens_workspace/output/.Spun out
PyAutoLabs/PyAutoFit#1414 —
Paths.preserve_in_ziponly adds a member absent from the zip and neverreplaces one, so a recomputed cache is written loose but the archived stale copy survives
restore(). Astale search therefore misses its cache on every run rather than once. Correctness is unaffected; only the
caching win is lost until the output is cleared. Recorded in the docstring.
Shipped behind Heart YELLOW
Score 52, no RED, acknowledged by the human. Reasons (none related to this change): workspace validation not
passing (10 failed, 2026-07-20); 58 stale parked scripts; 10 slow scripts; PyAutoCTI open PR 1390d old;
release validation stale (source moved since rehearsal).
Jammy2211 commentedon Jul 22, 2026
Shipped
Both PRs merged to
main:64790e7d(cache mask validation, primary fix)c37e478c(clear exception replacing the bareIndexError)All CI green on both (unittest 3.12 / 3.13, plus docs-build on PyAutoGalaxy).
No workspace change was required — the
double_einstein_ringscripts were never at fault, and noNEEDS_FIXmarker existed for them.Remaining follow-up: PyAutoLabs/PyAutoFit#1414 (
preserve_in_zipnever replaces an existing archivemember, so an invalidated cache is recomputed every run instead of being repaired on disk). Correctness is
unaffected by that; only the caching win is lost until the affected output is cleared.
Closing.