Affected Repositories

• PyAutoArray (primary)

Branch Survey

Suggested branch: feature/interferometer-sparse-data-vector

Implementation Steps

1. autoarray/inversion/inversion/interferometer/sparse.py — rewrite data_vector: resolve dirty_image as today (slim, masked-pixel indexed); for each linear object in self.linear_obj_list, if it is a Mapper build triplets with mapper_util.sparse_triplets_from(..., return_rows_slim=True, xp=self._xp) (slim rows index the slim dirty image; the existing _sparse_triplets_curvature_from uses extent-flat rows and is NOT the right indexing for this product — add a sibling helper _sparse_triplets_slim_from or a rows_slim flag) and compute the block as np.bincount(cols, weights=vals * dirty_image[rows], minlength=mapper.params) on NumPy and jnp.zeros(S).at[cols].add(vals * dirty_image[rows]) (or jax.ops.segment_sum with num_segments=S) on JAX; if it is an AbstractLinearObjFuncList keep xp.dot(linear_obj.mapping_matrix.T, dirty_image). Concatenate with xp.concatenate in the order of self.linear_obj_list so the result matches param_range_list_from(cls=LinearObj).
2. Confirm sparse_triplets_from includes the same sub_fraction normalisation as mapper_util.mapping_matrix_from (the curvature parity tests already rely on this); if the mapper has pix_sizes_for_sub_slim_index padding (invalid stencil slots with zero weight), the helper's masking must drop or zero them — verify, do not assume.
3. Audit per-call mapping_matrix use: abstract.py mapping_matrix cached property (now only reached from visualisation / mapped_reconstructed_data_dict), interferometer_numba/sparse.py (no touch per grep), _curvature_matrix_func_list_and_mapper (func-list matrices only — fine). Record findings in the PR body.
4. Tests in test_autoarray/inversion/inversion/interferometer/test_interferometer.py (and the numba subclass tests under interferometer_numba/): sparse data_vector equals mapping_matrix.T @ dirty_image to 1e-12 for (a) a single over-sampled Delaunay/rectangular mapper, (b) mapper + linear func list, (c) xp=jnp (importorskip jax), (d) a DatasetInterface whose sparse_dirty_image differs from the operator's. Keep the log_evidence parity tests between InversionInterferometerSparse and InversionInterferometerMapping green.
5. Witness measurement with the session scratchpad copy of the Aris evidence (aris_evidence/step8/bench.py --mesh step1 --mode numba style run against dataset_sparse_512.pkl, 5,746 vertices, 1 thread): record peak RSS before (7.17 GB) and after (target ≤ ~3.5 GB), steady s/call, and that the log likelihood at the same parameter vector is unchanged to 1e-8. Also record JAX jit peak RSS / XLA compile temp at 1,459 vertices (3.69 GB / 2.17 GB before) and attempt the 5,746-vertex JAX run that was previously infeasible (8.95 GB compile temp). Put the numbers in the PR body.

Key Files

• autoarray/inversion/inversion/interferometer/sparse.py — data_vector (line ~104), _sparse_triplets_curvature_from
• autoarray/inversion/mappers/mapper_util.py — sparse_triplets_from (return_rows_slim)
• autoarray/inversion/inversion/abstract.py — mapping_matrix hstack (line ~270)
• autoarray/inversion/inversion/interferometer_numba/sparse.py — numba subclass (audit only)
• test_autoarray/inversion/inversion/interferometer/test_interferometer.py — parity tests

Evidence

Prompt measurements (PROGRESS4 step8.4a): dense mapping matrix 1.07 GB @1459, 1.9 GB @2577, 4.2 GB @5746 vertices (+ hstack copy); route-independent. Cached dataset and bench script copied to this session's scratchpad aris_evidence/step8/ (temporary).

Interferometer sparse inversion: compute data vector without the dense mapping matrix

Type: feature
Target: PyAutoArray
Repos:

• PyAutoArray
  Difficulty: medium
  Autonomy: safe
  Priority: normal
  Memory: wiki/lensing/sources/dark-matter-substructure.md; reading-queue.md; wiki/galaxies/sources/cosmos-survey.md
  Status: formalised
  Consequence: glance
  Witness: on the Aris 512x512 setup (91,507 masked pixels, 5,746 Delaunay vertices, use_jax=False) peak RSS of one likelihood call drops from 7.2 GB to <= ~3.5 GB with log likelihood unchanged to 1e-8
  Review-minutes: 3
  Unattended: ready

autoarray/inversion/inversion/interferometer/sparse.py:104 computes
data_vector = xp.dot(self.mapping_matrix.T, dirty_image), which materialises the dense
(masked pixels x source pixels) mapping matrix via hstack — 4.2 GB at 91,507 x 5,746. The step only
needs the sparse mapping operator (the same triplets the curvature routes already use). A memory
trace on the 512x512 mask attributes all RSS growth from ~3 GB to 7.2 GB to this line, on both the
numba direct_conv and NumPy FFT routes. It probably also explains most of the ~9 GB XLA temp
memory of the JAX path at that size.

Do: compute the data vector from the sparse triplets (scatter/segment-sum: D_j = sum over nonzeros
in column j of val * dirty_image[row]) on both the NumPy and JAX paths, so no dense mapping matrix
is built in the likelihood. Check other places in the interferometer sparse inversion (and
the numba subclass) that touch mapping_matrix per call. Keep outputs identical; add a test.
Measure peak RSS and time before/after on the 512x512 cache (and JAX compile memory).

Evidence: /tmp/claude-1000/-home-jammy-Code-PyAutoLabs/50c8c6cd-6ac2-47c2-9290-7ee73f9c4315/scratchpad/aris/PROGRESS4.md and step8/ (dataset_sparse_512.pkl cache, bench.py). PyAutoArray ccddfba.