Repository navigation
campaign: interferometer large-mask likelihood — memory and routing (Aris 512×512) #62
Description
Interferometer large-mask likelihood — memory and routing (Aris 512×512)
Campaign: interferometer
Task: interferometer_large_mask_aris
Reviewed: 2026-10-15
Status: active
Filed 2026-10-15 from the Aris interferometer profiling session (Delaunay + AdaptSplit,
use_jax=False numba route and JAX jit, real-space masks 125×125 at 5,496 masked pixels and
512×512 at 91,507 masked pixels, 5.5M visibilities, 682×682 W~ extent). The human asked for the
three resulting PyAutoArray tasks to be logged and monitored as one Pulse speed-up campaign.
Mind owns the repository claims and PR lifecycle (issues below); this task records the
campaign intent, the baseline numbers the tasks must beat, and each check-in.
Members (Mind lifecycle, PyAutoArray; one claim at a time, in this order)
| # | Issue | Pass condition (Witness) | Mind state 2026-10-15 |
|---|---|---|---|
| 1 | PyAutoArray#626 sparse data vector | 512×512, 5,746 vertices, numba route: peak RSS 7.2 GB → ≤ ~3.5 GB, logL unchanged to 1e-8 | active, --auto, Heart RED development override |
| 2 | PyAutoArray#627 AdaptSplit log-det parity | numba vs JAX log_det_regularization_matrix_term agree < 1e-3 (now 2443.22 vs 2561.98), regression test |
queued behind #626 (planned.md) |
| 3 | PyAutoArray#628 numba route gate cost model | both masks pick the faster of direct_conv vs FFT at every tested vertex count | queued behind #627; DelaunayNN #624/#625 merged so no factory conflict |
Baseline measurements (s/call, 1 thread, laptop; PROGRESS4 steps 8.3–8.4)
512×512 mask, 91,507 masked pixels, regular-step Delaunay meshes:
| vertices (nnz/col) | NumPy FFT route | numba direct_conv | JAX jit (cores) | peak RSS |
|---|---|---|---|---|
| 1,459 (188) | 28.3 | 38.4 | 18.3 (1.59) | 4.13 GB FFT / 3.22 DC / 3.69 JAX |
| 2,577 (106) | 44.1 | 34.2 | 32.2 (1.75) | 4.34 / 3.41 / 5.02 |
| 5,746 (48) | 97.3 | 41.2 | not run (8.95 GB XLA compile temp) | 7.14 / 7.17 / — |
- Default gate 60 nnz/col mis-routes 2,577 vertices to FFT (1.29× slower); measured crossover ~140–150 nnz/col on this mask vs ~60 (Delaunay) / ~77 (rectangular) on 125×125 (autolens_profiling#226).
- Memory culprit:
sparse.py:104data_vector = mapping_matrix.T @ dirty_imagematerialises the dense (91,507 × S) matrix: 1.07 / 1.9 / 4.2 GB at 1,459 / 2,577 / 5,746 vertices, route-independent. - Backend parity: chi², curvature, data vector, curvature log-det identical to 1e-16 relative; only the AdaptSplit regularization log-det differs (125×125: 2443.22 numba vs 2561.98 JAX; 512×512 @1,459: 2576.59 vs 2574.73). ~1 in 4 draws non-PD (numba LinAlgError vs JAX -inf).
These are laptop timings under load and qualify no baseline; the campaign's after-measurements
re-run the same cached setups and are recorded here per task. Caches live in a session
scratchpad (temporary); the numbers above are the durable record.
Check-in 2026-10-15
- Filed: #626 active (worktree
interferometer-sparse-data-vector), #627 and #628 queued in Mind planned.md. Implementation worker running on #626; read-only root-cause investigation running for #627. - Next: after #626 PR-open and human
/prm, start #627 (/start_library), then #628. Record each task's before/after witness table here on close-out.
Activity
Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15 (evening): #627 root cause found before the claim freed — the backends build different AdaptSplit
Hbecauseadaptive_pixel_signals_fromcounts zero-weight mapping entries and scipy vs the JAX walk pick different triangles for on-vertex points; log-det factorizations agree to 2e-8. Fix locus moved to mapper_util; details on PyAutoArray#627. #626 implementation worker still running.Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15 (late): PyAutoArray#626 shipped to PR-open — PyAutoLabs/PyAutoArray#630 (decision-taken, Heart RED dev override; human /prm). Witness: numba 5,746 vtx peak RSS 7.17 → 3.14 GB; JAX jit 1,459 vtx 3.65 → 2.60 GB; XLA compile temp at 5,746 vtx 8.95 → 1.86 GB, JAX 5,746 now runs at 3.24 GB. Timings not claimed (machine load 2–14). #627 patch drafted and verified (diff on the issue), queued; #628 queued.
Jammy2211 commentedon Oct 15, 2026
Shipped (ledger rows)
#63 (merge
24749975) added the ledger taskinterferometer_large_mask_aristo the interferometer campaign and re-rendered the board. That change was ledger only. The library change it tracks, PyAutoLabs/PyAutoArray#630, has merged and is waiting for a release.This issue stays open as the campaign anchor for check-ins. GitHub auto-closed it on merge because the PR body contains the phrase "Closes #62" (in the sentence saying the PR does not close it), so it has been reopened.
Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15 (night): PyAutoArray#630 MERGED (7cc71238; pending release) — #626 closed, record organs/PyAutoMind/complete/2026/10/interferometer-sparse-data-vector.md. Ledger PR #63 MERGED (task row
interferometer_large_mask_arislive on the board; this issue stays open as the anchor). #627 started: worktree adaptsplit-logdet-backend-parity, patch applying, ship to PR-open in progress. #628 next.Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15 (late night): PyAutoArray#631 MERGED (0377eeaa; pending release) — #627 fixed Delaunay-only by the human's scoping decision (KNN/KNNBarycentric/rectangular unchanged). Witness: synthetic ill-conditioned AdaptSplit test — H parity 3.6e-15, numba/JAX log-dets 2.1e-9 apart, both within 1.7e-9 of a 50-digit reference; the Aris 125×125/512×512 pairs were NOT re-run on the branch (cache deleted for disk) — owed as a campaign after-measurement. Out of scope, unfiled: absolute 1e-8 jitter makes log-det H round-off-dominated at max(coeff)^4 ≳ 1e6. Next: #628 numba route-gate cost model takes the PyAutoArray claim.
Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15 (night): PyAutoArray#628 at PR-open — PyAutoLabs/PyAutoArray#632 (decision-taken, Heart RED dev override; human /prm). Fixed gate 60 nnz/col → two-constant cost model: t_fft = 7.62e-10·S·(pocketfft work on the padded (2ny,2nx) extent), t_dc = 1.77e-09·nnz·ny·nx, least-squares on 9 paired timings (6 bake-off cells from autolens_profiling#226 + 3 Aris 512 points); correct route at all 9, residual ratios 0.68–1.47. Live routing check on both Aris masks: 512 → FFT/numba/numba at 1,459/2,577/5,746 vtx (old gate mis-routed 2,577); 125 → numba at 1,405/2,947/5,526. Note: mask extents are 341² (512) and 83² (125); 682 was the padded grid. Phase 2 (autolens_profiling re-measurement record) follows merge. Timings not re-run.
Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15 (night): PyAutoArray#632 MERGED (17caf228; pending release) — all three campaign members now merged: #630 (sparse data vector), #631 (AdaptSplit log-det parity, Delaunay-only), #632 (route-gate cost model). Owed to this campaign, not yet filed as Pulse/Mind tasks: (1) Aris after-measurement of the log-det pair on both masks; (2) #628 phase 2 — autolens_profiling re-measurement record of the routing table under the default cost model (Mind prompt being filed at close-out); (3) copy the three before/after witness tables into tasks/interferometer_large_mask_aris.md via a Pulse PR.
Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15: the before/after witness tables for all three merged members (PyAutoArray#630, #631, #632) are recorded in #64. This issue stays open until the autolens_profiling phase-2 record lands.
Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15 (late): CPU speed-up research on the Aris 125×125 data at main 17caf228 (1 thread, use_positive_only_solver=False): full call 0.63 s @1,405 vtx / 2.0–2.4 s @2,947. Breakdown: direct_conv F 53% / 18%; np.linalg.solve(F+H) 18% / 40%; cholesky(F+H) for the log det 13% / 31%; data vector now 1 ms (#630). The two dense factorizations of the same SPD matrix are 31% / 72% of the call → Mind draft filed: draft/feature/autoarray/reuse_cholesky_factor_for_solve_and_log_det.md (one cho_factor; −20% / −40–45% per call, results to 1e-11). Also filed draft/feature/autoarray/direct_conv_symmetric_row_restriction.md (512×512 only; 2–2.5× on F, unconfirmed under load; needs #632 model refit). Biggest fixed-lens win remains the existing task interferometer_fixed_mapper_curvature_preload (with L1: ~6× @1,405, ~30× @512). Settings advice applied to the collaborator's runner: n_batch = 4·number_of_cores. Not worth it: float32 factor (0.027 nats error), numba threads inside 28 processes, Delaunay/H/positions (<3% each). Pulse PR #64 (witness tables) at PR-open; phase-2 profiling record in progress.
Jammy2211 commentedon Oct 15, 2026
Shipped (witness tables, #64): the three witness tables landed in
tasks/interferometer_large_mask_aris.md, the campaigns.yaml next-lines were refreshed and the board re-rendered (merge fc092a9). This issue stays open as the campaign anchor; the autolens_profiling phase-2 record (in flight) adds its table to the same task file via a later Pulse PR.Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15 (night): phase-2 record at PR-open — PyAutoLabs/autolens_profiling#420 (issue #419; decision-taken: timings carried, not re-timed; human /prm). Aris after-measurement DONE on PyAutoArray 17caf228, default settings, one fixed vector: numba vs JAX log_det_regularization_matrix_term agree to ≤4.4e-8 in all six cells (125: 1,405/2,947/5,526 vtx; 512: 1,459/2,577/5,746); the 125 original-vector pair is 2468.4463180 on both (was 2443.22 vs 2561.98). Routing: 6/6 cells pick the faster measured arm (512/1,459 → FFT; all others → direct_conv); the 512 pairs are in-sample for the cost model and the 125 FFT arm is a JAX-jit proxy — stated in the record. Follow-up (low): the decision-matrix note may still describe gate 60 as current. Next: copy this table into tasks/interferometer_large_mask_aris.md after #420 merges; L1 (Cholesky reuse) in development.
Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-15 (late): knowledge section (cost structure, routing model, ranked levers, collaborator settings) and the two human decision records (AdaptSplit Delaunay-only; route-gate cost model) at PR-open — #65 (ledger only; human /prm). This issue stays open as the anchor.
Jammy2211 commentedon Oct 15, 2026
Shipped (knowledge write-up + two decision records, #65): Knowledge section in tasks/interferometer_large_mask_aris.md; decisions/adaptsplit-zero-weight-delaunay-only.md and decisions/interferometer-route-gate-cost-model.md published and indexed in decisions.yaml; campaigns.yaml next-lines refreshed; board re-rendered. Campaign stays open — next: re-pin the six-cell table now that autolens_profiling #420 merged, at the next Pulse check-in.
Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-16 (early): L1 at PR-open — PyAutoLabs/PyAutoArray#634 (issue #633; Heart RED dev override; human /prm). One cached cho_factor of F+H serves the solve and the log det (NumPy, positive-only off, cholesky method; old path otherwise). Witness (laptop, 1 thread): solve+log-det 144.5 → 50.9 ms @1,405 vtx, 1,268 → 373 ms @2,947; full call −22–24% @1,405 (repeat/profiler; a first noisy run gave −8%), −51% @2,947. Log det bit-identical; logL within 1.1e-11 rel. Reconstruction differs 1.6e-3 rel @2,947 because cond(F+H)=7.7e14 there — the new solve is CLOSER to a refined reference (err 5.1e-4 vs 1.5e-3 for the old LU); witness clause to be reworded at /prm. Gate: PyAutoArray 2065 passed, Galaxy 1357, Lens 832; smoke 165/165; review CLEAN. Phase-2 #420 and Pulse #65 merged and closed out; shadow window 19/20.
Jammy2211 commentedon Oct 15, 2026
Check-in 2026-10-16: L1 MERGED — PyAutoArray#634 (e59d5fa9; pending release). The numba/NumPy path Aris runs now factorises F+H once: solve+log-det 144.5 → 50.9 ms @1,405 vtx, 1,268 → 373 ms @2,947; full call −22–24% / −51%. Witness accepted as reworded (reconstruction within conditioning at cond 7.7e14; log det bit-identical; logL ≤1.1e-11). Remaining levers on this campaign: L2 direct_conv symmetry (draft, unstarted, 512×512 only), fixed-mapper curvature preload (existing task; the big win for source_pix[1]). Four library PRs from this campaign are pending release: #630, #631, #632, #634.
Jammy2211 commentedon Oct 16, 2026
2026-10-16 check-in: L2 direct_conv symmetry kernel, quiet-core measurement (RAL 411833)
The 512-scale synthetic Aris setup: 341² extent, 91,361 masked pixels, nnz 274,083, 3-NN weights, symmetric W~ preload. Single-threaded numba on
euclid-ral-compute-1(ralpartition, 8 CPUs reserved, load ≈ 1). Library A is PyAutoArray maine59d5fa9(private checkout). C is the prototypekC: columns ranked by y, each convolution restricted to the rows later-ranked columns need and clipped to the mask's x-range, upper triangle gathered, then mirrored. Timed after a warm-up in A-C-C-A order.c_dcmust be refitted.Jammy2211 commentedon Oct 16, 2026
2026-10-16 check-in: L2 library kernel witness (PyAutoArray#635, RAL 411834 → 411835)
These runs use the library code itself:
curvature_direct_conv_symmetricagainst the unchangedcurvature_direct_conv. The branch was applied as a patch on a private PyAutoArraye59d5fa9. Serial numba oneuclid-ral-compute-1, load < 1, A-C-C-A order, same synthetic Aris-512 geometry as the earlier check-in.nullconfig value now means the default, and the factory only resolves the mask on NumPy.Jammy2211 commentedon Oct 21, 2026
Note for the upcoming profiling programme (not started): the matrix-free interferometer inversion (PyAutoArray#638, @HRSAstro, phases 1-4 merged 2026-10-21 as #645/#650/#651/#652, on main now) needs its dense-vs-matrix-free comparison as a campaign here — 50², 64², 128², 192² rectangular meshes, unconstrained and positive solves, time / peak RSS / evidence difference, one synthetic and one real interferometer dataset, laptop + RAL CPU legs. Runs against checkouts, so it is NOT release-blocked. Open measurement from phase 3: SLQ evidence error vs Powell-preconditioner quality (0.1% data-dominated, 14-40% on a prior-dominated toy) — decide whether the contributor's FFT preconditioner from the gridded uv weight density is needed. Mind prompt: draft/research/autolens_profiling/matrix_free_interferometer_phase5.md.