Repository navigation
Description
Overview
A non-finite traced source-plane Delaunay mesh currently reaches SciPy qhull through JAX pure_callback, where qhull raises and aborts the entire sequentially-vmapped multi-start batch. This affects ordinary broad-start Prodigy trajectories as well as resurrection and poisons persisted resumes.
The repair must isolate the invalid callback lane and preserve a loud NaN-at-likelihood contract. Returning sentinel connectivity alone is unsafe: the visibility walk can accept a padded [-1, -1, -1] simplex and downstream nearest-neighbour/degeneracy logic can launder the lane into finite pixel-0 weights.
Plan
- Guard the host Delaunay tables callback per vmap lane and return fixed-shape sentinel tables without invoking qhull for non-finite mesh points.
- Make the visibility walk reject padded/invalid simplices as successful locations and retain a live nearest-vertex fallback mapping, including split-point location.
- Gate the final JAX pixel weights on mesh validity so invalid meshes reach the inversion and figure of merit as NaN, while leaving the SciPy/NumPy production path unchanged.
- Pin the callback, mapping-index, poisoning-order, and finite-input no-op invariants in focused PyAutoArray tests.
- Extend imaging and interferometer production
_vmapparity scripts to assert NaN FOM isolation for poisoned lanes and bit-identical finite lanes; keep the existing finite likelihood literals and FD certification unchanged.
Detailed implementation plan
Work Classification
Both: PyAutoArray library repair first, then autolens_workspace_test parity coverage.
Affected Repositories
- PyAutoArray (primary)
- autolens_workspace_test (downstream parity assertions)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./PyAutoArray | main | clean |
| ./autolens_workspace_test | main | two unrelated point-source dataset JSON modifications |
The isolated task worktree avoids the unrelated workspace-test changes.
Suggested branch: feature/delaunay-nan-callback
Worktree root: ~/Code/PyAutoLabs-wt/delaunay-nan-callback/
Implementation Steps
- In
autoarray/inversion/mesh/interpolator/delaunay.py, add a host-sidenp.isfinite(points_np).all()guard toscipy_delaunay_tri_only. For invalid points, return-1-filledint32arrays at the documented(2N, 3),(2N, 3), and(N,)shapes without calling qhull. Because_jax_delaunay_tablesusesvmap_method="sequential", the check remains lane-local. - In
pix_indexes_delaunay_walk_from, track whether the current simplex row contains only non-negative vertex indices. An invalid/padded row must not setdone; it must terminate through the existing[seed, -1, -1]fallback. Apply the invariant to both primary query mapping andjax_delaunaysplit-point mapping. - In
pixel_weights_delaunay_from, after barycentric/nearest-neighbour selection, use the threadedxpbackend to make JAX weights NaN when any mesh vertex is non-finite. Keep the eager SciPy/NumPy production route unchanged for its existing exception-based diagnostics. The gate must be a bitwise no-op for finite meshes. - Add PyAutoArray unit coverage for sentinel table shapes/dtypes, invalid-simplex fallback mappings with at least one non-negative index per query, and finite-input no-op behavior. Cover an all-NaN mesh, a non-last poisoned vertex, and the last poisoned vertex to prevent negative-index IEEE-order laundering.
- Update
scripts/imaging/jax_likelihood/delaunay.pyandscripts/interferometer/jax_likelihood/delaunay.pywith productionFitness._vmapassertions: the poisoned lane produces a NaN FOM without raising, and finite lanes equal their unbatched/baseline results bit-for-bit. This covers dense and sparse inversion paths and the linked intermittent interferometer qhull failure surface. - Run focused PyAutoArray tests and the full PyAutoArray suite. Run the two likelihood parity scripts and
scripts/imaging/jax_grad/delaunay.pyunchanged; confirm existing finite likelihood literals and frozen-table FD gradients remain unchanged.
Key Files
autoarray/inversion/mesh/interpolator/delaunay.py— callback, visibility walk, split mappings, and final interpolation weights.test_autoarray/inversion/pixelization/interpolator/test_delaunay_walk.py— focused NumPy-side geometry invariants.scripts/imaging/jax_likelihood/delaunay.py— dense production-vmap regression.scripts/interferometer/jax_likelihood/delaunay.py— sparse production-vmap regression.scripts/imaging/jax_grad/delaunay.py— unchanged frozen-table gradient certification.
Acceptance Invariants
- No non-finite callback lane calls qhull or aborts sibling lanes.
- Every invalid callback mapping row retains at least one non-negative index, so NaN weights remain a live mapping contribution.
- Poisoned dense and sparse lanes reach NaN at the figure of merit.
- Finite lanes are bit-identical to solo/baseline evaluation and finite-input gradients remain FD-certified.
- No NumPy production exception contract is changed.
Original Prompt
Click to expand starting prompt
Delaunay tables callback crashes the whole JAX run when traced points contain NaN
Type: bug
Target: autoarray
Repos:
- PyAutoArray
Difficulty: medium
Autonomy: supervised
Priority: high
Status: draft
The bug (hit live on RAL, 2026-07-27, wsdev#117 campaign)
Under broad-start multi-start gradient descent (af.MultiStartProdigy, resurrect=True), a start's parameter vector can go non-finite mid-trajectory. For pure-JAX meshes that propagates through the likelihood for the search machinery to handle. For Delaunay, non-finite traced mesh points reach host qhull through _jax_delaunay_tables; qhull raises ValueError: Points cannot contain NaN, surfaced as a JaxRuntimeError, and one poisoned lane aborts all 16 starts.
The same crash occurred with resurrect=False, showing that ordinary broad-start descent can create the invalid mass configurations. Persisted NaN parameters also make dependent/resumed jobs fail immediately.
Fix sketch
Detect non-finite input inside the sequential callback lane and return fixed-shape sentinel tables without calling qhull. Ensure the downstream likelihood evaluates to NaN for that lane rather than laundering the sentinel result into finite weights. Validate via batched JAX likelihood evaluation with poisoned and finite lanes, plus the existing frozen-table finite-difference gradient certification.
Related: the intermittent interferometer Delaunay qhull NaN is likely the same missing guard on the sparse path.
Activity
Jammy2211 commentedon Jul 27, 2026
Implementation complete — ship checkpoint
The Delaunay NaN-lane repair is implemented and committed locally on
feature/delaunay-nan-callback:d802559e— per-lane callback guard, invalid-simplex fallback, JAX-only NaN weight gate, focused NumPy tests.1324100— dense/sparse production_vmap, raw-NaN likelihood, partial/all poisoning, and finite-lane value/gradient parity assertions.Evidence
pytest -x)._vmap: PASS; poisoned lane resampled, finite siblings bit-identical to the clean vmap; raw fit log-likelihood NaN._vmap: PASS with the same sparse-path assertions.jit(vmap(value_and_grad)); finite-lane values and nonzero gradients bit-identical to clean values.scripts/imaging/jax_grad/delaunay.py: PASS unchanged (14-parameter FD certification,rtol=1e-2).imaging/subhalo_recovery.pyexceeded its existing 300-second cap. It does not exercise Delaunay or the changed code.Human ship decision needed
The
--autofour-leg gate cannot open the PR automatically because of that unrelated smoke timeout and Heart's current unrelated YELLOW reasons:workspace validation not passing (13 failed, 2026-07-21T19-05-22Z)33 stale parked script(s)manifest drift: tenant firewall (organ code) — 5 mismatch(es) vs PyAutoMind/repos.yamlRecommendation: authorize push/PR for this task by acknowledging the exact YELLOW set above and accepting the unrelated
imaging/subhalo_recovery.pytimeout as non-causal. Otherwise the task remains parked until organism-wide smoke/Heart state is repaired.Jammy2211 commentedon Jul 27, 2026
Human ship acknowledgement received in the active session: the three Heart YELLOW reasons listed in the preceding checkpoint comment are acknowledged, and the unrelated
imaging/subhalo_recovery.py300-second smoke timeout is accepted as non-causal. Proceeding to push/open the pending-release PyAutoArray PR; merge remains manual.Jammy2211 commentedon Jul 27, 2026
Library PR created
PyAutoArray PR #411 is open with
pending-releaseand the full validation record. Tests are 906/906, review is CLEAN, and the human-acknowledged ship checkpoint is recorded.API changes: none — internal behavior hardening only.
Next: human merge of #411, then ship the already-committed
autolens_workspace_testdense/sparse regression follow-up (1324100) so CI consumes the merged library implementation. RAL mirrors remain untouched until the campaign owner selects a safe pull window.Jammy2211 commentedon Jul 27, 2026
Library merge and workspace handoff complete.
8c5e28ec210b6149ffa0d313f717884271ce722aafter all four CI jobs passed.pending-releaseroute and acknowledged Heart-YELLOW conditions; fix: isolate invalid Delaunay callback lanes #411 was mergeable/CLEAN with no requested changes.13241001454ab7420630659d0f20477ad7cc8fbf, labeledpending-release.Jammy2211 commentedon Jul 27, 2026
Shipped — 2026-07-27
PRs
Upstream PR
#411
Summary
The Delaunay callback now isolates invalid mesh points per sequential vmap lane instead of allowing qhull to abort the full batch, while preserving a NaN raw likelihood for the poisoned lane. PyAutoArray #411 merged as
8c5e28e; dense and sparse production regressions in workspace PR #226 merged asf96aed1after all four smoke jobs passed.Session Notes
The callback sentinel alone was insufficient because the walk could launder it into finite pixel-0 mappings, so the shipped design also rejects padded simplices and preserves NaN at the JAX weights boundary. The NumPy path remains unchanged for diagnostics, finite values and gradients remain bit-identical, and no RAL mirror was touched.