Skip to content

fix: isolate non-finite Delaunay callback lanes #410

Closed

Description

@Jammy2211

Overview

A non-finite traced source-plane Delaunay mesh currently reaches SciPy qhull through JAX pure_callback, where qhull raises and aborts the entire sequentially-vmapped multi-start batch. This affects ordinary broad-start Prodigy trajectories as well as resurrection and poisons persisted resumes.

The repair must isolate the invalid callback lane and preserve a loud NaN-at-likelihood contract. Returning sentinel connectivity alone is unsafe: the visibility walk can accept a padded [-1, -1, -1] simplex and downstream nearest-neighbour/degeneracy logic can launder the lane into finite pixel-0 weights.

Plan

  • Guard the host Delaunay tables callback per vmap lane and return fixed-shape sentinel tables without invoking qhull for non-finite mesh points.
  • Make the visibility walk reject padded/invalid simplices as successful locations and retain a live nearest-vertex fallback mapping, including split-point location.
  • Gate the final JAX pixel weights on mesh validity so invalid meshes reach the inversion and figure of merit as NaN, while leaving the SciPy/NumPy production path unchanged.
  • Pin the callback, mapping-index, poisoning-order, and finite-input no-op invariants in focused PyAutoArray tests.
  • Extend imaging and interferometer production _vmap parity scripts to assert NaN FOM isolation for poisoned lanes and bit-identical finite lanes; keep the existing finite likelihood literals and FD certification unchanged.
Detailed implementation plan

Work Classification

Both: PyAutoArray library repair first, then autolens_workspace_test parity coverage.

Affected Repositories

  • PyAutoArray (primary)
  • autolens_workspace_test (downstream parity assertions)

Branch Survey

Repository Current Branch Dirty?
./PyAutoArray main clean
./autolens_workspace_test main two unrelated point-source dataset JSON modifications

The isolated task worktree avoids the unrelated workspace-test changes.

Suggested branch: feature/delaunay-nan-callback

Worktree root: ~/Code/PyAutoLabs-wt/delaunay-nan-callback/

Implementation Steps

  1. In autoarray/inversion/mesh/interpolator/delaunay.py, add a host-side np.isfinite(points_np).all() guard to scipy_delaunay_tri_only. For invalid points, return -1-filled int32 arrays at the documented (2N, 3), (2N, 3), and (N,) shapes without calling qhull. Because _jax_delaunay_tables uses vmap_method="sequential", the check remains lane-local.
  2. In pix_indexes_delaunay_walk_from, track whether the current simplex row contains only non-negative vertex indices. An invalid/padded row must not set done; it must terminate through the existing [seed, -1, -1] fallback. Apply the invariant to both primary query mapping and jax_delaunay split-point mapping.
  3. In pixel_weights_delaunay_from, after barycentric/nearest-neighbour selection, use the threaded xp backend to make JAX weights NaN when any mesh vertex is non-finite. Keep the eager SciPy/NumPy production route unchanged for its existing exception-based diagnostics. The gate must be a bitwise no-op for finite meshes.
  4. Add PyAutoArray unit coverage for sentinel table shapes/dtypes, invalid-simplex fallback mappings with at least one non-negative index per query, and finite-input no-op behavior. Cover an all-NaN mesh, a non-last poisoned vertex, and the last poisoned vertex to prevent negative-index IEEE-order laundering.
  5. Update scripts/imaging/jax_likelihood/delaunay.py and scripts/interferometer/jax_likelihood/delaunay.py with production Fitness._vmap assertions: the poisoned lane produces a NaN FOM without raising, and finite lanes equal their unbatched/baseline results bit-for-bit. This covers dense and sparse inversion paths and the linked intermittent interferometer qhull failure surface.
  6. Run focused PyAutoArray tests and the full PyAutoArray suite. Run the two likelihood parity scripts and scripts/imaging/jax_grad/delaunay.py unchanged; confirm existing finite likelihood literals and frozen-table FD gradients remain unchanged.

Key Files

  • autoarray/inversion/mesh/interpolator/delaunay.py — callback, visibility walk, split mappings, and final interpolation weights.
  • test_autoarray/inversion/pixelization/interpolator/test_delaunay_walk.py — focused NumPy-side geometry invariants.
  • scripts/imaging/jax_likelihood/delaunay.py — dense production-vmap regression.
  • scripts/interferometer/jax_likelihood/delaunay.py — sparse production-vmap regression.
  • scripts/imaging/jax_grad/delaunay.py — unchanged frozen-table gradient certification.

Acceptance Invariants

  • No non-finite callback lane calls qhull or aborts sibling lanes.
  • Every invalid callback mapping row retains at least one non-negative index, so NaN weights remain a live mapping contribution.
  • Poisoned dense and sparse lanes reach NaN at the figure of merit.
  • Finite lanes are bit-identical to solo/baseline evaluation and finite-input gradients remain FD-certified.
  • No NumPy production exception contract is changed.

Original Prompt

Click to expand starting prompt

Delaunay tables callback crashes the whole JAX run when traced points contain NaN

Type: bug
Target: autoarray
Repos:

  • PyAutoArray
    Difficulty: medium
    Autonomy: supervised
    Priority: high
    Status: draft

The bug (hit live on RAL, 2026-07-27, wsdev#117 campaign)

Under broad-start multi-start gradient descent (af.MultiStartProdigy, resurrect=True), a start's parameter vector can go non-finite mid-trajectory. For pure-JAX meshes that propagates through the likelihood for the search machinery to handle. For Delaunay, non-finite traced mesh points reach host qhull through _jax_delaunay_tables; qhull raises ValueError: Points cannot contain NaN, surfaced as a JaxRuntimeError, and one poisoned lane aborts all 16 starts.

The same crash occurred with resurrect=False, showing that ordinary broad-start descent can create the invalid mass configurations. Persisted NaN parameters also make dependent/resumed jobs fail immediately.

Fix sketch

Detect non-finite input inside the sequential callback lane and return fixed-shape sentinel tables without calling qhull. Ensure the downstream likelihood evaluates to NaN for that lane rather than laundering the sentinel result into finite weights. Validate via batched JAX likelihood evaluation with poisoned and finite lanes, plus the existing frozen-table finite-difference gradient certification.

Related: the intermittent interferometer Delaunay qhull NaN is likely the same missing guard on the sparse path.

Activity

  1. Jammy2211 commented on Jul 27, 2026

    @Jammy2211
    CollaboratorAuthor

    Implementation complete — ship checkpoint

    The Delaunay NaN-lane repair is implemented and committed locally on feature/delaunay-nan-callback:

    • PyAutoArray d802559e — per-lane callback guard, invalid-simplex fallback, JAX-only NaN weight gate, focused NumPy tests.
    • autolens_workspace_test 1324100 — dense/sparse production _vmap, raw-NaN likelihood, partial/all poisoning, and finite-lane value/gradient parity assertions.

    Evidence

    • PyAutoArray: 906 passed (pytest -x).
    • Imaging Delaunay production _vmap: PASS; poisoned lane resampled, finite siblings bit-identical to the clean vmap; raw fit log-likelihood NaN.
    • Interferometer Delaunay production _vmap: PASS with the same sparse-path assertions.
    • Partial non-last vertex, last vertex, and all-NaN meshes: PASS under jit(vmap(value_and_grad)); finite-lane values and nonzero gradients bit-identical to clean values.
    • Existing scripts/imaging/jax_grad/delaunay.py: PASS unchanged (14-parameter FD certification, rtol=1e-2).
    • Review faculty: CLEAN.
    • Scoped autolens_workspace_test smoke: 20/21 passed; unrelated imaging/subhalo_recovery.py exceeded its existing 300-second cap. It does not exercise Delaunay or the changed code.

    Human ship decision needed

    The --auto four-leg gate cannot open the PR automatically because of that unrelated smoke timeout and Heart's current unrelated YELLOW reasons:

    1. workspace validation not passing (13 failed, 2026-07-21T19-05-22Z)
    2. 33 stale parked script(s)
    3. manifest drift: tenant firewall (organ code) — 5 mismatch(es) vs PyAutoMind/repos.yaml

    Recommendation: authorize push/PR for this task by acknowledging the exact YELLOW set above and accepting the unrelated imaging/subhalo_recovery.py timeout as non-causal. Otherwise the task remains parked until organism-wide smoke/Heart state is repaired.

  2. Jammy2211 commented on Jul 27, 2026

    @Jammy2211
    CollaboratorAuthor

    Human ship acknowledgement received in the active session: the three Heart YELLOW reasons listed in the preceding checkpoint comment are acknowledged, and the unrelated imaging/subhalo_recovery.py 300-second smoke timeout is accepted as non-causal. Proceeding to push/open the pending-release PyAutoArray PR; merge remains manual.

  3. Jammy2211 commented on Jul 27, 2026

    @Jammy2211
    CollaboratorAuthor

    Library PR created

    PyAutoArray PR #411 is open with pending-release and the full validation record. Tests are 906/906, review is CLEAN, and the human-acknowledged ship checkpoint is recorded.

    API changes: none — internal behavior hardening only.

    Next: human merge of #411, then ship the already-committed autolens_workspace_test dense/sparse regression follow-up (1324100) so CI consumes the merged library implementation. RAL mirrors remain untouched until the campaign owner selects a safe pull window.

  4. Jammy2211 commented on Jul 27, 2026

    @Jammy2211
    CollaboratorAuthor

    Library merge and workspace handoff complete.

  5. Jammy2211 commented on Jul 27, 2026

    @Jammy2211
    CollaboratorAuthor

    Shipped — 2026-07-27

    PRs

    Upstream PR

    #411

    Summary

    The Delaunay callback now isolates invalid mesh points per sequential vmap lane instead of allowing qhull to abort the full batch, while preserving a NaN raw likelihood for the poisoned lane. PyAutoArray #411 merged as 8c5e28e; dense and sparse production regressions in workspace PR #226 merged as f96aed1 after all four smoke jobs passed.

    Session Notes

    The callback sentinel alone was insufficient because the walk could launder it into finite pixel-0 mappings, so the shipped design also rejects padded simplices and preserves NaN at the JAX weights boundary. The NumPy path remains unchanged for diagnostics, finite values and gradients remain bit-identical, and no RAL mirror was touched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions