fix: test-mode-gate singular/non-PD inversion crash (release tail) #388

Activity

1. Jammy2211 commented on Jul 14, 2026

   

   Jammy2211

   on Jul 14, 2026

   CollaboratorAuthor

   More actions

   ⏸️ Parked for ship sign-off (--auto, supervised)

   Implementation is complete and committed locally on feature/inversion-testmode-singular-guard (commit 48fae1ac), not pushed — the supervised autonomy contract makes ship sign-off (push + PR-open) a human checkpoint.

   What was done

   The four inversion raise-sites now return a benign dummy only when autoconf.is_test_mode() is true, else re-raise exactly as before:

   Site	Normal mode (unchanged)	Test mode
   inversion_util.reconstruction_positive_negative_from	raise LinAlgError	return ones_like(data_vector)
   inversion_util.reconstruction_positive_only_from	raise InversionException	return ones_like(data_vector)
   abstract.log_det_curvature_reg_matrix_term	raise LinAlgError	return 0.0
   abstract.log_det_regularization_matrix_term	raise LinAlgError	return 0.0

   numpy-only by construction (the JAX linalg path returns NaN rather than raising). No conditioning floor — real-mode numerics are byte-for-byte unchanged.

   Diff: abstract.py +40/-… · inversion_util.py +20 · test_inversion_util.py +53 (3 files, +101/-12).

   Test evidence

   • 2 new numpy-only guard tests (singular matrix → raises normally; under monkeypatched PYAUTO_TEST_MODE → finite dummy).
   • test_autoarray/inversion/inversion/test_inversion_util.py + test_abstract.py: 41 passed.
   • Full test_autoarray/inversion/: 233 passed.

   The decision for you

   Sign off to ship? On approval, /ship_library runs the four-leg gate (full PyAutoArray suite, review, Heart) and pushes + opens the pending-release PR (merge stays human).

   • Approve → resume /ship_library on this task.
   • Change dummy semantics / sites → tell me what to adjust; the commit amends in place.
   • Prefer validation-tiering instead (exclude these scripts from the test-mode set rather than a library change) → say so and I'll revert the branch and re-scope to a PyAutoHeart harness change.

   Reproduction note: the reported LinAlgError/InversionException did not reproduce deterministically on clean main (TEST_MODE=1 and =2 both pass; the fragility shows as mapper_util divide-by-zero on near-singular samples). This fix removes the flaky crash regardless of which site tips over, without touching real inference.
2. Jammy2211 commented on Jul 14, 2026

   

   Jammy2211

   on Jul 14, 2026

   CollaboratorAuthor

   More actions

   🔴 Ship blocked — Heart RED (parked, nothing pushed)

   Human ship sign-off received, four-leg gate resumed. Tests green (test_autoarray/inversion/ 233 passed), but the Heart readiness gate is RED, which blocks commit/push/PR at every autonomy level. Verbatim RED reasons from pyauto-heart readiness:

   1. PyAutoArray: on branch feature/ticks-minus-in-math (not main)
   2. release validation FAILED (stage integrate)

   The corrective-PR exception for Heart RED is a contemporaneous human act, never reachable under --auto — this is an --auto run, so it does not apply. It would also be a poor fit: reason #1 is unrelated branch state, and reason #2 is the structural perf-flake integrate tail, which this fix does not causally repair (it removes flaky test-mode singular crashes; the integrate RED is cleared by the advisory-tier work + PyAutoArray returning to main).

   Parked. The fix is complete, tested, and committed locally (48fae1ac on feature/inversion-testmode-singular-guard) — not pushed. It ships via /ship_library once Heart clears (PyAutoArray back on main + integrate validation passing). No code was modified to make any leg pass.
3. Jammy2211 commented on Jul 14, 2026

   

   Jammy2211

   on Jul 14, 2026

   CollaboratorAuthor

   More actions

   ✅ Corrective-PR exception authorized — PR opened

   Human authorized the corrective-PR exception in-session (2026-07-14) for the named RED reason:

   release validation FAILED (stage integrate)

   PR #389 opened (pending-release) — the test-mode singular-inversion guard (commit 48fae1ac), reason-scoped, no mixed scope.

   Causal map: integrate FAILs slam.py + cpu_fast_modeling.py (run 29341418859) → unguarded test-mode singular inversion → 4-site is_test_mode() dummy guard → PR #389. Follow-up to PyAutoLens#607.

   Sibling RED remains: PyAutoArray: on branch feature/ticks-minus-in-math (not main) — Heart stays RED after any merge until PyAutoArray's main checkout returns to main and integrate validation re-runs on fresh wheels.

   Caveat (recorded): these FAILs did not reproduce deterministically on clean main; the fix is test-mode-gated. Causal confirmation comes from the post-merge re-validation, not before.

   Merge is a separate human act — not performed by this action. After a human merge: build fresh wheels → re-run release integration on main → obtain a new Heart verdict before any release decision.
4. Jammy2211 commented on Jul 14, 2026

   

   Jammy2211

   on Jul 14, 2026

   CollaboratorAuthor

   More actions

   ✅ Merged (corrective PR #389) — awaiting corrective validation

   PR #389 squash-merged to main by explicit human direction (commit 1ff90287fc7654f864fb4c51cba7ea69686a7b66, 2026-07-14). Issue kept open until corrective validation reruns.

   Recovery sequence (required before any release decision):

   1. Build fresh post-merge wheels.
   2. Re-run release integration on main.
   3. Obtain a new Heart verdict over that fresh evidence.

   Heart stays RED until the sibling reason is also cleared: PyAutoArray: on branch feature/ticks-minus-in-math (not main) (the main checkout must return to main), plus the perf-flake tail (no_run SLOW PRs alwt#169 / agwt#73). Release remains blocked throughout.

   Caveat carried forward: these FAILs did not reproduce deterministically pre-merge; whether the integrate stage actually flips is confirmed only by the re-validation above.
5. Jammy2211 commented on Jul 14, 2026

   

   Jammy2211

   on Jul 14, 2026

   CollaboratorAuthor

   More actions

   Corrective validation GREEN — closing

   The fresh mode=release validation run on the current main HEADs passed end-to-end:

   • Stage 2 TestPyPI rehearsal v2026.7.14.1.dev66001 — all 5 wheels built + uploaded.
   • Stage 3 wheel-based real-search integration — 542 passed, 0 failed, 0 timeout, 87 skipped.
   • Ingested → release_ready: true; Heart is now YELLOW (no RED reasons remain).

   This fix is on the validated main HEADs (commit_shas matched), so its corrective validation is satisfied. Closing.
6. Jammy2211

   closed this as completedon Jul 14, 2026