Affected Repositories

• PyAutoArray (primary)

Branch Survey

PyAutoArray is also claimed by magnification-maps-wiring (PyAutoLens#818, uncommitted edits to autoarray/plot/array.py only). On 2026-11-05 the human ruled to run this fix in parallel (no overlapping files).

Suggested branch: feature/numba-fork-safe-threading
Work Classification: Library
Worktree root: ~/Code/PyAutoLabs-wt/numba-fork-safe-threading/

Implementation Steps

1. autoarray/numba_util.py: add prepare_parallel_launch() plus module state _parallel_launch_pid and _threads_pinned_pid.
   • First call in a process lineage: if NUMBA_THREADING_LAYER is not in the environment and numba.config.THREADING_LAYER == "default", set numba.config.THREADING_LAYER to "tbb" if importlib.util.find_spec("tbb") finds it, else "workqueue". This must run before numba's first parallel launch, because the layer is fixed at that launch. Record os.getpid().
   • Later calls in a different pid (a forked child; module state is inherited): call numba.set_num_threads(1) once per child, unless the user set NUMBA_NUM_THREADS.
   • The docstring explains omppool's after-fork termination, the oversubscription problem, and workqueue's caveat (not safe for concurrent launches from several Python threads in one process; the search pools use processes).
2. autoarray/inversion/mesh/interpolator/sibson.py _sibson_mappings_weights_numba: call numba_util.prepare_parallel_launch() before kernel(...).
3. autoarray/inversion/inversion/interferometer_numba/sparse.py: call it before direct_conv_parallel_kernel()(*args). Do the same in test__parallel_kernel_equals_serial_kernel, or let the kernel wrapper call it.
4. Tests in test_autoarray/numba/test_numba_fork_safety.py (or next to the Sibson tests):
   • (a) Run a python -I subprocess: Sibson tables on a small random mesh, call in the parent, then mp.get_context("fork").Pool(2) calls it in children. Assert the children's outputs equal the parent's within a timeout, and that the children report numba.get_num_threads() == 1. A subprocess is used because the pytest process may already have fixed its layer. Skip when fork is unavailable or numba is missing.
   • (b) Unit tests: the layer choice respects the env var and an explicit config value, and a pid change triggers pinning (monkeypatch os.getpid).
   • (c) A control: the same subprocess with NUMBA_THREADING_LAYER=omp reproduces the failure, so the test is not vacuous (skip if omp is unavailable).
5. Run the PyAutoArray unit tests (Sibson parity, interferometer numba), then smoke the delaunay_nn path.

Key Files

• autoarray/numba_util.py: the new guard
• autoarray/inversion/mesh/interpolator/sibson.py: Sibson numba dispatch (_sibson_mappings_weights_numba)
• autoarray/inversion/mesh/interpolator/_sibson_numba.py: the prange kernel (unchanged)
• autoarray/inversion/inversion/interferometer_numba/sparse.py: the parallel direct_conv call site
• autoarray/operators/nufft_cpu.py: prior art (_openmp_pid)

Out of scope (noted)

• The parent's numba thread count defaults to the node's CPU count (252 on RAL), not the job's allocation. Capping it would change profiled timings, so it is left for a separate decision.

numba parallel=True kernels (Sibson, direct_conv) pick the GNU OpenMP threading layer, which kills every Nautilus worker forked after the likelihood warm-up

Type: bug
Target: autoarray
Repos:

• PyAutoArray
  Themes:
• source-reconstruction
• performance
  Difficulty: small
  Autonomy: supervised
  Priority: high
  Status: formalised
  Consequence: glance
  Witness: A PyAutoArray unit test runs the Sibson numba kernel (and the parallel direct_conv kernel) in the parent, then calls it from a fork-context multiprocessing pool, and gets the parent's result back within a timeout (today the default threading layer kills the children: TimeoutError); forked children run the kernel on one numba thread; and the RAL rerun of autolens_inference imaging/delaunay_nn/hst numba_cpu_sparse (job 414150's submit) gets past source_pix[1].
  Review-minutes: 3
  Unattended: ready
  Filed: 2026-11-05
  Updated: 2026-11-05

Finding

2026-11-05, likelihood-lifecycle phase 4 CPU batch, RAL job 414150 (both seeds), autolens_inference
imaging/delaunay_nn/hst numba_cpu_sparse_fp64, pin 2026.10.25.1, 8 cores. At SLaM stage
source_pix[1], PyAutoFit runs "the likelihood once before forking workers", then Nautilus forks its
pool. The children print Terminating: fork() called from a process already using GNU OpenMP, this is unsafe., the parent's result.get raises multiprocessing.context.TimeoutError, and the runner fails
the job (exit 1, 43 s). Log: /mnt/ral/jnightin/inference/autolens_inference/hpc/batch_cpu/error/error.414150_0.err.
The plain Delaunay numba leg (414149) is fine at the same stage. Its interpolator has no prange.

Mechanism

autoarray/inversion/mesh/interpolator/_sibson_numba.py is @numba.njit(parallel=True) with prange
over queries (likewise direct_conv_parallel_kernel in
inversion/interferometer_numba/inversion_interferometer_numba_util.py). With no TBB installed, numba's
default threading layer is omp, built on GNU OpenMP. Its omppool after-fork handler terminates any
child forked from a parent that has started the pool. The parent warm-up starts it, so every forked
worker dies.

Laptop repro (numba 0.68.1, Python 3.13): a prange kernel called in the parent, then
mp.get_context("fork").Pool(2).map_async(...). With the default and omp layers this gives
TimeoutError. With forksafe (falls back to workqueue without TBB) and workqueue, the children return
the correct results.

Second hazard: in a forked child numba starts NUMBA_NUM_THREADS threads, which defaults to the node's
CPU count (252 on RAL), not the job's allocation. With 8 Nautilus workers that heavily oversubscribes the
node.

Fix direction (library, never the workspace scripts)

Mirror the finufft guard in autoarray/operators/nufft_cpu.py (_openmp_pid). One shared helper used by
both parallel kernels:

1. Before the first parallel launch, if the user has not chosen a layer (NUMBA_THREADING_LAYER unset /
   numba.config.THREADING_LAYER == "default"), select a fork-safe layer (tbb if importable, else
   workqueue).
2. Record the pid of the first launch. In any other (forked) process, call numba.set_num_threads(1)
   before launching.
   Results must stay bit-identical: the parallel loops have no reductions, and the parity tests pin this.

Why it matters

Every NumPy-path DelaunayNN fit with number_of_cores > 1 dies on any machine without TBB, as does any
parallel direct_conv interferometer fit. It blocks the phase-4 delaunay_nn CPU validation leg (Pulse
likelihood-lifecycle campaign). Phase-2 profiled DelaunayNN per-call timings were single-process omp
runs, so an ingest note should record the layer the validation legs ran under.