Fix shared-memory ownership across Windows process handoff

[den-sq]

A Windows download worker can close its last shared-memory handle before the parent deserializes its result. The allocation then disappears and opening its name raises WinError 2, surfaced as BrokenProcessPool. The manager now owns a live handle before returning an allocation name and keeps it until the existing per-volume release point or run shutdown.

Closes #145. Based directly on main (97c4d4d), independently of #139 / PR #144.

Behavior

• Adapt lftomo's explicit non-owning serialization, idempotent/partial-construction cleanup, guarded destructor, and abnormal-owner-exit cleanup. Serialization excludes cached NumPy pixels and never transfers unlink authority.
• Replace the manager server's names-only tracker and Ouroboros's separate termed-name list with one owning registry. All array allocations occur at the server; inherited ShareableList registration acquires a server handle synchronously too.
• Preserve early release of termed buffers. Linux unlinks the name and closes the owner handle immediately; existing mappings remain usable. Windows closes the owner handle and releases storage when the remaining attachments close. Rejected removal leaves an unrelated supplied array usable.
• Clean up through the server's existing shutdown path. The old client-side shutdown() override was shadowed by BaseManager.start()'s finalizer. Use the parent-process sentinel for owner-death cleanup on Windows/POSIX and retain lftomo's POSIX process-group detachment.
• Add native lifecycle CI on Linux, Windows, and macOS with Python 3.11/3.12. Enable multiprocessing coverage and combine results in both existing coverage workflows, rather than excluding the new server implementation.

Validation

• Full Linux Python suite after the compatibility-audit expansion: 270 passed in 37.49 s; 99% helper coverage, above the unchanged 98% gate.
• Expanded native lifecycle CI: 34 passed on each Linux/macOS job and 17 passed, 1 skipped on each Windows job, for both Python 3.11 and 3.12. The skip is the POSIX process-group case; these focused tests do not use the CloudFiles dependency workaround.
• The deterministic handoff test fails against unmodified main's mem.py: after producer exit, pickle.loads() raises FileNotFoundError: [WinError 2]. It passes with this change. No scheduling delay is used to make the handoff succeed.
• Tests prove both run-scoped and termed allocations survive producer close/abrupt exit; per-buffer release drops server ownership; POSIX unlink preserves existing attachments; owner death releases allocations; failed client attachment/second-view construction rolls back ownership; and deserialization cannot inherit unlink authority.
• Native CI matrix and PR coverage passed on the audited implementation (8d92bc3): Linux, Windows, and macOS, each with Python 3.11 and 3.12. The first macOS run caught overlong explicit names in two test fixtures; those names were shortened without changing production code. The separate push coverage workflow also passed.
• git diff --check passed. No new warnings observed.

As requested, observer integration and real-data Linux/Windows slicing/backprojection and RAM-overhead validation remain open acceptance items in #139, under “Acceptance carried forward from #145 / PR #146”. Issue #145 marks both requirements as transferred, so they remain tracked after this PR closes it. They follow the memory-reporting branch rebase onto this change. That rebase must attach the existing observer to server-owned allocation/registration/release, preserve constructor observation for deserialized attachments, and remove the redundant __setstate__ observation path. The separate mapping-manifest publication failure remains work for #139. This PR does not claim scientific-pipeline or RAM-collector acceptance from lifecycle tests alone.

Shared-basis audit

• Facilities inspected: Ouroboros SharedNPArray, SharedNPManager, VolumeCache.remove_volume, and the slicing completion callback; lftomo lftomo/util/mem.py at 8d4c884e9f42ad913a38f6592f85ba9419f7a0f1; CPython SharedMemoryManager, SharedMemoryServer, tracker, and manager startup/finalization.
• Allocation authority: SharedMemoryServer.shared_memory_context now owns each live handle and its run-scoped/termed release policy. Previously CPython tracked run-scoped names and _termed_mem separately tracked termed names without owning handles. The global list, getter, list proxy registration, client proxy state, duplicate allocation path, and ineffective client shutdown loop are removed. list_segments, normal shutdown, owner-death cleanup, and explicit removal all derive from the single registry.
• Attachment authority: SharedNPArray continues to own only its process-local mapping, shape/views, and direct-creator unlink flag. Default whole-object serialization previously carried that flag and potentially cached pixels. The imported __reduce__ converts once to a name/shape/dtype/view boundary adapter, reconstructed as a non-owning attachment; it is not a second lifetime registry.
• Release authority: VolumeCache.remove_volume() and its existing completion callback retain the decision about when a termed buffer is no longer needed. The server atomically validates and executes release. No new buffer-retention schedule, sleep, retry, or duplicate completion state is introduced.
• Load-bearing decisions disclosed before implementation: server-owned handles replace names-only tracking for both allocation types; preserve per-volume release; import applicable lftomo lifecycle behavior without its unused role/reuse/storage architecture; use Python's parent sentinel instead of POSIX-only parent-PID polling. The server tracker/startup adaptation depends on CPython internals and is covered by the supported-version native CI matrix. Forced termination of the manager itself cannot run cleanup; existing OS/resource-tracker semantics still apply.
• Reporting boundary: this branch adds no competing observer or RAM record. Integration with Add portable, operation-scoped RAM accounting to shared run statistics #139's existing observation mechanism is explicitly deferred to its rebase.

The durable contract and integration notes are in documentation/docs/development/shared-memory-lifecycle.md.

Review follow-up

• Transferred Import lftomo memory lifecycle handling and fix Windows shared-buffer handoff ownership #145's observer-integration and real-data acceptance to explicit open items in Add portable, operation-scoped RAM accounting to shared run statistics #139 and cross-linked both issues. Preserved the Linux PSS gap and the USS-plus-live-segment proposal there as a metric decision requiring validation, including naming and double-counting checks; manager-held but untouched mappings alone do not close that gap.
• Documented the Python-cap upgrade audit for _run_server/startup/finalization, dispatch, and SharedMemoryServer.shared_memory_context, with the requirement to extend native lifecycle CI before supporting another Python version.
• These follow-ups change tracking and documentation only. The ownership implementation and approved lifecycle behavior are unchanged; no additional scientific or performance runs were needed for this edit.

Completed CPython compatibility audit

• Audited official CPython release sources 3.11.17, 3.12.15, 3.13.16, 3.14.8, and 3.15.0 at the three declared interfaces. No breaking change found in manager startup/finalization, dispatch request/result/error handling, or the server tracker interface; no production-code change was needed.
• Ran the expanded lifecycle suite under each of those Linux runtimes: 34 passed per version (170 total) with ResourceWarning promoted to an error and empty stderr. The probes cover fork, spawn, and forkserver, initializer argument forwarding, the installed shutdown finalizer, and both RPC/public error behavior.
• Identified the adjacent changes that matter for a future upgrade: SharedMemory(track=True) from 3.13, child-finalizer relocation in 3.13, and Linux's forkserver default from 3.14. The existing handoff and owner-death regressions now cover every available start method rather than only spawn.
• The newer Linux probes used isolated NumPy 2.5.4 environments; supported-version probes used NumPy 1.26.4. These are component compatibility results, not approval to raise the Python cap. Newer Windows/macOS runtimes, free-threaded builds, the full production dependencies, and scientific pipelines remain unvalidated for a cap bump. >=3.11,<3.13 and the supported CI version matrix are unchanged.
• Full findings, pinned source links, dependency versions, and reproduction command are recorded in documentation/docs/development/shared-memory-lifecycle.md, under “Compatibility audit (2026-10-17)”. The expanded native CI matrix passed on Python 3.11/3.12 for all three platforms, with no warnings in the lifecycle test summaries. The subsequent audit-results commit changes documentation only.

[tavateva]

Review of 87d8a41, graded against #145.

Verdict: approve. The fix targets the right owner. Allocation now happens in the manager server, which holds a live handle until the existing remove_volume release point or run shutdown. The names-only tracker and _termed_mem collapse into one owning registry that list_segments, shutdown, owner-death cleanup and explicit release all read. The diagnosis that the old client-side shutdown() override was shadowed by BaseManager.start()'s finalizer is correct.

Verification

• Full Linux suite: 253 passed. test_shared_memory_lifecycle.py: 17/17, also with -W error::ResourceWarning.
• Real Linux slice → backprojection through the live manager. This was the benchmark harness at 128³ with 3 pairs, used here as a smoke test, not for its timing figures.
  • Both operations completed and were pixel-equivalent across all runs.
  • The output pixel digests (7d7fabc3…, e1ddd105…) and the source digest are identical to a run of the same fixture on the pre-change pipeline (#140's head).
  • No leaked-segment warnings appeared in any run's stderr, and /dev/shm was empty afterwards.
• Relied on: the native Windows/macOS CI matrix and the author's Windows WinError 2 reproduction against unmodified main.

Non-blocking

• The closing keyword closes #145 with two acceptance items open. The body defers #145's real-data Linux/Windows slicing → backprojection validation and item 4 (integration with #139's observation facility) until the #144 rebase. Merging with Closes #145 closes the issue, and those requirements then live only in this PR body.
  • Minimum response: move both items onto #139, or a follow-up, and confirm they landed there before merge. Alternatively, change the keyword to Part of #145.
• This registry is the natural fix for the #144 Linux accounting gap. On Linux the server's handles are mapped but never touched, so they add nothing to PSS. Queued, untouched chunks therefore stay invisible to process_tree_pss_bytes after the rebase, as in the #144 early review. But _OwnedSharedMemoryTracker is now the single authoritative list of live operation segments and their sizes, which is exactly what a USS-plus-live-segments metric needs. Minimum response: none here. Noted for the #144 rebase.
• CPython internals are declared and pinned. _run_server mirrors BaseManager._run_server, dispatch is used directly, and the server's shared_memory_context is replaced. All of this is declared and covered by the 3.11/3.12 native matrix, and pyproject caps Python below 3.13. Any future Python cap bump should re-check these three seams.