Fix engine detection, shutdown and test process hygiene

docs/known-issues.mdx

@@ -120,22 +120,6 @@ On the peer that needs to connect, add a reachable IP address or hostname as
 **PAIR machine**, such as its Tailscale address; a saved address does not replace
 pairing or verification.
 
-## Leftover oMLX Launchers Count as Installed
-
-A leftover `~/.omlx` launcher can make PAIR report oMLX as installed even when
-`/Applications/oMLX.app` is absent, and PAIR retries starting it on each launch.
-This also affects 0.8.0. Restore the oMLX app or use the vendor's cleanup
-instructions for a stale installation; PAIR's installed status alone does not
-prove the app bundle is present.
-
-## Ollama Install Download Is Not Checksum-Pinned
-
-PAIR installs Ollama from the vendor's floating latest-download URL without a
-pinned checksum and logs `UNPINNED download`. This is unchanged since 0.8.0;
-download progress is not an integrity check. If a pinned artifact is required,
-install an independently verified Ollama release yourself instead of using
-PAIR's install action.
-
 ## The Terminal Interface Cannot Do Everything the Desktop App Can
 
 **Implemented in source, pending release testing.** `nvpair-tui` now shows

services/nvpair-engine-manager/README.md

@@ -109,8 +109,15 @@ On macOS arm64, the bundled manifest checks these CLI paths in order:
 2. `/opt/homebrew/bin/omlx`
 3. `/usr/local/bin/omlx`
 
+The manifest also requires `/Applications/oMLX.app/Contents/MacOS/omlx-cli`,
+the target of the app's launcher (`detect_requires`). A stale launcher without
+that target reports **not installed**; saved ON intent is retained but skipped
+at launch without an error. In desktop, use **Install oMLX**; in TUI Engines,
+select oMLX and press **i** to reinstall the app. PAIR does not delete or alter
+`~/.omlx` to repair detection.
+
 The Homebrew and `/usr/local` candidates are assumed to provide the same `omlx`
-CLI and `start`/`stop` semantics as the install.sh installation. This change does
+CLI and `start`/`stop` semantics as the app's launcher. This change does
 not verify vendor CLI behavior. No app-bundle executable is a candidate:
 `oMLX.app/Contents/MacOS/oMLX` is the GUI executable (also matched by lowercase
 `omlx` on case-insensitive macOS filesystems), and `omlx-cli` and other bundle
@@ -137,6 +144,39 @@ The bundled uninstalls remove only what an install put down: Ollama's app or
 install directory, and LM Studio's `~/.lmstudio/bin` and `llmster` runtime. LM
 Studio's conversations, credentials and settings are left where they are.
 
+### Unsupported-target build fallbacks
+
+FreeBSD can compile the engine manager, but is not a supported platform and has
+no bundled engine manifests. Its fallbacks refuse app install-identity claims
+and use ordinary process waiting rather than detached-downloader exit watching.
+This does not extend the supported platform list.
+
+### Fake-engine test cleanup
+
+Each fake-engine executable (including test copies) has a per-test owner under
+the run's `nvpair-em-test-*` directory. Before doing any work, the fake records
+its PID and executable identity there; detached children inherit registration.
+A `t.Cleanup` sweep force-kills owned process groups on Unix (process trees on
+Windows) and fails the test if any remain. `TestMain` also checks registered
+children and processes whose executables are under this run's directory, sweeps
+survivors, and fails the run if a per-test cleanup was missed. Sweeps verify
+executable identity before signaling PIDs and never target other test runs.
+`TestFakeEngineCleanupOnFailedTest` exercises cleanup after a fatal assertion.
+
+### Ollama: pinned initial download
+
+Fresh installs use Ollama **v0.40.15** from
+[its GitHub release](https://github.com/ollama/ollama/releases/tag/v0.40.15),
+not a floating latest URL. Each SHA-256 comes from that release's
+[`sha256sum.txt`](https://github.com/ollama/ollama/releases/download/v0.40.15/sha256sum.txt):
+`Ollama-darwin.zip` (both Mac architectures),
+`ollama-windows-amd64.zip`, `ollama-windows-arm64.zip`,
+`ollama-linux-amd64.tar.zst`, and `ollama-linux-arm64.tar.zst`.
+All five artifacts were independently downloaded into a fresh temporary directory
+and hashed without executing or installing them. v0.40.15 exceeds the 0.14.0
+minimum needed for `/v1/messages`. Ollama's own updates continue as before;
+the pin only selects the initial install, not a permanent runtime version.
+
 ### oMLX: the release disk image
 
 oMLX installs the way oMLX recommends: the `.dmg` from its GitHub releases,
@@ -367,6 +407,10 @@ lifecycle lock. A normal status read during an operation returns its current
 snapshot; explicit port checks and model sweeps wait within their caller's
 budget. Observations are discarded if a lifecycle operation or a newer state
 change intervened, so a delayed readiness reply cannot undo stop or set-port.
+Platform readers outside the engine operation lock copy the configuration under
+the state mutex before detection, actions, profile observations or health work;
+settings rollback cannot race those reads. Filesystem capability checks also run
+outside the state mutex.
 
 The five-second model sweep budget includes status probes and operation waits.
 `GET /v1/models` returns HTTP 503 if the sweep cannot complete in time, rather
@@ -554,6 +598,15 @@ SIGKILL escalation), and `taskkill /T /F` on Windows — where the windowless
 engines we spawn can't receive a graceful (non-`/F`) close, so a forced
 terminate is the only signal that actually stops them.
 
+Command-mode stop (including oMLX) gives the official stop CLI
+`max(grace_s, 5s)` to return, then up to `grace_s` (default 5s) for shutdown.
+It confirms three consecutive connection refusals **and**, when the host can
+resolve it, exit of the process observed on the port before stop. Closing the
+port alone does not mean app teardown is complete. The state mutex is never
+held across these waits; lifecycle operations remain serialized so an immediate
+start cannot overtake stop. A timeout leaves the engine running, not falsely
+stopped, and is returned to the caller.
+
 ### Adoption — start may attach to an engine it didn't launch
 
 Before spawning, `engine:start` **probes the chosen port's readiness

services/nvpair-engine-manager/actions.go

@@ -196,7 +196,8 @@ func (e *Executor) runRemovePathAction(ctx context.Context, st *engineState, act
 	root = expandPath(root)
 
 	if act.ModelResolution == modelResolutionLMSDiskPath {
-		cli := controlCLIPath(st.plat, st.installDir, runtime.GOOS)
+		plat, installDir := st.platformSnapshot()
+		cli := controlCLIPath(&plat, installDir, runtime.GOOS)
 		targets, err := e.lmsDiskPathsToDelete(ctx, cli, root, vars["model"])
 		if err != nil {
 			return nil, err
@@ -252,8 +253,9 @@ func (e *Executor) runCmdAction(ctx context.Context, st *engineState, act Action
 		}
 	}
 	vars["port"] = strconv.Itoa(port)
-	vars["install_dir"] = st.installDir
-	if cli := controlCLIPath(st.plat, st.installDir, runtime.GOOS); cli != "" {
+	plat, installDir := st.platformSnapshot()
+	vars["install_dir"] = installDir
+	if cli := controlCLIPath(&plat, installDir, runtime.GOOS); cli != "" {
 		vars["cli"] = cli
 	}
 

services/nvpair-engine-manager/capabilities.go

@@ -41,14 +41,18 @@ func capabilitiesFor(m *Manifest, platform string, adopted bool, installDir stri
 	if !ok {
 		return Capabilities{}
 	}
+	goos := strings.SplitN(platform, "/", 2)[0]
+	return capabilitiesForPlatform(m, p, goos, adopted, installDir)
+}
+
+func capabilitiesForPlatform(m *Manifest, p Platform, goos string, adopted bool, installDir string) Capabilities {
 	_, list := m.Actions["list_models"]
 	_, pull := m.Actions["pull_model"]
 	deleteAction, deleteModel := m.Actions["delete_model"]
 	uninstall := p.Uninstall != nil && (len(p.Uninstall.Run) > 0 || len(p.Uninstall.Remove) > 0) && !adopted
 	if uninstall && p.Runtime.modeOrDefault() == "command" {
 		// Uninstall declines a command-mode engine PAIR has no record of
 		// installing, so offering it would only lead to that refusal.
-		goos := strings.SplitN(platform, "/", 2)[0]
 		creates := ""
 		if p.Install != nil {
 			creates = p.Install.Creates

services/nvpair-engine-manager/capabilities_test.go

@@ -134,7 +134,7 @@ func TestSetPortCapabilityMatchesEngineSetPortRules(t *testing.T) {
 		{"external command with an official stop command", Runtime{Mode: "command", Start: [][]string{{"start", "{port}"}}, Stop: &StopSpec{Cmd: []string{"stop"}}}, true, true},
 	} {
 		t.Run(tc.name, func(t *testing.T) {
-			m := testEngineManifest(fakeEngineBin)
+			m := testEngineManifest(fakeEngineForTest(t))
 			m.Actions = nil // isolate set_port from the action flags
 			for key, p := range m.Platforms {
 				p.Runtime = tc.rt
@@ -174,7 +174,7 @@ func TestCapabilitiesUseEffectiveManifest(t *testing.T) {
 // EngineStatus carries the capabilities: engine:status, engine:get-installed,
 // and the engine:state-changed notification.
 func TestCapabilitiesOnEveryStatusPath(t *testing.T) {
-	m := testEngineManifest(fakeEngineBin)
+	m := testEngineManifest(fakeEngineForTest(t))
 	for key, p := range m.Platforms {
 		p.Detect = nil        // nothing installed
 		p.Runtime.Ready = nil // no probe, so no listener adoption can occur
@@ -229,7 +229,7 @@ func TestCapabilitiesOnSetPortResult(t *testing.T) {
 // registry with no block for this host: get-installed and status still list it,
 // and report every flag false rather than a guess from another platform.
 func TestUnsupportedPlatformReportsNoCapabilities(t *testing.T) {
-	m := testEngineManifest(fakeEngineBin)
+	m := testEngineManifest(fakeEngineForTest(t))
 	m.Platforms = map[string]Platform{
 		"unsupported/platform": m.Platforms[runtime.GOOS+"/"+runtime.GOARCH],
 	}

services/nvpair-engine-manager/cli_resolution_test.go

@@ -31,6 +31,10 @@ func isolatedCommandRegistry(t *testing.T, engine, root string) *Registry {
 			plat.Detect[i] = filepath.Join(root, path)
 		}
 	}
+	for i, path := range plat.DetectRequires {
+		plat.DetectRequires[i] = filepath.Join(root, path)
+		resolutionFile(t, plat.DetectRequires[i])
+	}
 	// Presence/HTTP adoption is covered separately. Keep these filesystem tests
 	// independent of any real service on the engine's default port.
 	plat.Runtime.Port, plat.Runtime.Ready, plat.Runtime.Health = 0, nil, nil
@@ -146,6 +150,9 @@ func TestCommandCLIOverrideIsAuthoritative(t *testing.T) {
 			// shipped macOS platform on any test host and disable real probes.
 			m, _ := reg.Get("omlx")
 			p, _ := m.PlatformFor("darwin", "arm64")
+			for i, path := range p.DetectRequires {
+				p.DetectRequires[i] = filepath.Join(home, path)
+			}
 			p.Runtime.Port, p.Runtime.Ready, p.Runtime.Health = 0, nil, nil
 			m.Platforms = map[string]Platform{runtime.GOOS + "/" + runtime.GOARCH: *p}
 			want := ""
@@ -197,7 +204,7 @@ func TestCommandLifecycleAndActionsExecuteResolvedCLI(t *testing.T) {
 		t.Run(map[bool]string{false: "detected candidate", true: "explicit override"}[override], func(t *testing.T) {
 			root := t.TempDir()
 			cli := filepath.Join(root, "control-cli"+exeExt())
-			copyFile(t, fakeEngineBin, cli)
+			copyFile(t, fakeEngineForTest(t), cli)
 			startMarker, stopMarker := filepath.Join(root, "started"), filepath.Join(root, "stopped")
 			p := Platform{Detect: []string{filepath.Join(root, "missing"), cli}, Runtime: Runtime{
 				Mode:  "command",
@@ -221,7 +228,7 @@ func TestCommandLifecycleAndActionsExecuteResolvedCLI(t *testing.T) {
 				if err := os.MkdirAll(filepath.Join(root, "fake"), 0o700); err != nil {
 					t.Fatal(err)
 				}
-				copyFile(t, fakeEngineBin, filepath.Join(root, "fake", "control-cli"+exeExt()))
+				copyFile(t, fakeEngineForTest(t), filepath.Join(root, "fake", "control-cli"+exeExt()))
 			}
 			t.Cleanup(ex.StopAll)
 			if err := ex.Start(context.Background(), "fake"); err != nil {

services/nvpair-engine-manager/desiredstate.go

@@ -153,5 +153,9 @@ func (e *Executor) restoreEnabled(ctx context.Context, engine string) error {
 	if err != nil || !known || !enabled {
 		return err
 	}
+	// Retain ON intent for a future reinstall, but never launch a missing engine.
+	if installed, err := e.Detect(engine); err != nil || !installed {
+		return err
+	}
 	return e.doStart(ctx, st, engine, startOpts{})
 }

services/nvpair-engine-manager/desiredstate_test.go

@@ -46,13 +46,13 @@ func TestDesiredOnSurvivesShutdownAndExplicitOffDoesNot(t *testing.T) {
 	baseDir := t.TempDir()
 	ctx := context.Background()
 
-	first := desiredTestExecutor(t, testEngineManifest(fakeEngineBin), baseDir)
+	first := desiredTestExecutor(t, testEngineManifest(fakeEngineForTest(t)), baseDir)
 	if err := first.Start(ctx, "fake"); err != nil {
 		t.Fatalf("start: %v", err)
 	}
 	first.StopAll()
 
-	second := desiredTestExecutor(t, testEngineManifest(fakeEngineBin), baseDir)
+	second := desiredTestExecutor(t, testEngineManifest(fakeEngineForTest(t)), baseDir)
 	t.Cleanup(second.StopAll)
 	if err := second.RestoreEnabled(ctx); err != nil {
 		t.Fatalf("restore ON: %v", err)
@@ -64,7 +64,7 @@ func TestDesiredOnSurvivesShutdownAndExplicitOffDoesNot(t *testing.T) {
 		t.Fatalf("explicit stop: %v", err)
 	}
 
-	third := desiredTestExecutor(t, testEngineManifest(fakeEngineBin), baseDir)
+	third := desiredTestExecutor(t, testEngineManifest(fakeEngineForTest(t)), baseDir)
 	if err := third.RestoreEnabled(ctx); err != nil {
 		t.Fatalf("restore after OFF: %v", err)
 	}
@@ -75,7 +75,7 @@ func TestDesiredOnSurvivesShutdownAndExplicitOffDoesNot(t *testing.T) {
 
 func TestRestoreEnabledExcludesUnreconciledEngineWithoutChangingIntent(t *testing.T) {
 	baseDir := t.TempDir()
-	exec := desiredTestExecutor(t, testEngineManifest(fakeEngineBin), baseDir)
+	exec := desiredTestExecutor(t, testEngineManifest(fakeEngineForTest(t)), baseDir)
 	t.Cleanup(exec.StopAll)
 	if err := exec.desired.set("fake", true); err != nil {
 		t.Fatal(err)
@@ -100,13 +100,13 @@ func TestRestoreEnabledExcludesUnreconciledEngineWithoutChangingIntent(t *testin
 
 func TestFailedRestoreRetainsDesiredOnForRetry(t *testing.T) {
 	baseDir := t.TempDir()
-	seed := desiredTestExecutor(t, testEngineManifest(fakeEngineBin), baseDir)
+	seed := desiredTestExecutor(t, testEngineManifest(fakeEngineForTest(t)), baseDir)
 	if err := seed.Start(context.Background(), "fake"); err != nil {
 		t.Fatalf("seed start: %v", err)
 	}
 	seed.StopAll()
 
-	failing := testEngineManifest(fakeEngineBin)
+	failing := testEngineManifest(fakeEngineForTest(t))
 	key := runtime.GOOS + "/" + runtime.GOARCH
 	platform := failing.Platforms[key]
 	platform.Runtime.Args = []string{"noserve"}
@@ -116,7 +116,7 @@ func TestFailedRestoreRetainsDesiredOnForRetry(t *testing.T) {
 		t.Fatal("expected restore failure")
 	}
 
-	retry := desiredTestExecutor(t, testEngineManifest(fakeEngineBin), baseDir)
+	retry := desiredTestExecutor(t, testEngineManifest(fakeEngineForTest(t)), baseDir)
 	t.Cleanup(retry.StopAll)
 	if err := retry.RestoreEnabled(context.Background()); err != nil {
 		t.Fatalf("retry restore: %v", err)
@@ -147,7 +147,7 @@ func TestDeclinedAdoptedStopStillPersistsOffIntent(t *testing.T) {
 
 	// The httptest listener runs in this test process, whose image is not the
 	// fake engine binary, so the stop is declined (never terminated).
-	m := testEngineManifest(fakeEngineBin)
+	m := testEngineManifest(fakeEngineForTest(t))
 	key := runtime.GOOS + "/" + runtime.GOARCH
 	p := m.Platforms[key]
 	p.Runtime.Port = port
@@ -192,14 +192,14 @@ func TestConfirmedStopPersistsOffDespiteCommandExitError(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	manifest := testEngineManifest(fakeEngineBin)
+	manifest := testEngineManifest(fakeEngineForTest(t))
 	key := runtime.GOOS + "/" + runtime.GOARCH
 	platform := manifest.Platforms[key]
 	platform.Runtime = Runtime{
 		Mode:  "command",
 		Port:  port,
 		Ready: &Probe{HTTP: srv.URL, Status: http.StatusOK},
-		Stop:  &StopSpec{Cmd: []string{fakeEngineBin, "failmark", marker}, GraceS: 2},
+		Stop:  &StopSpec{Cmd: []string{fakeEngineForTest(t), "failmark", marker}, GraceS: 2},
 	}
 	manifest.Platforms[key] = platform
 

services/nvpair-engine-manager/download_completion_unix_test.go

@@ -29,7 +29,7 @@ func TestLMSCompletionStopsDownloaderChildAfterLauncherExit(t *testing.T) {
 		}
 	}}
 	go func() {
-		_, err := runLMSDownloadCommand(ctx, []string{fakeEngineBin, "shutdown-download-fixture", marker, "exit-parent", childMarker}, output)
+		_, err := runLMSDownloadCommand(ctx, []string{fakeEngineForTest(t), "shutdown-download-fixture", marker, "exit-parent", childMarker}, output)
 		done <- err
 	}()
 	select {

services/nvpair-engine-manager/download_shutdown_test.go

@@ -77,7 +77,7 @@ func testRunStopsDownloader(t *testing.T, ignore, midCancel, tree, exitEarly, us
 	go func() {
 		defer close(pullDone)
 		_, _ = ex.trackedPull(ctx, "lmstudio", "fixture", "shutdown-pull", true, func(ctx context.Context) (json.RawMessage, error) {
-			argv := []string{fakeEngineBin, "shutdown-download-fixture", marker, mode}
+			argv := []string{fakeEngineForTest(t), "shutdown-download-fixture", marker, mode}
 			if tree {
 				argv = append(argv, childMarker)
 			}

services/nvpair-engine-manager/download_wait_other.go

@@ -0,0 +1,14 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build !darwin && !linux && !windows
+
+package main
+
+// Unsupported targets have no non-reaping exit watcher. Use ordinary Wait;
+// this is a build fallback, not supported detached-downloader supervision.
+func watchDownloadExit(w *downloadWait) error {
+	w.reap()
+	go func() { <-w.finished; close(w.exited) }()
+	return nil
+}

services/nvpair-engine-manager/e2e_test.go

@@ -133,7 +133,7 @@ func TestE2EPrepareShutdownCancelsStartingEngine(t *testing.T) {
 	cfg := t.TempDir()
 	home := t.TempDir()
 	pidFile := filepath.Join(t.TempDir(), "fake.pid")
-	manifest := testEngineManifest(fakeEngineBin)
+	manifest := testEngineManifest(fakeEngineForTest(t))
 	platform := manifest.Platforms[hostKey()]
 	platform.Runtime.Env["FAKE_PID_FILE"] = pidFile
 	platform.Runtime.Env["FAKE_START_DELAY"] = "1m"
@@ -256,7 +256,7 @@ func (m *e2eManager) stop(t *testing.T) {
 }
 
 func writeFakeManifest(t *testing.T, dir string) {
-	writeE2EManifest(t, dir, testEngineManifest(fakeEngineBin))
+	writeE2EManifest(t, dir, testEngineManifest(fakeEngineForTest(t)))
 }
 
 func writeE2EManifest(t *testing.T, dir string, manifest *Manifest) {

services/nvpair-engine-manager/enginenative_test.go

@@ -136,7 +136,7 @@ func TestEngineNativeStatsReportsTheEngineView(t *testing.T) {
 // with no omlx engine has no admin API to ask. That is a normal state, not a
 // failure, so the answer is "unsupported" with no error.
 func TestEngineNativeStatsReportsUnsupportedWhenTheEngineHasNoAdminAPI(t *testing.T) {
-	ex := newTestExecutor(t, testEngineManifest(fakeEngineBin))
+	ex := newTestExecutor(t, testEngineManifest(fakeEngineForTest(t)))
 	got := ex.EngineNativeStats(context.Background())
 	if got.Supported {
 		t.Fatalf("supported = true, want false (%+v)", got)
@@ -330,7 +330,7 @@ func TestEngineNativeActivityReportsTheEngineRows(t *testing.T) {
 // activity view follows the stats view — no admin API means "unsupported",
 // with no error.
 func TestEngineNativeActivityReportsUnsupportedWhenTheEngineHasNoAdminAPI(t *testing.T) {
-	ex := newTestExecutor(t, testEngineManifest(fakeEngineBin))
+	ex := newTestExecutor(t, testEngineManifest(fakeEngineForTest(t)))
 	got := ex.EngineNativeActivity(context.Background())
 	if got.Supported {
 		t.Fatalf("supported = true, want false (%+v)", got)
@@ -347,7 +347,7 @@ func TestEngineNativeActivityReportsUnsupportedWhenTheEngineHasNoAdminAPI(t *tes
 // ride the em surface, so they are served on the same listener behind the
 // same two gates — plaintext for loopback only, mTLS for pinned peers only.
 func TestEngineNativeSurfaceIsGatedLikeTheModelList(t *testing.T) {
-	ex := newTestExecutor(t, testEngineManifest(fakeEngineBin))
+	ex := newTestExecutor(t, testEngineManifest(fakeEngineForTest(t)))
 	plainMux, tlsMux := emSurface(ex, clustertrust.Open(t.TempDir()))
 
 	for _, path := range []string{engineStatsPath, engineActivityPath} {
@@ -408,7 +408,7 @@ func TestEngineNativeSurfaceIsGatedLikeTheModelList(t *testing.T) {
 // loopback. The mTLS personality does not serve the path at all, so no peer
 // can read another node's live engine rows.
 func TestTheLiveActivityPathIsNotServedToPeers(t *testing.T) {
-	ex := newTestExecutor(t, testEngineManifest(fakeEngineBin))
+	ex := newTestExecutor(t, testEngineManifest(fakeEngineForTest(t)))
 	plainMux, tlsMux := emSurface(ex, clustertrust.Open(t.TempDir()))
 
 	// The local read still works: this node's own proxy reads it over