From 659bbab2a1728b38e353e08b473a3351216325a1 Mon Sep 17 00:00:00 2001 From: Jared Ping <13906915+JaredP94@users.noreply.github.com> Date: Fri, 6 Nov 2026 09:16:05 +0200 Subject: [PATCH 1/2] docs(changelog): refresh unreleased notes and validation scope The unreleased notes stopped before the latest routing, engine settings and model-download work. Summarize the merged changes and the upgrade requirements while retaining the existing entries and release boundaries. Record the owner's packaged source-build validation with its untested cases and follow-ups, and put user-facing limitations in known issues. Leave version changes and release naming to the separate release-preparation change. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com> --- CHANGELOG.md | 32 ++++++++++++++++++- docs/known-issues.mdx | 24 +++++++++++++++ docs/release-readiness.mdx | 63 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 118 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca381e3aa..72fd33448 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,12 +12,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +**Upgrade notes:** **Breaking:** A PAIR peer counts for routing, identity, models, +telemetry and workloads only after proving itself over its pinned cluster +connection. A PAIR machine added by address that is not in your cluster shows +"Not in your cluster" with a **Pair** action and is not routed to. Upgrade every +machine: older peers appear unverified until upgraded. To use another machine's +engine without pairing, add the engine's own address as **Engine only**; on a +PAIR machine, its owner must also allow LAN engine binding with +`engine_allow_lan_bind`. Saved manual entries now record **PAIR machine** or +**Engine only** intent; entries saved before this release become **PAIR machine**, +never a routable bare engine without your explicit choice. JSON-RPC additions +for engine settings, model load profiles, download cancellation and LAN-bind +permission, plus peer-verification and progress fields, are additive; no method +was removed. Relay protocol stays v2. + ### Security +- **Only proven peers supply cluster state (`nvpair-node-scanner`, `nvpair-manual-nodes`, `nvpair-ui-broker`)**: Discovery and public inventory are previews, not proof of a machine's identity; PAIR accepts a peer's services, models and telemetry only through its pinned cluster connection (adapted from NVIDIA/Personal-AI-Router#179). Removed members and expired certificates cannot keep cached routing or scheduling authority, and delayed replies from retired workers cannot restore it. - **Pairing rejects out-of-order messages (`eap-noob`, `nvpair-cluster-manager`)**: A completion before the PIN was applied could reach Registered with an empty-key MAC, a key exchange before negotiation crashed either side, and the joiner accepted EAP-Success without a verified completion. A failed pairing message now also releases its pending session. - **Trust-store paths are confined (`nvpair-cluster-manager`)**: A peer identifier such as `../identity` in an endorsed roster entry could write a pin outside `trusted/`. Unsafe identifiers are refused. - **Bounded reads and fail-closed peers**: Node-info, model inventories, error-store ingest and remote progress frames are size-limited; a proxy whose peer pin has gone refuses the dial instead of falling back to an unpinned transport. +### Changed +- **Discovery reuses its sockets (`nvpair-node-scanner`, discovery responders)**: mDNS scans reuse sockets and refresh them on network changes rather than recreating them every scan. A three-minute synthetic test on one Mac measured 371 → 7 UDP socket opens, including startup and shutdown; this is not a long-duration reliability guarantee (NVIDIA/Personal-AI-Router#19, NVIDIA/Personal-AI-Router#59). + ### Fixed +- **Stalled nodes no longer delay every request (`ollama-proxy`, `lmstudio-proxy`, `omlx-proxy`)**: A node that stalls before headers moves behind eligible peers for one response-header timeout, then returns to normal on expiry or success. A dropped response can retry before its first body bytes without creating another visible workload; a committed stream is never retried (NVIDIA/Personal-AI-Router#6, NVIDIA/Personal-AI-Router#144). +- **Local port conflicts are reported (`services`)**: Another program listening on `127.0.0.1` is now detected when PAIR starts or changes a port. A separately running Ollama app on `127.0.0.1:11434`, for example, is reported as a conflict instead of silently receiving local traffic intended for PAIR. +- **Engine controls stay responsive (`nvpair-engine-manager`)**: Slow status probes no longer block start, stop or port changes, and an over-budget model inventory leaves peers with their last-good list rather than an empty one. Shutdown stops engines concurrently and cleans up detached daemons even when their start command fails or is cancelled (NVIDIA/Personal-AI-Router#84, NVIDIA/Personal-AI-Router#180, NVIDIA/Personal-AI-Router#13, NVIDIA/Personal-AI-Router#81). +- **Copied endpoints and download errors are clearer (`desktop`)**: LM Studio and oMLX copy OpenAI-compatible URLs ending in `/v1`; Ollama keeps its native API root. Recognized model-download failures keep the engine's error and add next steps, rather than guessing a cause for ambiguous errors (NVIDIA/Personal-AI-Router#152). +- **Departed machines do not leave jobs running (`nvpair-ui-broker`)**: In-flight workloads on a disappearing unverified machine are marked failed instead of remaining active indefinitely. - **Uninstalling an engine keeps your models and settings (`nvpair-engine-manager`)**: LM Studio uninstall ran `rm -rf ~/.lmstudio` and oMLX `rm -rf ~/.omlx`. Each manifest now declares its model store, removal is confined and preserves it, and only the runtime is removed. PAIR uninstalls a vendor-installed engine only if it recorded installing it (engines installed before this release: use the vendor's uninstaller). - **Stopping a leftover engine works on macOS (`nvpair-engine-manager`)**: The process-image lookup read `/proc`, so stopping an engine PAIR lost the handle to was always refused. - **Manual nodes survive worker restarts (`nvpair-ui-broker`, `desktop`)**: The broker owns `configs/manual-nodes.json` and replays it into every new manual-nodes worker, so Tailscale and VPN peers are no longer lost when the worker restarts or the terminal interface drives the broker. @@ -25,9 +48,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Workload frames are broadcast in order (`nvpair-workload-manager`)**: A removal can no longer overtake its upsert and resurrect a job on a peer, and a failed emit's retry is no longer swallowed as a duplicate. - **mDNS sends from UDP 5353**: Queries and responses use the RFC 6762 source port. - **The local node's inventory refreshes (`nvpair-node-scanner`)**: VRAM, CPU and memory in the discovery snapshot no longer stay at their startup values. -- Also: IPv6 link-local pairing keeps a locally valid zone; LM Studio and oMLX chat actions and every engine's model load (locally and from a peer) wait up to 10 minutes for headers; engine health probes reuse connections; every GPU shows on a node card; the renderer CSP is tighter; a closed stdout no longer crashes the app; model search finds Ollama models newer than the bundled catalog. +- Also: IPv6 link-local pairing keeps a locally valid zone; LM Studio and oMLX chat actions and every engine's model load (locally and from a peer) wait up to 10 minutes for headers; engine health probes reuse connections; every GPU shows on a node card; the renderer CSP is tighter; a closed stdout no longer crashes the app; model search finds Ollama models newer than the bundled catalog; macOS packages declare Local Network and Bonjour permission keys (not a discovery fix); broker shutdown joins workers concurrently within its deadline; install stages show without invented percentages. ### Added +- **Editable engine launch settings (`desktop`, `nvpair-ui-broker`, `nvpair-engine-manager`)**: Set ports, launch arguments and environment variables for this device and verified peers, including `OLLAMA_*` variables (NVIDIA/Personal-AI-Router#154). Bind-address overrides are rejected and a failed apply restores the previous settings, reporting any restoration failure; oMLX shows only its proxy port because its server settings live in the oMLX app. +- **Anthropic Messages through every proxy (`ollama-proxy`, `lmstudio-proxy`, `omlx-proxy`)**: `/v1/messages` routes directly to engines with native support: Ollama 0.14.0+, LM Studio 0.4.1+ and oMLX 0.1.0+, including streaming. Adapted from NVIDIA/Personal-AI-Router#27 and NVIDIA/Personal-AI-Router@7b21d1f; PAIR does not translate protocols or promise Anthropic feature parity. +- **LM Studio load options per model and device (`desktop`, `nvpair-engine-manager`)**: Save an allowlisted set of tuning options, including GPU use, context length and parallelism, locally or on verified peers for the next load (NVIDIA/Personal-AI-Router#29). PAIR never unloads a model automatically: "Loaded with different options — reload to apply" means explicitly **Eject**, then **Load**, to apply the saved options. +- **Model download progress and cancellation (`desktop`, `nvpair-tui`, `nvpair-engine-manager`)**: Ollama and LM Studio show live progress locally and on verified peers, accept cancellation asynchronously and confirm when it stops; interrupted downloads can be retried. Adapted from NVIDIA/Personal-AI-Router#111 with the asynchronous cancellation of NVIDIA/Personal-AI-Router#115, but PAIR deliberately keeps partial data for resuming and never deletes cache files; oMLX has no PAIR download cancellation. +- **Saved nodes by IP address or hostname (`desktop`, `nvpair-tui`, `nvpair-manual-nodes`)**: Choose **PAIR machine** to save an address and pair, or **Engine only** to use an engine directly without trusting PAIR identity or telemetry claims (NVIDIA/Personal-AI-Router#10, NVIDIA/Personal-AI-Router#92). Failed pairing keeps the entry for retry or removal, and PAIR intent survives restarts without becoming an engine-only route. +- **Clear peer-verification labels (`desktop`, `nvpair-tui`)**: Unverified machines show "Not in your cluster" with **Pair**; direct engine entries show "Engine only — added by address" without peer controls or telemetry. Previews stay separate from verified machines even if they claim the same identity. +- **Optional first-body deadline (`ollama-proxy`, `lmstudio-proxy`, `omlx-proxy`)**: `--first-body-timeout` or `NVPAIR_PROXY_FIRST_BODY_TIMEOUT` opts into retry and cooldown when an engine sends headers but no body. It is disabled by default so legitimate long prefill can wait; a short deadline can duplicate work on another node. - **oMLX installs from its release disk image (`nvpair-engine-manager`, `desktop`)**: Installing oMLX from PAIR now follows oMLX's recommended path: the checksum-pinned `.dmg` from its GitHub releases (the macOS 26–27 or macOS 15 build, whichever fits), copied to `/Applications` and opened so the app writes its `~/.omlx/bin/omlx` CLI shim; the app updates itself from then on. The previous `install.sh` source no longer exists. An `oMLX.app` already in `/Applications` is opened, not replaced, and is never offered for uninstall; uninstalling one PAIR installed removes the app and its two shims and keeps the rest of `~/.omlx`. After oMLX's in-app updater replaces the bundle, PAIR no longer offers uninstall. The desktop's oMLX links now point at the oMLX repository and releases. - **Configurable proxy header timeout**: `--response-header-timeout` or `NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT` (default unchanged at 120 s) for requests that queue or wait on slow model loads. diff --git a/docs/known-issues.mdx b/docs/known-issues.mdx index 4d7b7b068..e2081a5e1 100644 --- a/docs/known-issues.mdx +++ b/docs/known-issues.mdx @@ -112,6 +112,30 @@ second-machine reproduction is still needed before choosing a remedy; no speculative fix has been made. Until then, restart PAIR on the affected Mac. This remains open in [release readiness](release-readiness.mdx#known-limitations-spec-9). +## Roaming Peers Need a Saved Address + +A cluster member keeps the LAN address recorded when it paired. If that machine +changes networks, its peer may no longer reach it without a saved address. +On the peer that needs to connect, add a reachable IP address or hostname as +**PAIR machine**, such as its Tailscale address; a saved address does not replace +pairing or verification. + +## Leftover oMLX Launchers Count as Installed + +A leftover `~/.omlx` launcher can make PAIR report oMLX as installed even when +`/Applications/oMLX.app` is absent, and PAIR retries starting it on each launch. +This also affects 0.8.0. Restore the oMLX app or use the vendor's cleanup +instructions for a stale installation; PAIR's installed status alone does not +prove the app bundle is present. + +## Ollama Install Download Is Not Checksum-Pinned + +PAIR installs Ollama from the vendor's floating latest-download URL without a +pinned checksum and logs `UNPINNED download`. This is unchanged since 0.8.0; +download progress is not an integrity check. If a pinned artifact is required, +install an independently verified Ollama release yourself instead of using +PAIR's install action. + ## The Terminal Interface Cannot Do Everything the Desktop App Can **Implemented in source, pending release testing.** `nvpair-tui` now shows diff --git a/docs/release-readiness.mdx b/docs/release-readiness.mdx index f234dfba6..203d33686 100644 --- a/docs/release-readiness.mdx +++ b/docs/release-readiness.mdx @@ -10,6 +10,69 @@ notice audit. It describes the current checkout, not a certification of the next release. No release was published and no installed app was replaced by this audit. +## Packaged-app validation (2026-11-05) + +**Owner's source build, not certification of a published artifact.** Notes dated +2026-11-05/06 cover a macOS arm64 package built from `44611f9`, ad-hoc signed +and unnotarized, still labelled desktop `0.8.0` / service product `2.6.0`. +It was installed over existing 0.8.0 data on two of the owner's Macs +(macOS 27.2, arm64, 128 GB), each fully backed up first. Validation drove the +packaged broker through the same JSON-RPC calls as the desktop and relaunched +the app; it was **not a desktop UI click-through**. + +### Validated + +- Broker and all twelve workers started with relay protocol 2. Existing manual + nodes and cluster files survived; a saved entry without a kind migrated to + `pair`. +- Ollama installed with increasing download progress. PAIR started and stopped + Ollama, LM Studio and oMLX; the oMLX installations were development builds. +- Model downloads showed progress, accepted cancellation without deleting + partial data, became interrupted after engine-manager was killed and + automatically restarted, and retried to completion. +- Ollama inference returned HTTP 200 through `/api/chat`, + `/v1/chat/completions` and `/v1/messages`, streaming and non-streaming. +- Local and remote launch settings applied a harmless change, restarted the + engine, rejected an `OLLAMA_HOST` override and restored the exact settings. + LM Studio load-option validation accepted allowed options and rejected + execution-directory controls, unknown flags and short flags; no saved-profile + model load was exercised. +- A controlled listener on `127.0.0.1` made a proxy-port change fail up front + with an occupied-port error, leaving settings unchanged. +- With two machines in one cluster using a saved Tailscale address, one verified + the other over pinned cluster mTLS. Remote engine start and all six inference + paths above reached the peer while the local Mac had no Ollama; remote + download cancellation and re-download worked. The saved entry stayed `pair` + across a manual-nodes worker restart. + +### Still unvalidated + +Windows/Linux runtime, Intel Macs, an actual LM Studio load with a saved +profile, oMLX installation from its release disk image, real-machine unpairing, +and verification in the reverse direction remain untested. Rollback after an +engine has already stopped has unit coverage only: the port-conflict test +refused the change before anything stopped. Desktop UI interaction and +published release assets were not validated. + +### Follow-ups + +The validation notes record these as non-blocking follow-ups, not fixes: + +- [Roaming peer addresses](known-issues.mdx#roaming-peers-need-a-saved-address), + [leftover oMLX launchers](known-issues.mdx#leftover-omlx-launchers-count-as-installed) + and [Ollama's unpinned install download](known-issues.mdx#ollama-install-download-is-not-checksum-pinned) + have user-facing limitations and workarounds. +- Starting oMLX 0.7.4.dev1 immediately after stopping it failed; waiting briefly + allowed it to start. A startup scanner probe can warn before node-info is up. +- Windows downloads use job objects, but engine stopping still uses + `taskkill`; runtime validation and cleanup unification remain follow-ups. +- JSON-RPC notifications have no write deadline, so a stalled stdout reader can + stall lock-holding callers. Engine-manager tests can leave fake-engine + helpers running, and engine-manager does not build for FreeBSD. + +llama.cpp is deferred to a later release because it needs a thirteenth proxy +worker, unsigned-binary review and its own download-cancellation path. + ## Documentation and Installer Maintenance - [x] Backfill `0.7.0-rc.1` in the changelog using its published notes and source; From 7d69dbaf02ddb32c877fb3d5c4206d4034855e21 Mon Sep 17 00:00:00 2001 From: Jared Ping <13906915+JaredP94@users.noreply.github.com> Date: Fri, 6 Nov 2026 09:33:07 +0200 Subject: [PATCH 2/2] docs(changelog): correct LM Studio load limit and note unpair cleanup Review follow-up. LM Studio loads now run under a 4-minute CLI limit rather than the 10-minute header wait, and removing a machine from the cluster forgets its saved PAIR address. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com> --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 72fd33448..a62b2f1a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,14 +48,14 @@ was removed. Relay protocol stays v2. - **Workload frames are broadcast in order (`nvpair-workload-manager`)**: A removal can no longer overtake its upsert and resurrect a job on a peer, and a failed emit's retry is no longer swallowed as a duplicate. - **mDNS sends from UDP 5353**: Queries and responses use the RFC 6762 source port. - **The local node's inventory refreshes (`nvpair-node-scanner`)**: VRAM, CPU and memory in the discovery snapshot no longer stay at their startup values. -- Also: IPv6 link-local pairing keeps a locally valid zone; LM Studio and oMLX chat actions and every engine's model load (locally and from a peer) wait up to 10 minutes for headers; engine health probes reuse connections; every GPU shows on a node card; the renderer CSP is tighter; a closed stdout no longer crashes the app; model search finds Ollama models newer than the bundled catalog; macOS packages declare Local Network and Bonjour permission keys (not a discovery fix); broker shutdown joins workers concurrently within its deadline; install stages show without invented percentages. +- Also: IPv6 link-local pairing keeps a locally valid zone; LM Studio and oMLX chat actions and Ollama and oMLX model loads (locally and from a peer) wait up to 10 minutes for headers, and LM Studio loads run under a 4-minute limit; engine health probes reuse connections; every GPU shows on a node card; the renderer CSP is tighter; a closed stdout no longer crashes the app; model search finds Ollama models newer than the bundled catalog; macOS packages declare Local Network and Bonjour permission keys (not a discovery fix); broker shutdown joins workers concurrently within its deadline; install stages show without invented percentages. ### Added - **Editable engine launch settings (`desktop`, `nvpair-ui-broker`, `nvpair-engine-manager`)**: Set ports, launch arguments and environment variables for this device and verified peers, including `OLLAMA_*` variables (NVIDIA/Personal-AI-Router#154). Bind-address overrides are rejected and a failed apply restores the previous settings, reporting any restoration failure; oMLX shows only its proxy port because its server settings live in the oMLX app. - **Anthropic Messages through every proxy (`ollama-proxy`, `lmstudio-proxy`, `omlx-proxy`)**: `/v1/messages` routes directly to engines with native support: Ollama 0.14.0+, LM Studio 0.4.1+ and oMLX 0.1.0+, including streaming. Adapted from NVIDIA/Personal-AI-Router#27 and NVIDIA/Personal-AI-Router@7b21d1f; PAIR does not translate protocols or promise Anthropic feature parity. - **LM Studio load options per model and device (`desktop`, `nvpair-engine-manager`)**: Save an allowlisted set of tuning options, including GPU use, context length and parallelism, locally or on verified peers for the next load (NVIDIA/Personal-AI-Router#29). PAIR never unloads a model automatically: "Loaded with different options — reload to apply" means explicitly **Eject**, then **Load**, to apply the saved options. - **Model download progress and cancellation (`desktop`, `nvpair-tui`, `nvpair-engine-manager`)**: Ollama and LM Studio show live progress locally and on verified peers, accept cancellation asynchronously and confirm when it stops; interrupted downloads can be retried. Adapted from NVIDIA/Personal-AI-Router#111 with the asynchronous cancellation of NVIDIA/Personal-AI-Router#115, but PAIR deliberately keeps partial data for resuming and never deletes cache files; oMLX has no PAIR download cancellation. -- **Saved nodes by IP address or hostname (`desktop`, `nvpair-tui`, `nvpair-manual-nodes`)**: Choose **PAIR machine** to save an address and pair, or **Engine only** to use an engine directly without trusting PAIR identity or telemetry claims (NVIDIA/Personal-AI-Router#10, NVIDIA/Personal-AI-Router#92). Failed pairing keeps the entry for retry or removal, and PAIR intent survives restarts without becoming an engine-only route. +- **Saved nodes by IP address or hostname (`desktop`, `nvpair-tui`, `nvpair-manual-nodes`)**: Choose **PAIR machine** to save an address and pair, or **Engine only** to use an engine directly without trusting PAIR identity or telemetry claims (NVIDIA/Personal-AI-Router#10, NVIDIA/Personal-AI-Router#92). Failed pairing keeps the entry for retry or removal, and PAIR intent survives restarts without becoming an engine-only route. Removing a machine from the cluster also forgets its saved PAIR address; Engine only entries stay until you remove them. - **Clear peer-verification labels (`desktop`, `nvpair-tui`)**: Unverified machines show "Not in your cluster" with **Pair**; direct engine entries show "Engine only — added by address" without peer controls or telemetry. Previews stay separate from verified machines even if they claim the same identity. - **Optional first-body deadline (`ollama-proxy`, `lmstudio-proxy`, `omlx-proxy`)**: `--first-body-timeout` or `NVPAIR_PROXY_FIRST_BODY_TIMEOUT` opts into retry and cooldown when an engine sends headers but no body. It is disabled by default so legitimate long prefill can wait; a short deadline can duplicate work on another node. - **oMLX installs from its release disk image (`nvpair-engine-manager`, `desktop`)**: Installing oMLX from PAIR now follows oMLX's recommended path: the checksum-pinned `.dmg` from its GitHub releases (the macOS 26–27 or macOS 15 build, whichever fits), copied to `/Applications` and opened so the app writes its `~/.omlx/bin/omlx` CLI shim; the app updates itself from then on. The previous `install.sh` source no longer exists. An `oMLX.app` already in `/Applications` is opened, not replaced, and is never offered for uninstall; uninstalling one PAIR installed removes the app and its two shims and keeps the rest of `~/.omlx`. After oMLX's in-app updater replaces the bundle, PAIR no longer offers uninstall. The desktop's oMLX links now point at the oMLX repository and releases.