Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,22 +12,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

**Upgrade notes:** **Breaking:** A PAIR peer counts for routing, identity, models,
telemetry and workloads only after proving itself over its pinned cluster
connection. A PAIR machine added by address that is not in your cluster shows
"Not in your cluster" with a **Pair** action and is not routed to. Upgrade every
machine: older peers appear unverified until upgraded. To use another machine's
engine without pairing, add the engine's own address as **Engine only**; on a
PAIR machine, its owner must also allow LAN engine binding with
`engine_allow_lan_bind`. Saved manual entries now record **PAIR machine** or
**Engine only** intent; entries saved before this release become **PAIR machine**,
never a routable bare engine without your explicit choice. JSON-RPC additions
for engine settings, model load profiles, download cancellation and LAN-bind
permission, plus peer-verification and progress fields, are additive; no method
was removed. Relay protocol stays v2.

### Security
- **Only proven peers supply cluster state (`nvpair-node-scanner`, `nvpair-manual-nodes`, `nvpair-ui-broker`)**: Discovery and public inventory are previews, not proof of a machine's identity; PAIR accepts a peer's services, models and telemetry only through its pinned cluster connection (adapted from NVIDIA/Personal-AI-Router#179). Removed members and expired certificates cannot keep cached routing or scheduling authority, and delayed replies from retired workers cannot restore it.
- **Pairing rejects out-of-order messages (`eap-noob`, `nvpair-cluster-manager`)**: A completion before the PIN was applied could reach Registered with an empty-key MAC, a key exchange before negotiation crashed either side, and the joiner accepted EAP-Success without a verified completion. A failed pairing message now also releases its pending session.
- **Trust-store paths are confined (`nvpair-cluster-manager`)**: A peer identifier such as `../identity` in an endorsed roster entry could write a pin outside `trusted/`. Unsafe identifiers are refused.
- **Bounded reads and fail-closed peers**: Node-info, model inventories, error-store ingest and remote progress frames are size-limited; a proxy whose peer pin has gone refuses the dial instead of falling back to an unpinned transport.

### Changed
- **Discovery reuses its sockets (`nvpair-node-scanner`, discovery responders)**: mDNS scans reuse sockets and refresh them on network changes rather than recreating them every scan. A three-minute synthetic test on one Mac measured 371 → 7 UDP socket opens, including startup and shutdown; this is not a long-duration reliability guarantee (NVIDIA/Personal-AI-Router#19, NVIDIA/Personal-AI-Router#59).

### Fixed
- **Stalled nodes no longer delay every request (`ollama-proxy`, `lmstudio-proxy`, `omlx-proxy`)**: A node that stalls before headers moves behind eligible peers for one response-header timeout, then returns to normal on expiry or success. A dropped response can retry before its first body bytes without creating another visible workload; a committed stream is never retried (NVIDIA/Personal-AI-Router#6, NVIDIA/Personal-AI-Router#144).
- **Local port conflicts are reported (`services`)**: Another program listening on `127.0.0.1` is now detected when PAIR starts or changes a port. A separately running Ollama app on `127.0.0.1:11434`, for example, is reported as a conflict instead of silently receiving local traffic intended for PAIR.
- **Engine controls stay responsive (`nvpair-engine-manager`)**: Slow status probes no longer block start, stop or port changes, and an over-budget model inventory leaves peers with their last-good list rather than an empty one. Shutdown stops engines concurrently and cleans up detached daemons even when their start command fails or is cancelled (NVIDIA/Personal-AI-Router#84, NVIDIA/Personal-AI-Router#180, NVIDIA/Personal-AI-Router#13, NVIDIA/Personal-AI-Router#81).
- **Copied endpoints and download errors are clearer (`desktop`)**: LM Studio and oMLX copy OpenAI-compatible URLs ending in `/v1`; Ollama keeps its native API root. Recognized model-download failures keep the engine's error and add next steps, rather than guessing a cause for ambiguous errors (NVIDIA/Personal-AI-Router#152).
- **Departed machines do not leave jobs running (`nvpair-ui-broker`)**: In-flight workloads on a disappearing unverified machine are marked failed instead of remaining active indefinitely.
- **Uninstalling an engine keeps your models and settings (`nvpair-engine-manager`)**: LM Studio uninstall ran `rm -rf ~/.lmstudio` and oMLX `rm -rf ~/.omlx`. Each manifest now declares its model store, removal is confined and preserves it, and only the runtime is removed. PAIR uninstalls a vendor-installed engine only if it recorded installing it (engines installed before this release: use the vendor's uninstaller).
- **Stopping a leftover engine works on macOS (`nvpair-engine-manager`)**: The process-image lookup read `/proc`, so stopping an engine PAIR lost the handle to was always refused.
- **Manual nodes survive worker restarts (`nvpair-ui-broker`, `desktop`)**: The broker owns `configs/manual-nodes.json` and replays it into every new manual-nodes worker, so Tailscale and VPN peers are no longer lost when the worker restarts or the terminal interface drives the broker.
- **Jobs that share a number stay apart (`desktop`, `nvpair-tui`)**: Jobs are keyed by origin, engine, run and id, so an Ollama job and an oMLX job with the same number no longer share a card or a row.
- **Workload frames are broadcast in order (`nvpair-workload-manager`)**: A removal can no longer overtake its upsert and resurrect a job on a peer, and a failed emit's retry is no longer swallowed as a duplicate.
- **mDNS sends from UDP 5353**: Queries and responses use the RFC 6762 source port.
- **The local node's inventory refreshes (`nvpair-node-scanner`)**: VRAM, CPU and memory in the discovery snapshot no longer stay at their startup values.
- Also: IPv6 link-local pairing keeps a locally valid zone; LM Studio and oMLX chat actions and every engine's model load (locally and from a peer) wait up to 10 minutes for headers; engine health probes reuse connections; every GPU shows on a node card; the renderer CSP is tighter; a closed stdout no longer crashes the app; model search finds Ollama models newer than the bundled catalog.
- Also: IPv6 link-local pairing keeps a locally valid zone; LM Studio and oMLX chat actions and Ollama and oMLX model loads (locally and from a peer) wait up to 10 minutes for headers, and LM Studio loads run under a 4-minute limit; engine health probes reuse connections; every GPU shows on a node card; the renderer CSP is tighter; a closed stdout no longer crashes the app; model search finds Ollama models newer than the bundled catalog; macOS packages declare Local Network and Bonjour permission keys (not a discovery fix); broker shutdown joins workers concurrently within its deadline; install stages show without invented percentages.

### Added
- **Editable engine launch settings (`desktop`, `nvpair-ui-broker`, `nvpair-engine-manager`)**: Set ports, launch arguments and environment variables for this device and verified peers, including `OLLAMA_*` variables (NVIDIA/Personal-AI-Router#154). Bind-address overrides are rejected and a failed apply restores the previous settings, reporting any restoration failure; oMLX shows only its proxy port because its server settings live in the oMLX app.
- **Anthropic Messages through every proxy (`ollama-proxy`, `lmstudio-proxy`, `omlx-proxy`)**: `/v1/messages` routes directly to engines with native support: Ollama 0.14.0+, LM Studio 0.4.1+ and oMLX 0.1.0+, including streaming. Adapted from NVIDIA/Personal-AI-Router#27 and NVIDIA/Personal-AI-Router@7b21d1f; PAIR does not translate protocols or promise Anthropic feature parity.
- **LM Studio load options per model and device (`desktop`, `nvpair-engine-manager`)**: Save an allowlisted set of tuning options, including GPU use, context length and parallelism, locally or on verified peers for the next load (NVIDIA/Personal-AI-Router#29). PAIR never unloads a model automatically: "Loaded with different options — reload to apply" means explicitly **Eject**, then **Load**, to apply the saved options.
- **Model download progress and cancellation (`desktop`, `nvpair-tui`, `nvpair-engine-manager`)**: Ollama and LM Studio show live progress locally and on verified peers, accept cancellation asynchronously and confirm when it stops; interrupted downloads can be retried. Adapted from NVIDIA/Personal-AI-Router#111 with the asynchronous cancellation of NVIDIA/Personal-AI-Router#115, but PAIR deliberately keeps partial data for resuming and never deletes cache files; oMLX has no PAIR download cancellation.
- **Saved nodes by IP address or hostname (`desktop`, `nvpair-tui`, `nvpair-manual-nodes`)**: Choose **PAIR machine** to save an address and pair, or **Engine only** to use an engine directly without trusting PAIR identity or telemetry claims (NVIDIA/Personal-AI-Router#10, NVIDIA/Personal-AI-Router#92). Failed pairing keeps the entry for retry or removal, and PAIR intent survives restarts without becoming an engine-only route. Removing a machine from the cluster also forgets its saved PAIR address; Engine only entries stay until you remove them.
- **Clear peer-verification labels (`desktop`, `nvpair-tui`)**: Unverified machines show "Not in your cluster" with **Pair**; direct engine entries show "Engine only — added by address" without peer controls or telemetry. Previews stay separate from verified machines even if they claim the same identity.
- **Optional first-body deadline (`ollama-proxy`, `lmstudio-proxy`, `omlx-proxy`)**: `--first-body-timeout` or `NVPAIR_PROXY_FIRST_BODY_TIMEOUT` opts into retry and cooldown when an engine sends headers but no body. It is disabled by default so legitimate long prefill can wait; a short deadline can duplicate work on another node.
- **oMLX installs from its release disk image (`nvpair-engine-manager`, `desktop`)**: Installing oMLX from PAIR now follows oMLX's recommended path: the checksum-pinned `.dmg` from its GitHub releases (the macOS 26–27 or macOS 15 build, whichever fits), copied to `/Applications` and opened so the app writes its `~/.omlx/bin/omlx` CLI shim; the app updates itself from then on. The previous `install.sh` source no longer exists. An `oMLX.app` already in `/Applications` is opened, not replaced, and is never offered for uninstall; uninstalling one PAIR installed removes the app and its two shims and keeps the rest of `~/.omlx`. After oMLX's in-app updater replaces the bundle, PAIR no longer offers uninstall. The desktop's oMLX links now point at the oMLX repository and releases.
- **Configurable proxy header timeout**: `--response-header-timeout` or `NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT` (default unchanged at 120 s) for requests that queue or wait on slow model loads.

Expand Down
24 changes: 24 additions & 0 deletions docs/known-issues.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,30 @@ second-machine reproduction is still needed before choosing a remedy; no
speculative fix has been made. Until then, restart PAIR on the affected Mac.
This remains open in [release readiness](release-readiness.mdx#known-limitations-spec-9).

## Roaming Peers Need a Saved Address

A cluster member keeps the LAN address recorded when it paired. If that machine
changes networks, its peer may no longer reach it without a saved address.
On the peer that needs to connect, add a reachable IP address or hostname as
**PAIR machine**, such as its Tailscale address; a saved address does not replace
pairing or verification.

## Leftover oMLX Launchers Count as Installed

A leftover `~/.omlx` launcher can make PAIR report oMLX as installed even when
`/Applications/oMLX.app` is absent, and PAIR retries starting it on each launch.
This also affects 0.8.0. Restore the oMLX app or use the vendor's cleanup
instructions for a stale installation; PAIR's installed status alone does not
prove the app bundle is present.

## Ollama Install Download Is Not Checksum-Pinned

PAIR installs Ollama from the vendor's floating latest-download URL without a
pinned checksum and logs `UNPINNED download`. This is unchanged since 0.8.0;
download progress is not an integrity check. If a pinned artifact is required,
install an independently verified Ollama release yourself instead of using
PAIR's install action.

## The Terminal Interface Cannot Do Everything the Desktop App Can

**Implemented in source, pending release testing.** `nvpair-tui` now shows
Expand Down
63 changes: 63 additions & 0 deletions docs/release-readiness.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,69 @@ notice audit. It describes the current checkout, not a certification of the next
release. No release was published and no installed app was replaced by this
audit.

## Packaged-app validation (2026-11-05)

**Owner's source build, not certification of a published artifact.** Notes dated
2026-11-05/06 cover a macOS arm64 package built from `44611f9`, ad-hoc signed
and unnotarized, still labelled desktop `0.8.0` / service product `2.6.0`.
It was installed over existing 0.8.0 data on two of the owner's Macs
(macOS 27.2, arm64, 128 GB), each fully backed up first. Validation drove the
packaged broker through the same JSON-RPC calls as the desktop and relaunched
the app; it was **not a desktop UI click-through**.

### Validated

- Broker and all twelve workers started with relay protocol 2. Existing manual
nodes and cluster files survived; a saved entry without a kind migrated to
`pair`.
- Ollama installed with increasing download progress. PAIR started and stopped
Ollama, LM Studio and oMLX; the oMLX installations were development builds.
- Model downloads showed progress, accepted cancellation without deleting
partial data, became interrupted after engine-manager was killed and
automatically restarted, and retried to completion.
- Ollama inference returned HTTP 200 through `/api/chat`,
`/v1/chat/completions` and `/v1/messages`, streaming and non-streaming.
- Local and remote launch settings applied a harmless change, restarted the
engine, rejected an `OLLAMA_HOST` override and restored the exact settings.
LM Studio load-option validation accepted allowed options and rejected
execution-directory controls, unknown flags and short flags; no saved-profile
model load was exercised.
- A controlled listener on `127.0.0.1` made a proxy-port change fail up front
with an occupied-port error, leaving settings unchanged.
- With two machines in one cluster using a saved Tailscale address, one verified
the other over pinned cluster mTLS. Remote engine start and all six inference
paths above reached the peer while the local Mac had no Ollama; remote
download cancellation and re-download worked. The saved entry stayed `pair`
across a manual-nodes worker restart.

### Still unvalidated

Windows/Linux runtime, Intel Macs, an actual LM Studio load with a saved
profile, oMLX installation from its release disk image, real-machine unpairing,
and verification in the reverse direction remain untested. Rollback after an
engine has already stopped has unit coverage only: the port-conflict test
refused the change before anything stopped. Desktop UI interaction and
published release assets were not validated.

### Follow-ups

The validation notes record these as non-blocking follow-ups, not fixes:

- [Roaming peer addresses](known-issues.mdx#roaming-peers-need-a-saved-address),
[leftover oMLX launchers](known-issues.mdx#leftover-omlx-launchers-count-as-installed)
and [Ollama's unpinned install download](known-issues.mdx#ollama-install-download-is-not-checksum-pinned)
have user-facing limitations and workarounds.
- Starting oMLX 0.7.4.dev1 immediately after stopping it failed; waiting briefly
allowed it to start. A startup scanner probe can warn before node-info is up.
- Windows downloads use job objects, but engine stopping still uses
`taskkill`; runtime validation and cleanup unification remain follow-ups.
- JSON-RPC notifications have no write deadline, so a stalled stdout reader can
stall lock-holding callers. Engine-manager tests can leave fake-engine
helpers running, and engine-manager does not build for FreeBSD.

llama.cpp is deferred to a later release because it needs a thirteenth proxy
worker, unsigned-binary review and its own download-cancellation path.

## Documentation and Installer Maintenance

- [x] Backfill `0.7.0-rc.1` in the changelog using its published notes and source;
Expand Down
Loading