Repository navigation
Serve llama.cpp through its own proxy worker - #61
Merged
Merged
Conversation
Copy the LM Studio worker without changing its behavior, retaining routing, streaming, retry, workload and peer-ingress coverage. Give llama.cpp its own service identity and port 1237, exact model IDs and both model-list paths mapped to /models. Retain native Messages support verified in b12171. Adapt model-list semantics from NVIDIA/Personal-AI-Router@6d2d67f. Co-authored-by: Terve <ntervalon@nvidia.com> Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Validate --model and -m through the closed b12171 launch policy, resolving both cache and file symlinks and requiring a contained regular GGUF. Preserve no-model router starts, and use the selected model path as the single-model readiness and health identity. Publish the server inventory through the existing read-only list_models action. Engine-manager remains 1.5.0, the additive bump already present relative to v0.9.0. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
A cache-relative filename could be parsed as a file URI without containing ://. Reject explicit URI schemes in non-absolute selections even when a same-named cache file exists. TestLlamaModelPathValidation was observed failing for file:model.gguf before this guard and passing afterward. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Supervise llamacpp-proxy alongside the existing proxies, with manual hang restarts, bounded writes, settings convergence, telemetry and terminal peer ingress. Advertise lc, preserve per-engine model attribution, and schedule llama.cpp work independently. Extend explicit bare-engine and verified-peer projections without granting previews routing authority. Remove the zero-proxy-port settings exception and expose the reported proxy port in desktop and TUI. Update the 13-worker and 15-executable inventories, installers, CI routing coverage and generated contracts. Keep existing release evidence separate from the new source behavior. Add broker, discovery, manual probe and cross-process coverage; the latter is CI-only. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Open selected GGUF files read-only without following a final symlink, recheck the cache boundary and device/inode, and inherit only the verified descriptor into llama-server. Keep previews and saved selections separate from the child descriptor path and align owned-process probes with that path. Reject split filenames and split metadata because b12171 derives sibling paths for shards. The fake-engine inode regressions fail on bf50b64 both before and after spawn. Engine-manager race tests and darwin/linux/windows vet pass. Its existing unreleased 1.5.0 minor bump since v0.9.0 covers this fix. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Use one fallback exclusion builder for llama.cpp, Ollama, LM Studio and oMLX. Reserve every engine server port from the static configured-ports inventory, including stopped and custom engines, plus accepted, persisted and live proxy ports and node-service listeners. Preserve per-engine port-operation ownership and release settings locks across inventory calls, revalidating Ollama plans before committing fallback state. The stopped-engine regression fails on bf50b64 for all four proxies, including oMLX configured on 1237. Broker and all subpackage race-test binaries pass, along with darwin/linux/windows vet and SPDX. The existing unreleased broker 2.6.0 minor bump since v0.9.0 covers this fix. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Count the llama.cpp proxy in the broker inventory and build/test descriptions: thirteen workers and fifteen service executables. The worker table already includes llamacpp-proxy; keep that canonical row rather than adding a duplicate. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Opening only the model's final component without following symlinks still allowed a cache parent to redirect both pathname resolutions to the same outside inode after validation. Resolve the cache once, walk its canonical ancestors from the filesystem root with no-follow openat directory descriptors, then walk the selected cache-relative model from the pinned cache descriptor. Validate the opened regular GGUF directly and retain ExtraFiles inheritance without a second pathname resolution. Close every intermediate descriptor on all paths. Add parent-swap regressions before and after canonical cache resolution, nested-model child inode and descriptor-leak coverage, and traversal checks. Keep settings and previews unchanged. The engine-manager's existing unreleased minor version remains sufficient for this fix. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
The macOS runner exposed an EBADF race in os.ReadDir on /dev/fd: a transient descriptor can close before the directory reader stats its entry. Count Readdirnames results instead, closing the directory before checking errors. Descriptor counting needs names only, not per-descriptor metadata. Keep the leak assertion strict. A deliberate retained model descriptor made the test fail with before=6 and after=7; removing it restored the passing nested-model launch and failed-launch coverage. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Reserve other engines' default, saved, accepted and live proxy ports without excluding the requesting engine's own ports. Keep all engine-server and node-service reservations, including stopped engines, and retain the port-operation serialiser and unlocked inventory calls. Try a usable persisted proxy port before selecting a reported fallback so an upgrade does not move oMLX from 1236 or advance its settings revision unnecessarily. Cover all four proxies, the live oMLX regression, and isolated four-proxy startup in the cross-process suite. No additional version bump: nvpair-ui-broker already has an unreleased minor bump (2.6.0 versus 2.4.1 in v0.8.0), per the implementation brief. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
This was referenced Nov 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
llama.cpp PR 2 of 4: serve llama.cpp through PAIR. PR 1 (#60) added it as a managed engine on macOS arm64 without routing; this adds its own proxy worker, discovery and scheduling, so requests reach it locally and from verified peers.
llamacpp-proxy(services/llamacpp-proxy, default port 1237), copied fromlmstudio-proxywithout changing that worker: model routing and strict routing, streaming passthrough, stall retry and cooldown with the opt-in first-body deadline, bounded bodies, workload identity and telemetry, cluster ingress over pinned mTLS with verified peers only, terminal peer ingress, and the shared listen helper for loopback port conflicts. llama.cpp has its own service identity, exact model IDs, and both/v1/modelsand/modelslists./v1/messagesis routed, because llama-server b12171 serves it natively (tools/server/server.cpp:190).--model/-m, accepted only for a regular.gguffile inside PAIR's llama.cpp model cache ({install_dir}-models), with both the cache and file symlinks resolved and URI-shaped values rejected. Starting without a model (router mode) still works.Co-authored-by credit for the upstream design: NVIDIA/Personal-AI-Router@6d2d67f.
Scope
New services/llamacpp-proxy; nvpair-ui-broker, nvpair-engine-manager, nvpair-node-scanner, nvpair-manual-nodes, nvpair-job-scheduler, shared, TUI, desktop, docs, build and installer scripts. Deferred: model hub, downloads and delete (PR 3); remaining desktop/TUI/release integration (PR 4).
Versions:
llamacpp-proxy1.0.0 (new); job-scheduler 1.2.0, manual-nodes 1.4.0, node-scanner 1.4.0; other touched components keep their existing unreleased bumps; product/installer unchanged.Validation
--modelvalidation covers inside/outside the cache, symlink escapes, non-GGUF files and URI-shaped values.Risk
A 13th worker adds a process and a listener on 1237. llama.cpp's macOS build is not notarized (see PR 1). Windows and Linux remain unsupported for llama.cpp.
Checklist
git commit -s), certifying the Developer Certificate of Origin.services/versions.json.Live check
2026-11-07 re-run, macOS 27.2 arm64, locally packaged PR head
f21d3ac838cd335cbfe780aef870154d782e6f88(not a released-asset certification). Built in detached/tmp/pair-work/live61bwithnpm ci --no-audit --no-fund && npm run -s build:mac:arm64. ZIP SHA-256 matched locally and remotely:6426ba7c26829aeb52a9c922568e0cf489c199d37fce47843612b5c968756579. Python 3.9.6 isolated-mode driver used the packaged broker with the desktop's worker-path flags, including--llamacpp-proxy-path.The previous run at
f77de1ffound oMLX moved from 1236 to 1238 and its saved revision advanced 1 → 2. This re-run confirms that regression is fixed inf21d3ac: oMLX stays on 1236 at revision 1.lsofrecorded Ollama 11434, LM Studio 1234, oMLX 1236 and llama.cpp 1237, each on127.0.0.1,::1and IPv6*.engine:get-settingsafter startup: Ollama revision 8/server 11435/proxy 11434; LM Studio revision 1/server 1235/proxy 1234; oMLX revision 1/server 1235/proxy 1236. No proxy-port or revision change./propsHTTP 200 with rolerouter; settings server 8082, proxy 1237, empty launch text. No model downloaded./v1/modelsand/models: HTTP 200, zero models. Made-up-model chat: HTTP 502 in 1 ms,no available node advertises the requested model. Ollama 11434 and LM Studio 1234: baseline HTTP 503,model inventory unavailable; oMLX 1236: HTTP 200, 7 models.engine:preview-settingsrejected each of--model ../outside.gguf,--model /etc/hosts,--model https://x/y.ggufand--model missing.gguf, each witherrors.launchText:model must be an existing regular .gguf file inside the PAIR llama.cpp model cache.Restored the original app and app data with
dittofrom fresh, verified~/pair-live61b/backup/(retained; older backups untouched).diff -rqpassed (app-data check excludesSingleton*and logs); independent bytewise file, symlink, structure and mode verification matched before relaunch. Published PAIR 0.9.0 relaunched with broker + 12 live worker children;lsofconfirmed the restored oMLX proxy on 1236.~/pair-live61b/incomingremoved. Local evidence:/tmp/pair-review/live-llamacpp/evidence-61b.jsonl. No live-check failures.Proxy startup regression follow-up
Commit
f21d3ac8preserves each proxy's own default/saved port while retaining other engines' reservations, including stopped engines, and all engine-server/node-service exclusions. Upgrading no longer moves a free oMLX listener from 1236 or advances its settings revision unnecessarily. A usable persisted port is tried before a reported fallback; the port-operation serializer and unlocked worker inventory calls are retained.Validation: broker race test binary passed twice; broker subpackage race tests, darwin/Linux/Windows vet, gofmt and SPDX checks passed. New regressions fail on
f77de1fand pass with the fix. The isolated four-proxy startup test passed on Linux and macOS. CI run 42393452547, attempt 2, has all six jobs passing; the final Linux artifact has 127 passing tests and zero skips. The first Linux attempt hit an unchanged peer-discovery timeout inTestModelsHTTPEnrichment; it passed on retry. Cross-process tests were not run locally.No additional service version bump: the broker already has the unreleased 2.6.0 bump versus 2.4.1 in v0.8.0, per the implementation brief.