server: a slot pages to named host RAM and back; checkpoints keep a byte budget; a page carries its draft KV by joelteply · Pull Request #70 · CambrianTech/llama.cpp

joelteply · 2026-10-24T05:24:47Z

Continuum SERVING-ENGINE.md slice S1 (plan line 3b): a mind switch costs a memcpy, not a 66-100 s disk save or a 70-160 s re-prefill. One commit, 971962df4.

Base: fable/page-checkpoint-subset (= e7e1077, continuum's submodule pin). master lacks 84 commits of the pin, so a PR to master would carry them too. Retarget once the pin's stack lands on master.

What it adds

POST /slots/{id}?action=page_out&page=<name>: the slot's target KV, MTP draft KV, speculative tail state and context checkpoints move into a server-owned RAM page store. The slot is freed.
?action=page_in&page=<name>: restores the page into any slot with its tokens, so the next request's prefix matches. The page leaves the store.
?action=spill&page=<name> with body {"filename", "drop"}: writes the page in the slot-save format (.bin + .ckpt + .lora + new .dft) on the HTTP thread, off the engine thread. drop frees only that generation of the page.
?action=page_drop&page=<name>: frees the page.
page_out and page_in wait while the slot is busy, as restore does.
Errors are typed: 404 for an unknown page, 503 past --page-ram-bytes, 400 for an empty slot or a bad name. A failed page_out leaves the slot intact; a failed page_in leaves the page in the store.
/props reports slot_page_version: 1.
save/restore: unchanged, except that save writes .dft (the draft state, counted in save-size) and restore reads it. With no .dft, the draft sequence is cleared and the cold start is logged.
--ctx-checkpoints-bytes: one byte budget across all slots, in place of the per-slot count. The slot holding the most bytes drops its oldest checkpoint after its first. Every eviction is logged and counted.
/metrics: page_store_bytes, page_store_pages, ctx_checkpoint_bytes, page_{out,in,failed}_total, ctx_checkpoints_{evicted,skipped}_budget_total.
llama_state_seq_save_file_from_data (llama-ext.h), so libllama stays the only owner of the .bin format. The sidecar writers move out of the save arm into server-pages.cpp, shared by save and spill.

New files: server-pages.{h,cpp} (111 / 301 lines) and server-checkpoints.{h,cpp} (28 / 42). server-context.cpp carries only the wiring.

Measured

Qwen3.8-27B Q4_K_M, MTP draft, q8_0 KV, FA on, M5 Metal, 35,001 tokens. 2 slots, -c 131072, unified KV, ubatch 2048, checkpoints 32 with min step 2728. Other agents were building on the machine at the same time.

	median of 3	bytes
page_out	39 ms (first run 92 ms)	2,113,724,788 total: target 1,376,189,600, draft 144,064,164, 2 checkpoints 593,471,024
page_in	29 ms	the same
disk save	588 ms (522-1000)	2,113,864,940 on disk
disk restore	232 ms (208-293); the file was still in the page cache, so a cold read is slower
spill	780 ms	2,113,864,940

After page_in, the next completion prefilled 11 tokens: the same as a slot restored from disk and as a slot that never paged.
Output is identical to the never-paged slot (96 tokens, temperature 0).
MTP accepted 66/85 in all three cases (paged, disk with .dft, never paged), so the draft is warm after page_in.
Decode ran at 15.8 / 14.2 / 13.3 tok/s.
Prefill measured 96 tok/s while the builds ran.

59k: not measured. My script restored a second copy of a 68,850-token page into a 131,072-cell context, and the restore failed (it should have). The script is fixed (one resident copy during the disk comparison); the rerun waits for a machine window.

Qwen3.5 0.8B hybrid, 19,794 tokens: page_out 10.4 ms, page_in 7.8 ms for 304 MB, against save 134 ms and restore 37 ms.

Tests

tools/server/tests/unit/test_slot_page.py (5 tests) and test_slot_save.py (11): 16 passed.

The hybrid checkpoint-budget test runs when LLAMA_SERVER_TEST_HYBRID_GGUF points at a small hybrid GGUF.
The identity test pages out of slot 1, reuses slot 1, pages into slot 0, and compares prompt_n and content against a slot that never paged.

Not in this slice

A listing of pages.
Spilling before sleep: pages are dropped, with a warning, when the model is freed.
Re-applying the checkpoint budget at page_in.
The core consuming the verbs (S4, Codex).

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…hole generations Review fixes on the RAM page verbs (Codex on #70): - One count: server_page_bytes is what page_out admits (measured before any copy, speculative tail state included) and what the store charges; put re-checks the ceiling on exactly the page it stores, so no pre-count can get a page past it. - The ceiling counts live ownership: a stored page carries a charge released when its LAST owner drops it, so a page a spill still holds after page_in, page_drop, replacement or a model unload keeps counting. Admission subtracts a replaced page only when the store is its sole owner. - page_out moves the slot's prompt only into a stored page: put inserts first and moves the prompt back on any refusal, so a refused or failed page_out leaves the slot's sequence intact. - Page files (server-page-files.{h,cpp}, moved out of server-pages.cpp): save and spill publish `<file>` + .ckpt/.lora/.dft as one generation. Every file is written to `..page-staging/<pid>-<n>/` and synced, a commit record is renamed into place, then the files move over the committed ones and a sidecar the generation lacks is removed. A failure before the record leaves the committed generation untouched and nothing staged; restore reads through server_page_reader, which completes a publication cut short after its record, removes stages no live publication owns, and holds commits off while it reads. A save whose sidecar cannot be written now fails instead of leaving a mixed set. - /slots page results report n_bytes_spec. Tests: tests/test-server-pages.cpp (admission = charge with speculative state, spill-held pages stay counted, refusal keeps the incumbent, failed and interrupted publications keep the committed generation and leave nothing staged); test_slot_page.py adds a crashed-publication restore and an MTP page at an exact ceiling (hybrid GGUF). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…he model A relaunch is a pause, not a reload (continuum SERVING-ENGINE.md §3, slice S2, plan line 3b). POST /reconfigure {"n_ctx", "n_parallel", "cache_type_k", "cache_type_v", "force"}: admission closes (requests that arrive queue), the slots reach a ubatch boundary, every occupied slot's target KV, draft KV, speculative tail and checkpoints move to host RAM (S1's page capture), the draft and target contexts are freed and rebuilt on the SAME llama_model (weights and mmap stay), the slots are rebuilt for the new n_parallel, the states go back, and serving resumes. - A request in flight keeps its slot and finishes as if nothing happened (test: identical output). A request on a slot past the new n_parallel drains first. A request whose tokens do not fit the new window refuses the reconfigure; it is never truncated. - An idle mind with no seat moves into a free slot, or stays a named RAM page (page_in brings it back) with a typed reason; one larger than the new window is likewise kept whole. - A KV type change is an intentional cold start: refused without "force"; with it, resident states, the page store and the prompt cache are dropped, logged loudly and counted. - A geometry libllama cannot build rolls back to the old one with every state restored. - Receipt: drain, page out, free, init, page in and total ms; bytes; per-slot fate. /props context_geometry and total_slots follow the change (routes read an atomic meta snapshot); /metrics reconfigure_total, reconfigure_failed_total. A wake from sleep reloads the new geometry. - libllama: one sequence's state reads into a cache split into a different number of streams (its cells all land in its own stream); the whole-cache read still requires the same layout. - common: common_init_result::reinit_context/free_context and the speculative init result's rebuild the context on the loaded model; the context, sampler and threadpool setup are shared with the first build rather than copied. common_kv_cache_type_from_str is public. - The mechanism is tools/server/server-reconfigure.{h,cpp} behind server_reconfigure_host; server-context.cpp implements the host over its slots and wires the task, the route and the admission gate. Slot init is one function shared by load_model and the reconfigure. Measured (M5 Metal, this build): - Qwen3.5 0.8B hybrid, lane geometry (174,592 ctx, q4_0 KV, 32 checkpoints), two minds of 25,226 and 17,883 tokens: pause 72-104 ms (page out 6-13, free 3-5, init 57-80, page in 6) for 271 MB; median 78 ms over 40 cycles. Each mind's next act prefilled 15 tokens. - Footprint: 100 grow/shrink cycles (2 -> 3 slots, n_ctx x2 and back) 395 MB -> 327 MB, never growing; 20 cycles at lane geometry 1,685 -> 1,617 MB. - Qwen3.8 27B with the M5 lane's flags (mmproj, MTP draft model, q4_0 KV, 2 x 87,296), same two minds: 2 -> 3 slots 826 ms, 3 -> 4 (4 x 59,392) 802 ms, back to 2 626 ms, 2.25 GB paged; init (context + MTP context allocation) is 0.5-0.6 s of it. Next acts prefilled 15 tokens (6.4 s and 7.2 s wall, against 248 s and 173 s to seat them). CONTENDED: a second 27B serving lane started on the node during the run, so these are upper bounds; the 20-cycle 27B pauses (0.54-4.6 s, median 0.70 s) ran beside it too. Tests: tools/server/tests/unit/test_reconfigure.py: 5 passed (attention and hybrid), plus the opt-in 100-cycle footprint validator (LLAMA_SERVER_TEST_RECONFIGURE_CYCLES); with test_slot_page, test_slot_save, test_speculative: 27 passed. Full unit suite: 395 passed, 185 failed, the same 185 as S1's binary (test_tool_call real models not cached offline, one test_completion). Rebased on #70's review fixes: the pre-check counts each kept page with server_page_measure (the store's own count, speculative state included); a put the store refuses after a failed restore drops that state with a typed reason instead of aborting the reconfigure; a cold start clears the store (a spill-held page stays counted until its spill ends). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…e move; a status error is never absence Codex's two #70 findings at c179351: - Windows: sync_dir was a successful no-op, so roll_forward could drop the commit record before the moved names were durable. Every name a generation depends on (staged files, the commit record, the installed files, the older sidecars it retires) is now placed by one place(): MoveFileExW with MOVEFILE_WRITE_THROUGH on Windows, which returns only once the move is on disk; rename plus sync_dir on POSIX. Older sidecars are moved into the stage rather than deleted, because a delete has no write-through form. - roll_forward read fs::exists(staged, ec) == false on an I/O or permission error as "already moved", then dropped the record. present() throws on any status error that is not not_found, so the record and stage keep custody. The test seam becomes server_page_files_inject_fault(op, path) for sync and status. The failed-publication scenario gains a status fault on a staged member after the commit: publish refused, reader refused, record kept, then a clean reader completes the generation. The Windows server job now builds and runs test-server-pages, so the repeated-publication fixture executes on NTFS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…yte budget; a page carries its draft KV A mind switch costs a memcpy instead of a 66-100 s disk save or a 70-160 s re-prefill (continuum SERVING-ENGINE.md, slice S1, plan line 3b). - /slots/{id}?action=page_out&page=<name>: the slot's target KV, draft (MTP) KV, speculative tail state and context checkpoints move into a server-owned RAM page store, and the slot is freed. page_in puts them into any slot with its tokens, so the next request's prefix matches; the page leaves the store. spill writes a page in the slot-save format (.bin + .ckpt + .lora + new .dft) on the HTTP thread, off the engine thread; "drop": true then frees that page's generation only. page_drop frees a page. page_out/page_in defer while the slot processes, as restore does. Errors are typed (404 unknown page, 503 over --page-ram-bytes, 400 empty slot); a failed page_out leaves the slot intact, a failed page_in leaves the page in the store. /props slot_page_version 1. - save writes the draft state to `.dft` and counts it in its quote; restore reads it, so MTP is warm after a restore (no `.dft`: the draft seq is cleared and the cold start logged). - --ctx-checkpoints-bytes: one budget across all slots in place of the per-slot count; the slot holding the most bytes drops its oldest checkpoint after its first. Every eviction logs and counts. - /metrics: page_store_bytes/pages, ctx_checkpoint_bytes, page_{out,in,failed}_total, ctx_checkpoints_{evicted,skipped}_budget_total. - llama_state_seq_save_file_from_data (llama-ext.h): libllama stays the only owner of the sequence-file format; the sidecar writers move from the save arm into server-pages.cpp, shared by save and spill. Measured (Qwen3.5 0.8B hybrid, M5 Metal, 19,794 tokens, 2 checkpoints, median of 5): page_out 10.4 ms, page_in 7.8 ms, 304 MB; save 134 ms, restore 37 ms (page-cache warm), 304 MB on disk; spill 101 ms. The paged slot's next completion processed 6 prompt tokens and its output matched a slot that never paged. Tests: tools/server/tests/unit/test_slot_page.py (5, hybrid case via LLAMA_SERVER_TEST_HYBRID_GGUF) + test_slot_save.py (11): 16 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…hole generations Review fixes on the RAM page verbs (Codex on #70): - One count: server_page_bytes is what page_out admits (measured before any copy, speculative tail state included) and what the store charges; put re-checks the ceiling on exactly the page it stores, so no pre-count can get a page past it. - The ceiling counts live ownership: a stored page carries a charge released when its LAST owner drops it, so a page a spill still holds after page_in, page_drop, replacement or a model unload keeps counting. Admission subtracts a replaced page only when the store is its sole owner. - page_out moves the slot's prompt only into a stored page: put inserts first and moves the prompt back on any refusal, so a refused or failed page_out leaves the slot's sequence intact. - Page files (server-page-files.{h,cpp}, moved out of server-pages.cpp): save and spill publish `<file>` + .ckpt/.lora/.dft as one generation. Every file is written to `..page-staging/<pid>-<n>/` and synced, a commit record is renamed into place, then the files move over the committed ones and a sidecar the generation lacks is removed. A failure before the record leaves the committed generation untouched and nothing staged; restore reads through server_page_reader, which completes a publication cut short after its record, removes stages no live publication owns, and holds commits off while it reads. A save whose sidecar cannot be written now fails instead of leaving a mixed set. - /slots page results report n_bytes_spec. Tests: tests/test-server-pages.cpp (admission = charge with speculative state, spill-held pages stay counted, refusal keeps the incumbent, failed and interrupted publications keep the committed generation and leave nothing staged); test_slot_page.py adds a crashed-publication restore and an MTP page at an exact ceiling (hybrid GGUF). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…mit record; its byte charge dies last Codex's review of e08b11e: - sync_dir discarded open/fsync failures, so roll_forward could drop the commit record and report a publication whose moved names were never proven durable. It now throws; the stage sync fails the publication before the commit, the record dir and destination syncs leave the record in custody for the next reader. Only the final sync after the record's removal warns: the generation is durable by then and a resurrected record names a stage already moved. - server_page::charge was declared last, so it was destroyed first and released the bytes before the buffers were freed; it is declared first now. Tests: test_a_failed_dir_sync_never_reports_or_loses_a_generation (stage, commit-record and destination sync faults, via a test seam) and test_repeated_publication_replaces_every_file (rename onto existing main + sidecars, four generations). test-server-pages OK on macOS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…e move; a status error is never absence Codex's two #70 findings at c179351: - Windows: sync_dir was a successful no-op, so roll_forward could drop the commit record before the moved names were durable. Every name a generation depends on (staged files, the commit record, the installed files, the older sidecars it retires) is now placed by one place(): MoveFileExW with MOVEFILE_WRITE_THROUGH on Windows, which returns only once the move is on disk; rename plus sync_dir on POSIX. Older sidecars are moved into the stage rather than deleted, because a delete has no write-through form. - roll_forward read fs::exists(staged, ec) == false on an I/O or permission error as "already moved", then dropped the record. present() throws on any status error that is not not_found, so the record and stage keep custody. The test seam becomes server_page_files_inject_fault(op, path) for sync and status. The failed-publication scenario gains a status fault on a staged member after the commit: publish refused, reader refused, record kept, then a clean reader completes the generation. The Windows server job now builds and runs test-server-pages, so the repeated-publication fixture executes on NTFS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…ver branches on the OS Joel: an OS should be only the diff of its nuance in an adapter, never a copy. server-page-files.cpp carried five #if _WIN32 blocks (pid, liveness, file sync, dir sync, durable move) interleaved with the publication logic, and the test repeated the pid branch. page_os (server-page-os.h) is the one signature each; server-page-os-posix.cpp and server-page-os-win32.cpp hold only their own calls, chosen by CMake. The publication logic and the test call page_os:: and contain no OS branch. Net: 96 lines removed from the logic, adapter added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

… server-context, as llama-server does) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

joelteply · 2026-10-28T11:32:14Z

Rebased onto the live line. This PR is now stacked on #79 (fable/draft-kv-once, which sits on the live pin's branch 889580fc/seq-state-across-stream-counts). Head: 6465508. In the one conflict, the draft sidecar, this PR's version wins: it carries the speculative state, and its restore clears a stale draft. #79's checkpoint change, where checkpoints skip a draft that rolls back by trimming, is kept.

Measured on the live model

M5 Max, 2026-10-28 11:22-11:31Z. Qwen3.6-35B-A3B Q4_K_M, MTP draft, the live flags (-c 636928 --parallel 4, q4_0 KV, ctx-checkpoints 32), --page-ram-bytes 20e9. One mind of 97,526 tokens, built over 40 turns, is moved while slot 1 streams a 2,500-token completion. Median gap between that stream's tokens: 20 ms.

operation on the other mind	op time	worst token gap on the streaming slot
disk `save` (today's path, on the loop)	1428 ms	1403 ms
`page_out` to RAM (2.02 GB: draft 202 MB, 18 checkpoints 1.19 GB)	123 ms	110 ms
`page_in` into slot 2	89 ms	53 ms
`page_out` again	78 ms	63 ms
`spill` to disk (HTTP thread)	1466 ms	46 ms
`page_in` again	101 ms	58 ms

After page_in, slot 2 reused 97,507 tokens and prefilled 9. Decode ran at 31.0 t/s with 73 of 74 drafts accepted. The greedy continuation is identical to the disk-restored mind from the server: a page carries the draft's KV once; checkpoints stop copying it #79 run.
/metrics: page_out_total 2, page_in_total 2, page_failed_total 0, ctx_checkpoint_bytes 1.25e9.
Script and output: ~/.continuum/evidence/2026-10-28-page-save-repro/ram_pages_window.sh, ram-pages-window.txt (the M5).

A mind switch now freezes the other minds for about 0.1 s instead of 1.4 s, and the disk write no longer freezes them at all.

🤖 Generated with Claude Code

…d draft (Codex/Cormac on #79); needs a hybrid MTP GGUF to run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

github-actions Bot added the server label Oct 24, 2026

github-actions Bot added the testing label Oct 24, 2026

github-actions Bot added the devops label Oct 24, 2026

This was referenced Oct 25, 2026

server: the checkpoint budget's eviction gives up what a saved page gives up first, never oldest-first #74

Open

server: tail_tools are callable without entering the rendered head (restacked on the engine pin) #75

Merged

joelteply and others added 6 commits October 28, 2026 06:21

tests: the page publication test builds on Windows too (it links only…

6465508

… server-context, as llama-server does) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

joelteply force-pushed the fable/ram-pages branch from da27da5 to 6465508 Compare October 28, 2026 11:32

joelteply changed the base branch from fable/page-checkpoint-subset to fable/draft-kv-once October 28, 2026 11:32

tests: a malformed or torn draft sidecar restores the page with a col…

a76ac59

…d draft (Codex/Cormac on #79); needs a hybrid MTP GGUF to run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

Navigation Menu

Repository navigation

server: a slot pages to named host RAM and back; checkpoints keep a byte budget; a page carries its draft KV - #70

server: a slot pages to named host RAM and back; checkpoints keep a byte budget; a page carries its draft KV#70
joelteply wants to merge 7 commits into
fable/draft-kv-oncefrom
fable/ram-pages

joelteply commented Oct 24, 2026

Uh oh!

joelteply commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

joelteply commented Oct 24, 2026

What it adds

Measured

Tests

Not in this slice

Uh oh!

joelteply commented Oct 28, 2026

Measured on the live model

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant