Skip to content

server: the checkpoint budget's eviction gives up what a saved page gives up first, never oldest-first - #74

Open
joelteply wants to merge 8 commits into
fable/ram-pagesfrom
164bbff9/server-a-saved-page-keeps-checkpoints-ju
Open

server: the checkpoint budget's eviction gives up what a saved page gives up first, never oldest-first#74
joelteply wants to merge 8 commits into
fable/ram-pagesfrom
164bbff9/server-a-saved-page-keeps-checkpoints-ju

Conversation

@joelteply

@joelteply joelteply commented Oct 25, 2026 •
edited
Loading

Copy link
Copy Markdown

Lever (2) of Fable's checkpoint work (card 164bbff9), stacked on fable/ram-pages (#70) as Fable chose.

What it fixes

The M5 on 2026-10-25 (a hybrid model, 4 residents paging over 3 lanes) re-prefilled 20k+ tokens a turn. Once it hit "full re-prefill: no checkpoint passed the search (raw lcp 9710)". A slot's next prompt diverges about 11k tokens in, but the checkpoints it kept were all near the end of its 35–40k prompt.

Change

  • select_page_checkpoints(n_tokens, keep, raw_lcp) decides what a saved page keeps:
    • the first checkpoint;
    • up to min(2, keep/2) of the newest at or below the slot's last raw common prefix;
    • the newest.
      It keeps at most 1 + keep, so memory is unchanged.
    • resume_raw_lcp is engine-owned: set where the restore search computes the raw lcp, and reset per request and on page restore and erase. It is never cached_tokens, which is n_past after a restore and 0 on a miss.
  • The byte budget's eviction (server: a slot pages to named host RAM and back; checkpoints keep a byte budget; a page carries its draft KV #70's server_checkpoints_fit) asks the same selector. server_checkpoint_victim gives up the checkpoint the selector leaves out when it keeps all but one. A list keeps longest its head, its newest under the divergence and its newest, never dropping oldest-first. With no divergence known, it behaves as before.

Tests

tests/test-page-checkpoints:

  • the subset with and without a known divergence;
  • the budget never grows;
  • the eviction victim is never the head, the newest, or one of the two under the break;
  • across two slots, the fuller list evicts until the new checkpoint fits and keeps its two under the break;
  • an over-budget checkpoint drops nothing.

Validation

This PR has not been compiled locally: IntelMac serves and builds nothing beside its lanes, so fork CI is the compile. The direction was decided by a throwaway simulation posted in #continuum-dev: over 300 turns at 0.5–1 page per turn, mean re-prefill fell from 32–37k to about 26.6k tokens, and near-full re-prefills from 154–300 to 1. The live receipt follows the continuum pin bump. It will correlate each page's save and restore with the next request's raw lcp, and log the candidates.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

joelteply and others added 8 commits October 23, 2026 22:55
@joelteply @claude
server: a slot pages to named host RAM and back; checkpoints keep a b…
971962d 
…yte budget; a page carries its draft KV

A mind switch costs a memcpy instead of a 66-100 s disk save or a 70-160 s re-prefill
(continuum SERVING-ENGINE.md, slice S1, plan line 3b).

- /slots/{id}?action=page_out&page=<name>: the slot's target KV, draft (MTP) KV,
  speculative tail state and context checkpoints move into a server-owned RAM page store,
  and the slot is freed. page_in puts them into any slot with its tokens, so the next
  request's prefix matches; the page leaves the store. spill writes a page in the slot-save
  format (.bin + .ckpt + .lora + new .dft) on the HTTP thread, off the engine thread;
  "drop": true then frees that page's generation only. page_drop frees a page.
  page_out/page_in defer while the slot processes, as restore does. Errors are typed
  (404 unknown page, 503 over --page-ram-bytes, 400 empty slot); a failed page_out leaves
  the slot intact, a failed page_in leaves the page in the store. /props slot_page_version 1.
- save writes the draft state to `.dft` and counts it in its quote; restore reads it, so
  MTP is warm after a restore (no `.dft`: the draft seq is cleared and the cold start logged).
- --ctx-checkpoints-bytes: one budget across all slots in place of the per-slot count; the
  slot holding the most bytes drops its oldest checkpoint after its first. Every eviction
  logs and counts.
- /metrics: page_store_bytes/pages, ctx_checkpoint_bytes, page_{out,in,failed}_total,
  ctx_checkpoints_{evicted,skipped}_budget_total.
- llama_state_seq_save_file_from_data (llama-ext.h): libllama stays the only owner of the
  sequence-file format; the sidecar writers move from the save arm into server-pages.cpp,
  shared by save and spill.

Measured (Qwen3.5 0.8B hybrid, M5 Metal, 19,794 tokens, 2 checkpoints, median of 5):
page_out 10.4 ms, page_in 7.8 ms, 304 MB; save 134 ms, restore 37 ms (page-cache warm),
304 MB on disk; spill 101 ms. The paged slot's next completion processed 6 prompt tokens
and its output matched a slot that never paged.
Tests: tools/server/tests/unit/test_slot_page.py (5, hybrid case via
LLAMA_SERVER_TEST_HYBRID_GGUF) + test_slot_save.py (11): 16 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply @claude
server: page RAM ceiling counts every live page; page files publish w…
e08b11e 
…hole generations

Review fixes on the RAM page verbs (Codex on #70):

- One count: server_page_bytes is what page_out admits (measured before any copy,
  speculative tail state included) and what the store charges; put re-checks the
  ceiling on exactly the page it stores, so no pre-count can get a page past it.
- The ceiling counts live ownership: a stored page carries a charge released when
  its LAST owner drops it, so a page a spill still holds after page_in, page_drop,
  replacement or a model unload keeps counting. Admission subtracts a replaced page
  only when the store is its sole owner.
- page_out moves the slot's prompt only into a stored page: put inserts first and
  moves the prompt back on any refusal, so a refused or failed page_out leaves the
  slot's sequence intact.
- Page files (server-page-files.{h,cpp}, moved out of server-pages.cpp): save and
  spill publish `<file>` + .ckpt/.lora/.dft as one generation. Every file is
  written to `..page-staging/<pid>-<n>/` and synced, a commit record is renamed
  into place, then the files move over the committed ones and a sidecar the
  generation lacks is removed. A failure before the record leaves the committed
  generation untouched and nothing staged; restore reads through
  server_page_reader, which completes a publication cut short after its record,
  removes stages no live publication owns, and holds commits off while it reads.
  A save whose sidecar cannot be written now fails instead of leaving a mixed set.
- /slots page results report n_bytes_spec.

Tests: tests/test-server-pages.cpp (admission = charge with speculative state,
spill-held pages stay counted, refusal keeps the incumbent, failed and interrupted
publications keep the committed generation and leave nothing staged);
test_slot_page.py adds a crashed-publication restore and an MTP page at an exact
ceiling (hybrid GGUF).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply @claude
server: a page's directory sync failure is an error and keeps the com…
c179351 
…mit record; its byte charge dies last

Codex's review of e08b11e:
- sync_dir discarded open/fsync failures, so roll_forward could drop the commit record and report a
  publication whose moved names were never proven durable. It now throws; the stage sync fails the
  publication before the commit, the record dir and destination syncs leave the record in custody
  for the next reader. Only the final sync after the record's removal warns: the generation is
  durable by then and a resurrected record names a stage already moved.
- server_page::charge was declared last, so it was destroyed first and released the bytes before
  the buffers were freed; it is declared first now.

Tests: test_a_failed_dir_sync_never_reports_or_loses_a_generation (stage, commit-record and
destination sync faults, via a test seam) and test_repeated_publication_replaces_every_file (rename
onto existing main + sidecars, four generations). test-server-pages OK on macOS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply @claude
server: every name a page generation depends on is placed by a durabl…
be000af 
…e move; a status error is never absence

Codex's two #70 findings at c179351:

- Windows: sync_dir was a successful no-op, so roll_forward could drop the
  commit record before the moved names were durable. Every name a generation
  depends on (staged files, the commit record, the installed files, the older
  sidecars it retires) is now placed by one place(): MoveFileExW with
  MOVEFILE_WRITE_THROUGH on Windows, which returns only once the move is on
  disk; rename plus sync_dir on POSIX. Older sidecars are moved into the stage
  rather than deleted, because a delete has no write-through form.
- roll_forward read fs::exists(staged, ec) == false on an I/O or permission
  error as "already moved", then dropped the record. present() throws on any
  status error that is not not_found, so the record and stage keep custody.

The test seam becomes server_page_files_inject_fault(op, path) for sync and
status. The failed-publication scenario gains a status fault on a staged
member after the commit: publish refused, reader refused, record kept, then
a clean reader completes the generation. The Windows server job now builds
and runs test-server-pages, so the repeated-publication fixture executes on
NTFS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply @claude
server: page publication's OS calls live in one adapter; the logic ne…
42ef0d5 
…ver branches on the OS

Joel: an OS should be only the diff of its nuance in an adapter, never a copy.
server-page-files.cpp carried five #if _WIN32 blocks (pid, liveness, file
sync, dir sync, durable move) interleaved with the publication logic, and
the test repeated the pid branch.

page_os (server-page-os.h) is the one signature each; server-page-os-posix.cpp
and server-page-os-win32.cpp hold only their own calls, chosen by CMake. The
publication logic and the test call page_os:: and contain no OS branch.
Net: 96 lines removed from the logic, adapter added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply @claude
tests: the page publication test builds on Windows too (it links only…
da27da5 
… server-context, as llama-server does)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply @claude
server: a saved page keeps checkpoints just under the slot's last div…
77921e1 
…ergence

`keep_newest_checkpoints` kept the oldest checkpoint and the newest N. On the Continuum M5
(Qwen3.6 hybrid, 4 residents paging over 3 lanes, 2026-10-25) a 35-40k prompt diverged
from its next request at about 11k, so a restored page held nothing under the break and the
turn re-prefilled 20k+ tokens, once from 0 ("full re-prefill: no checkpoint passed the
search", raw lcp 9710).

select_page_checkpoints keeps, within the same budget (at most 1 + keep): the oldest, up to
min(2, keep/2) of the newest checkpoints at or below the slot's last raw common prefix, and
the newest. server_slot.resume_raw_lcp records that raw lcp (engine tokens, before any
checkpoint fallback) per request; it is reset per request, and on restore and erase, so a
stale divergence never selects. Unknown raw lcp keeps today's subset exactly.

Test: tests/test-page-checkpoints.cpp.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply @claude
server: the checkpoint budget's eviction gives up what a saved page g…
d7c6d1a 
…ives up first, never oldest-first

The byte budget across slots dropped each list's oldest checkpoint after its first: exactly the
checkpoints under the slot's last divergence, which a resumed prompt needs. server_checkpoint_victim
asks select_page_checkpoints which checkpoint it leaves out when it keeps all but one, so a list
keeps longest its head, the newest at or below the slot's last raw common prefix, and its newest;
with no divergence known it drops the oldest after the head, as before. server_checkpoints_fit
takes each slot's resume_raw_lcp beside its list.

Tests (test-page-checkpoints): the victim is never the head, the newest, or one of the two under
the break; across two slots the fuller list evicts until the new checkpoint fits and keeps its
two under the break; a checkpoint larger than the budget drops nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply
joelteply changed the base branch from master to fable/ram-pages October 25, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Apple Metal CUDA devops ggml model mtmd server SYCL testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@joelteply