feat(qwen4exp): fit the context to the devices, prefix snapshots in system memory

[davide221]

Qwen3.8-Flash-Next now gets the largest context the box holds, without a hand-picked --max-ctx, and its prefix cache works at that context with four requests at once.

What changes

1. Prefix snapshots in system memory beside a discrete GPU. Snapshots and concurrent prefix checkpoints were copies in the target GPU's memory, so in split mode they competed with the KV caches on the R9700. At 131,072 tokens with four slots their allowance was 473 MB: most captures failed, and in one run a request came out different after a cross-slot restore. They now go to system memory through create_snapshot_backend, as the other models' snapshots do, and the server's resident limit bounds them. On the gfx1151 iGPU they stay in its memory, which is the system's, under the planner's allowance as before.
2. --max-ctx auto. The server starts from the model's trained context (262,144 tokens) and the qwen4exp backend lowers it, in steps of 4096, to the largest at which every slot's cache, the resident graphs, a floor-sized prompt chunk and, with snapshots on the target, one full snapshot per slot fit. The server then serves that context and /props reports it. The chunk planner returns what it measured (Qwen4ExpChunkPlan); Qwen4ExpContextFit shrinks a miss by its shortfall, then interpolates between the largest fit and the smallest miss. An unpark keeps the fitted context. Other architectures refuse auto.
3. Split serving keeps one prompt chunk's memory on the target. At 131,072 tokens with four slots, a long prompt ran the R9700 out of memory. The chunk planner did not count two kinds of growth. First, every slot kept its own split schedulers and so its own prompt-chunk buffers (about 1.9 GB each); the slots now share the first slot's. Second, the CUDA pool keeps every block it frees while a chunk's temporaries grow with its position; split prompt chunks now release the cached blocks, and a failed graph allocation retries once after releasing them.
4. --profile qwen-next-r9700-strix uses --max-ctx auto; a number on the command line still wins.

Results

UD-Q4_K_XL, adaptive MTP, greedy, R9700 + Strix Halo unless noted. Every run starts in about 15 s.

Layout	Context	4 short prompts together	Same output alone vs together
Split, 4 at once, before (131,072 set by hand)	131,072	76.6 tok/s	3 of 4
Split, 4 at once, auto	131,072	78.2 tok/s	4 of 4
Split, 1 at once, auto	262,144		4 of 4
Strix Halo alone (192 GB), 4 at once, auto	262,144	48.8 tok/s	4 of 4

Split, 4 at once, prefix cache on:

• Two ~6K prompts beside two short ones: outputs identical alone and together (4 of 4).
• A resent 16K prompt prefills in 0.3 s instead of 10.4 s, with the same answer.
• A 6K-token checkpoint restores from system memory in 106 ms.
• Every capture is saved.
• A 96K prompt beside three short requests completes in 72 s. The R9700 keeps at least 2.15 GB free through all of these runs; before the fix in (3) it fell to 0 and the server died.

Strix Halo alone, 262,144 tokens with four slots: a 159K prompt beside three short requests completes, and a resent 16K prompt prefills in 0.2 s.

Validation

• test_qwen4exp_prefix_cache: CPU checks, then on the Strix Halo and in split mode (system-memory snapshots, hot experts across park/unpark).
• test_qwen4exp_seq_engine: prefix store and prefill cuts with checkpoints in device memory and in system memory, the QSA boundary, and exact distinct concurrent streams.
• test_server_unit: 650 of 650 pass, including the new launch-profile check.
• test_qwen4exp_chunk: the context search.

The split's four-at-once ceiling is the R9700's memory: 262,144 tokens for four slots needs part of the KV cache on the Strix Halo, which is the next step.

🤖 Generated with Claude Code

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed 82397fd2 · 0 candidate issues checked, kept at ≥70% confidence · add the code-review label to review again}