Repository navigation
Conversation
…uns in a context with rollback slots Serving keeps recurrent rollback slots for drafts (n_rs_seq > 0); the exact walk refused any context that had them, which is what forced a second, rollback-free training context beside serving. llm_graph_context::rs_keep() is now the one rule every family's state write and every snapshot-producing op reads (delta-net, mamba, lfm2, bailing; each decided it separately before): serving with rollback keeps n_rs_seq + 1 states, a training graph keeps one, the live state in group 0, which is the same bytes in both layouts. The walk now refuses only a sequence with an undo PENDING (llama_memory_recurrent::rollback_pending), and refuses by name a model whose state write does not report its exit to the walk, where the gradient across chunks was silently dropped before. Measured on the M5 (Metal), base 1782249: - serving unchanged: 19 recurrent/hybrid archs x n_rs_seq {0, 4}, random weights, 100-token prefill + 4 single decodes + (where rollback is live: qwen35, qwen35moe, deepseek4, nemotron_h, nemotron_h_moe, bailingmoe3, lfm2, lfm2moe) a 2-token undo and redo: logits and full sequence state byte-identical to the base, 38 of 38 rows. - training: test-walk-exact on qwen3.5-4b Q4_K_M, the exact walk in a context with n_rs_seq 4 vs 0: cosine 1.0000, norm ratio 1.0000; every other check in the file unchanged (exit 0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… so training stops falling back to the CPU A LoRA training epoch on the M5 spent most of its time on the CPU: Metal's OUT_PROD refused (1) any ne[2]/ne[3] above 1, which is every batched backward with grouped heads (the 4B's delta-net backward), and (2) any src0 width not a multiple of 16, which is every LoRA parameter gradient at rank 8. The 16-multiple rule only matters for the dequantizing copy of a quantized src0; it now applies only there. The 2D path becomes a per-slice helper, and the op loops over dims 2-3 with the CPU backend's broadcast rule (dst per src0 = ne2/ne02, ne3/ne03), reusing one slice of scratch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ed copy-and-transpose path is deleted OUT_PROD was composed from a dequantizing copy, two transposing copies, a mul_mm and an add per row block, over scratch reserved after dst. Every pass wrote device memory and read it back, so the backward matmul ran at about half the forward's rate. kernels/out_prod.metal is mul_mm with the contraction along the rows of both operands. Each thread dequantizes 16 consecutive elements of one weight row in the weight's own block order and stages them transposed in threadgroup memory. The gradient is read with its own strides, so both layouts work, and staging is float. Batching and broadcast come from the grid. Deleted with it: the row blocks, OUT_PROD_BLOCK_BYTES, the scratch after dst and its get_alloc_size hook, and the synthetic-view helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… so a LoRA training graph stays on the GPU The ops a Metal training graph still sent to the CPU after OUT_PROD, each refused by supports_op: - PAD with left padding. The kernel now reads src[i - lp] per dim, through the source's element stride (a training graph pads a view). - RMS_NORM_BACK and SOFT_MAX_BACK on strided rows (a per-head norm). Rows are now addressed through each tensor's dim 1-3 strides, and each row stays contiguous. - GET_ROWS_BACK (no kernel). This is the gradient of the output-row gather. One threadgroup per dst row sums, in id order, the source rows whose id is that row: no atomics, and the same summation order as the CPU reference. Tests: left-pad cases per dim on contiguous, non-contiguous and permuted sources, and an RMS_NORM_BACK over row views of a twice-wide tensor. Not in this change: ADD with a transposed src1 (a strided read in the hot binary kernel; it measured about 1% of a 4B epoch). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…o every ubatch P1 step 1 (continuum ggml-org#5032 section 9.6): training is inference plus extras that switch on and off, so the switch is in the batch. llama_batch gains `label`: the target token at that position, -1 for none, and NULL for an inference batch, which is unchanged. llama_batch_init allocates it filled with -1. The allocator carries labels into each ubatch the same way it carries the output flags, counts them (get_n_labels), and refuses by name a labelled position that is not an output, since the loss reads that position's logits row. Next (step 2): decode() runs a ubatch with labels through ggml_opt in the same loop, and llama_context::opt_walk::train_chunk, the second copy of decode's batch loop, is deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…k's copy of the batch loop is gone P1 step 2 (continuum ggml-org#5032 section 9.6). process_ubatch has one switch: a ubatch whose rows carry labels, on a context with a trainable adapter, is built fresh and runs through ggml_opt (llama_context::opt_ubatch: the loss over the labelled rows, the backward, the optimizer step at a period's end, and the active walk's taps). Every other ubatch runs graph_compute as before. opt_walk::train_chunk, which repeated decode's batch loop, now fills a chunk's labels and calls llama_decode. - cparams.training joins the graph reuse key. It was missing, so a serving decode with the same shape as a training ubatch could reuse the training graph. - Every batch view (server, speculative, mtmd, batched-bench, perplexity, parallel) carries its slice of the labels. - test-walk-exact: the device-refusal case is skipped on a CPU-only run, because the budget caps GPU memory only (ggml_opt_alloc). Its old skip test (graph size == 0) never fired: the CPU graph is measured in host buffers. Measured (fork bdd3cd5 + 32b8657 + this, M5, --device none, qwen3.5-4b hybrid Q4_K_M, 2026-10-14T04:46Z-06:01Z): every exact row is 1.0000 through the new path (exact walk vs one graph, checkpointed state, f16 and q8_0 caches, context + reply, recompute, rollback slots). The plain walk's rows (0.8376, 0.6710) are close to the pre-P1 Metal values (0.8378, 0.6714); the difference is backend, not path. There is no pre-P1 CPU run to compare against. Still owed: the Metal run, for the device-refusal case (the 4B, P1 vs bdd3cd5, in a quiet window). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ed run is restored P1 step 3a (continuum ggml-org#5032 section 9.6). The objective is each labelled run (a reply) given everything before it as served: only labelled rows see the shadow (the adapter being trained), so the context before a run carries no gradient and is never trained. The exact walk's backward now covers the reply only, not the window. - The shadow is added on the labelled rows by row selection (llm_graph_context::shadow_add): those rows of the base and of the shadow's term are gathered, added, and written into a copy of the base, so every other row is the stable graph's byte for byte. A ubatch with no label never applies the shadow; the label count is in the graph reuse key. (A 0/1 mask multiply was rejected in review: inf or NaN times zero is NaN, and x + 0 turns -0 into +0.) - SET_ROWS gains the destination's gradient in ggml backward (the result's gradient except at the replaced rows), which row selection needs. - The walk goes run by run: the context before a run decodes under her stable set; the run decodes under the shadow, trains (exact: in reverse, with the K/V and recurrent-state adjoints inside the run only; plain: a step per chunk), and is then restored, its cells and recurrent state decoded again under her stable set. One checkpoint helper and one rewind serve both walks; the plain walk's scratch sequence (n_seq_max >= 2) is gone. - decode_span no longer toggles cparams.training: process_ubatch sets it per ubatch (one of the four forward differences in section 9.6 removed). - The batch allocator refuses, before any mutation, a labelled row that is not an output or has more than one sequence, and a sequence whose labelled rows are not one run ending its rows in the batch. A training ubatch whose output rows are not exactly its labelled rows fails by name: the loss is their mean. - Tests: test-walk-plan for the per-run marks; test-walk-exact case 8, the cache after either walk byte-identical to a fresh stable decode; test-backend-ops SET_ROWS with grad=1. Measured: in the follow-up receipts (CPU and Metal, this SHA). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ot the window P1 step 3b (continuum ggml-org#5032 section 9.6). Under the served-prefix objective only a run's own positions carry K/V gradient, so the host accumulator is sized to the longest run and rebased to each run's start (llama_walk_acc::start_run, cparams.walk_acc_base). Before, it was sized to the window: at her served window on a 27B that alone was gigabytes of host memory. - The in-place (flash) path taps the window's cells but keeps only the run's rows before the chunk: ggml_opt_accumulate_grad_rows adds rows [row0, row0 + n) of a tensor's gradient into a caller's span; the rest stays on the device for its consumers. What later chunks put on a chunk's own rows reaches it through the surrogate, now one helper for both attention paths (build_walk_surrogate). - The host budget's refusal names the reply, which is what the K/V gradient grows with. Measured (this commit, M5 Metal, test-walk-exact, 2026-10-14): qwen2.5-0.5b 12:56:43-12:57:41Z and qwen3.5-4b hybrid 12:57:41-13:01:53Z, both OK; every exact row 1.0000/1.0000; the cache after either walk byte-identical to her stable decode; device refusal 3 retries, 4 refusals, 0 ubatch failures. Owed: the 27B at her served window with a 2k reply, accumulator bytes and memory peak measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…s no vocabulary-wide one-hot P1 (continuum ggml-org#5032 section 9.6, the precision and memory rows). A training step's labels were a one-hot F32 tensor the shape of the logits: [vocab x rows], 485 MiB at a 248k vocabulary and a 512-row chunk, beside the logits themselves. - CROSS_ENTROPY_LOSS and its backward accept I32 labels, one target per row, -1 for a row that carries no loss (ggml_cross_entropy_labels_ok names the two forms). CPU, Metal and CUDA kernels; SYCL refuses the I32 form until it has one. - ggml-opt: GGML_OPT_LOSS_TYPE_CROSS_ENTROPY_TARGETS (appended to the enum) builds an I32 labels input and counts accuracy against it directly. llama trains with it and writes each output row's target token. - GGML_OPT_TOP=N lists a training graph's N largest tensors at allocation: where its memory goes, measured. - test-backend-ops: CROSS_ENTROPY_LOSS with I32 targets (masked rows included), forward and finite-difference gradient. test_out_prod no longer crashes gradient mode (it built its weight in a null context), which had stopped every grad run. Measured (M5 Metal, 2026-10-14): test-backend-ops CROSS_ENTROPY_LOSS test and grad OK on CPU and MTL0 ([10,5,4,3] and [30000,2,1,1]); test-walk-exact OK on qwen2.5-0.5b (13:17-13:18Z) and qwen3.5-4b hybrid (13:18-13:23Z), every exact row 1.0000/1.0000 and the cache after either walk byte-identical to her stable decode. qwen3.5-4b chunk graph at window 8192, chunk 512, recompute (13:23-13:24Z): 2,417 -> 2,108 MiB on the flash path, 2,265 -> 2,037 MiB explicit. CUDA: not run here; owed on the 5090. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
P1 (continuum ggml-org#5032 section 9.6, the memory rows). The logits of a chunk and their gradient were the two largest tensors of its graph, [vocab x rows] F32 each (485 MiB apiece on the 4B at a 248k vocabulary and 512 rows), and their size does not shrink with the model. - ggml_opt_prepare_alloc_head: given the hidden state and the outputs (a row-wise function of it: MUL_MAT, MUL, ADD, SCALE, as a language model's head with its adapters and scales is), the head leaves the forward graph and is built again for each slice of rows; the loss is the mean over every row, each slice weighted by its share. Under recompute only one slice's logits and their gradient exist at a time. A head with any other node is not sliced. - llama trains with it, OPT_HEAD_SLICE_BYTES (64 MiB) of logits per slice: 67 rows at a 248k vocabulary. - ggml-opt rebuilt a per-step graph's loss into its predecessor's tensors when it chained the slices, and kept pred/ncorrect from the previous graph when a graph had none: both reset per graph now (the second read a freed tensor). Measured (M5 Metal, 2026-10-14): test-walk-exact OK on qwen2.5-0.5b (13:45:52-13:46:41Z) and qwen3.5-4b hybrid (13:46:41-13:52:10Z): every exact row 1.0000/1.0000, the cache after either walk byte-identical to her stable decode, the forward loss identical to the unsliced head (0.5B chunk 512: 4.122972 before and after). qwen3.5-4b chunk graph at window 8192, chunk 512, recompute (13:52:10-13:53:28Z): 2,108 -> 1,623 MiB on the flash path, 2,037 -> 1,552 explicit; the largest tensors are now the flash-attention backward's score tiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… and Metal P1 (continuum ggml-org#5032 section 9.6, the memory rows). FLASH_ATTN_EXT's backward was composed from generic ops over 4096-position tiles, three passes, keeping [tile x rows x heads] F32 score tensors (128 MiB each on the 4B at chunk 512) and adding up to 96 graph nodes per tile per layer, so it grew with the window. - ggml_flash_attn_back replaces the dead stub of the same name (it aborted: "TODO: adapt to ggml_flash_attn_ext"): q, k, v, mask in flash_attn_ext's layouts, d the output's gradient; one F32 result holding dQ, dK, dV (ggml_flash_attn_back_offset) and each query row's log-sum-exp and D. Scores are recomputed, never kept. - CPU: the reference (a pass over query rows, then over KV positions with the GQA group summed). - Metal: kernel_flash_attn_back_q / _kv for F32, F16 and q8_0 caches, one simdgroup per row; dq = scale (sum p dp k - D sum p k), so two sweeps over the window instead of three. - FLASH_ATTN_EXT's backward builds the op when the forward's back tile is GGML_FA_BACK_NATIVE; llama selects it when every GPU it computes on has the kernel (probed at the model's attention shape and the cache's types in opt_init), else the composed tiles (CUDA until its kernel). - test-backend-ops: FLASH_ATTN_BACK each backend against the CPU (F16, F32, q8_0, GQA, head 256, odd rows); FLASH_ATTN_EXT gradients through either backward, by finite difference. Measured (M5, 2026-10-14): test-backend-ops FLASH_ATTN_BACK 4/4 on MTL0 against CPU. Finite differences: the native and composed backward pass the direct check and sit at the same level on the SUM-wrapped one (1e-4 to 7e-4 against a 1e-4 bound, varying with the random init, for both): that check does not separate them; the walk test below does. test-walk-exact (native selected) OK on qwen2.5-0.5b (14:04:36-14:05:28Z) and qwen3.5-4b hybrid (14:05:28-14:11:14Z): every exact row 1.0000/1.0000, the cache byte-identical after either walk; the flash path against the F32 explicit gradient went from 0.9999/0.9995 to 1.0000/1.0000 on the 0.5B at an f16 cache. qwen3.5-4b chunk graph, window 8192, chunk 512, recompute (14:11:14-14:12:43Z): 1,623 -> 1,306 MiB on the flash path. SLOWER for now: the 4B walk test took 346 s against 329 s with the composed backward (the kernel is dot products per row, not simdgroup matrices); the matrix version is next. CUDA: no kernel yet (the composed path runs there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
Assisted-by: OpenAI Codex
…owners Assisted-by: OpenAI Codex
Assisted-by: OpenAI Codex
Assisted-by: OpenAI Codex
Share row projection and bounded head replay between direct decode and the walk. Keep shadow scatter coordinates explicit and preserve replayed operator precision. Cover sparse outputs and head adapters against the independent optimizer objective. Assisted-by: OpenAI Codex
PR64 (999d682) built the zero tensor of ggml_acc_or_set with ggml_fill over a new tensor, to remove 0 * inf = NaN. A tensor made by ggml_new_tensor is a graph leaf, and leaves stay allocated for the whole graph, so every such zero (cache-window-sized ones included) stayed resident. The zero is now a one-element filled leaf repeated to src's shape: the same NaN-free value, freed after its use. Measured (M5 Metal, qwen3.5-4b hybrid, WALK_MEASURE window 8192, chunk 512, recompute, 2026-10-14): chunk graph 2,118.6 MiB at PR64's head -> 1,286.6 MiB (the base 88c5c1e measures 1,305.6); explicit path 2,324.8 -> 1,551.8. Bisected: 88c5c1e + PR64's ggml.c alone 2,118.6; reverting only this hunk 1,305.6. test-walk-exact OK on qwen2.5-0.5b (22:25:13-22:26:04Z) and the 4B (22:26:04-22:36:01Z): every exact row 1.0000/1.0000, cache byte-identical after the walk, 4B chunk graph 666.8 MiB with recompute (PR64 head: 904.1). test-opt-dynamic-accum OK; test-backend-ops grad SET_ROWS and FLASH_ATTN_EXT on CPU: 0 failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… dO scaled for half tiles FLASH_ATTN_BACK's Metal kernel was one simdgroup per row with no reuse: 196 ms at a training shape (head 256, 16 heads over 4 KV heads, 256 rows, 2048 window, f16 cache) where the forward takes 2.5 ms. Now three passes (FlashAttention-2 style; the per-row kernel stays for K and V heads of different sizes): - stats (query blocks of 8): each row's log-sum-exp and D in one sweep, online; - dkv (KV blocks of 16): dK += dS^T Q and dV += P^T dO over the KV head's whole group; - dq (query blocks of 8): dQ += dS K. S and dP are 8x8 simdgroup tiles of half inputs accumulated in F32; staging steps rows with threads across columns (no division per element); partial blocks write through threadgroup memory. dO is scaled by the power of two that brings its largest |value| to [4, 8) (ggml_flash_attn_back_scale, recorded in the result after D by every backend): training gradients sit below half's normal range, and unscaled the half tiles cost the gradient its precision (4B against the F32 gradient at an f16 cache: 0.9943/0.9946 unscaled, 1.0000/0.9998 scaled). test-backend-ops: a perf case beside the forward at the same shape, with the backward's FLOPs. Measured (M5 Metal, 2026-10-14): FLASH_ATTN_BACK 4/4 against the CPU (f16, f32, q8_0, GQA, head 256, 300 positions x 37 rows). Perf at the shape above: 196 ms (per row) -> 67 ms per call, 321 GFLOPS (the forward: 2.5 ms, 3.45 TFLOPS): faster, not yet near the forward; the next work. 4B training measure at window 8192 (both resident slots busy, confounded): 77.8 s against 78 s for the composed backward, chunk graph 1,286.6 MiB against its 1,623. test-walk-exact OK on qwen2.5-0.5b and the 4B hybrid (23:02:52-23:08:04Z): every exact row 1.0000/1.0000, against the F32 gradient back to the composed path's values, cache byte-identical after the walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… threadgroup memory Cormac's review of 8ec4f9d: the tiled path's threadgroup memory is 26,176 bytes at head 256 but 50,752 at head 512, over Apple's 32 KiB, and the only guard was D == Dv && D % 32 == 0. The route now also requires the computed size to fit the device's maximum; otherwise the per-row path runs. test-backend-ops gains a head-512 case, which takes the per-row path. Measured (M5 Metal, 2026-10-15): FLASH_ATTN_BACK 5/5 against the CPU, the head-512 case included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ng context P2 (continuum ggml-org#5032 section 9.6). The engine half of "the server batches learning with serving": a reply trains on the sequence that served it, in the serving context, with no second context and no second copy of her window. - llama_opt_learn(ctx, seq, n_cached, tokens, labels, n, result): the sequence holds her context exactly as served ([0, n_cached)); attention cells past it are dropped, and a recurrent part must already end there (restore the served-prefix checkpoint first: a recurrent state cannot be cut back), refused by name otherwise, as is a pending rollback. The reply decodes with labels (the shadow on labelled rows only, PR64's physical-row path), then is restored under her stable set from the recurrent snapshot. Her context is never decoded again; other sequences are not touched. One batch is one exact step; a longer reply steps per batch. - The one-ubatch-per-batch rule moves from llama_opt_init into the walk (a named refusal there), so a serving context with -b > -ub can learn. - test-walk-exact case 9: two sequences in one context, her reply learned in place on sequence 0. Measured (M5 Metal, 2026-10-15): test-walk-exact OK. qwen2.5-0.5b: learned in place vs the walk's exact step cosine 1.0000, norm ratio 0.9997; her sequence byte-identical to a stable decode of the reply; the neighbour byte-identical. qwen3.5-4b hybrid (05:35:29-05:44:47Z): cosine 1.0000, norm ratio 1.0000, her sequence and the neighbour byte-identical; every other row unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…re touching the cache Cormac's review of 6fc7bc6: null tokens or labels crashed in the batch fill, and a reply past the sequence's context failed inside the training decode (and then the restore) after her reply cells were already dropped. Both are refused by name before any change. Each call also clears the failure state a previous refusal left, so a refusal never refuses the next call. test-walk-exact case 9 gains the refusals: null labels, past the context, a prefix the sequence does not hold; each refused, her sequence byte-identical after all three. Measured (M5 Metal, qwen2.5-0.5b, 2026-10-15): the three refused with her sequence byte-identical; the following live learn unchanged (cosine 1.0000, norm ratio 0.9997 against the walk's exact step; her cache and the neighbour byte-identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…); older GPUs take the per-row path Raised by Cormac's lane H on IntelMac: the tiled route checked head sizes and threadgroup memory but not has_simdgroup_mm. Measured (M5 Metal, 2026-10-15): FLASH_ATTN_BACK 5/5 against the CPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… serving context P2, server half (continuum ggml-org#5032 section 9.6). A learning turn is a task on the slot loop (SERVER_TASK_TYPE_LEARN), run between that slot's turns: no second context, no copy of her window. - POST /learn {id_slot, n_prefix, reply | reply_tokens, lora_id, lr} (only with --train-dir). The served prompt is what the slot holds; her recurrent state comes back from a checkpoint one token before the prompt's end, which a learning lane keeps (the prompt checkpoint offsets gain 1 when --train-dir is set); the last prompt token is re-decoded labelled with the reply's first token; then the slot returns to her whole served prompt (checkpoint + that token decoded again), so her next turn reuses it. The shadow joins her adapter set for the step only, at scale 1; llama_opt_init runs once (AdamW, the request's learning rate), one shadow per server run for now. - llama_opt_learn: trains to the last labelled row (an unlabelled row after the run is refused in one batch), takes exactly ONE optimizer step per call (manual period end at the ubatch that completes its labelled rows, each ubatch weighted by its share), and reserves no host logits for rows that feed only the loss (opt_outputs_internal; the walk too): a 512-row chunk at a 248k vocabulary was 485 MiB of host RAM, and the serving context's output cap tripped on it. - KV training admission: a context with a stream per slot may train with flash attention when the batch touches one stream (was: refused whenever the cache had several streams, which is how a serving lane runs). - ggml-opt: a caller-driven step's loss is recorded unscaled by opt_period, so the result reports it right (was 4x at -b 2048 -ub 512). - test-walk-exact case 9 runs on a unified F32 cache with explicit attention AND a stream per slot with an f16 cache and flash attention. Measured (M5 Metal, 2026-10-15): test-walk-exact case 9 on qwen2.5-0.5b: unified 1.0000/0.9997, streams + flash + f16 1.0000/1.0000 against the walk's exact step, her sequence and the neighbour byte-identical, the three refusals refused with her sequence unchanged. test-opt-dynamic-accum OK. llama-server, qwen3.5-4b hybrid, 2 slots, -fa on, --train-dir: a served turn, then POST /learn of a 17-token correction four times: loss 2.860 -> 2.525 -> 1.769 -> 1.111, 2.3-2.6 s per learn (slow: GET_ROWS_BACK and OUT_PROD are not on Metal in this branch; the Metal training-op branches merge next). Serving after learning unchanged on her slot (the shadow applies to labelled rows only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ed row backward, GET_ROWS_BACK on Metal) into the P2 line So a learning step stays on the GPU. Conflicts were adjacent additions (FLASH_ATTN_BACK beside GET_ROWS_BACK), both kept. Measured (M5 Metal, 2026-10-15): test-backend-ops MTL0 against the CPU: FLASH_ATTN_BACK 5/5, OUT_PROD 414/414, GET_ROWS_BACK 4/4, PAD 30/30, SET_ROWS 172/172, CROSS_ENTROPY_LOSS 5/5, RMS_NORM_BACK 20/20, SOFT_MAX_BACK 24/24. llama-server qwen3.5-4b, POST /learn of a 17-token correction: 2.3-2.6 s -> 298-410 ms per learn, losses unchanged (2.860, 2.525, 1.769, 1.111). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
Cormac's review of 018ab27 (P1-b): the re-decode of the last prompt token ran with the learn set (shadow at scale 1). The shadow was already off for a batch without labels (llama-graph.cpp lora_on: n_shadow_rows is the ubatch's label count, llama-context.cpp graph_params), so the restored state was hers; her slot's own set is now applied before the restore so this does not depend on that gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… works Measured on the M5 (qwen3.5-4b hybrid, 2 slots, the 4B test server; her live lane on the same GPU was busy throughout, so compare the pairs, not the runs): learns run back to back took the other slot from p50 218 / p95 450 ms per token to 686 / 2,567. Learning must barely touch the flow. While any other slot is processing, a learn waits out a rest of t*(1/LEARN_DUTY - 1) after a learn of t (LEARN_DUTY = 0.10); with every other slot idle it runs at once. A deferred learn is bound to her slot (task.id_slot), so it comes back when a slot frees: learning lands in the gaps. Measured after (06:13:32-06:13:53Z): the other slot p50 46.8 -> 46.1, p95 73.3 -> 73.8 ms per token with learning on; the two learns waited for the gap (p50 10.4 s wall) instead of the GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…lock, on SIMD dot products FLASH_ATTN_BACK's CPU kernel converted every f16 K and V row again for each query row (twice in pass 1), accumulated its dot products as scalar doubles, and in pass 2 regathered the strided query and output-gradient rows for every KV position. It is the reference other backends are checked against and the backward on machines with no usable GPU. Now pass 1 works blocks of 8 query rows of one head and pass 2 blocks of 16 KV positions of one KV head: each K and V row is converted once per block, each query and dO row gathered once per block, the dot products and accumulations run on ggml_vec_dot_f32 / ggml_vec_mad_f32, and D stays a double. Scratch per thread: max(8*(M + D + Dv), 16*(D + Dv)) + D + Dv floats (ops.h), sized by the work planner (ggml-cpu.c). Measured (IntelMac i7-8850H, 12 threads, test-backend-ops perf, 2026-10-15): at head 256, 16 heads over 4 KV heads, 256 rows, 2048 window, f16 cache: 2,195.5 -> 316.2 ms/call (9.78 -> 67.92 GFLOPS); the forward at the same shape is 51.6 ms. Exactness: the old and new kernels on identical seeded inputs (causal mask, GQA, f16 cache, dO at 1e-3) agree to float rounding on every part of the result: NMSE dq <= 1.97e-13, dk <= 1.49e-13, dv <= 1.00e-13, lse <= 6.6e-16, D <= 4.7e-14, scale identical, at (128,8,2,512,64), (64,4,2,300,37) and (256,16,4,2048,256). test-backend-ops grad: the native backward's MUL case OK; its SUM finite-difference case misses 1e-4 on both the old and new kernels (3/5 and 4/5 runs, random inputs), and goes to inf with dV negated, so it reaches this kernel. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…l holds its prompt Learning usually comes after review, when her slot may have served other turns since: n_prefix alone said only that the slot held that many tokens, not that they were this turn's. /learn now also takes `messages` (the conversation she was served, its last message her reply or the correction): rendered through the same chat template serving framed it with, the served prompt is everything before the reply, and her slot must hold exactly those tokens, else the learn is refused with the first differing position. One renderer: /train's render_example becomes train_render, shared by both routes, and server_trainer exposes its serving renderer. Measured (M5 Metal, qwen3.5-4b hybrid test server, 2026-10-15): a turn served through /v1/chat/completions (30 prompt tokens), then /learn with the same messages plus a corrected assistant reply, three times: loss 1.573 -> 1.085 -> 0.474, 632-687 ms each. A conversation her slot never served: refused, "slot 0 no longer holds this turn's prompt (it differs at token 23 of 30): serve the prompt again, then learn". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
ggml_cross_entropy_loss_weighted(logits, targets, weights): row i contributes w_i * -log softmax(logits_i)[t_i] / rows and its gradient scales by w_i, so how much a row counts is the caller's to say (her relevance on what she perceived), not only whether. A zero weight skips the row's softmax entirely. The weights are F32, one per row: 4 bytes a row beside a vocabulary-wide logits row. CPU and Metal implement it; CUDA refuses the weighted form in supports_op (the scheduler runs it on the CPU) until its kernel takes the weights. Measured (M5, 2026-10-15): test-backend-ops CROSS_ENTROPY_LOSS on MTL0 7/7, gradients (finite difference, weighted rows including zeros) OK on MTL0 and CPU at [10,5,4,3] and [30000,2,1,1]; full grad mode 19815/19815 on both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
Assisted-by: Codex
Assisted-by: Codex
…times Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
…ssage) A hybrid (recurrent) or SWA model resumes a slot only from a saved checkpoint. The server checkpoints at the last user message, at user messages at least --checkpoint-min-step past the previous checkpoint, and near the prompt end. A client whose stable prefix ends EARLIER than the last user message (an append-only context followed by per-request framing and the latest tool exchange) loses up to min-step tokens on every request: the next request rolls back to the last checkpoint before the divergence and re-reads from there. checkpoint_user_message (index among the request's user messages, -1 = none) names where the stable prefix ends; the prefill breaks there and checkpoints unconditionally, like the last user message. One code path, every backend. Measured (M5, Qwen3.5-0.8B Q8_0, Metal, -c 16384, --ctx-checkpoints 32, --checkpoint-min-step 4096; two requests sharing a 40-message body, then per-request framing + an assistant call + a user tool result): second request cached 8,895 / fresh 4,511 without the anchor; cached 11,995 (the body's end) / fresh 1,411 with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continuum card db6f418b. Built on the deployed pin 50b02b8 (not yet on master), so this diff stacks on that line.
Problem. A hybrid (Gated DeltaNet) or SWA model resumes a slot only from a saved checkpoint. The server checkpoints at three places: the last user message, user messages at least
--checkpoint-min-steppast the previous checkpoint, and near the prompt end. Continuum's minds send an append-only context, then per-request framing, then the latest tool exchange (assistant call plus user result). So the stable prefix ends EARLIER than the last user message, and every request rolls back up to min-step tokens and re-reads them. Live on the M5, Qwen3.8-27B with--checkpoint-min-step 5808: consecutive prompts shared ~34k tokens, but the engine cached 21,729.Change.
checkpoint_user_messageis an index among the request's user messages (-1 = none). It names where the stable prefix ends. The prefill breaks there and checkpoints unconditionally, the same way it treats the last user message. It is one code path, so it works on every backend. +26/-2.Measured. M5, Qwen3.5-0.8B Q8_0, Metal,
-c 16384 --ctx-checkpoints 32 --checkpoint-min-step 4096. Two requests share a 40-message body, then per-request framing, an assistant call and a user tool result:Not yet measured: the paged-slot restore path (checkpoint sidecars saved with KV pages) on the 27B. That receipt comes from the live deploy.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo