[bong-water-water-bong]

Follows the pin-side change (1bit-MONSTER/llama.cpp #137): the next-layer routing predictor that fits Qwen3.8-Flash-Next's hyper-connections. Nothing here changes a running configuration — --moe-predict does nothing without --moe-slots, which this build refuses until the HRX streamer lands.

Why

The prefetch that hides a streamed model's expert reads used the gate-ahead router. On Flash-Next that predictor is wrong by construction: with four 2560-wide residual streams, layer l+1's router input is l+1's own mixer applied to the residual after layer l's MoE, so applying l+1's router to layer l's FFN input mixes through the wrong weights. It was right about two times in three, and every wrong guess is a read from a drive that is already the limit: 133 s per 512-token pass without prefetch, 247 s with depth 1 at 4,608 slots (docs/moe-streaming.md, "Flash-Next, streamed").

What lands

• docs/moe-streaming.md, new subsection "The next layer's own block (Flash-Next; untested on device)": what the predictor computes (l+1's PLE injection, its attention or DeltaNet block, its combine, then its FFN mixer and router; ranking only, deeper than the layer's top-k), why an attention successor needs no state suppression and the recurrent/conv/PLE histories do (scratch), the flag and its measurement variables (_DEPTH, _MARGIN, _MIN, _LAYERS, _OUT, _STATS), and the explicit statement that it is untested on device and what is missing (the prefetch consumer, i.e. the streamer). The "Next" list item 3 now points at the built path.
• docs/serve.md: the synopsis line and a short paragraph under the streaming section, with the same "untested on device" note.
• app/serve.cpp: --moe-predict off|router|block, refused if the value is unknown, passed through as ONEBIT_MOE_PREDICT beside ONEBIT_MOE_PREFETCH inside the ONEBIT_MOE_STREAMING_HRX block (the only place a streamed run reads it).

Verification

Docs-only + one option in serve; the engine is not built here (the HRX toolchain is not part of this host-side run, and no device run was made — that is the next step after the lane opens). The pin-side branch is built and its unit test passes on this box.

Device plan (from the private notes, in this order)

1. ONEBIT_MOE_PREDICT=router vs block, resident Flash-Next, hit rate of the next layer's experts; falsifier: block not >= 90 %.
2. Streamed, --moe-slots 4608 and 9216, --moe-prefetch 1 --moe-predict block: pass time against the 133 s baseline; falsifier: not below it at 4,608 slots (a correct predictor with no drive slack — then gate on resident share).
3. Long context 64K/128K for the indexer's per-layer cost.

🤖 Generated with Claude Code

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit af54be8)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 137 - Partially compliant Compliant requirements: MoE expert slot grouping by size classes implemented Memory usage reduced from 34.4 GiB to 27.0 GiB for Flash-Next --moe-predict flag added with appropriate validation Documentation updated with new predictor behavior and usage Non-compliant requirements: The PR does not include the actual implementation of the prefetch consumer (streamer) that is needed to fully test the new predictor on device Requires further human verification: The claim that the new predictor is "untested on device" needs verification that the streamer implementation is indeed missing Memory usage measurements should be confirmed on actual Strix Halo hardware

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing validation for --moe-predict dependency The --moe-predict flag is added without proper validation that it requires --moe-slots to be set. While the documentation mentions this dependency, the code does not enforce it at runtime, which could lead to unexpected behavior or errors when the flag is used without proper configuration. else if (a == "--moe-predict") { o.moe_predict = next(); if (o.moe_predict != "off" && o.moe_predict != "router" && o.moe_predict != "block") { throw std::runtime_error("--moe-predict takes off, router or block, not '" + o.moe_predict + "'"); } } Unverified claim about predictor performance The documentation states that the new predictor is "untested on device" and that the predictor and its hit-rate table are built and unit-tested host-side. However, the PR description claims that the Coder-30B chat replay runs normally with 95.2% hits, which contradicts the "untested on device" claim. This discrepancy needs clarification. ### The next layer's own block (Flash-Next; untested on device) The gate-ahead predictor above does not fit Qwen3.8-Flash-Next: its layers keep four 2560-wide residual streams (hyper-connections), so layer l+1's router input is l+1's own mixer applied to the residual *after* layer l's MoE, and applying l+1's router to layer l's FFN input mixes through the wrong weights. The measured result is in "Flash-Next, streamed" below: right about two times in three, and a loss on a drive that is already the limit. The predictor that does fit runs layer l+1's own token-mixer block on the current residual, before layer l's MoE output exists: l+1's PLE injection, its attention or DeltaNet block, its combine, then its FFN mixer and its router. Only the ranking of the router logits is used, and more candidates than the layer routes to are kept so the near misses are read too. The output of the model does not change: the block only decides which experts a prefetch reads, and whatever a token really routes to is still waited for. State writes of that speculative pass are suppressed where the exact pass would read them: the DeltaNet recurrent state and the conv histories are written into scratch tensors, and an attention successor needs nothing, because the exact block rewrites the same cache cell before anything reads it. The speculative nodes are also left out of the fused-op manifest, so they cannot make a fused dispatch claim the wrong tensor. `--moe-predict off|router|block` (env `ONEBIT_MOE_PREDICT`) selects it, and takes effect with `--moe-prefetch 1` once the HRX llama.cpp runs the streamer; `router` keeps the cheap residual-only form for comparison. `ONEBIT_MOE_PREDICT_DEPTH`, `_MARGIN` (read a deeper candidate only while its logit is within the given distance of the best one), `_MIN`, `_LAYERS` and `_OUT` (a trace of predictions and the routing that was really used) tune and measure it, and `ONEBIT_MOE_PREDICT_STATS=1` prints the hit rate per layer when the process exits. The graph side is `src/onebit-moe-predict.*` plus one hook in the `qwen4exp` graph in the llama.cpp pin, and it is **untested on device**: the predictor, its readback and the hit-rate table are built and unit-tested host-side (`tests/test-onebit-moe-predict.cpp`), and the prefetch consumer (the streamer's in-graph op) is the step that needs the GPU lane.

[bong-water-water-bong]

Review (orchestrator): serve flag passthrough + docs, pairs with fork #137; draft until that one's device numbers are in, then both merge with the pin.

[github-actions]

Persistent review updated to latest commit b27c692

[github-actions]

Persistent review updated to latest commit b27c692

[bong-water-water-bong]

Triage: needs work — not the code, the claim under it.

What the diff is (3 files, +61/-3): Options::moe_predict at app/serve.cpp:164, pushed into the child env at app/serve.cpp:983 (if (!o.moe_predict.empty()) env.push_back("ONEBIT_MOE_PREDICT=" + o.moe_predict);, inside the same slots > 0 block as --moe-slots), arg parsing at app/serve.cpp:2693-2695 validating off|router|block, plus docs/serve.md and docs/moe-streaming.md.

The flag is inert against every pin we have. Nothing in the pin reads the variable:

$ git grep ONEBIT_MOE_PREDICT e7e5cd08          # empty
$ git ls-tree e7e5cd08 src/ | grep onebit        # empty
$ git grep ONEBIT_MOE_PREDICT 40c4ccb76e48       # empty (the #610 bump target)

and docs/moe-streaming.md:61-62 (added by this PR) says the graph side is src/onebit-moe-predict.* plus one hook in the qwen4exp graph "in the llama.cpp pin". That is false today and would stay false through the #610 bump.

The consumer is llama.cpp #137 (1bit/moe-predict-block), and it cannot reach a pin as it stands: it is a draft, its base is master, and its commit sits on upstream master (5f0d4619a, parent c35b66744, 11,511 commits of upstream divergence from our pin) — not on the fork's 1bit/hrx-vulkan-patched lineage the engine pins. It also has no prefetch consumer: the PR says llama.cpp still runs all 48 layers in one graph, so the prediction is computed and read back but nothing is prefetched with it.

So: either (a) wait for #137 to be rebased onto the fork branch, merged and bumped into a pin before this docs text goes in, or (b) land this as an explicitly forward-declared stub and change that sentence to "not in the pin yet — engine #551 / llama.cpp #137". The arg parsing, validation and off default are all fine; off is also the current behaviour, so the environmental half is harmless either way. Marked "untested on device" in the PR body — that is the honest state and I am not asking for that here.

[github-actions]

Persistent review updated to latest commit d6d0eed

[github-actions]

Persistent review updated to latest commit a782ed6

[github-actions]

Persistent review updated to latest commit af54be8