[twu3202]

Required human contributor answers

1. What outcome are you trying to achieve?

   Run faster on native qwen3.5 workers for requests that ask several questions about the same thing.

2. What is the highest-leverage variable for that outcome?

   How many prefix tokens each request prefills.

3. How will this work move that variable rather than create busywork?

   qwen3_5: run prompts that share prefixes once per prefix #99 runs a shared prefix once and continues each branch from its state. qwen3_5: add a GEMM handle with one algorithm per weight shape #98 is the pre-req of qwen3_5: run prompts that share prefixes once per prefix #99, making a GEMM row's result no longer depends on how many rows are in the call.

4. Who is the customer or user, and what do they actually need?

   agent asking several questions about one same thing such as screenshot, and they need faster answers.

After answering the four questions: is a PR needed at all to achieve this outcome?

Yes, to answer #85.

For a bug fix: link actual observed failure evidence.

N/A, not a bug fix.

Purpose

For #85, the shared executor step of the plan (§3; PR 3 in §7), as revised in this update. It now builds on #96's prefix continuation and on #98, so it should be merged after #98; until then the diff here includes #98. This PR's own change is 2ff1396; the merges on top bring in #98 and today's main. No worker change; #113 builds on it for Decider.

Model::forward_shared runs prompts that share token prefixes: a request prefix, then groups with their own prefix and branches. Each prefix runs once, and each branch continues from the state after it, with positions and the causal mask continuing too. It returns one final hidden state per branch, in order.

• State: each prefix is captured with jev_vl: add native inference and prefix caching #96's run_window into a PrefixState: per linear-attention layer the float32 recurrent state and the last three conv inputs, per full-attention layer the prefix keys and values. Each branch then runs as its own pass from that state. The request and group states persist across calls and grow only for a longer prefix; their contents are valid only within one call, which runs eagerly and synchronizes its stream before returning, also on error.
• Split points: each prefix ends at its last multiple of 64 tokens, and its remaining tokens run with every group or branch after it. With a split inside a chunk, the gated delta rule rounds differently: up to 1.25e-2 relative L2 in the final hidden state on 4B and 1.20e-2 on 9B against forward_fixed, in an earlier version of the tests below that also ran exact splits. The result would also depend on the prompt's siblings, since they set the common prefix. Rounding down avoids both, at the cost of up to 63 extra tokens per branch.
• GEMMs: the shared runs use the fixed-algorithm handle from qwen3_5: add a GEMM handle with one algorithm per weight shape #98 (reference M 64), so each result equals forward_fixed, a plain pass over the full prompt with the same handle and the same run_window, bit for bit, however the prompts are grouped. It differs from forward, which picks algorithms per row count, by rounding.
• Changes to run_window: it takes the GEMM handle; it can capture after a continuation, which a group prefix needs; and for text positions it offsets the rotary tables by the window start, which no caller reached before. forward_multimodal_capture and forward_multimodal_continue behave as before, and forward, forward_batch, forward_multimodal and CUDA Graph replay are unchanged.

docs/architecture.md, the runtime README and the backend README now describe the entry point, its state and its extra memory.

Test Plan

System1-Omni Version / Commit: 18cbd99, which merges main 3e9cc33 into cb8ddb3; cb8ddb3 merges #98 (d8cd04f, with main bd81983) into 2ff1396; 2ff1396 is on #98 (d717e49), on main 0d25210.

CPU: the five Rust steps in CI (format, Clippy, frontend tests, workspace tests, release build), mkdocs build --strict, and a test of how prompts split into runs.

GPU, one RTX 6000 Ada (sm_89), on the Cua-S1 4B and Open-Jev-9B exports, with random token ids:

• forward_shared against forward_fixed on every full prompt: request prefixes of 0, 1, 63, 64, 65, 300 and 1,000 tokens, groups with 0, 1, 70 and 128 prefix tokens, and branches of 1 to 200 tokens;
• state isolation: the same request twice, after a longer request that grows the buffers, and with groups and branches reversed; forward before, between and after the shared runs; invalid requests (no groups, an empty group or branch, out-of-vocabulary ids anywhere) refused, after which the model still gives the same results;
• the same with CUA_S1_GRAPH=1 on 9B, where the plain forward between shared runs replays a CUDA Graph;
• all GPU kernel tests, including jev_vl: add native inference and prefix caching #96's prefix-state test with the OmniJev-4B export as the multimodal checkpoint;
• the plain forward on this branch against main, Cua-S1 and Open-Jev-9B workers side by side;
• timing of M1-like layouts on 9B.

Test Result

CPU. All six commands pass on macOS, including the split test.

forward_shared against forward_fixed. Every branch is bit-identical, at every request prefix length and group shape above, on 4B and 9B, and on 9B with CUA_S1_GRAPH=1. Against forward, the difference is that of the GEMM algorithms alone: up to 1.84e-2 (4B) and 1.53e-2 (9B) relative L2 on these random prompts. That is above the plan's 1e-2 hidden-state gate and inside #92's native vs reference spread (p99 0.023), so the update proposes how to treat that comparison.

State isolation. Every check passes, and the model gives the same results after the refused requests. All of the invalid requests fail validation before any GPU work, so the error path's synchronization isn't exercised by these tests.

Kernel tests. 13 of 13 pass.

Plain forward unchanged: Cua-S1 567 of 567 byte-identical (134 of them forward passes) in eager and CUDA Graph modes, and Open-Jev-9B 253 of 253 identical apart from timing, against main 0d25210.

Timing (9B, random tokens, one warm call each, one request prefix and 38-token branches):

Request prefix	Tokens rerun per branch	Branches	forward_shared	One forward per branch
3,072	0	255	6.24 s	124.99 s
3,100	28	255	6.97 s	131.02 s
3,135	63	255	7.47 s	131.85 s
1,024	0	32	0.86 s	5.41 s
1,050	26	32	0.89 s	5.47 s
256	0	255	5.35 s	15.27 s

A single pass with the fixed GEMMs takes 3% less time at 64 tokens, 8% more at 1,024 and 17% more at 3,072 than forward; the prefix pass carries that cost once per request. These are synthetic layouts with random tokens. M1's real 3,093-token prefix would rerun 21 tokens per branch. The plan's measurement on real Open-Jev requests comes with the worker PR.

Demo / evidence

No demo: no worker uses the entry point yet. Test output and scripts are kept locally.

Self-review

• I have reviewed the full diff and addressed the issues I found.
• I have checked that the change follows the project's architecture and stays focused on the stated purpose.
• I have run the checks appropriate to this change and reported commands, results, and anything I could not verify above.
• I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

[Levius-Fubuki]

Integration note: reject unsupported execution profiles before device work

forward_fixed() and forward_shared() were introduced for the merged-weight path. When integrating the unmerged/reference profiles from #118/#143, explicitly reject unsupported combinations before allocations or CUDA calls.

Preserve the prefix-state, scratch-growth and Graph-shape guards during conflict resolution.

Add a CPU regression that proves early rejection of unsupported profiles. Device qualification should cover merged 4B and 9B, Graph off/on, reordered groups/branches, growth, ordinary-forward interleaving and recovery after invalid input.

Shared-prefix output must match a full pass using the same fixed handle exactly. Any comparison with the default handle’s different arithmetic should remain a separately reported metric.

---

Reviewed head: 18cbd992f422faad139ca0cf01832d20b3ec8bb8.

[twu3202]

Agreed. Main has no unmerged or reference profile yet, so if #118 or #143 lands first I'll make forward_fixed/forward_shared refuse those profiles before any allocation and add a CPU test for it. The device checks you list are the ones in the test plan (Graph on for 9B only), and the default-handle difference stays a separately reported number.

[Levius-Fubuki]

@twu3202 Dependency update for the execution-profile guard we discussed: #117 is now in main (0410e841d3ec72c1462609fa7721d0e424ac356c), and I have also followed up in #98 about the combined-interface ABI update. Please refresh #99 after #98, preserving prefix-state, scratch-growth and Graph-shape guards.

When integrating the unmerged/reference profiles from #118/#143, keep the agreed explicit rejection in forward_fixed/forward_shared before allocations or CUDA calls, with a CPU regression. Retain merged 4B/9B same-fixed-handle exact comparisons, reordered groups/branches, growth, ordinary-forward interleaving and invalid-input recovery; preserve the reported Graph coverage boundary (9B in the existing run). Differences from the default handle remain a separate numerical metric.

#118 is marked merged into codex/qwen-vision-27b, but is not yet included in current main. Its source-pinned A800 ABI8 JEMM evidence does not qualify fixed/shared-prefix execution. These reminders do not change the existing gates or expand the historical hardware claims.