Repository navigation
decider: reuse request prefixes with conservative auto selection - #113
hsliuustc0106 merged 49 commits into
Conversation
Add CUDA operations that continue a sequence after a shared prefix, for request-local prefix reuse (InferMatrixLab#85): the Gated DeltaNet conv with the inputs of the three positions before it, the chunked prefill from and to a float32 state, gated attention for queries after cached keys, and a pitched device-to-device row copy. The existing conv, prefill and gated attention are these operations without history, state or cached positions. Bump the ABI to 6 and add the Rust bindings and GPU tests against the unsplit calls.
cuBLASLt's heuristic picks an algorithm per M, so a row's result can change with the number of rows in the call, and a prefix and its branch run separately round differently from one pass over the same tokens. Add cs1_gemm_create_fixed: one algorithm per weight shape (N, K, ldy), the heuristic's first choice at a reference M among algorithms without split-K, for every M. The default handle is unchanged. Bump the ABI to 7 and add a GPU test that each projection's rows match across M.
|
@hsliuustc0106 Follow-up to your latest review. The contributor has explicitly confirmed completion of self-review, and all four self-review declarations are now checked in the PR description. The four human contributor answers and PR-necessity decision remain recorded. This follow-up updates the PR description only; the source head remains Please re-check the updated description and reassess the checklist-only readiness item, then re-review this PR. |
hsliuustc0106
left a comment
There was a problem hiding this comment.
Incremental re-check at new head e790c3cfc4: one new docs commit since my last check — "Document current Decider ABI 7 device qualification" — and, decisively, the four-item self-review checklist is now checked (4/4), completing the post-#138 template gate that was the sole blocker (human answers and PR-necessity were already recorded).
The new commit is reviewed and it improves honesty rather than weakening it: it records the 2026-10-13 ABI-7 device qualification (RTX 4090, five frozen requests, 20 rows, eager/packed/Graph/fixed fidelity controls, retirement and capture-fallback checks) with explicit limits stated — "two timing samples per mode are only descriptive", "neither campaign establishes task quality, every supported GPU or universal bit-identical agreement with Transformers", and rebuild-together requirements per consumer generation. Fork CI at this exact head: rust/cuda-contract/benchmarks/docs all pass.
No actionable findings remain. APPROVED. Unverified checks: the 2026-10-13 qualification itself is author-run (no GPU on this host); the historical 2026-10-07 campaign remains archive-based; Rust suite covered by the exact-head CI run; broader HTTP/performance coverage stays outside the campaign as the docs state. Merge in stack order (#94 → #110 → #111 → #112 → #113).
|
@hsliuustc0106 one note on merge order: this branch also carries #98 ( |
FindingsP3 — Forced modes fail hard with a request 503; prep ops before enabling them in production
P3 — M-independence of the fixed algorithm rests on one GPU check
|
hsliuustc0106
left a comment
There was a problem hiding this comment.
Incremental re-check at new head a631b8b95c after the rebase wave (prior review: APPROVED at e790c3cfc). Wave-verification note (this pass): the rebase wave was verified systematically — every own-commit series is patch-identical per range-diff; every post-pin merge resolution was diffed against both parents with zero dropped lines; approved-content markers verified present in every tree; the key new merge 3be46fa (shared-vision line × jemm v2 tip) carries #143\u2019s new test files intact and is ABI 8/8 consistent at every descendant head.
The three merge resolutions were verified: the approved decider docs (recipe/decider/README.md + validation.md) survive intact with pure additions from the #111-line ancestry; the laya test files are unioned, not dropped; ABI 7 remains consistent (ops.h = manifest) at this head; the prefix-workloads consumer and tests/decider/data/README.md are present. Upstream CI (full suite incl. rust/build/benchmarks) green at this exact head.
APPROVED (verdict carries). Unverified: the 2026-10-13 device qualification remains author-run (no GPU on this host); Rust suite covered by exact-head CI.
Preserve graph-loading ownership and diagnostics, combine fixed and reference GEMM profiles under ABI 9, and retain model-specific numerical paths and deployment recovery guidance.
Add a per-process sequence to the PID and clock nonce and cover simultaneous fixture creation with a shared timestamp. The full workspace run observed AlreadyExists with the original helper.
hsliuustc0106
left a comment
There was a problem hiding this comment.
Incremental re-check at new head 5e340cab24 after the rebase/merge wave onto main@32bdd2a (prior review: APPROVED at a631b8b95c). Own-commit series patch-identical (git range-diff, named refs, cat-file verified). New since that approval: one test-only commit 5e340cab "test(cua-s1): prevent concurrent export fixture directory collisions" — reviewed: prevents parallel test runs from colliding on the shared export fixture directory; no production code touched. The merge resolutions carry the absorbed jemm/decider lineage, consistent with the pass-57 wave verification. Upstream CI at this exact head: full suite incl. rust/build/benchmarks green.
APPROVED (verdict carries). Unverified: the 2026-10-13 device qualification remains author-run (no GPU on this host); Rust suite covered by exact-head CI.
Fresh ABI 9 A800 GPU qualification — 2026-10-22Ran the integrated source Evidence release and all five assets · verification guide Archive: 433,822 bytes, SHA256
Extract and run Scope: this A800/ABI9 campaign supersedes the previous device-validation handoff for the bounded matrix above. The remaining14 optional cases are explicitly listed, and full other-model/27B/Hopper/other-GPU qualification remains unverified. There were zero selected measured timing samples and no speedup, throughput, latency, peak-memory or broad model-quality claim. Original source and numerical gates were unchanged; launcher/cache/auditor setup failures and their corrections are retained. All campaign workers exited and final recorded GPU use was0MiB. This is execution evidence, not a new human attestation or maintainer approval. |
ABI9 A800 follow-up: all remaining14 optional cases completed — 2026-10-22Exact source is unchanged:
CLM used genuine pinned weights and the documented16-tensor export oracle; fresh text-oracle regeneration was byte-identical. Laya's three tests are CPU-only: pinned206-tensor conversion, official17-case packing and legacy-temperature inventory all pass. OmniJev used genuine heads/tokenizer against committed references. Jev-VL checked all12 frozen R1 image entries with the original restored manifest; this is a tokenizer/prefix check, not a full-VL-model export. Open-Jev used the complete pinned27B export, because the supported9B contract disables packing. All18 original base shards and six trained checkpoint files were verified. The maintained exporter produced17 files/51,269,477,178 bytes; before/after hashes were unchanged across the main GPU test. The regression passed candidate isolation/order/state/shape checks; a separately planned trace replication additionally observed 10 actual captures +4 replays, zero fallback. Both GPU tests used ABI9 library SHA256 Cumulative frozen optional inventory at this head: 57/57 unique cases (43 GPU +14 CPU), 64 Rust pass executions. The follow-up contributes15 executions:14 new cases plus one Graph trace replication. This is functional regression coverage, not blanket full-model quality or other-hardware qualification. Zero benchmark samples; no performance/speedup claim. All636 recorded source files remain unchanged; GPU memory returned to0 MiB. Complete additive evidence, exact14-case catalog, raw commands/logs/oracles/pins, portable audit and independent review. The prior release and its210 raw records remain immutable and are included verbatim under |
|
Compatibility with #149 is now resolved downstream at #149
Exact combined-head CI · Docs are green. This is targeted qualification of the combined head, not a rerun/relabeling of this unchanged PR's earlier57-case qualification; that historical evidence and source pin remain intact. The new integrated source needs fresh maintainer review; previous approvals are not extended by this comment. Immutable evidence, reconstruction patch, review map, raw records, independent review and portable audit. Archive 1,297,304 bytes, SHA256 |
hsliuustc0106
left a comment
There was a problem hiding this comment.
State correction: my re-approval of this PR at this exact head (5e340cab24, pass 73) was posted with the wrong review event type (COMMENT instead of APPROVE) due to a scripting error in my posting script — the full re-approval verdict and its verification are in that review body directly below and are unchanged: zero actionable findings, upstream full-suite CI green at this head. This review corrects the recorded state to match that verdict. Unverified checks as listed there: 2026-10-13 device qualification author-run; Rust suite via exact-head CI.
hsliuustc0106
left a comment
There was a problem hiding this comment.
PR #113 incremental review — verdict carried — pass 84 (head b3cd06a27)
**Verdict: APPROVED (carried).
Pinned state (re-verified)
- PR head:
b3cd06a27f0995cd7719c8148b8b479e11e44a53(git rev-parse origin/pr/113==gh pr view 113headRefOid). Non-draft, basemain, title "decider: reuse request prefixes with conservative auto selection". - Previously reviewed head:
5e340cab2(my APPROVE of 2026-10-22T10:07Z, a state correction of the pass-73 re-approval; zero actionable findings on record — re-read both review bodies via the reviews API, confirmed). - First-parent chain (verified with
rev-parse ^1/^2, all SHAscat-file -eOK):
b3cd06a2(authored) ←782db6f4(merge) ←267614bf(merge) ←5e340cab2. git merge-base --is-ancestor 5e340cab2 origin/pr/113→ yes. No own-series commit was rewritten (the previously approved head is the direct first-parent ancestor); range-diff gate therefore trivially satisfied, not run.origin/maintip5ba6a3ba6is the merge-base; merge267614bfabsorbs3a289bd1(codex/pr112-main-20261023, contains main) and merge782db6f4absorbsf54b3cdf(prefix-shared-executor, contains main). Tree delta5e340cab2..head: 12 files, +592/−36.
Merge hygiene (both merges had conflicts; every resolution hunk read)
267614bf(5e340ca × 3a289bd): syntheticgit merge-tree --write-tree=232801baa, actual =c26126a36→ differs in 4 conflicted files. Resolutions are real side-choices, no novel content:recipe/open_jev/native.mdkeeps PR-side ABI-9 wording;qwen3_5/README.mdhunk 1 takes side-branch ABI-9 paragraph, hunk 2 takes PR-side shared-executor text;gemm.cudedups a duplicatedreference_mfield and takes the side-branch'splan_forscheme logic;tests/qwen3_5/kernels.rskeeps the PR-side shape list (2B/4B/9B/27B).782db6f4(267614b × f54b3cd): synthetic14582ba2f, actual790d43945→ differs in 5 files.docs/supported-models.mdkeeps the Decider-2B row (correct for this PR);recipe/open_jev/native.mdandqwen3_5/README.mdkeep the fuller PR-side wording;Cargo.tomlfinal state is byte-identical to5e340cab2(union incl. bothgraphandprefixtest registrations — nothing lost);model.rstakes the canonicalensure_fixed_profile()?helper inforward_fixed/forward_sharedand keeps the already-approved PR-sideDrop(unconditional scratch/shared clear) and field order.- Tree-wide conflict-marker grep at head (
.rs .md .toml .py .cu .c): clean.
Authored commit b3cd06a2 ("Align profile rejection regression and Graph recipe with canonical executor")
2 files, test+docs only:
tests/qwen3_5/graph_loading.rs(4 lines): assertion text"require merged weights"→"only merged weights", matching the canonical guard message atsrc/models/qwen3_5/native/src/model.rs:1658("fixed-algorithm GEMMs and shared prefixes support only merged weights with the default numerics"). Not vacuous: the test still asserts bothforward_fixedandforward_sharedreturn errors and that theloading_liveallocation metric is unchanged across the failed calls.recipe/decider/validation.md(3 lines): pins the single named testordered_shape_replay_current_ids_growth_and_bounded_cachein the--test graphinvocation. The test exists (tests/qwen3_5/graph.rs:7) and its body matches the doc's description (ordered-shape replay, fresh-ID upload before replay, shape growth, FIFO eviction beyond 64 cached shapes, 0 fallbacks).- No behavior change smuggled in: the
5e340cab2..headdelta ofmodel.rsis a pure refactor — inlineensure!(identical predicatelora.is_none() && reference.is_none() && fp32_gdn.is_none()) moved into theensure_fixed_profilehelper plus a restored doc comment; the production guard is retained.
12-file delta (full new state read)
tests/qwen3_5/model.rs(+226, new): substantive profile-rejection regression. Generates a CPU fake CUDA library whose stubs are derived fromcuda.rs'sapi!block at build time (cannot drift from the ABI), compiles it withcc(#[cfg(unix)]), loads a tiny 2-layer LoRA checkpoint, and assertsforward_fixed/forward_sharedare refused with zero CUDA calls (call counter unchanged) and zero buffers (scratchnone,fixed_gemmnull,sharedempty) across LoRA, FP32-GDN and reference profiles, and that the guard passes with default numerics. Wired via#[cfg(test)] #[path]atsrc/model.rs:2402.tests/qwen3_5/profile_fake.c(+20, new): counting CPU fake for the above;PROFILE_FAKE_ABIinjected at compile time from the parsed source constant.benchmarks/bench.py(+68) andbenchmarks/README.md(+26):--score-decimals {2,3,4}(default 4),score_allowanceworst-case bound0.5·10^-sd + 0.5·10^-4·Σk + 1e-9(math checks out for independently rounded probabilities), exact score level-range check, complete-consistent zero-fit exception for Decider's all-zero Score,config.jsonrecordsscore_decimals+validation_protocol. Every README claim ("exact level range", "records it for reproducibility", accepted byread_answers/request_one/replay) matches the code.tests/benchmarks/test_score_wire.py(+225, new): 8 tests incl. a randomized property test (counts 2–10 × decimals 2/3/4 × 32 samples), zero-fit completeness/negatives, transport integration viaMockTransport, and end-to-endbench.runconfig recording. In CI via.github/workflows/ci.yml:44.src/backends/cuda/qwen3_5/gemm.cu(14): head state is byte-identical to canonicalf54b3cdf; reference scheme exemption moved fromfirst_choiceintoplan_for— semantically identical on theplan_forpath; side effect:fixed_choice's explicitNONE(line 98) is no longer overridden toMASKfor reference handles, which is strictly more correct for a function that then verifiessplits==1 && scheme==NONE. Informational; upstream-validated content.src/backends/cuda/contract.md(2): example-manifestabi_version7→4 inside a block the doc explicitly labels "illustrative, not the current Qwen manifest" — reduces impersonation of a real ABI. Informational.docs/architecture.md(7): new sentence "refuse an unmerged adapter, reference numerics or FP32 GDN before any device work" matchesensure_fixed_profileexactly (and is now machine-checked by the new profile test).tests/qwen3_5/graph_loading.rs,recipe/decider/validation.md: as above.- Decider runtime admission / Graph lifetime production code: unchanged in this delta (authored commit is test+docs; merges carried canonical upstream executor content, resolved as verified above).
PR body (template + evidence pins)
- New 2026-10-23 section pins head
b3cd06a27…and tree608ccac86a6f…— verified equal togit rev-parse b3cd06a2^{tree}. Evidence explicitly re-pinned "on this head"; CI/Docs links match the runs observed below. Not stale. - Body explicitly discloses the authored repair ("stale constructor-test error substring after #99's guard wording change; both assertions repaired with the production guard retained") — consistent with the commit.
- Four human contributor answers + the "is a PR needed" follow-up present; 4/4 self-review checkboxes checked; scope limits and author-run disclaimers preserved.
CI at the exact head (observed)
gh pr checks 113: rust 2m22s pass, benchmarks pass, cuda-contract pass, build pass, deploy skipping (intentional, documented).- Actions API
head_sha=b3cd06a2…on both repos returns the same two runs, both completed/success: CI39886014613, Docs39886014628(InferMatrixLab URLs; ThinkFlowLab API lists the same runs).
Host checks I ran (no cargo/GPU per host rule; /Users/hsliu/miniconda3/bin/python on a detached worktree at b3cd06a2, removed after)
python -m unittest discover -s tests/benchmarks -p 'test_*.py'→ 65 tests OK (includes the new score-wire suite).python -m unittest discover -s tests/decider -p 'test_*.py'→ 22 tests OK.python -m mkdocs build --strict→ clean build, no warnings.
Findings
Zero P0–P3 findings. Informational notes only:
src/backends/cuda/qwen3_5/gemm.cu:98—fixed_choice's explicitCUBLASLT_REDUCTION_SCHEME_NONEis now honored for reference handles (previously overridden toMASKbyfirst_choice); strictly more correct given the followingsplits==1 && scheme==NONEverification. Content is byte-identical to the canonical upstream branch.src/backends/cuda/contract.md:54— example manifest now saysabi_version: 4; self-disclaimed as illustrative.
Prior-finding status
No open findings carried: my last two reviews (COMMENTED 2026-10-22T02:04Z; state-correction APPROVE 2026-10-22T10:07Z) recorded zero actionable findings at 5e340cab2, and the two merges since then introduced no unresolved resolution issue. Standing unverified items unchanged in kind: device/A800 qualification remains author-reported (no GPU on this host); the Rust suite is covered by exact-head hosted CI (green), not run locally.
Verdict rationale
Non-draft + zero findings + template answers present + evidence re-pinned to the exact new head (tree hash verified) + exact-head CI green on both workflows. No downgrade trigger; prior APPROVE carries to b3cd06a27f0995cd7719c8148b8b479e11e44a53.
|
Additional independent review at I checked complete-row validation, exact token-prefix grouping, nonempty suffixes, question/candidate reconstruction, auto-selection's saved-work threshold, mode exclusivity, health snapshots and error synchronization. Personally run: 21 Decider CPU tests passed / 9 ignored. The two shared-prefix GPU regressions also passed using this branch's own compiled Qwen consumer, with bit-identical results to independent fixed-algorithm full prompts. The generic GPU input was derived from already-cached Strict docs passed. The native backend build inputs match the compiled ABI9 library used in the GPU run. I did not load the pinned trained Decider checkpoint or repeat its HTTP, task-output, full-model retirement or performance campaigns; generic-backbone results do not re-certify those author records. This is a review/validation comment, not an approval. Review provenance and scope
|
|
@hsliuustc0106 Conflicts with main are resolved and pushed; local checks pass. Could you please re-review? Thanks! |
Conflict resolution against main — 2026-10-24
Head
f285b8081300906cb62fc1e95a21ce46e57ec08bnormally merges main67b867b04b23acfe735c381559f7b2510f1e784e, retaining the original branch history. The only manual conflict resolution is the Qwen test manifest: Graph, prefix, constructor/stream-ownership, image-decoding and device-feature targets are all retained without duplicate registrations. ABI remains 9.Current-head local macOS checks passed: formatting; strict workspace/all-target Clippy; workspace tests (310 passed, 0 failed, 65 ignored); workspace release build; 22 Decider Python tests; CUDA text contract (0 errors, existing Laya ABI warning); strict MkDocs; and whitespace/merge checks. Device/checkpoint cases were skipped; no new CUDA compilation, GPU parity or performance qualification was performed. Historical device evidence remains pinned to its original source. Hosted checks for this new head are reported by GitHub separately. The original PR remains open.
Current-main integration and A800 validation — 2026-10-23
Current head
b3cd06a27f0995cd7719c8148b8b479e11e44a53, tree608ccac86a6f61377d22940fc252934a3e4d20d9, integrated with main5ba6a3ba668cb33028eb0bce2c6d17cdc53aebe7through normal merges. Original history is retained. Includes refreshed #112 and canonical #99 profile/ownership changes. Current main owns all20 CUDA build inputs; ABI remains9.Source review found a stale constructor-test error substring after #99's guard wording change; both assertions were repaired with the production guard retained. The documented Graph selector and duplicate Decider navigation were also fixed. Initial failure and focused/full repaired results are preserved.
Current-source local checks: 269 Rust workspace tests passed,0 failed,57 optional ignored; all format/strict Clippy/release/Python/contracts/smoke/strict-docs/whitespace stages passed. Four Python suites total163 tests on this head. Across both heads all26 configured local stages passed; optional workspace skips are not device passes.
**Fresh A80080GB/sm80 qualification:**13 selected GPU Rust executions plus2 GPU-hidden contract executions on this head. Default and cuDNN libraries were freshly compiled as ABI9, capability/dependency receipts checked, and consumers rebuilt separately. Fresh pinned reference20 requests/56 rows, all six modes120 parity records on this head, original protocol2 numerical/schema gates unchanged. Maximum probability/per-level-fit drift0.0138, Score drift0.04. Shared/fixed logits+responses and long-auto/fixed are exact with real sharing; matched packed4/Graph4 responses are exact with positive captures/replays and zero unexpected fallbacks. No equality across differently packed controls or performance claim.
Six-mode HTTP162 logical checks,12 concurrent requests per mode with observed client overlap/direct-proxy body parity; six projection-failure retirement modes with503/unavailable health/two later refusals and sampled model memory release;18 capture-fallback requests all passed on this head. The combined serial campaign has60 successful stages; every process stage ends at0MiB device use.
Independent agent source review inspected both complete effective diffs and all repairs, with no unresolved actionable source finding. The separate offline evidence review recomputed retained raw results and checked full source identity. This is author-side agent review, not maintainer approval or a new human attestation. Existing contributor-intent declarations and self-review history below are preserved verbatim; earlier sections remain historical evidence for their own source pins.
CI success · Docs success at this exact current head.
Full report, protocol, raw source/CPU/GPU/HTTP evidence, independent reviews and portable audit. All21 public assets and their checksums were downloaded and verified; downloaded audit passed. The GPU evidence was downloaded, hashed and audited before platform shutdown; fresh SSH became unavailable after shutdown. Both original PRs remain open and unmerged; main was not changed. Scope is bounded correctness on one A800, not all optional tests, broad task quality, other models/hardware, throughput, latency or peak-memory profiling.
Compatibility with PR149 verified — 2026-10-22
This PR remains
5e340cab24b3e55704bfc674a725b5ed82238710. Compatibility is implemented in the original #149 branch at428f834fad970f7e943180fb749871e04cbdb3e0, which retains this113 head as a parent. No source change here and no GitHub PR merge. Graph/fixed/shared/auto ABI9 modes and runtime admission/deadline controls are preserved together.The exact combined head passed local305 tests (57 optional ignored), all standard build/docs checks and80 Python tests. A800 qualification passed20 reference/120 mode comparisons, six-mode HTTP,42 lifecycle scenarios, six failure retirements and18 capture-fallback cases; corrected server CPU/doc71 +startup4 and15 targeted device/checkpoint executions also passed. CI · Docs are green.
New combined-head evidence, scope, raw records and portable independent audit. All five downloaded assets and11,886 public files verified; numerical gates unchanged, no performance claim. This targeted combined-head campaign does not relabel the earlier57-case qualification below as rerun, and does not extend old approvals to the new integration. Existing human declarations remain preserved.
ABI9 A800 qualification completed — 2026-10-22
Qualified source remains
5e340cab24b3e55704bfc674a725b5ed82238710; no source change was needed. One A80080GB/sm80, driver595.71.05, CUDA13.0.88. All14 follow-up cases and scope · complete additive evidence and portable audit.The complete additive archive is 568,456 bytes, SHA256
c48bba3b167f2791bb93f72b70a3c265300708cb6d5688502ccfe07b738548f1. It includes323 public files, the prior225-file pack verbatim and all210 earlier sealed raw records. All five downloaded assets were byte/checksum verified; the downloaded portable audit and independent review passed, including636 source hashes. Source remains unchanged and GPU memory returned to0 MiB. Preparation and auditor errors/corrections are retained. Large external model/tokenizer/package inputs are omitted with provenance, not measurement logs.The initial ABI9 campaign and its immutable archive preserve the earlier43/57 snapshot. There are zero benchmark samples and no performance, broad model-quality or other-hardware claim. Offline checks review recorded author execution; they do not independently authenticate GPU execution or rerun device math. Existing human declarations below are preserved without adding a new human attestation or maintainer approval.
Current-main fixed/shared integration — 2026-10-22
Current head:
5e340cab24b3e55704bfc674a725b5ed82238710. Normal merge1f25992fpreserves original PR #113 heada631b8b9as first parent and maintained #1123de420ccas second parent, including main32bdd2aade0e0f3f0452d04ca5022fcaaa4783c9. The original #98/#99 commits remain ancestors. No other contributor PR was changed or closed.The integration reconciles fixed/shared GEMM with main's reference GEMM, unmerged LoRA, FP32 GDN and multimodal Graph paths. The combined Qwen CUDA contract is ABI 9: rebuild the matching library and all Qwen consumers together. Header, manifest, loader, fake libraries and current deployment recipes agree. Fixed/shared profiles reject unsupported reference/FP32/adapter combinations before resource allocation. Model-owned prefix and fixed handles preserve synchronization and lifetime rules, together with #112's explicit Graph controls and single final-field stream owner.
A full workspace run exposed a pre-existing Cua test fixture collision when parallel tests received the same clock tick. A separate focused test-only commit adds a process-local sequence to temporary directory names and an actual eight-thread, identical-timestamp filesystem regression. The original failure and subsequent passing results are retained; production Cua behavior is unchanged.
Executed validation
Final macOS arm64 source: 268 workspace tests passed / 0 failed / 57 ignored. Formatting, strict Clippy, release build, strict docs and whitespace checks passed. Python benchmark/Decider/CUDA-checker/backend suites passed 58/22/68/8 cases, and smoke validated four requests. Targeted Qwen host tests passed 24 cases with 29 device cases ignored. Source contract has zero errors and one existing Laya declaration warning. A fresh CPU-only Docker CUDA 13 default build for sm89 passed; symbol/dependency inspection and the CPU ABI probe confirmed ABI 9. This compiles CUDA source but executes no GPU kernels.
Independent full-diff review and a follow-up review of the fixture repair found no unresolved actionable findings. Final effective diff against main: 53 files, +3025/-242 (3267 total changed lines; 2303 source/test lines). Reviewed diff SHA256:
1e773225c702f530c5544016e259ec74427fd2187f1a525dabe77b1947bcc8ea. Final commit tree exactly matches the validated tree.Updated device qualification — 2026-10-22: the fresh A800/sm80 ABI9 campaign above now validates the bounded fixed/shared/auto/Graph, HTTP and lifecycle matrix with default and matching cuDNN builds. The initial43/57 snapshot has now been completed by the follow-up above:57/57 unique optional Rust cases (43GPU +14CPU), with the complete pinned Open-Jev27B packed/Graph regression included. Other GPUs/toolchains, broader full-model and Hopper qualification, and broad quality/performance remain outside this campaign. Historical standalone ABI7 and main/combined ABI8 results below keep their original source scope. Rebuild the matching ABI9 library and all Qwen consumers together. Current-head hosted CI and Docs passed; PR documentation deployment is intentionally skipped.
Existing contributor declarations below are preserved; this update reports agent execution and review without adding human attestation or maintainer approval. Historical base/ABI/hold statements below retain their named revisions and are superseded by this heading for the current integration state.
CPU maintenance and merge handoff — 2026-10-15
Current head:
a631b8b95cb4a2853d821f6dde576717d479e461.Actual target:
mainatf2d3d32897c1fe190fecf853a82a9456d3791462; the PR target was not changed.This branch contains the original shared changes
d717e497(#98) and2ff1396(#99), not the latest #98/#99 tips. Preferred handoff: coordinate #98 → #99, then refresh this branch while preserving its Decider Graph/cache/lifetime changes and recheck the resulting diff; #112 can land independently first. A maintainer may instead choose this cumulative branch first and coordinate duplicate PR closure with their author. No other contributor's PR was modified or closed.The standalone branch remains ABI 7; main/#112 remain ABI 6, and the separate shared-vision stack uses ABI 8. Run
fixed_gemm_rows_do_not_depend_on_mon each deployment GPU. The documented HTTP503/unavailable-health alert and rebuild/restart/warmup recovery also apply if auto mode selects a shared plan that subsequently fails. These are documented requirements, not new device passes.No source was changed in this pass. Inspected the current source/dependency history and completed existing self-review record; previously observed current-head hosted checks were successful. Prior personally-run Rust/device records below keep their original source and scope; no Rust suite or GPU execution was repeated in this body-only update.
This pass follows the contributor's 2026-10-15 instruction to maintain these existing PRs, complete CPU-only fixes and merge preparation, and keep #117 unmerged. Existing contributor declarations are preserved; this update adds no new human attestation or maintainer approval. No PR was merged and no GPU/model/performance run was performed. Source-pinned historical results remain historical. Current-head hosted workflow links are recorded separately below when available.
Historical records below
Earlier maintenance/review/test sections retain their named revisions and environments. Their old ABI or build commands are historical reproduction instructions, not commands for deploying this current head. Use the current branch recipes for builds; the CPU checks above do not qualify skipped device tests.
Historical maintenance — 2026-10-15
Current head:
a631b8b95cb4a2853d821f6dde576717d479e461.Synced onto maintained #112 at 03bbb61, including main after #111's merge. Added the per-device fixed_gemm_rows_do_not_depend_on_m command and its exact one-test acceptance rule, the actual HTTP503/unavailable-health alert condition, same-ABI rebuild guidance and restart/warmup recovery procedure. Source inspection confirmed automatic mode also shares the failure/retirement contract when it selects a shared plan; the documentation now states that rather than treating auto as an execution-error fallback. Production retirement behavior and all original numerical gates are unchanged. #98/#99 remain separate open dependencies: their merge ordering is recorded for maintainer coordination, and no other contributor's PR was modified or merged.
Personally run: format, strict workspace Clippy, workspace tests (206 passed, 0 failed, 45 ignored), release build, benchmark tests, CUDA checker/backend tests, source contract check, smoke validation and strict MkDocs all passed. Device-dependent tests were compiled and skipped on this macOS host. No new GPU, numerical-parity or performance pass is claimed.
Current-head hosted workflows: ci.yml, docs.yml. Linked hosted results are separate from personally executed local checks. Earlier evidence sections retain their named original revisions and scope; they are not new qualification claims.
No PR merge was performed. #117 remains on its explicit merge hold.
Purpose
Add default-off request-local shared-prefix execution and its independent fixed-GEMM control. Reuse aligned request/question prefixes while preserving branch state, positions, final-token readout and original response order. Keep main’s separate text/multimodal Graph lifetimes, both external GPU regression modules, and the ABI 7 fixed-GEMM interface. Prefix buffers persist, while their contents are valid only within one request.
#94/#110/#111 prerequisites are now merged into main. Review the remaining #112 → #113 increments in order; #113 also carries the #98/#99 shared-executor dependencies, whose merge order remains a maintainer coordination item.
Required human contributor answers
These answers reuse the contributor's explicit 2026-10-12 maintenance instruction:
keep the existing PRs mergeable and CI passing; eliminate conflicts and review
issues; serve maintainers who need clear, verified changes that are easy to merge;
repair, validate and push to the existing PRs. The source is the contributor's
local maintenance conversation on that date; private conversation details are
not published.
What outcome are you trying to achieve?
Keep the existing PRs mergeable and CI passing.
What is the highest-leverage variable for that outcome?
Eliminate merge conflicts and resolve review issues.
How will this work move that variable rather than create busywork?
Repair the identified conflicts and review issues, validate the affected
integration, and push the verified changes to the existing PR branches.
Who is the customer or user, and what do they actually need?
Repository maintainers, who need clear, verified changes that are easy to merge.
After answering the four questions: is a PR needed at all to achieve this outcome?
The contributor explicitly directed updates to the existing PRs. Maintain these
PRs; no new PR is needed for the maintenance work.
Historical Current integration
Head:
e790c3cfc4bdc1106fe1aa9cfe118d045ab63c41(documentation-only follow-up to runtime source4c7d437023c47c777afc7b82e6cb95a399468785). Target: main at3e9cc33a61ddf110d97512fb6f296cf540377c61, refreshed during this maintenance. Main and published prior heads remain ancestors through normal merges.Target diff: 52 files, 2566 authored changed lines / 3436 total text changed lines (additions plus deletions). Counts exclude Markdown, licenses/notices, lockfiles, static fixtures and the example request from authored lines; the ABI manifest is included.
The predecessor merge conflict was reproduced before repair. The resolution retains persistent shared-prefix state and fixed-GEMM handle destruction together with both Graph caches and both root test modules. The current-main worker contracts and GPU tests remain present. Workspace recipe commands include
omni-laya/serve.Current main and the independent #111/#112 consumers require ABI 6. This complete #113 branch requires ABI 7, including
cs1_gemm_create_fixed. Rebuild the library and every consumer together. The #113 recipe’s ABI 7 command applies to that branch; it does not imply ABI 7 is already in main. Combining this chain with #135’s ABI 8 changes is a separate cumulative integration and validation task; the standalone evidence here does not qualify it.Runtime-source CPU checks and documentation follow-up
The following CPU-only macOS checks passed on runtime-source predecessor
4c7d437023c47c777afc7b82e6cb95a399468785(Rust 1.98.0):Workspace: 201 passed, 0 failed, 45 ignored; Decider Python helpers: 22 passed. The workspace ignored device/checkpoint cases; separate evidence below states which were later executed. Release builds for #112/#113 were rebuilt after clearing only affected package artifacts to ensure the exact worktree code was compiled. Python 3.12 with pinned documentation/benchmark requirements was used for broader host checks. Strict documentation build passed for #111 and #113; the shared benchmark suite passed 23 tests and validated the four-request smoke manifest. #113 also passed 68 CUDA checker tests, 8 backend helper tests and the ABI manifest check (0 errors; the existing Laya declaration warning remains).
The current upstream PR checks for published runtime-source predecessor
4c7d437023c47c777afc7b82e6cb95a399468785completed successfully:PR documentation deployment is intentionally skipped. The CUDA-contract CI job runs host checks and does not compile CUDA or execute GPU tests. These are upstream checks for the published runtime-source predecessor; earlier CI URLs in the prior description are historical. The new documentation-only head has passed local strict MkDocs and
git diff --check; its hosted checks passed at the new head, linked below.Demo / evidence and limits
Fresh device qualification — 2026-10-13
The designated GPU runner executed qualified runtime source
4c7d437023c47c777afc7b82e6cb95a399468785with the freshly compiled ABI 7 library on one NVIDIA GeForce RTX 4090, sm_89. Toolchain: CUDA 13.0.88, Rust 1.98.0, Python 3.12.3, Torch 2.12.1+cu130, Transformers 5.17.0. Checkpoint/tokenizer/calibration: Mapika/decider-2b533964dae8be954c5b5e19fa4948e48408094c1e; author reference:50d0be0d7cb43d2066965ce5fa7f3fe4e489a60f. All five model files and five imported reference files were hash-checked. Optional FLA, causal-conv1d and flash-attention packages were absent; the reference used the documented Transformers fallbacks. The existing Python environment was not changed.The frozen synthetic corpus has 5 requests / 20 independent rows / 18,394 processed tokens, with a longest reference-compared row of 3,541 tokens: short mixed Choice/Score/Noul, long shared mixed, reordered short questions, short input after long shared work, and empty questions. The original gates were fixed before execution: probability/per-level-fit drift ≤0.02, Score drift ≤0.1, matching Choice decision at reference margin ≥0.05, exact token/candidate/readout/usage/identity comparisons, and validation protocol version 2 finite typed-schema/formula/rounding checks.
Native diagnostic SHA-256:
936d9cd20ec5881d8d89ea7d1ce7db6d15d2f5ada885f07371a6bb5bab56f2ea; ABI 7 library:2e6d15a1726314440e8547986fb7cd56a45fdc5931dcd56694d78883a20dff28; frozen reference records:965e57ba64275f6db5ec0d6597da73111309e812070149a79b14dd434b67ac36. A 31-path pre-GPU inventory, complete raw records, protocols, logs and failure wrappers are retained in the maintenance evidence archive. Both local and server CPU reviews recomputed the raw responses with the unchanged validator. Fresh raw evidence is retained in the local maintenance archive; no new public asset has been published.Reproduction uses
recipe/decider/download_weights.py,src/backends/cuda/qwen3_5/build.sh ... 89, the pinned author reference,tests/decider/verify_reference.py --cases <frozen-cases.json>,verify_modes.py --plan <frozen-plan.json>,verify_retirement.pyandverify_graph_fallback.py. The frozen case/plan files and hashes are included in the retained archive. Every native consumer must be rebuilt with the matching library.Only two diagnostic roundtrip timing samples per mode were recorded; they are descriptive and do not establish a speedup. No frontend latency, throughput, task-quality, other-GPU, auto-prefix, 36,864-token limit or cumulative ABI 8 claim is made by this campaign. The five-request reference corpus and selected tests delimit this qualification.
Historical evidence
The original Decider followups archive and
recipe/decider/validation.mdretain the earlier 2026-10-07 campaigns and their own implementation/ABI provenance. Those results are historical; the fresh results above are tied to the repaired head and current environment. Historical Torch 2.14.0 measurements are not presented as measurements of the current Torch 2.12.1 campaign.Review map and self-review
Trace prepared complete rows through batching and response reconstruction, then review Graph/scratch/prefix lifetimes, synchronization, error retirement and Rust/CUDA ABI declarations. Processing and adapters stay model-owned, shared device operations stay under Qwen/CUDA, admission stays per complete request, and tests remain under root tests/. Generated measurement output and agent plans are excluded from source changes.
The documentation follow-up changes only
recipe/decider/README.mdandvalidation.mdto date the fresh qualification, preserve historical provenance and distinguish standalone ABI 7 from main ABI 6 and cumulative ABI 8.srcandteststree hashes match the qualified runtime source exactly (b97905de9812612fafe32b9fcc1b9bd316cf1364andecdc18b8e31e6e9628bc54075b3fa03d8fd354ee); manifests, CI, navigation, dependencies and download helper also remain byte-identical. The fresh device evidence therefore applies to this unchanged runtime, while the new source head is explicitly a documentation-only qualification note.The complete current-target diff and predecessor increments received local self-review and independent review; no semantic findings remained in that review. Device evidence has the scope stated above and does not replace contributor or maintainer review. The contributor explicitly confirmed completion of self-review and authorized this checklist/template update on 2026-10-13 in the maintenance conversation. This records the human confirmation; it does not turn unverified measurements into passes or replace maintainer review.
Documentation-head hosted verification
Exact documentation head
e790c3cfc4bdc1106fe1aa9cfe118d045ab63c41: CI and Docs passed. Source/test/build-input identity to the qualified runtime head is recorded above.Cumulative ABI 8 qualification — 2026-10-13
Combined integration branch, final head
18e1a5a6f68b9d11d3e11c7ea7b939af87272f53, integrates the twelve source-qualified maintenance heads frozen for the earlier 2026-10-13 campaign through normal merges. The later #117 ancestry-only follow-up leaves its complete source tree unchanged. This is separate review material, with no new PR and no main merge. It includes a substantive cross-feature fix: model-owned transient vision output survives its borrow, then retires only after both producer and consumer streams drain. A production-entry injected-copy failure confirms the matching output allocation is released before error return and subsequent requests recover.262 Rust CPU tests (59 ignored), 163 Python tests, strict Clippy/format, release, docs and ABI contracts pass. Fresh cuDNN ABI8 GPU qualification passes 17 Rust device invocations, all eleven fixed-seed JEMM reference operators with exact Graph replay, and five Decider controls (five requests/20 rows each); maximum probability drift 0.012, Score drift 0.01. Fixed/shared logits and complete responses are exact; actual Graph replay and shared-prefix savings are observed. Runtime validation source is
b909398d; the final head changes only active recipe ABI wording, with source/test/build identity verified. No new performance or full 27B/Hopper claim is added.Exact-head cumulative CI and Docs passed. Raw protocols, responses, failures and hashes are retained in the local maintenance evidence archive.