dspark: cut per-cycle CUDA overheads (single-draft decode, GPU confidence probe)

[vincenzopalazzo]

Summary

Two CUDA-side DSpark cycle-cost reductions, measured on a DGX Spark (GB10):

• decode single-token drafts directly instead of running a one-row batch
  verify that cannot check anything (commit_drafts is unconditionally 1 for
  one drafted token) plus a per-layer frontier snapshot; the single-token
  decode path keeps the decode-graph kernels and leaves exactly the state
  plain decode would have left
• evaluate the proposer's confidence probe on the GPU: one small kernel plus
  a 4-byte readback instead of a 28 KB device-to-host hidden copy and a
  7424-wide host matvec per drafted token, twice per cycle on average
• keep the CPU confidence path as the fallback (Apple, non-Q8_0 markov
  weights, DS4_DSPARK_NO_GPU_CONFIDENCE as the diagnostic switch), and make
  it re-derive markov_state before any CPU markov consumer runs — this also
  covers a pre-existing case where reuse_first_confidence could leave the
  CPU markov fallback reading a stale markov_state

Root cause

DS4_DSPARK_STATS=1 on GB10 shows the cycle economics: target eval ~60 ms,
batch verify ~85 ms, propose ~20 ms. With confidence pruning most proposals
are a single token, and for those the verify pass re-derives state the
ordinary decode produces directly, at batch-kernel prices (no decode graphs,
tiny-batch kernel tiers). On the propose side, an nsys trace shows the
confidence gate doing a blocking 28 KB D2H read plus a host matvec per
drafted token, serialized in the middle of the propose loop.

Performance (main 84cc882 vs PR, DGX Spark GB10, CUDA sm_121)

DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
(81 GiB) + DeepSeek-V4-Flash-DSpark-support.gguf, --dspark --temp 0 --nothink --tokens 128, defaults otherwise, 3 runs, median generation t/s:

Prompt	main gen	PR gen	Delta
prose (mmap/TLB explainer)	17.35 t/s	17.77 t/s	+2.4%
code (C SPSC ring buffer)	18.42 t/s	19.10 t/s	+3.7%

Isolating the two commits with the diagnostic switch on the PR build
(DS4_DSPARK_NO_GPU_CONFIDENCE=1 leaves only the single-draft change):

Prompt	main	+ single-draft	+ GPU confidence
prose	17.35	17.39	17.77
code	18.42	18.71	19.10

Propose-stage breakdown (DS4_DSPARK_STATS=1, whole 128-token prose run):

Metric	CPU confidence	GPU confidence	Delta
propose	577–655 ms	396–467 ms	about −30%
prop_conf0	112–124 ms	1.6–1.7 ms	−98%

The GPU and CPU probes made identical gating decisions in every run (same
proposed/accepted counters and histograms). The logit differs from the CPU
matvec only by floating-point reduction order, and it only gates proposal
length; the verifier re-checks every proposed token against the target
argmax, so committed output cannot change.

tests/dspark_acceptance_fixture.sh (64-token cases, PR build, defaults):

Case	output_match	baseline t/s	DSpark t/s
hello	1	16.98	21.04
redis	1	17.80	19.03
math (no drafts)	1	15.75	16.01
python_reverse	1	18.03	20.74
c_add	1	17.38	24.80

The math case (proposer never fires) was previously a small net loss; the
cheaper probe makes it neutral.

Plain decode is untouched by both commits (ds4-bench runs without
--dspark, so the changed code never executes). README-style sweep
(promessi_sposi, 2048..32768, 128 greedy tokens), run in both orders to
control for the box warming up — whichever build runs first wins by the same
fraction of a percent, so the residual deltas are run-ordering noise, not
code:

Generation t/s (main first / PR second, then PR first / main second):

Context	main (1st)	PR (2nd)	PR (1st)	main (2nd)
2048	17.37	17.30	17.39	17.36
4096	14.87	14.90	15.00	14.92
8192	14.71	14.64	14.72	14.66
16384	14.64	14.54	14.64	14.52
32768	14.05	13.95	13.97	13.96

Prefill behaves the same way (798-806 t/s at 2048, 878-896 t/s above,
order-dependent within ~1%).

Test plan

• make cuda-spark clean on DGX Spark (CUDA 13.0, sm_121)
• make (Metal) clean on macOS
• make cuda-regression on the Spark
• tests/dspark_acceptance_fixture.sh: 5/5 output_match=1, zero
  verifier errors, zero replay fallbacks
• A/B with DS4_DSPARK_NO_GPU_CONFIDENCE=1: identical DSpark stats
  counters, so the GPU probe changes timing only
• before/after ds4-bench CSVs on the same box, same power state

Analysis and patches developed with Claude (see commit trailers); all
numbers measured on real hardware as described.