Support AProjQ4 GGUFs: Q4_K dense attention projections

[GiorgioOppo]

The AProjQ4 DeepSeek V4 Flash GGUFs quantize the five dense attention projections per layer (attn_q_a, attn_q_b, attn_kv, attn_output_a, attn_output_b) as Q4_K instead of Q8_0. Loading already accepted them (tensor_expect_dense_quant_layout), but the decode graph read the Q4_K blocks through the hardcoded Q8_0 kernels and generated garbage (BOS loops), and the CPU reference died with "expected a 2D Q8_0 tensor".

• Metal decode graph: gate the fused Q8_0 q_a/kv pair kernel and the plain Q8_0 matvec fallbacks on the actual tensor type, dispatching through the existing generic dense-quant path for Q4_K. Q8_0 models keep the exact kernels they used before, so their output stays bit-identical.
• CPU reference: add a dense Q4_K matvec/matmul family (activations prequantized to Q8_K, rows reduced with ds4_vec_dot_q4_K_q8_K), with grouped, decode-scratch and prefill-batch variants, and dispatch the attention projection call sites on tensor type.
• CUDA decode-TP attention output split now refuses non-Q8_0 output projections with a clear error instead of computing garbage.

Verified on Apple M1 Pro 16 GB with --metal --ssd-streaming on DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix: coherent greedy output, --decode-consistency max_abs=0 rms=0, and no regression on the AProjQ8 gguf.

[GiorgioOppo]

#618

[GiorgioOppo]

DS4_METAL_STREAMING_EXPERT_NOCACHE=1 DS4_METAL_STREAMING_EXPERT_PREAD_SPLIT=4 ./ds4 --metal --ssd-streaming --ssd-streaming-cache-experts 448 -m gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix.gguf

[GiorgioOppo]

@antirez ,
Could you please review this post in case you accept it or close it and the related post on hf (https://huggingface.co/antirez/deepseek-v4-gguf/discussions/22)?

[adamlawi]

Volunteering as a tester for the CUDA path — this is the configuration your table lists as implemented but untested.

Machine: stock NVIDIA DGX Spark, GB10 (sm_121), 128 GB unified memory, aarch64, Ubuntu 24.04.4, driver 580.173.02, CUDA 13.0.88. Built with make cuda-spark. Not a Mac and not SSD streaming — the model is fully resident, so this exercises the decode path under different pressure than the M1 Pro run.

What I can run, using a harness already in place from measuring ds4f-q2 vs ds4f-q2-q4 on this box:

• --decode-consistency and greedy A/B against the AProjQ8 baseline (correctness first)
• ds4-bench sweeps at 2k/4k/6k/8k, 128 generated tokens per frontier — the AProjQ8 baseline here is 872 t/s prefill / 14.69 t/s decode at 8k
• full-window prefill (194k tokens) with free-memory sampling every 2 s, reporting the minimum at peak
• perplexity on a fixed sample, to put a number on the quality delta on this hardware

Why the memory result is the interesting one here. Resident ds4f-q2 leaves very little headroom on a 128 GB Spark: at --ctx 200000, filling 97% of the window leaves 2.5 GB free at peak; --ctx 220000 leaves 0.94 GB; --ctx 300000 cannot be filled past ~52% (0.7 GB at 155k tokens). The dense tensors account for 8.20 GiB of the startup span phase on this build, so cutting them to Q4_K should show up directly as usable context. If it frees ~2 GiB, that plausibly moves the ceiling from 200k to 250–300k — a benefit orthogonal to the SSD-streaming speedup you measured.

What blocks me: I could not find an AProjQ4 GGUF published (not in antirez/deepseek-v4-gguf, and none under your HF account), and gguf-tools/deepseek4-quantize only builds from --hf DIR with the original safetensors — ~568 GB, which does not fit here alongside the ~78 GB output (644 GB free). Your write-up mentions wiring --requant-gguf to consume a dense imatrix, but I do not see that tool on any of the three branches of your fork.

So: could you publish DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf, or share the requant path that derives it from an existing GGUF? Either way I can turn results around quickly.

Context for the numbers above: #705.

[GiorgioOppo]

Hi, the download is blocked by a pr on hf, https://huggingface.co/antirez/deepseek-v4-gguf/discussions/22 You should be able to download the template from https://huggingface.co/antirez/deepseek-v4-gguf/blob/refs%2Fpr%2F22/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf

[adamlawi]

Tested on a stock NVIDIA DGX Spark (GB10, sm_121, 128 GB unified, CUDA 13.0.88, driver 580.173.02), as offered above. Short version: the CUDA path is correct, the memory win is bigger than expected, but prefill regresses ~16x, which is a blocker for resident use on this machine.

Setup: branch aprojq4-dense-attention at 41aefd2c, make cuda-spark, no local changes — builds clean in 2m52s with zero warnings. Model is the GGUF you linked (5ca1abfe, 78.62 GiB). Baseline everywhere below is IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731, run with the same binary, so the comparison isolates the quant rather than the code version.

One caveat up front: the AProjQ4 file is from an older checkpoint (no -0731) and I did not download a matching AProjQ8 baseline, so memory and correctness numbers are clean, but speed and perplexity carry a checkpoint difference that I cannot separate out.

1. Correctness: the CUDA path looks good

--decode-consistency 64:

model	max_abs	rms
AProjQ8 (baseline)	1.476	0.293
AProjQ4	1.118	0.191

Top-1 matches, top-10 sets match. The divergence is smaller than on the existing Q8_0 path, so the new kernel is not introducing numerical error.

2. Memory: a bigger win than the file-size delta suggests

--inspect shows 215 tensors moved Q8_0 → Q4_K (345 → 130 Q8_0, plus 215 Q4_K), everything else byte-identical. Startup spans and the memory plan both drop by exactly the file-size delta:

• startup model preparation: 8.20 GiB → 6.06 GiB, and no time penalty (2.12s → 1.70s)
• planned at --ctx 200000: 85.21 GiB → 83.07 GiB
• aligned artifacts: 474 / 78.71 GiB → 259 / 74.16 GiB (the Q4_K tensors get no repack builder)

But free memory after startup improves by ~11 GB, well beyond the 2.14 GiB the file shrank:

ctx	AProjQ8 free	AProjQ4 free
200000	9.8 GB	21.3 GB
250000	8.7 GB	20.1 GB
300000	8.1 GB	19.1 GB

My guess is that the Q4_K tensors no longer exist simultaneously as an aligned artifact and a mapping, which accounts for roughly 6.7 GiB of it — I have not verified the rest, so treat the exact figure with caution. On GB10 (CPU and GPU share one 128 GB pool) this is a real change: --ctx 300000 now starts with more headroom than --ctx 200000 had before.

3. Prefill: ~16x slower, and this is the blocker

ds4-bench, same prompt file, 128 generated tokens per frontier:

ctx	prefill Q8	prefill Q4	decode Q8	decode Q4
2048	817.32	53.98	17.55	17.28
4096	892.89	54.20	15.08	14.86
6144	883.59	54.18	14.97	14.78
8192	877.75	54.17	14.83	14.65

Decode is essentially free (−1.2%) — impressive for halving the precision of the attention projections. All the cost is in prefill, exactly where your comment above matmul_q4_K_dense_kernel predicts it ("correct-but-unoptimized for prefill chunks"; every token re-reads the weight rows). With a 4096-token prefill chunk this dominates.

Practical effect for a long-context server: a full 194681-token prefill went at a steady 54.6 t/s, i.e. ~59 minutes instead of ~4.5 (I aborted the run at 17%). So on this machine the patch trades a context ceiling we can now afford for one we can no longer fill — a 15k-token page would go from ~17s to ~4.5 minutes.

That makes the dequant + GEMM prefill path you mention in the comment the deciding factor here, rather than a later optimization. Is that something you're planning? If you push a prefill path, I'm happy to re-run this whole sweep on the same machine — the harness is set up and a rebuild plus full sweep is under an hour.

4. Perplexity — reported for completeness only

Same 1882-token sample: AProjQ4 12.730 (avg_nll 2.5439) vs AProjQ8-0731 11.108 (avg_nll 2.4077). Please do not read this as a +14.6% quantization cost — it is dominated by the checkpoint difference, and it is inconsistent with your own +0.99% measured within one checkpoint. I mention it only so the number isn't misread if it surfaces elsewhere. If you publish an AProjQ8 GGUF from the same older checkpoint, I'll rerun and give you a clean quality delta.

Also worth noting for anyone reading this from the DGX Spark side: this is a different issue from #705, which is about routed Q4_K experts having no aligned repack builder. This PR touches dense tensors only and does not OOM.

[GiorgioOppo]

@adamlawi,
Thanks for the feedback, it's hard to test on cuda without it :) Anyway, it should be optimized now and the prefill should be faster. Let me know if the prefill is accelerated or unchanged. In the next few days I'll update gguf with the new version of deepseek that wasn't available.

[adamlawi]

Retested 6c4d769d on the same DGX Spark. Prefill is fixed — ~15.5x faster than before, now within ~4-5% of the Q8_0 baseline. Same binary, same prompt file, same model as my previous comment.

ctx	prefill 41aefd2c	prefill 6c4d769d	AProjQ8 baseline	gap to baseline
2048	53.98	783.60	817.32	−4.1%
4096	54.20	850.37	892.89	−4.8%
6144	54.18	841.37	883.59	−4.8%
8192	54.17	837.14	877.75	−4.6%

Decode is unchanged (17.32 / 14.90 / 14.82 / 14.67 vs 17.55 / 15.08 / 14.97 / 14.83 for Q8_0, i.e. −1.1%), so the MMQ path did not cost anything on the decode side.

Two more checks, since an optimization like this could plausibly trade accuracy for speed — it did not:

• --decode-consistency 64: rms 0.182 / max_abs 1.088, versus 0.191 / 1.118 on 41aefd2c and 0.293 / 1.476 on the Q8_0 baseline. Still better than the baseline path.
• perplexity on the same 1882-token sample: 12.730174, versus 12.729817 before the change — identical to four decimal places.

No ds4_mmq_q4_K_dense returned ... falling back lines appeared in any run, so MMQ accepted every shape in this model.

Memory is unchanged from my previous report, as expected: startup spans 6.06 GiB (vs 8.20 GiB for AProjQ8), 259 aligned artifacts / 74.16 GiB, and ~11 GB more free memory at every context size. So on a 128 GB GB10 this now looks like a straight win: ~2 GiB smaller resident model plus ~11 GB more free memory, for ~4.5% prefill and ~1% decode — and --ctx 300000 starts with more headroom than --ctx 200000 had with AProjQ8.

Nice work — that was a fast turnaround.

One question about the other half of the commit, --requant-gguf: does it work from an existing GGUF as input? Our production model is the -0731 checkpoint, and the blocker for us so far has been that deepseek4-quantize only builds from --hf DIR (~568 GB of safetensors, which does not fit next to the output on this box). If requantizing an existing AProjQ8-...-0731.gguf into AProjQ4 is supported, I'd be glad to try it and report back — that would let us run this variant on the checkpoint we actually serve, and would also give you a clean same-checkpoint quality delta, which my perplexity numbers above cannot provide.

[adamlawi]

Standing offer, so you don't have to ask: when you publish the GGUF for the newer DeepSeek checkpoint, I'll run the full suite on the DGX Spark and post the numbers here — --inspect, --decode-consistency, the ds4-bench prefill/decode sweep, perplexity, and the startup-span / free-memory measurements at 200k / 250k / 300k context. Same harness as the two runs above, so the results will be directly comparable.

That run would also finally give a clean same-checkpoint quality delta, which neither of my perplexity numbers so far can provide (the AProjQ4 file I tested is from the older checkpoint, and I don't have a matching AProjQ8 baseline for it).

The same applies to --requant-gguf whenever you get a chance to answer: if it accepts an existing GGUF as input, I'll requantize our production AProjQ8-…-0731.gguf and report back on that too.

No rush from my side — the machine is a stock DGX Spark that stays available, and a rebuild plus the whole measurement suite takes well under an hour. Just ping me here.

[GiorgioOppo]

The new gguf is online https://huggingface.co/antirez/deepseek-v4-gguf/tree/8268ad9384d11cbb63f472d097e499a9c59709ec. It's made by requantizing the q8. I know it's not the best choice, but on my hardware it's the only thing I can do.

[GiorgioOppo]

try use DS4_CUDA_NO_Q4_ATTN_GROUPED_TOK8=1 DS4_CUDA_NO_Q4_QKV_PAIR=1 in the next run

[adamlawi]

Ran 8b84e878 both ways on the DGX Spark. The two new paths cost ~38% of prefill — setting both env vars restores the 6c4d769d numbers exactly. Same binary, same prompt file, same model throughout.

ctx	A: default (new paths on)	B: both env vars =1	6c4d769d	AProjQ8 baseline
2048	495.15	797.21	783.60	817.32
4096	521.72	851.00	850.37	892.89
6144	518.51	843.21	841.37	883.59
8192	516.13	838.35	837.14	877.75

So DS4_CUDA_NO_Q4_ATTN_GROUPED_TOK8=1 DS4_CUDA_NO_Q4_QKV_PAIR=1 is a clean revert to the previous prefill performance — the regression is fully attributable to those two paths, not to anything else in the commit.

They also hurt numerics, which I did not expect:

variant	--decode-consistency 64 rms	max_abs
A: new paths on	0.307	1.529
B: env vars =1	0.185	0.942
6c4d769d	0.182	1.118
AProjQ8 baseline	0.293	1.476

With the new paths active the divergence is worse than the Q8_0 baseline; with them off it is back to the 6c4d769d value. That looks like a correctness signal on the grouped/paired path, not just a scheduling tradeoff.

The good news — decode improved, and it is not from these two paths:

ctx	decode A	decode B	6c4d769d	AProjQ8 baseline
2048	18.21	18.28	17.32	17.55
4096	15.59	15.64	14.90	15.08
6144	15.50	15.52	14.82	14.97
8192	15.35	15.37	14.67	14.83

Decode is ~+5% over 6c4d769d in both variants, i.e. the gain survives disabling the two paths and comes from the rest of the commit. That is the best decode I have measured on this machine — above the Q8_0 baseline, which is a notable result on its own.

One more thing worth a look: perplexity on the same 1882-token sample is 12.7836 (A) and 12.7904 (B), versus 12.7302 on 6c4d769d. So unlike prefill and rms, ppl does not return to the previous value when the env vars are set — a ~0.47% regression that appears to come from elsewhere in the commit. Also note A and B differ by 0.0068 here, while two runs of 6c4d769d agreed to within 0.0004, so there may be some nondeterminism in this build.

Summary from this machine: keep whatever gave the decode win, and either fix or gate the grouped-tok8 / QKV-pair paths — on GB10 they cost 38% of prefill and worsen decode-consistency. Happy to re-run anything on request.

(And the --requant-gguf question from my previous comment still stands whenever you have a moment — if it accepts an existing GGUF as input, I'd like to requantize our production -0731 model and give you a clean same-checkpoint comparison.)

[GiorgioOppo]

@adamlawi Feel free to try the requantization test

(And the --requant-gguf question from my previous comment still stands whenever you have a moment — if it accepts an existing GGUF as input, I'd like to requantize our production -0731 model and give you a clean same-checkpoint comparison.)

[GiorgioOppo]

@adamlawi could you run ./gguf-tools/quality-testing/score_official to compare the two quantizations?
these are my results on macos
q4.tsv
q8.tsv

[adamlawi]

--source-gguf works, and the result is better than I expected. I requantized our production -0731 model directly from GGUF — no safetensors needed — and then ran score_official on both. Summary: on a same-checkpoint comparison the AProjQ4 quality cost is +0.64% avg_nll, while decode gets faster than Q8_0 and the model frees ~13 GB of memory.

1. Requantization

./deepseek4-quantize \
  --source-gguf  DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf \
  --template     <same file> \
  --attention-proj q4_k \
  --out          ...-AProjQ4-SExpQ8-OutQ8-chat-v2-0731-requant.gguf

Converted 215 tensors q8_0 -> q4_K, output 84,420,584,032 bytes — 224 bytes off your published AProjQ4 file, so the layout matches exactly. --inspect confirms it: 130 × q8_0 (1.59 GiB), 215 × q4_K (2.41 GiB), everything else byte-identical to the source.

Two notes for anyone repeating this:

• The tool streams 16 rows at a time, so memory use is negligible — it ran fine alongside a live 80 GB ds4-server on the same box, only costing disk I/O.
• No imatrix was used. write_imatrix_kvs only records the imatrix filename and entry count in the KV block, not the vectors, so a requantized GGUF cannot recover them from its source. This is therefore a double quantization (Q8_0 → f32 → Q4_K) with unweighted Q4_K, i.e. a worse case than your imatrix-based path — which makes the quality numbers below a pleasant surprise.

2. score_official — as requested, plus your files for comparison

data/flash/manifest.tsv, ctx 4096, both env vars set to 1 (so the grouped-tok8 / QKV-pair paths from my previous comment are out of the picture):

	avg_nll	top1	first_match	avg_lcp
yours: Q8 (macOS, older ckpt)	0.632957	82.79%	44	3.830
yours: Q4 (macOS, older ckpt)	0.653162	82.88%	41	3.420
mine: Q8 -0731 (CUDA)	0.404811	85.86%	55	5.150
mine: Q4 requant -0731 (CUDA)	0.407383	85.34%	56	5.540

Q8 → Q4 delta: +3.19% for your pair, +0.64% for mine.

Please don't compare the absolute values across our two pairs — the data/flash fixtures were collected from the official DeepSeek V4 Flash 0731 API, so they match my checkpoint and not the older one. That alone explains most of the level difference (0.405 vs 0.633). Only the within-pair deltas are meaningful, and even those differ in backend and checkpoint.

Worth noting: on my pair first_match and avg_lcp actually go up slightly with Q4 (56 vs 55, 5.540 vs 5.150), while top1 drops 0.5 pp — so greedy behaviour is essentially preserved.

Classic perplexity on my own 1882-token sample agrees: 11.108 → 11.316, i.e. +1.87%. That also settles the confusion from my earlier comments — the +14.6% I reported before was almost entirely the checkpoint difference, not the quant.

3. Speed and memory on the requantized -0731

ctx	prefill Q8	prefill Q4	decode Q8	decode Q4
2048	817.32	789.63	17.55	18.41
4096	892.89	850.92	15.08	15.71
6144	883.59	842.99	14.97	15.61
8192	877.75	839.80	14.83	15.44

Decode is ~5% faster than Q8_0 (that gain is from the rest of 8b84e878, as reported earlier), prefill is ~4% slower. Startup spans 8.20 → 6.06 GiB, aligned artifacts 474/78.71 GiB → 259/74.16 GiB, and free memory after startup:

ctx	Q8 -0731	Q4 requant -0731
200000	9.8 GB	23.1 GB
300000	8.1 GB	20.3 GB

So on a 128 GB GB10 this variant now gives us --ctx 300000 with more headroom than --ctx 200000 had before, at +0.64% avg_nll and with faster decode. That is a straightforward win for long-context serving, and it is only possible because --source-gguf exists — thanks for adding it.

Happy to re-run any of this, or to try an imatrix-guided requantization if you think it is worth measuring the difference.

[adamlawi]

Thanks for the push — the variables are there now and I measured 0c20bfb2 on the DGX Spark (GB10, CUDA, Q4+imatrix, --temp 0, 128 tokens, same 1,574-token prompt, DS4_DSPARK_STATS=1, two runs per configuration).

Short version: DS4_CUDA_DSPARK_EXACTN does exactly what it promises and the end-to-end result does not move. The cost is relocated, not removed — and I think I can show you where it goes.

configuration	decode t/s	vs baseline
baseline, no DSpark	19.28 / 19.30	—
DSpark, defaults (conf 0.7)	13.03 / 12.96	−32.6%
DS4_CUDA_DSPARK_EXACTN=1	13.10 / 13.06	−32.2%
DS4_CUDA_ENABLE_DSPARK_NONCAUSAL_ONLINE=1	12.98 / 12.97	−32.7%
both	12.61 / 12.65	−34.5%
both + --dspark-confidence 0.9	16.18 / 16.22	−16.0%

All non-timing counters are byte-identical within each pair, so the ordering is solid.

Where the time goes

EXACTN is a genuine win on both instrumented stages:

stage	defaults	EXACTN=1
verify_layer	2452 ms	876 ms (−64%)
replay	3886 ms	1172 ms (−70%)
sum of the two	6338 ms	2048 ms (−4290 ms)
verify (total)	2472 ms	5123 ms
spec_total	6387 ms	6319 ms
net_saved	−3127 ms	−3052 ms

In the default run verify is essentially all verify_layer (2472 vs 2452). With EXACTN=1, verify is 5123 ms while verify_layer is 876, verify_upload 16, verify_read 0.3, verify_head 0 — leaving ≈4230 ms inside verify that no counter accounts for.

That unaccounted 4230 ms is almost exactly the 4290 ms you saved on verify_layer + replay. So the exact-N path looks like it is absorbing the work rather than eliminating it, and it has no timer of its own, which is why spec_total barely moves (6387 → 6319) and decode doesn't move at all.

Is that path instrumented anywhere I missed? If you add a counter for it I'll re-run immediately — right now the biggest single cost in the speculative path is invisible in the stats output.

Two details that may help localize it: the speculation counters are identical to the default run (cycles=51, proposed=116, avg_accept=1.510), so nothing about what gets speculated changed — only replay_fallbacks (30 → 13) and the cuda_exactn_* family (attempt=26, full=17, fallback=9, partial_fallback=9). Note that 9 of 26 attempts fall back; if the fallback re-does work already attempted, that would fit the shape of the missing time.

The other two flags

DS4_CUDA_ENABLE_DSPARK_NONCAUSAL_ONLINE activates but is performance-neutral: verify 2490 ms, replay 3844 ms, both within noise of the defaults. It does change numerics — cycles 51 → 52, proposed 116 → 112, avg_accept 1.510 → 1.462 — deterministically across both runs. Enabling both flags is worse than either alone (12.63 t/s, spec_total 6627 ms).

DS4_METAL_DSPARK_EXACT_ROWS_ASYNC_TAILS I did not test — this is a CUDA machine, and the flag isn't in the CUDA binary.

Best result so far, with a caveat I want to flag myself

both + --dspark-confidence 0.9 at 16.18 / 16.22 t/s is the best DSpark figure I have measured on this machine (previous best 15.15 on 5e05a6c7), and net_saved improves from −1733 ms to −1137 ms. But I only measured conf 0.9 with both flags on, so I cannot separate the flags from your antirez/main merge. At conf 0.9 EXACTN only fires 8 times, which makes me suspect the merge is doing most of that work — but that's a guess, not a measurement. I'll settle it next time I have the machine down.

The direction from my previous comment still holds: every configuration that speculates less does better, and even the best case is net-negative.

Unrelated, for your information

Baseline decode is unchanged after the merge (19.29 vs 19.32 on 5e05a6c7), so nothing regressed on the path I run in production. Baseline prefill, however, is 758.69 / 760.34 against ~773 on c3c862d6 — −1.7%, outside the 0.2% run-to-run spread. It's a cross-build comparison with an identical prompt, so I'd call it a signal rather than a finding, but you may want to know it before the merge goes further.

Usual caveats: one synthetic prompt, one machine, --temp 0.

[GiorgioOppo]

Thanks for testing it, I'll try something else. After this attempt, I'm out of ideas and will close the PR. I'll reopen it with a clean version using only the Q4 implementation.

[GiorgioOppo]

@adamlawi Can I ask you for a comparison between q4 and q8 with and without dispark for troubleshooting?

[adamlawi]

Here's the Q4 vs Q8 comparison you asked for, plus a correction to something I got wrong in my previous comment.

Same machine and method throughout: DGX Spark (GB10), CUDA, --temp 0, 128 tokens, same 1,574-token prompt, DS4_DSPARK_STATS=1, two runs per configuration, all in one window on 0c20bfb2. Non-timing counters are byte-identical within each pair.

Q4 vs Q8, with and without DSpark

model	no DSpark	with DSpark	regression
Q4 (AProjQ4 requant + imatrix)	19.27 / 19.16	12.86 / 12.84	−33.1%
Q8 (AProjQ8-SExpQ8-OutQ8 + imatrix)	18.27 / 18.29	12.46 / 12.22	−32.5%

The regression is essentially identical on both quantizations, so this isn't a Q4-specific problem.

What makes that more interesting is that draft quality is not the same on the two models:

	Q4	Q8
cycles	51	66
proposed	116	99
accept_rate	66.38%	61.62%
avg_accept	1.510	0.924
verify	2505 ms	2300 ms
replay	3921 ms	3297 ms
spec_total	6456 ms	5624 ms
target	2726 ms	3727 ms
net_saved	−3178 ms	−3147 ms

Q4 produces 63% better drafts than Q8 (avg_accept 1.510 vs 0.924) and ends up with the same regression, and net_saved lands within 1% of each other (−3178 vs −3147 ms). The speculative path costs about the same in both cases; what differs is only how much target decode it is being compared against.

For reference on the non-DSpark path: Q4 is 5.1% faster than Q8 in decode (19.22 vs 18.28 avg) and 2.3% slower in prefill (759.7 vs 777.5).

Your confidence 0 configuration

I ran the exact configuration you posted (DS4_CUDA_DSPARK_EXACT2=0, DS4_CUDA_DSPARK_EXACTN=1, DS4_CUDA_DSPARK_PROPOSER_BLOCK_MAX=5, DS4_DSPARK_SSD_VERIFY_BLOCK_MAX=5, --dspark-confidence 0, --ctx 4096, --prefill-chunk 128, --nothink), adding only --prompt-file and -n 128 so it terminates.

It produces the best draft quality I have measured on this machine and the worst end-to-end result:

cycles=27 proposed=131 accept_rate=77.10% avg_accept=3.741 no_draft=0
cuda_exactn_attempt=25 cuda_exactn_full=13 cuda_exactn_fallback=12
verify=7424.8ms verify_layer=1347.3ms replay=1845.8ms
spec_total=9298.7ms target=1463.5ms net_saved=-4406.5ms
decode: 11.24 / 11.28 t/s

avg_accept of 3.741 is six times better than the 0.622 I measured on c3c862d6 and more than double the 1.510 at default confidence. Every draft-quality metric is the best in the series. And spec_total is 9299 ms against 1463 ms of target decode — the speculative path costs 6.4× the work it replaces.

(Its prefill reads 269.78 / 270.44 rather than ~848 — that's --prefill-chunk 128, not a regression. Not comparable with the other rows.)

I think this is the cleanest evidence so far for the point I've been making across the last few rounds: on this hardware, better drafts do not help, because the bottleneck is the per-cycle cost of speculating at all.

Correction: I was wrong about where the improvement came from

In my last comment I reported 16.18 / 16.22 t/s at --dspark-confidence 0.9 with both flags on, and said I suspected your antirez/main merge rather than the flags. I measured it, and it's the opposite.

conf 0.9	decode	verify_layer	replay	spec_total
5e05a6c7 (before your optimizations)	15.15	—	—	—
0c20bfb2, flags off	14.94 / 15.01	752 ms	1588 ms	2356 ms
0c20bfb2, flags on	16.18 / 16.22	85 ms	476 ms	1799 ms

The merge alone is −1.1%. Your flags are worth +8.2%. At conf 0.9 they cut verify_layer by 89% and replay by 70%, and this time the saving is not eaten by the unaccounted cost I described last time — spec_total drops by 24% and it shows up in decode.

So the picture is more specific than "the optimizations don't help":

• at conf 0.7 (heavy speculation) the added cost cancels the saving — 13.03 → 13.10, no change
• at conf 0.9 (light speculation) the saving survives — 14.98 → 16.20, +8.2%
• at conf 0 (always speculate) it's worst of all — 11.26

Your work measurably helps in the low-speculation regime. That's a narrower claim than you were probably hoping for, and DSpark is still net-negative overall here (best case net_saved −1137 ms). But it isn't nothing, and I wanted you to have that number before you decide what to do with the PR — it's a different result from the one I sent you this morning.

If it's useful before you close: I'm happy to run the full --dspark-confidence sweep (0.5 / 0.7 / 0.8 / 0.9 / 0.95) with the flags on, which would show exactly where the crossover sits. That's one window on my side.

Usual caveats: one synthetic prompt, one machine, --temp 0.

[adamlawi]

Retracting the prefill observation from my earlier comment — it was an artifact of my own method, not something your merge did.

I said baseline prefill looked ~1.7% lower after the merge (758.69 / 760.34 against ~773 on c3c862d6). I've now measured both binaries in the same window, alternating prod / test / prod / test, same model, same prompt, same minute — the only variable being the binary:

binary	prefill	decode
823186e2 (what I run in production)	764.13 / 760.45	18.70 / 18.76
0c20bfb2 (after your merge)	768.77 / 765.02	19.28 / 19.26

Prefill is 0.6% higher after the merge, not lower. Nothing regressed.

What misled me: both binaries measured ~762–767 in that afternoon window, while the same 0c20bfb2 had measured 757–761 in the morning. Prefill drifts about 1% between windows on this machine even though within-pair spread is 0.2–0.5%. I compared a number from one window against a number from two days earlier and read the difference as a property of the code. That comparison wasn't valid.

Incidentally, and opposite to what I expected: decode on 0c20bfb2 is 2.9% faster than on 823186e2 (19.27 vs 18.73 avg), and my 823186e2 figure matches what I measured on 08-08 (18.71) to the decimal — so that difference does reproduce across days, unlike the prefill one.

Sorry for the noise on that point.

[adamlawi]

I ran the full --dspark-confidence sweep I offered, with both your flags enabled. It turned up something that I think changes the diagnosis, so I'm sending it even though I see you've just pushed e4e195c6 — all of this is measured on 0c20bfb2, and I'm happy to repeat it on the new commit.

Method as before: DGX Spark (GB10), CUDA, Q4+imatrix, --temp 0, 128 tokens, same 1,574-token prompt, DS4_CUDA_DSPARK_EXACTN=1 + DS4_CUDA_ENABLE_DSPARK_NONCAUSAL_ONLINE=1 throughout, two runs per point, baseline measured in the same window.

The curve

confidence	decode t/s	vs baseline	proposed	spec_total
baseline (no DSpark)	19.31 / 19.39	—	—	—
0.5	10.99 / 11.04	−43.1%	148	8532 ms
0.7 (default)	12.59 / 12.67	−34.7%	112	6634 ms
0.8	15.45 / 15.41	−20.3%	73	3592 ms
0.9	16.23 / 16.28	−16.0% ← best	39	1803 ms
0.95	16.15 / 16.10	−16.7%	26	1441 ms
0.99	16.12 / 16.11	−16.7%	3	157 ms

The curve does not converge to baseline. It flattens at −16.7%.

At confidence 0.99 the module barely speculates at all: 3 proposals across 124 cycles, verify = 0.000 ms, replay 157 ms, spec_total 157 ms. The speculative path costs essentially nothing — and decode is still 16.12 against a 19.35 baseline. That's 3.2 t/s that spec_total cannot account for.

Where it comes from — it's the flag, not the memory

I ran three states in one window, interleaved, two runs each:

state	decode	prefill
A — baseline	19.31 / 19.29	764.18 / 766.80
B — support model loaded via --mtp, but no --dspark	19.31 / 19.29	764.58 / 763.98
C — --dspark --dspark-confidence 0.99	16.19 / 16.01	846.48 / 846.45

B is indistinguishable from A, to two decimal places, in both runs. Loading the 5.6 GB support model costs nothing at all (B also emits no DSpark stats, confirming the module is inactive).

The entire ~16% appears the moment --dspark is raised, while the module is proposing 3 drafts in 124 cycles. And it comes with a second change in the same step: prefill jumps from ~765 to ~846, +10.7% — also at conf 0.99, also with nothing being speculated.

So: not memory, not the cost of speculating. Enabling --dspark appears to switch the execution path itself — paying ~16% on decode and gaining ~11% on prefill before any draft is produced.

What I think this means for the PR

This reframes the last three rounds. Both of us have been treating the cost as proportional to speculation — which is why EXACTN looked like the right lever, and why it was reasonable to expect that cutting verify/replay would eventually bring DSpark to parity. On this hardware it can't: even with the speculative path at ~0 ms, there's a fixed ~16% floor underneath it.

That's not a reason to abandon the work — it's a different target. Your flags do measurably help in the low-speculation regime (+8.2% at conf 0.9, as I reported earlier). But the thing standing between DSpark and break-even here isn't verify or replay; it's whatever --dspark changes globally on entry. The prefill improvement in the same step suggests the two are the same mechanism seen from opposite ends — something is being reconfigured for speculative execution and it happens to suit prefill and hurt decode.

If you want a starting point: whatever --dspark disables or reroutes in the decode path when it initializes, independent of dspark_confidence_threshold.

One counter question

net_saved doesn't reconcile at the top of the range. At conf 0.99 it reads −1205 ms while spec_total is 157 ms and verify is 0.000 ms. Whatever it's subtracting isn't in the other counters — that may be the same missing accounting as the ~4230 ms I reported inside verify last time.

Caveats

• All on 0c20bfb2; e4e195c6 is newer and touches exactly this area. Say the word and I'll rerun the sweep and the A/B/C test on it — it's about 20 minutes on my side.
• The C pair spread 1.1% (16.19 vs 16.01), wider than the usual 0.2–0.5% here. Doesn't affect the conclusion — the gap to baseline is 3.2 t/s — but noting it.
• Best operating point in this build is --dspark-confidence 0.9, not higher; above it the curve dips slightly and flattens.
• One synthetic prompt, one machine, --temp 0.

[GiorgioOppo]

oh wow 850 t/s on prefill side is really impressive

[adamlawi]

Heads-up: the GB10 fast path changes numerics — the code comment says it doesn't

Setup: DGX Spark (GB10), DeepSeek V4 Flash 0731, own AProjQ4 requant + imatrix, --ctx 300000, quality via gguf-tools/quality-testing/score_official on the 100-case flash manifest.

ds4_gpu_matmul_q4_K_hc_expand_available() says the default GB10 candidate "keeps the ordinary Q4_K MMVQ result". It does not — the results move, and one env var moves them back:

build	avg_nll	avg_lcp
823186e2	0.396432403	6.090
0c20bfb2	0.396432403	6.090
94fe9e89 default	0.395886931	5.880
94fe9e89 + DS4_CUDA_NO_Q4_GB10_FAST=1	0.396432403	6.090

Why this is solid: score_official is fully deterministic here — it reproduces to the 9th decimal across days, and two different builds with a 35-commit antirez/main merge between them give bit-identical results. The variance floor is zero, so any difference is real. Toggling that one flag switches the numbers back and forth. Per-case, avg_nll differs in 100/100 cases.

This is not a quality regression — I want to be precise about that:

• agreement with the API's top-1 token over all 2313 positions improves: 1990 → 1992
• avg_lcp drops only because 2 cases out of 100 flip an early argmax, and greedy_lcp is a prefix metric, so one early flip forfeits the whole tail
• both flips are stylistic, at razor-thin margins: Here's → Here is (0.540 → 0.273 nats) and inline \( 2x \) → display math (0.093 → 0.018 nats)

So the fast path looks fine quality-wise, and it's a big win: decode 18.73 → 25.11 t/s (vs 20.58 with the fast path off). The only issue is that the comment promises bit-exactness that isn't there — worth either relaxing the comment or checking whether the epilogue has an unintended precision difference (accumulation order?).

[ivanfioravanti]

Thanks for the feedback @GiorgioOppo and @adamlawi
I will deep dive on it to understand what's going on and move the PR to draft in the meantime.

[GiorgioOppo]

Thanks @adamlawi, I'll review the code.
In the meantime, @ivanfioravanti , could you try our version on a more powerful Mac than my 16GB one?

[GiorgioOppo]

@adamlawi Here are a couple of optimizations, I hope. Could you rerun the q4 vs q8 vs main antirez tests?

[GiorgioOppo]

I've pushed the physical limits of my hardware by generating at 3 t/s on my 16GB MacBook Pro. I don't think I'd suggest any other optimizations. I encourage everyone to continue with the Q4 because it has good prospects. Compared to the Q8, it doesn't seem to lose quality, but rather brings benefits.

SUMMARY:

1. On the MacBook Pro with a 16GB SSD streaming, there's a clear improvement of 2x3.
2. A 10% improvement even on the MacBook without SSD streaming.
3. Better memory utilization in DGX, allowing for increased context
4. increase generation speed on DGX to 25 t/s
5. small lapse in correctness

[adamlawi]

Thanks — ran the q4 vs q8 vs main antirez comparison you asked for, on ad8149df with no opt-in flags set (DS4_CUDA_ENABLE_Q8_FOLD and DS4_CUDA_ENABLE_Q4_K1024_PERSISTENT present in the binary but unset in the environment).

Good news: the numerics fix in 9ba80f06 didn't cost the speed.

Setup: DGX Spark (GB10), CUDA, same 1,574-token prompt, score_official on the same 100-case manifest (2,313 target tokens), decode via ds4-bench interleaved A-B-A-B in one window against our current production binary.

	avg_nll	avg_lcp	first_match	top1_rate	decode @ctx2048 t/s
ad8149df (this fix, default)	0.396432403	6.090	60/100	0.860355	24.55–24.86
our current production (94fe9e89, default)	0.395886931	5.880	—	—	24.57–24.97
ad8149df Q8 (reference)	0.404811251	5.150	55/100	0.858625	—

avg_nll/avg_lcp/first_match/top1_rate on ad8149df match our earlier canonical numbers (the ones we got with DS4_CUDA_NO_Q4_GB10_FAST=1 on the old build) to the 9th digit, so the row-packed epilogue is confirmed out of the normal decode path. Speed across four context frontiers (2048/4096/6144/8192) is within noise (<0.3%, both directions) of what we're running in production today — so whatever recovered the throughput, it isn't either of the two opt-in flags in this commit, since both were off for this run.

main antirez doesn't load the model on this test (unsupported type 12, cuda prefill failed) — expected, since the Q4_K attention path (case 12u) only exists on this branch.

This looks like a strict improvement over what we're running now: same speed, correct numerics. We haven't switched production to it yet — want to let it sit for a bit given how fresh it is, same caution as last time. Let us know if you want anything else measured.

[GiorgioOppo]

@adamlawi can you add also prefill speed, and the comparison with the q8 in the main branch of Antirez?
This would allow us to have a complete picture to ask for the merge

[adamlawi]

Sure — prefill from the same run, plus the main-branch Q8 comparison.

Prefill (t/s, avg of the two interleaved runs, same 1,574-token prompt):

ctx frontier	ad8149df (fix)	our production (94fe9e89)
2048	849.72	850.02
4096	922.60	920.94
6144	906.56	908.69
8192	904.87	905.17

Same story as decode: within noise, no regression from the fix.

main branch (antirez/ds4, 84cc8823) + Q8, same manifest/prompt:

Q8 doesn't touch the Q4_K attention path, so it loads fine on main (unlike Q4, which needs case 12u from this branch).

	avg_nll	avg_lcp	first_match	top1_rate
main + Q8	0.404811251	5.150	55/100	0.858625162
our fork + Q8 (from the earlier table)	0.404811251	5.150	55/100	0.858625162

Bit-identical to the 9th digit — our fork's Q8 path matches main exactly, so the fork isn't introducing any drift there. The only thing that changed anything numerically was the new Q4_K attention path, and 9ba80f06 fixed that.

main + Q8 speed for reference (same 4 frontiers): prefill 839.25 / 924.28 / 926.63 / 928.63 t/s, decode 17.58 / 15.10 / 14.99 / 14.87 t/s — this is the Q8 baseline your Q4 work is competing against, consistent with what we measured on our own Q8 file back in August.

We switched production to ad8149df a couple hours ago given how clean this looked (same speed as what we had, correct numerics). Will keep watching it over the next few days.

[GiorgioOppo]

@DanteCpp would you like to test this implementation?

[marcodelpin]

Happy to. Since @adamlawi has already covered CUDA thoroughly on GB10, let me say up front what
this board actually adds rather than repeating his work: it is the only non-GB10 CUDA target in
the thread.

Jetson AGX Thor T5000 - Blackwell, compute capability sm_110 (not sm_121), 2560 CUDA cores /
96 tensor cores, 128 GB LPDDR5X at 273 GB/s, CUDA 13.0.48, L4T R38.4.

Why the arch difference is not cosmetic:

• sm_110 sits outside the sm_120/sm_121 MXFP4 allowlist in the Makefile, so it takes different
  compile-time paths from GB10 by construction.
• ds4_gpu_matmul_q4_K_hc_expand_available() is a GB10 candidate path, so this board should not be
  taking it. That makes it a useful control for the numerics discrepancy at the end of the thread:
  if the shift reproduces on sm_110 it is in the shared code, if it does not it is GB10's.

So the useful output from here is whatever separates sm_121-specific from CUDA-in-general.

Two runs, because they answer different questions

1. Reproduce adamlawi's runs on the published file. Same GGUF, same tests, different arch - that
isolates architecture as the only variable. This is the cross-arch control his numbers currently
lack.

2. Requantize our own -0731 and test that. Neither published AProjQ4 is the -0731 checkpoint,
so run 1 cannot give a same-checkpoint delta against our baseline. Following adamlawi's recipe with
--source-gguf on the -0731 AProjQ8 we already hold locally gives a clean same-checkpoint
comparison on sm_110, and skips a 78 GiB download.

Both matter: run 1 is comparable to his results, run 2 is comparable to ours.

The baseline for run 2 is already published and on the same box: canonical ds4-bench sweep over 32
contexts, idle machine verified at the start and again at the end, plus --logprob-vectors,
--server and make cuda-regression green on sm_110 (#788). At 78.6 GiB the AProjQ4 is fully
resident here, so neither run involves streaming.

One practical note: the AProjQ4 file is not on the main branch of antirez/deepseek-v4-gguf, and a
HEAD against the plain main path returns a 15-byte body - which reads as a broken URL rather than as
a pending PR. You gave adamlawi the pinned-commit link; it may be worth putting it in the PR
description too until HF discussion #18 lands.

Which commit should I build? The thread mentions f5662897, then 0c20bfb2, 823186e2, 94fe9e89
and b4922c96, and the last of those comes with an open numerics question - so I would rather take
the one you consider current than guess from the discussion.

Timing: the board is part-way through an unrelated benchmark queue, so this will not be immediate. I
will post numbers rather than progress updates, and I will say which results are inconclusive rather
than filling the gaps with GB10 figures.

[adamlawi]

@marcodelpin ad8149df is the one I'd consider current — that's what we're running in production and what the q4/q8/main-antirez and prefill numbers I posted above are from. It's the tip of GiorgioOppo's aprojq4-dense-attention branch (11.08, "preserve canonical Q4 GB10 numerics"), which supersedes 823186e2/94fe9e89/b4922c96 mentioned earlier in the thread — those were intermediate states before the numerics fix.

Looking forward to the sm_110 control — if the avg_lcp shift reproduces there it points at shared code rather than GB10-specific.