speed-bench: Jetson AGX Thor (sm_110) numbers

[marcodelpin]

speed-bench: Jetson AGX Thor (sm_110) numbers

Adds speed-bench/jetson_agx_thor.csv and its _ts.svg. No Jetson/Thor numbers were in the
corpus.

Hardware

NVIDIA Jetson AGX Thor T5000 developer kit - Blackwell, compute capability sm_110, 2560 CUDA
cores / 96 tensor cores, 128 GB LPDDR5X at 273 GB/s, 14-core Arm Neoverse-V3AE, CUDA 13.0.48,
L4T R38.4.

How it was run

Built from an unmodified tree - sm_110 needed no patch:

make cuda CUDA_ARCH=sm_110

The command from the README, plus --cuda for the CUDA backend, on an idle machine:

./ds4-bench -m ds4flash.gguf --cuda \
  --prompt-file speed-bench/promessi_sposi.txt \
  --ctx-start 2048 --ctx-max 65536 --step-incr 2048 --gen-tokens 128

Model: DeepSeek V4 Flash, IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731 (80.8 GiB),
fully resident in RAM.

Numbers

The absolute numbers, and the same rows against gb10.csv - same 128 GB LPDDR5X at 273 GB/s,
6144 CUDA cores against Thor's 2560, i.e. Thor has 41.7%:

ctx	prefill t/s	generation t/s	vs GB10 prefill	vs GB10 gen
2048	389.68	12.16	47.2%	67.4%
8192	416.63	10.75	47.2%	70.7%
16384	406.64	10.61	46.6%	70.3%
32768	378.96	9.99	44.3%	69.2%
65536	358.53	9.61	43.6%	69.4%

Across the full 32-context sweep: prefill 356.52 - 424.58 t/s, generation 9.48 - 12.16 t/s.

Prefill lands close to the CUDA-core ratio; generation sits consistently above it, as one would
expect if decode is partly bandwidth-bound and the bandwidth is identical on both.

kvcache_bytes matches gb10.csv byte-for-byte at every one of the 32 contexts.

Correctness on sm_110

Per CONTRIBUTING, on the CUDA machine:

make cuda-regression CUDA_ARCH=sm_110
  -> ds4: CUDA backend initialized on NVIDIA Thor (sm_110) dev=0
  -> cuda-regression: top-k n_comp=32768 n_tokens=32 elapsed=0.003s
  -> cuda long-context regression: OK

ds4_test --logprob-vectors also passes against the official DeepSeek continuation vectors in
tests/test-vectors/flash-0731/official.vec, and ds4_test --server passes.

No failures to report.

Power and efficiency

Sampled with tegrastats during the sweep, 305 samples, at the factory-default power mode:

total board draw	66.1 W mean, 104.3 W peak
GPU temperature	64.4 C peak
generation	0.184 t/s per W
prefill	5.90 t/s per W

Raising the power mode above the 120 W default bought +5.8% generation and cost 11.6% prefill, so
the default is the better operating point for this workload and these numbers are from it.

There are no power figures anywhere in speed-bench/ today, so this is a starting point rather
than a comparison - someone has to measure the first one. The method is stated above precisely so
it can be repeated on any other board: sample tegrastats (or the platform equivalent) for the
duration of the same sweep, at the default power mode, and report the mean and the peak of total
board draw rather than a reading taken before or after the run.

For a module whose whole point is being deployable, t/s per W is arguably the axis that matters
most, and right now nobody has published one.

Cost per token

Prices as of 2026-08, US list, ex-VAT, for the smallest configuration that fits the model:

platform	price	gen t/s	$ per gen t/s
MacBook Pro 14in M5 Max, 128GB	6,699	39.35	170
DGX Spark Founders Edition	4,699	18.05	260
Jetson AGX Thor dev kit	3,499	12.16	288

Thor is last on this axis, and the gap to the Mac is 1.7x. Worth stating plainly rather than
leaving implied: its case is not cost per token but that it is an embedded, deployable module with
the Jetson stack, at 66 W - a different product class from a laptop or a desk appliance.

Prices move: the Spark figure is the post-February-2026 MSRP, up 18% from its $3,999 launch price
on LPDDR5X supply. Anyone reading this later should re-check rather than trust the table.

---

Note: DS4_CUDA_HAVE_MXF4 gates the indexer, not the weights

No code change proposed.

Adding sm_110 to the Makefile allowlist does not work - ptxas rejects it:

Instruction 'mma with block scale' not supported on .target 'sm_110a'

The flag's five consumers are all in ds4_cuda.cu, all on the lightning-indexer path that needs
that block-scaled MMA. The MXFP4 weight kernels in cuda/mmq/ reference the flag nowhere, run
on __dp4a, and are already compiled into the stock sm_110 build: ds4_mmq_mxfp4_dense, _moe,
_moe_pair and _moe_vec are present in the binary, while indexer_mxf4* is absent.

They are also numerically correct there: make test-mxfp4-cuda on sm_110 reports
MXFP4 routed MMQ and MXFP4 routed MMVQ at 0/1536 failures, both fused decode paths
clean, and MXFP4 CUDA parity: PASS.

So the allowlist is correct as it stands. Only the name reads wider than the thing it gates, and a
one-line comment at the conditional would make that clear.

Whether a real MXFP4 model runs on Thor is untested here.

[GiorgioOppo]

Hi @marcodelpin, would you like to try our implementation #621 with a slightly different GGUF file on your Jetson AGX Thor board (sm_110)?