POWER10/11 MMA acceleration for all quantized formats (silicon-validated)

[mavin2009]

What this is

MMA (Matrix-Multiply Assist) acceleration for every quantized format this fork ships on POWER10/POWER11 — the Bonsai formats (Q1_0/Q2_0) plus the K-quants, the IQ grid codebooks, the legacy types, TQ1_0/TQ2_0, IQ4_NL/XS and MXFP4/NVFP4. Today only Q4_0/Q8_0 and the float types ride MMA; everything else falls back to scalar/VSX vec_dot.

Integration is via llamafile_sgemm behind #if defined(__MMA__): zero impact on any other architecture. The series adds ten kernel translation units, a tensor-keyed weight-pack cache (packs once per tensor per model lifetime, parallel first-touch across the op's threads), and small-n dispatch policy that routes each format to whichever path measured faster on silicon.

The branch is 21 commits, one per patch of the numbered series it was developed as; each commit message carries its measured result.

Measured on POWER10

IBM 9105-42A LPAR (4 cores/SMT2), RHEL 9.7, GCC 11.5. Baseline = the same tree built -mcpu=power9 (MMA compiled out), same machine, minutes apart.

Prompt processing: 3.7×–48× across every format tested. Highlights (pp128 @8t): Q2_0 1.7B 8.7→416 t/s (48×), IQ3_XS 1.5B 33→331 t/s (10×), TQ1_0 13→332 t/s (26×), Q4_K 27B 5.9→21.9 t/s (3.7×).

Token generation: parity or better everywhere. Q2_0 4.6× (its fallback is scalar); GER-GEMV over the cached tiles buys TQ1_0 3.1×, IQ2_XXS +42%, IQ3_XXS/S +38%, IQ1_S +23%, IQ1_M +58%; formats whose vec_dot measured better (TQ2_0, IQ2_S/XS, K-quants, IQ4, legacy) keep vec_dot at small n by explicit dispatch policy.

How it was verified

• End-to-end: temperature-0 greedy token-identity gates (MMA build vs no-MMA build, same tree) across 18 model probes covering every accelerated format family — all PASS. Where bit-identity is impossible by construction, a three-tier gate certifies divergences against the machine's measured cross-codegen envelope (a -mcpu=power8 control build containing none of this code).
• Decoders: bit-exact against ggml's own dequantize_row_* — every grid/sign table compared exhaustively, ~7.6M random dequantized elements at maxrel = 0.
• Kernels: 15 standalone suites vs exact float64 references (random data, ragged shapes, multi-slab, n=1), UBSan clean, run under qemu and natively on POWER10.

Full methodology, the validation harness, benchmark data and the complete engineering log (including the experiments that lost and stayed in-tree as documented negative space) are in the companion repo: https://github.com/mavin2009/ppc-mma-kernels — start with docs/VALIDATION-POWER10.md and docs/REVIEW.md.

Caveats, stated plainly

• Validated on one POWER10 machine. Power11 is untested (no P11-specific paths; it should simply inherit). If you have Power silicon, scripts/validate-on-power.sh in the companion repo runs the whole protocol against a checkout of this branch and emits a paste-ready report — independent results very welcome.
• MXFP4/NVFP4 are kernel-level + decoder-cross-check verified; they cannot be produced by requantization in this fork, so no end-to-end gate exists for them yet.
• The pack cache holds decoded int8 weights alongside the mmap'd model (defaults to a 2 GiB cap, PPC_MMA_PACK_CACHE_MB to change, refusals print loudly). Migrating it into the repack buffer-type machinery is designed but deliberately sequenced later; the design note explains why.

Happy to split this into smaller PRs (kernels first, cache second, dispatch policy third) if that's easier to review.

[khosravipasha]

nice, this is pretty cool. I missed this somehow.
It might be better to upstream to main llama.cpp speically for Q1_0 everything is already in the upstream.

For Q2_0 we are still in middle a migration but that will also eventually be upstreamed.

[Copilot]

Pull request overview

Adds POWER10/11 MMA acceleration for quantized CPU matrix multiplication.

Changes:

• Adds MMA kernels for K-quants, IQ formats, legacy formats, and Q1/Q2.
• Introduces cached weight packing and format-specific dispatch.
• Registers the new implementation with llamafile SGEMM.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ggml/src/ggml-cpu/CMakeLists.txt	Registers new POWER sources.
llamafile/sgemm.cpp	Dispatches quantized operations to MMA kernels.
llamafile/qbit_ppc_mma.{cpp,h}	Implements Q1_0/Q2_0 kernels.
llamafile/q2k_ppc_mma.cpp	Implements Q2_K kernels.
llamafile/q3k_ppc_mma.cpp	Implements Q3_K kernels.
llamafile/q4k_ppc_mma.cpp	Implements Q4_K kernels and GEMV.
llamafile/q5k_ppc_mma.cpp	Implements Q5_K kernels.
llamafile/q6k_ppc_mma.cpp	Implements Q6_K kernels.
llamafile/iq4_ppc_mma.cpp	Implements IQ4/MXFP4 kernels.
llamafile/iq_grid_ppc_mma.cpp	Implements grid-codebook and NVFP4 kernels.
llamafile/iq_grids_ppc.h	Provides IQ lookup tables.
llamafile/legacy_ppc_mma.cpp	Implements legacy quantized kernels.
llamafile/kquants_ppc_mma.h	Declares MMA and cache APIs.
llamafile/ppc_pack_cache.cpp	Implements the packed-weight cache.

Comments suppressed due to low confidence (3)

ggml/src/ggml-cpu/llamafile/ppc_pack_cache.cpp:98

• The cache identity omits lda, although llamafile_sgemm explicitly permits any row stride at least k. Calling the kernel for two views with the same base pointer, m, k, and format but different row strides reuses the first packed layout and produces incorrect output. Include the source stride (and immutable tensor/lifetime identity) in the key and all acquire/publish calls.

extern "C" void * ppc_apack_cache_acquire(const void * key, int64_t m, int64_t k,
                                          int variant, size_t bytes, int * fresh) {

ggml/src/ggml-cpu/llamafile/ppc_pack_cache.cpp:164

• This invalidation function is never called by the model or backend-buffer teardown paths, so admitted packs survive model unload and remain resident until process exit. Repeatedly loading models can consume the 2 GiB process-wide cap with unreachable packs, after which later models permanently fall back to per-call packing. Tie cache entries to the owning buffer/model lifetime rather than relying on an uncalled global clear.

// explicit invalidation for embedders that unload models
extern "C" void ppc_apack_cache_clear(void) {

ggml/src/ggml-cpu/llamafile/ppc_pack_cache.cpp:219

• An unready hit is assumed to belong to the same ggml operation. If two graph executions use the same tensor concurrently, the second execution also receives fresh=1; its threads duplicate row slices and decrement the first fill's pending, potentially publishing the buffer before every unique slice is packed. Add a per-fill operation identity, or make unrelated executions wait for readiness while only the creating operation joins the fill.

        if (hit) {
            void * b = hit->buf;
            if (!hit->ready) *fresh = 1;   // join the parallel fill
            pthread_mutex_unlock(&g_mu);
            return b;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mavin2009]

Thanks for taking a look and for the feedback.

I'll go ahead and upstream it if you think that's the best place for this.

[khosravipasha]

@mavin2009 yes please upstream and we can pick up the changes from there.
Main thing to note taht Q1_0 is equivalent in both places, but Q2_0 ours is group size 128 and the official one is group size 64 so make sure to test that for correctness.

You can use the ggufs ending with Q2_0_g64.gguf in our huggingface to test with the official Q2_0 llama.cpp.