flash-decoding

Here are 3 public repositories matching this topic...

JohnScheuer / flash-decoding-bench

Benchmark comparing Flash-Decoding-style chunked KV-parallel attention against standard SDPA at decode time (q_len=1) on RTX 2070, measuring the crossover point where KV-parallelism outperforms fused attention kernels.

benchmarking cuda pytorch transformer decode attention kv-cache long-context llm-inference flash-attention flash-decoding mlsystems

Updated Jul 19, 2026
Python

8b-is / transformers-ultra

Sponsor

Star

Ultra-High-Throughput Transformers Engine: NVIDIA FP8/TMA, Zero-Allocation Slotted Static KV Cache, Sub-5µs CUDA Graphs, Fused Logits Sampling, Tree Speculative Decoding, Split-KV Flash-Decoding & Apple Silicon Metal (MSL) Simdgroup Kernels

transformers pytorch zero-allocation tma apple-silicon fp8 llm-inference speculative-decoding flash-decoding cuda-graphs metal-msl

Updated Sep 11, 2026
Python

Sergasgr / CodeAlign-Runtime

Star

C++/CUDA inference engine for Qwen2.5-Coder-0.5B, written from scratch: INT4 GEMV/GEMM kernels, Flash-Decoding, a 24-layer decode engine and lossless speculative decoding. Verified against Hugging Face; benchmarked against bandwidth ceilings, PyTorch and llama.cpp.

benchmarking cpp cuda low-latency cuda-kernels quantization gpu-programming pytorch-extension gemv int4 llm-inference qwen speculative-decoding flash-decoding

Updated Sep 28, 2026
Python

Add this topic to your repo

To associate your repository with the flash-decoding topic, visit your repo's landing page and select "manage topics."

Learn more