Popular repositories Loading

1. Annotated-LLM-Runtime Annotated-LLM-Runtime Public

   From-scratch, heavily-annotated CUDA inference runtime for Qwen2.5-Coder-7B on H100 (sm_90). Custom INT4 packer, fused GEMV, paged KV, split-KV attention, CUDA graph decode — every hot path comment…

   Cuda 21 3

2. Qwen-jev Qwen-jev Public

   Turns a small open-source Qwen LLM into a fast, single-pass classifier — swap the word-guessing head for a classification head, fine-tune only the head, then get instant labels + confidence instead…

   Python 7 1

3. WarpGroup-backend WarpGroup-backend Public

   A high-performance C++ backend for extreme-context LLM inference. It replaces item-count batching with dynamic, VRAM-aware First-Fit Decreasing (FFD) bin packing. By using PyBind11 for async queuei…

   Python 5 1

4. swarmkv swarmkv Public

   C++ inference runtime for llama.cpp that shares a single document KV-cache prefill across multiple analytical branches via snapshot fan-out. Eliminates redundant GPU compute and dramatically reduce…

   C++ 3 1

5. VRAM-Conductor VRAM-Conductor Public

   C++17 + CUDA orchestrator for running multiple LLM agents on one constrained GPU. `lmxd` daemon does NVML-seeded admission control to stop llama.cpp OOM crashes; `LayerStreamer` + `PinnedHostPool` …

   C++ 3

6. inter-llm-tokf inter-llm-tokf Public

   Inter-LLM knowledge handover framework leveraging Open Knowledge Format (OKF) for Qwen family. Bypasses text paring and tokenization by passing binary token arrays directly into model embedding lay…

   Python 3 2