KV cache compression via block-diagonal rotation. Beats TurboQuant: better PPL (6.91 vs 7.07), 28% faster decode, 5.3x faster prefill, 44x fewer params. Drop-in llama.cpp integration.

TurboQuant KV Cache Compression for llama.cpp — 5.2x memory reduction with near-lossless quality | Implementation of Google DeepMind's TurboQuant (ICLR 2026)

No description available

TurboQuant for GGML: 4.57x KV Cache Compression with 72K+ Context for Llama-3.3-70B on Consumer GPUs.

No description available

1bit llama.cpp gguf weights paired with turboquant 4 bit kv cache

TurboQuant KV cache compression for llama.cpp — HIP/ROCm port for AMD RDNA3 (gfx1100)

TurboQuant llama.cpp fork with optimized turbo4 kernels for Gemma 4 D=256/512 heads — lazy K/V, batch decode, warp-cooperative write. 120 t/s with 3.8x KV compression on RTX 3090.

Complete local AI coding pipeline: Qwen3.5-35B-A3B + llama-server + TurboQuant + OpenCode + Context7 MCP + Chrome DevTools. 188 t/s on RTX 5090, zero cloud APIs.

No description available

One-click LLM server with TurboQuant Llama CPP engine

Practical guide: TurboQuant KV-cache quantization for llama.cpp. Run 122B models on consumer GPUs.

Fused Triton kernels for TurboQuant KV cache compression — 2-4 bit quantization with RHT rotation. Drop-in HuggingFace & vLLM integration. Up to 4.9x KV cache compression for Llama, Qwen, Mistral, and more.

Simple all in one build script for llama-cpp-turboquant on Windows 11.

🚀 Run any LLM on any hardware. 130% faster MoE inference with ExpertFlow + TurboQuant KV compression. Ollama-compatible API. Built on llama.cpp.

A TurboQuant implementation with Llama.cpp for AMD with Vulkan runtime

World's first TurboQuant KV cache compression for llama.cpp on AMD ROCm (RX 9070 / gfx1201)

🦙 Docker template for running llama.cpp llama-server in router mode with NVIDIA CUDA and AMD Vulkan GPU acceleration. Features TurboQuant KV cache optimization, long context support (up to 256K tokens), and optimized configurations for 24GB+ VRAM cards.

NEXUS: Production C++ inference engine for Apple Silicon. Run 400B+ LLMs on your Mac via layer streaming, Metal GPU compute, TurboQuant KV compression, NXF format, MoE routing, and Neural Engine speculative decoding. Faster than AirLLM, more capable than llama.cpp.

Experimental TurboQuant implementation and llama.cpp-style integration path for long-context inference

Pre-compiled Windows binaries and CMake fixes for the experimental TurboQuant branch (with Gemma 4 support)

Model Switcher & Benchmark Tool for llama-server with TurboQuant KV-Cache

llama.cpp fork: Qwen 3.5 hybrid GGUF + loader fixes; syncs with ggml-org/llama.cpp

TurboQuant Run larger AI models with longer context on your GPU — powered by Google's TurboQuant KV cache compression.

llama.cpp fork with TurboQuant quantization (turbo2/3/4) and TriAttention GPU-accelerated KV cache pruning. 75 tok/s on Qwen3-8B / RTX 3080.

Run Large Language Models on CPU with up to 8× less RAM using advanced KV cache compression.

a llama.cpp fork combining PrismML's 1-bit kernels with TurboQuant KV cache compression.

Pre-built llama-server with pmem-tier + TurboQuant KV cache compression for JAC

A lightweight desktop application for managing and interacting with llama.cpp models through a clean, modern interface - Support TurboQuant technology

TurboQuant Vulkan: 3-bit KV cache quantization for llama.cpp using Lloyd-Max Gaussian codebooks. 4.57x compression, Vulkan GPU support (AMD/Intel/NVIDIA). Hobby project.