For makers and artists, the bottleneck is no longer compute power, but memory bandwidth. When running long-context models, the Key-Value (KV) cache-the memory storing attention history for every token-can grow to dwarf the model’s own weights. This forces the system to wait on data streams rather than performing calculations, creating latency that kills creative flow. The solution lies in shrinking this cache without losing quality. Recent 2026 research has converged on ultra-low-bit quantization, with three distinct contenders leading the charge.
Consider Llama-3.1-70B in BF16. Its KV cache consumes roughly 0.31 MB per token. At 128K tokens, this swells to approximately 40 GB; at 1M tokens, it exceeds 300 GB-more than double the 140 GB required for the weights themselves. Reducing this footprint is the primary lever for cutting both operational cost and decode latency.
Current methods generally fall into five categories: token eviction, quantization, low-rank projection, merging, and architectural sharing. The latest work focuses heavily on quantization. Google and NYU’s TurboQuant (ICLR 2026) and Together AI’s OSCAR attack the problem from opposing angles, while Apple’s EpiCache addresses a specific gap neither covers.
The core challenge for quantizers is outlier channels: specific channels with disproportionately large magnitudes that dominate the quantization range, squeezing the rest of the signal into a few levels. This is why naive INT2 quantization (four levels) often collapses to near-zero accuracy.
KIVI established the baseline here. It demonstrated that key vectors possess fixed outlier channels across tokens, whereas value vectors do not. Consequently, it quantizes keys per-channel and values per-token. This tuning-free 2-bit recipe reduces end-to-end peak memory by about 2.6×, serving as the reference point for newer methods.
TurboQuant: data-oblivious and theoretically optimal
TurboQuant handles outliers without inspecting your data, using a two-stage approach:
- Stage one: each vector is randomly rotated so its coordinates become nearly independent and approximately Gaussian, allowing an optimal precomputed scalar (Lloyd–Max) quantizer to be applied per coordinate.
- Stage two: a 1-bit Quantized Johnson–Lindenstrauss (QJL) transform is applied to the residual, providing a provably unbiased estimate of attention logits with no normalization-constant overhead.
The theoretical selling point is that TurboQuant’s distortion is provably within a small constant factor (≈ 2.7×) of the information-theoretic lower bound. In practice, it achieves essentially full-precision recall on Needle-in-a-Haystack tasks at 4× compression. The paper reports absolute quality neutrality at 3.5 bits and only marginal degradation at 2.5 bits per channel. Because it requires no calibration, it works on any model untouched and also functions as a fast vector-database quantizer.
One caveat to note: the widely repeated claim of “8× faster attention on H100” originates from Google’s blog, not the academic paper. That figure refers to a narrow attention-logit microbenchmark. TurboQuant’s documented sweet spot remains the 3–4 bit near-lossless regime.
OSCAR: attention-aware and deployment-ready
OSCAR takes the opposite approach. Its premise is that at INT2’s four levels, data-oblivious rotation is insufficient; blindly smoothing ranges fails when there is almost no precision to spare. Instead, OSCAR computes an attention-aware rotation from a one-time offline calibration pass. Keys are rotated into the eigenbasis of the query covariance, while values are rotated into the score-weighted value covariance. A Hadamard transform plus a bit-reversal permutation then spreads channel importance evenly across quantization groups.
What sets OSCAR apart is that it ships as a complete system, not just an algorithm:
- Mixed-precision paged cache: recent tokens remain in BF16 while history compresses to INT2. At 128K context, only ~0.24% of tokens stay in BF16.
- Fused Triton kernels with full SGLang integration (supporting paged-attention and prefix-cache compatibility).
- Precomputed rotations (a “RotationZoo”) available for Qwen3-4B/8B/32B, GLM-4.7-FP8, and MiniMax-M2.7, requiring no recalibration.
Operating at an effective 2.28 bits, OSCAR lands within 1.42 points of BF16 on Qwen3-8B and is essentially on par on Qwen3-32B (a 0.02-point gap). On GLM-4.7-FP8, where naive INT2 collapses to zero and data-oblivious baselines reach only low single digits, OSCAR matches BF16 and edges slightly ahead on reported benchmarks. Together AI reports up to 7.83× job-level throughput and roughly 8× KV-cache memory reduction at 100K context, with up to ~3× faster decoding.
So which one wins?
Neither, and that is the honest answer. For deployable INT2 at 128K tokens on supported models, OSCAR is currently the only demonstrated option that does not collapse, complete with production-ready SGLang support. For training-free, model-agnostic quantization in the 3–4 bit regime, TurboQuant offers far broader generality.
OSCAR’s paper notes that TurboQuant drops by more than 40 points at a comparable budget, but that evaluation runs inside OSCAR’s own framework, quantizes all layers, uses a single random seed, and operates well below TurboQuant’s intended bit-width. That is a weak basis for a head-to-head verdict. A more interesting possibility is that the two are complementary: pairing a calibration-aware rotation with an optimal scalar quantizer is a promising combination nobody has shipped yet. (Both teams have publicly noted the same idea.)
The third axis: EpiCache
TurboQuant and OSCAR are both built for a single long context. Neither handles extended multi-turn conversations, where history piles up across many exchanges. Apple’s EpiCache is a training-free KV-cache management framework aimed exactly at that gap:
- Block-wise prefill processes history in blocks to keep peak memory bounded.
- Episodic clustering segments the conversation into coherent semantic “episodes,” each with its own compressed cache.
- Episode-matched retrieval routes each query to the most relevant episode at inference time.
- Adaptive layer-wise budget allocation measures each layer’s sensitivity to eviction and distributes the memory budget accordingly.
Across LongMemEval, RealTalk, and LoCoMo, EpiCache reports up to 40% higher accuracy than eviction baselines, near-full-cache accuracy at 4–6× compression, and up to 3.5× lower peak memory (and ~2.4× lower latency). Because it decides which tokens to keep rather than how precisely to store them, it composes directly with OSCAR or TurboQuant for compounding savings.
Key takeaways
- TurboQuant pushes the theoretical, model-agnostic frontier, the go-to for 3–4 bit near-lossless compression on any model.
- OSCAR leads on deployable INT2, with up to 7.83× throughput and ~8× memory reduction at 100K context on supported models.Source Read original →
