Why INT2 KV Cache Quantization is Hard

KV activations contain channel-wise outliers. A small subset of channels holds extremely large values, while most are well-behaved. When quantized to 2 bits (INT2), the quantizer wastes most of its range on rare spikes, compressing normal values into just one or two effective levels. This degrades attention quality significantly.

What OSCAR Does Differently

The key observation is that the rotation applied before quantization should be derived from attention statistics, rather than raw KV activations. For keys and values, OSCAR estimates empirical covariance matrices to align channels with important attention directions.

The final composed rotations are:

RK = UQ · HHad · Pbr

RV = US · HHad · Pbr

Each of these factors addresses a distinct failure mode of per-group low-bit quantization:

• UQ / US: Aligns channels with attention-importance directions. This diagonalizes the error-weighting matrix so that important directions are identifiable.
• HHad: Equalizes channel importance exactly by compressing peaky eigenspectra to a uniform value across all channels.
• Pbr: Reorders channels so each group receives one representative from each level of the importance hierarchy, ensuring balanced quantization error distribution.

The research team proves UQ and US are optimal under a frozen-error surrogate objective with diagonal residual assumptions. They provide Theorem 1 to support these claims.

The Serving System: Mixed-Precision Cache Layout

OSCAR integrates into SGLang’s production serving stack as an INT2 KV-cache mode with full compatibility for paged attention. The KV cache layout uses three regions per request:

• Sink tokens (first 64 tokens): Stored in BF16 and used as attention sinks.
• Recent tokens (last 256 tokens before current position): Stored in BF16 for recent context.
• History tokens: Quantized to INT2 after OSCAR rotation and clipping. For values, the relevant error is in attention output SV; directions that remain large after aggregation by S are quantization error propagators. The value rotation RV is absorbed into model weights offline.

The write path involves rotating tokens, clipping them to a calibration-derived percentile threshold (e.g., cK = 0.96 for keys and cV = 0.92 for values), then quantizing with per-token asymmetric INT2 at a default group size of GK = 64 channels per group. The read path dequantizes, inverse-rotates, and passes results to the attention kernel in one fused pass.

Outcome

The research team evaluated OSCAR on four model configurations: Qwen3-4B-Thinking-2507, Qwen3-8B, Qwen3-32B, and GLM-4.7-FP8 (358B parameters). Benchmarks include AIME25, GPQA-Diamond, HumanEval, LiveCodeBench v6, and MATH500, all at 32K maximum generation length.

The research team also compared against channel-wise methods on AIME25, showing OSCAR at 2.38 bits per KV element achieves above KIVI-KV2* and Kitty in terms of accuracy.

The research team compared long-context robustness (RULER-NIAH) and throughput (H100, 100K context, batch size 1), showing OSCAR matches the BF16 curve through 128K context.

The speedup in throughput at batch size 32 reaches 6.17× on Qwen3-4B-Thinking and 7.83× on GLM-4.7-FP8, reducing KV memory by 8× directly.