Nunchux AI has released VC-Attention, a training-free low-bit attention kernel for video Diffusion Transformers designed to tackle value quantization error and slow softmax calculations simultaneously.
The video generation bottleneck
Video DiTs process clips by flattening them into sequences of spatiotemporal tokens and applying full self-attention at every layer. A 5-second 720p clip generated by Wan2.2-14B produces approximately 70,000 tokens. On an RTX 5090, the attention mechanism consumes more than 64% of the total generation time. The research team notes that attention accounts for roughly two-thirds of every denoising step on a single B200 when running MiniMax-H3.
While low-bit Tensor Cores accelerate the two matrix products involving QK and PV, two obstacles remain. First, existing methods such as SageAttention2 smooth queries and keys. Following QK smoothing and rotation, the value term contributes 82% of the output error on Wan2.2. Second, the softmax operation between the products continues to run in FP32. On B200 and H200 hardware, that exponential calculation and its FP8 cast form the longest stage in the pipeline.
V-Smooth: Handling value outliers
Value outliers appear in specific tokens, and their channels shift across heads, layers, and steps. A Hadamard rotation preserves token norms, so it fails to remove them. Rotating V changes value error by only 0.2%.
V-Smooth takes a different approach:
- Group: An online k-means algorithm clusters value tokens per batch and head. Keys and values are permuted together, ensuring non-causal attention output remains unchanged.
- Demean: Each 128-token hardware block subtracts its mean. Only the residual is quantized, using per-channel E4M3 at 8 bits or NVFP4 at 4 bits.
- Restore: The mean is added back using the row sum online softmax already in place. No second pass or extra buffer is required.
Averaged over 100 Wan2.2 heads, the block mean removes 8% of block energy in sequence order. It removes 12% under DeltaQuant’s static cube and 36% after sorting. Each mean operation costs 0.125 bit per value element.
Grouping runs only on the first 25% of denoising steps. The permutation is reused across 4 adjacent steps. Averaged over the full schedule, grouping costs 3 to 4% of attention time.
ExpCast-FP8: Removing the softmax bottleneck
An E4M3 byte is already close to a logarithm of the value it stores. Read as an integer, it equals roughly 8 log2(v) + 56. ExpCast-FP8 writes the byte directly from the log-domain score with one fused multiply-add. The constant β = -0.35 centers the leftover error, and no constant is fitted per model.
The direct path writes the same byte as the FP32 exponent-then-cast path on 79.6% of each doubling. Elsewhere it lands one code away. The paper proves a per-row total variation bound under 3.64%, plus any underflow tail. Across 204,800 Wan2.2 attention rows, the measured average is 1.6%. ExpCast-FP8 applies only to the 8-bit kernel, since NVFP4 has no single affine log-to-code map.
Hand-written CuTe/CUDA fusion of the preprocessing chain cuts one V-Smooth call from 42.2 ms to 4.8 ms on B200.
Explainer: How VC-Attention works
Benchmarks
Tests cover four open-weight video DiTs: Wan2.2-T2V-A14B, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3. Fidelity is scored against BF16 FlashAttention-4 outputs over 100 prompts.
| GPU (Wan2.2) | Precision | Attention speedup | End-to-end speedup |
|---|---|---|---|
| B200 | 8-bit | 1.59× | 1.19× |
| H200 | 8-bit | 1.46× | 1.13× |
| RTX PRO 6000 | 4-bit | 2.27× | 1.36× |
| RTX 5090 | 4-bit | 3.58× | 1.70× |
On B200, VC-Attention is 6.02× faster than SageAttention2, which ships no Blackwell kernel. On H200, the gap is 1.16×. On workstation cards, 4-bit V-Smooth matches SageAttention3 on the RTX PRO 6000. It stays within 5% on the RTX 5090, so fidelity separates them.
Fidelity results:
- At 8 bits, V-Smooth adds 2.3 dB PSNR over SageAttention2 on Wan2.2 and 2.8 dB on HunyuanVideo-1.5.
- Adding ExpCast-FP8 gives back 0.7 to 2.1 dB but still beats SageAttention2 on all four models.
- At 4 bits, V-Smooth beats SageAttention3 by 2.9 dB on Wan2.2 and 3.6 dB on LongCat-Video.
- Run training-free, Attn-QAT falls 3.4 to 6.7 dB below SageAttention2.
On MiniMax-H3 at 1344×768, attention runs 1.60× faster than BF16 FlashAttention-4 on B200. PSNR is 20.2 dB versus 19.9 dB for SageAttention2. On B300, the paper reports 1.47× versus 1.31× for a naive FP8 kernel. The blog chart lists 1.51× for B300.
Nunchux Attention, the company’s proprietary extension, reaches 1.91× on B200 and 1.83× on B300 for MiniMax-H3 attention.
The method changes only per-interaction cost. So it can compose with sparse attention like Sparse VideoGen and Radial Attention, and distillation and multi-GPU execution. Nunchux says free MiniMax-H3 access is coming through its Modelverse waitlist.
What it means
For people generating video, this change targets the specific hardware bottlenecks that slow down rendering on high-end GPUs. By handling quantization errors more effectively and replacing expensive softmax steps with simpler arithmetic, the system reduces the time spent waiting for calculations to complete. The results show that users can achieve faster generation speeds on consumer and professional cards without sacrificing the quality of the output.
