Zyphra has released ZYFA, an open source library of FP8 and BF16 attention kernels for AMD Instinct MI355X GPUs. The code is available under Apache 2.0 and targets long-context prefill operations. On real model activations, the FP8 kernels run 1.16 to 1.76 times faster than AITER, AMD’s default kernel library. At a head dimension of 128, ZYFA’s FP8 error is 62 to 78 per cent lower than AITER’s.
TL;DR
- The tool is a kernel library supporting causal forward attention in FP8 e4m3 and BF16 formats. It handles head dimensions of 128 and MLA shapes of 192/128.
- It runs on AMD Instinct MI355X hardware (CDNA4, gfx950) with ROCm 7.2.3 and PyTorch 2.12. The interface is a drop-in replacement for flash_attn_func.
- Performance gains on real activations range from 1.16 to 1.76 times faster than AITER in FP8. BF16 performance is roughly equal to the baseline, ranging between 1.01 and 1.02 times.
- At 32K tokens, the kernel achieves 2,672 TFLOP/s in FP8. This represents 53 per cent of the FP8 datasheet peak, compared to AITER’s 39 per cent.
- On Mistral-7B activations, the speedup is only 1.16 times. The FP8 clock drops to 1,628 MHz versus AITER’s 2,108 MHz.
What is ZYFA
ZYFA is distributed via a technical paper and accompanying code from Zyphra. It powers the attention engine within the Zyphra Optimization Engine, which runs Zyphra Cloud. The BF16 kernels are written in HIP, while the FP8 kernels consist of hand-scheduled assembly emitted by a Python generator.
Why does FP8 attention underperform on MI355X
FP8 matrix instructions run at twice the peak speed of BF16 instructions. However, FP8 only accelerates the two matrix products, QKᵀ and PV. The softmax operation between them runs on the vector unit at the same speed in both precisions.
For 32 queries against 128 keys, FP8 matrix work takes 1,024 cycles. The exact FP8 softmax takes 896 vector cycles. The kernel requires near-perfect overlap to keep matrix units busy. Zyphra’s thread traces show AITER’s matrix pipe idle for 31.7 per cent of cycles.
The MI355X does not track matrix instruction completion in hardware, so instruction order must create that overlap.
How does ZYFA work
ZYFA uses three techniques to improve efficiency.
Lean P: one multiply-add instead of an exponential
An e4m3 bit pattern, read as an integer, grows by eight each time the value doubles. The code for 2^x is roughly eight times the value plus a constant. Lean P writes each FP8 probability with one multiply-add and one integer conversion. This cuts softmax vector cycles per block from 896 to 384.
Two-wave ping-pong schedule
Two waves share each SIMD unit. One issues matrix instructions while the other computes probabilities, then they swap. With lean P, the matrix pipe is busy 86 per cent of steady-state cycles, up from 60 per cent with exact P.
Per-block scaling and a rounded row sum
Each 64-key block of probabilities is scaled by a power of two before rounding. CDNA4’s scaled matrix instruction removes the factor at no cost. ZYFA also divides by the sum of the rounded probabilities, computed with one extra matrix instruction per block.
How fast is ZYFA
Real-activation FP8 speedups at 16K and sustained clocks are 1.16 times for Mistral-7B, 1.17 times for Qwen3-8B, and 1.76 times for DeepSeek-V2-Lite. On Gaussian benchmarks at 32K, the head-dim-128 FP8 kernel reaches 2,672 TFLOP/s against AITER’s 1,944. At the MLA shape it reaches 2,464 against 1,237, a 1.99 times ratio.
On captured Mistral-7B and Qwen3-8B layers, ZYFA delivers 1.94 and 1.91 TFLOP/J. AITER delivers 1.60 and 1.56.
Why are real-model gains smaller than benchmark gains
The MI355X clock depends on how many operand bits flip. AITER allows more small probabilities to become zeros or subnormals. Those switch fewer bits, so its clock runs higher. On the Mistral-7B layer, ZYFA runs at 1,628 MHz while AITER runs at 2,108 MHz. ZYFA still needs fewer cycles overall. The Zyphra team recommends benchmarking on real activations at sustained clocks.
How accurate is ZYFA’s FP8 attention
Against fp64 attention, ZYFA’s FP8 error is 0.22 times AITER’s on Mistral-7B and 0.38 times on Qwen3-8B. It sits within a few per cent of Composable Kernel and Triton. In full Mistral-7B, KL divergence from fp64 is 3.4×10⁻³ with ZYFA and 5.9×10⁻³ with AITER. ZYFA’s outputs were also bitwise identical across 7,200 launches.
Most full-model FP8 error comes from quantizing Q, K and V. A Hadamard rotation of Q and K roughly halves that input error. On Mistral-7B it adds about 0.1 per cent to attention time.
How does ZYFA compare with other attention kernels
| Feature | ZYFA FP8 | AITER FP8 | FlashAttention-3 | FlashAttention-4 |
|---|---|---|---|---|
| Maker | Zyphra | AMD | Dao-AILab et al. | Dao-AILab et al. |
| Target GPU | AMD MI355X | AMD MI300X to MI355X | NVIDIA H100/H800 | NVIDIA B200 (plus Hopper) |
| Headline throughput | 2,672 TFLOP/s FP8 | 1,944 TFLOP/s FP8 (Zyphra-measured) | Close to 1.2 PFLOP/s FP8 | 1,613 TFLOP/s BF16 |
| Share of peak | 53% FP8 | 39% FP8 (Zyphra-measured) | Not disclosed (FP8) | 71% BF16 |
| Softmax approach | Lean P, no exponential | Hardware exponential | Overlap via async Tensor Cores | Software-emulated exponential |
| FP8 accuracy claim | 62-78% lower error than AITER | Baseline | 2.6x lower error than baseline FP8 | Not disclosed |
| Scope | Causal prefill forward only | Prefill, decode, MLA, paged | Forward and backward, FP8 forward | Forward and backward |
| License | Apache 2.0 | MIT | BSD-3-Clause | BSD-3-Clause |
Cross-vendor numbers come from different inputs and clocks. Zyphra did not measure NVIDIA GPUs.
What it means
The improvements are specific to the MI355X GPU and the prefill stage of a model. Users of other hardware or those relying on decoding performance will see no change. The library is open source for developers who need to optimise long-context generation on this specific chip.