**What Happened:**
A new paper titled “Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion” was posted on Reddit by user Franck_Dernoncourt. The paper describes a method where a trainable diffusion attention module is injected into each layer of an AR Transformer model, which has been frozen. Both heads share one key-value (KV) cache for parallel token generation. This approach results in up to 7.8× throughput improvements and matches the accuracy of base models like Qwen3-8B.
**Why It Matters:**
This work is significant because it demonstrates a novel method that achieves high efficiency without compromising on performance or requiring additional training. By freezing the backbone model, Orthrus maintains all the benefits of the original architecture while introducing new mechanisms for parallel token generation via diffusion and verification heads. The key findings include up to 7.8× throughput improvements over existing methods like speculative decoding (EAGLE-3 and DFlash) without any initialization overhead or TTFT penalties.
**Takeaways:**
1. **High Efficiency**: Orthrus achieves remarkable performance gains in parallel token generation, with up to 7.8× throughput improvement.
2. **Model Stability**: By freezing the underlying model, Orthrus inherits its strengths while adding new mechanisms for efficient parallelization.
3. **Versatility**: The approach can be applied to various downstream tasks and models without requiring modifications to existing architectures.
Originally published at reddit.com. Curated by AI Maestro.
Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise.