Stanford and UC Berkeley researchers have released TokenGate, a module designed to speed up the initial processing of long text inputs for large language models. The system uses a learned mechanism to skip irrelevant parts of the input, reporting a 1.4x to 11.3x reduction in the time to generate the first token without losing accuracy. Peak memory usage for prefill drops by 1.5x to 3.4x. The code and model checkpoints are available on GitHub under an Apache-2.0 license.
Quick facts
- Architecture: A small MLP gate and LoRA (r=16) integrated into Qwen3-1.7B or Qwen3.5-2B. Testing covered contexts up to 128k tokens.
- Hardware: Benchmarks run on a single A100 with a batch size of 1. Training required four A100 GPUs.
- Performance: The method proved Pareto-optimal across five tasks when compared to six baselines.
- Top result: On the NIAH 16k task, the system achieved an 11.3x speedup in time-to-first-token and reduced memory usage by 3.42x. Gains reached 13.0x at 128k context length.
- Lowest result: Speedup fell to 1.4x on specific tasks.
What is TokenGate
TokenGate functions as a drop-in component that adds trainable pruning to existing pretrained models. Standard transformers route every token through every layer, causing attention costs to rise quadratically with length. Previous pruning methods that do not require retraining are cheap but degrade accuracy when pruning rates are high. Structural changes to the model often necessitate training from scratch.
How the pruning mechanism works
The forward pass divides into three distinct phases:
- Standard forward: Layers below the gate layer process all N tokens normally.
- Gate module: An MLP scores each token based on its post-attention residual at layer L. A top-K selector retains only the highest-scoring tokens.
- Pruned forward: Layers above L process only those K tokens and never create key-value entries for the discarded ones.
Only the document portion of the prompt is eligible for pruning. The question is never removed.
Token-gated attention explained
Direct top-K selection is not differentiable, which prevents end-to-end training. TokenGate adapts the Mixture-of-Experts router technique. Each kept token’s score becomes a bias on attention logits above the gate. This bias provides a gradient path back to the scorer.
A learnable per-head gain and a learnable temperature refine this process. The temperature adjustment accounts for each query seeing fewer keys. Training uses standard next-token loss. Each step averages a full-context loss and a random-budget loss. Consequently, a single trained gate functions across different budget settings.
Benchmark performance
The team tested Qwen3-1.7B on RepoBench-P, 2WikiMultihopQA, MultiNews, BioDEX, and needle-in-a-haystack (NIAH). Baselines included FastKV, SlimInfer, DASH, GemFilter, FlexPrefill, and MInference, all running on the same LoRA backbone.
- Against full prefill: 1.4x to 11.3x faster time-to-first-token and 1.5x to 3.4x lower memory.
- Against the next-best baseline: 1.2x to 6.4x faster time-to-first-token and up to 2.1x lower memory.
- 2WikiMultihopQA: At 95% of full-context accuracy, the prefill runs 3.0x faster with 2.7x less memory.
- Longer contexts: A gate trained at 16k reaches 12.8x speedup at 32k, 13.0x at 64k, and 128k. Memory savings remain near 3.43x.
The learned selector outperforms FastKV’s heuristic at the same layer. Even random-K selection scores higher on the TokenGate backbone. The authors attribute this to backbone co-adaptation.
Hybrid attention compatibility
Yes, Qwen3.5-2B mixes Gated DeltaNet and full-attention layers. On NIAH 32k, TokenGate at layer 3 matched full-prefill accuracy. It ran 3.8x faster than full prefill and 2.1x faster than FastKV. Gating at layers 1 or 2 reached 7.8x, within 6% accuracy. FastKV can only prune attention layers.
Comparison with other methods
| Feature | TokenGate | FastKV | SlimInfer | GemFilter | DASH |
|---|---|---|---|---|---|
| Token selection | Learned MLP gate | Attention from last queries | Block attention, multi-stage | Last-query attention at filter layer | Hidden-state stability |
| Training needed | Yes (gate + LoRA r=16) | No | No | No | No |
| Pruning layer in TokenGate’s Qwen3-1.7B runs | 1 | 13 (default) | 8, 17, 25 | 12 | 11 |
| Prunes at Gated DeltaNet layers | Yes | No (attention layers only) | Not disclosed | Not disclosed | Not disclosed |
| Best self-reported prefill gain (own setup) | 11.3x TTFT, 3.4x memory | 1.82x prefill | 2.53x TTFT (LLaMA3.1-8B, RTX 4090) | 2.4x speed, 30% less GPU memory vs SOTA | Not disclosed |
| Code | GitHub | GitHub | GitHub | GitHub | Announced |
Self-reported gains use different setups, so they are not directly comparable. Pruning layers come from TokenGate’s Appendix C.2.
Deployment details
The GitHub repository requires Python 3.10+ and CUDA PyTorch 2.9+. An optional FlashAttention-2 key-bias kernel speeds up dense Qwen3. It costs 1.0x to 1.1x a plain call. Without it, inference falls back to FlexAttention.
A {state} placeholder marks the prunable document. generate takes a max_kept_prefill_tokens budget. Checkpoints cover all five tasks on Qwen3-1.7B, plus a Qwen3.5-2B NIAH gate. Example: liana313/tokengate-niah-qwen3-1.7b.
Training mirrors a TRL/PEFT recipe. Each step is 1.3x slower than plain LoRA. Runs converged faster on four of five tasks. Mean training time was 1.8 versus 2.2 A100 hours.
Summary
TokenGate adds learned token pruning to existing LLMs via a small gate and LoRA. Prefill runs 1.4x to 11.3x faster with 1.5x to 3.4x lower peak memory. Token-gated attention makes top-K selection trainable, using an MoE-router style approach. It beats the next-best pruning baseline by up to 6.4x on TTFT. It works on hybrid Qwen3.5-2B, where FastKV cannot prune DeltaNet layers.
