DeepSeek has released DeepInfer, the open-source inference framework powering its DeepSeek-V4.1-Flash model. The code is available under the Apache 2.0 licence. A separate MIT-licensed project, deepseek-recipe, handles token encoding and decoding. This release provides developers with a rare look at the serving stack used by a frontier AI lab.
Key details
- The framework currently supports only DeepSeek-V4.1-Flash, which has a 1M-token context window.
- It runs on NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs. Benchmarks tested the system using 4 GPUs for prefill and 64 for decode.
- DeepSeek claims a single cluster processes 155 trillion tokens daily.
- Performance peaks at over 25,000 output tokens per second per GPU with 256 requests per rank.
- Limitations include offline inference only; the system lacks a KV cache, P/D disaggregation, and load balancing.
What is DeepInfer?
DeepInfer is an inference engine built by DeepSeek for its own models. The GitHub README states it powers large-scale production services. This release covers offline inference with the core features required at scale. The engine was originally built on vLLM, incorporating ideas from SGLang and TensorRT-LLM.
A deployment Q&A document provides production context. A single cluster serves 155 trillion tokens per day. DeepSeek reports overall revenue of US$38 per GPU-hour at valley pricing.
How does DeepInfer work?
The engine combines DeepSeek’s custom kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. Five system-level ideas sit on top of these.
Parallelism: Attention DP plus Wide-EP
Attention uses data parallelism. MoE layers use expert parallelism with an expert-parallel load balancer (EPLB). Decode runs EP64 to EP128 inside a supernode. Prefill uses the smallest expert parallelism that memory allows, typically EP2 to EP8.
DSpark: confidence-scheduled speculative decoding
A draft model proposes five tokens per step. A Confidence Head predicts each draft token’s acceptance rate. The scheduler then verifies only the most valuable prefix. According to the Q&A, this cuts verified draft tokens by about 30%, moving the figure from 5.00 to 3.2 to 4.4. Accepted length barely moves, staying between 3.29 and 3.20.
SWA bounded replay
On a cache hit, DeepInfer recomputes only the last 128 tokens of sliding-window attention state. This makes recomputation cost constant, regardless of context length. In the decoder layers, DeepSeek says this nearly halves total prefill compute.
Asynchronous scheduling, determinism and structured output
CPU batch scheduling overlaps with GPU execution, so the GPU never idles between steps. Batch-invariant inference and deterministic sampling are available on NVIDIA hardware only. Structured outputs support regex, JSON schema and EBNF grammars.
How fast is DeepInfer?
DeepSeek publishes Pareto curves for DeepSeek-V4.1-Flash at 131,072 context. Prefill ran on 4 SM100 GPUs; decode ran on 64. Figures below are read from those charts and are self-reported.
- Decode (EP64DP64, nextn=3, 75% acceptance): Above 25,000 tokens/s/GPU at 256 requests per rank, near 100 tokens/s per user. At 8 requests, per-user speed passes 300 tokens/s.
- Prefill (EP4DP4, 140-token bounded replay): About 60,000 non-cached tokens/s/GPU at 4,096 tokens per rank. Throughput nears 90,000 at 65,536 tokens, with roughly 730 ms per step.
- Wide-EP payoff: An EP8 vs EP64 chart shows EP64 ahead at every batch size.
DSpark’s dynamic verification adds 4% to 7% throughput at about 60 sequences per rank. At 16 sequences per rank it is 1.5% slower than fixed 5-token verification.
How does DeepInfer compare with vLLM, SGLang and TensorRT-LLM?
The table below compares DeepInfer with other major inference frameworks.
| Feature | DeepInfer | vLLM | SGLang | TensorRT-LLM |
|---|---|---|---|---|
| Maintainer | DeepSeek | vLLM project | SGLang project | NVIDIA |
| License | Apache 2.0 | Apache 2.0 | Apache 2.0 | Apache 2.0 |
| Model coverage | 1 (DeepSeek-V4.1-Flash) | 200+ architectures | LLMs, VLMs, diffusion | Support matrix |
| Hardware | NVIDIA SM100, Huawei Ascend 950 | NVIDIA, AMD, Intel, CPUs, plugins incl. Ascend, TPU | NVIDIA, AMD, TPU, Intel, Apple, Ascend, Moore Threads | NVIDIA GPUs |
| Expert parallelism | Yes, with EPLB | Yes | Yes | Wide-EP |
| Speculative decoding | DSpark, confidence-scheduled | n-gram, suffix, EAGLE, DFlash | Yes, via SpecForge drafts | Yes |
| P/D disaggregation | Not in this release | Yes (prefill, decode, encode) | Via llm-d, Dynamo and others | Yes |
| Online serving | Offline inference only | OpenAI-compatible server | Yes | Yes |
What are DeepInfer’s limitations?
A full service still needs a KV cache system and VE/Prefill/Decode disaggregation. It also needs load balancing across data-parallel replicas. The DeepSeek team states those components are tightly coupled to internal infrastructure.
Hardware support is narrow. Builds require Linux, GCC 13 or later, Python 3.12, and CUDA 13 or CANN 9.2. The smallest efficient production unit is 64 prefill GPUs plus 64 decode GPUs. The internal FlashMLA decoding kernel is 5% to 10% faster than the public one. End-to-end, DeepSeek puts the gap at 5% or less.
What it means
For teams running DeepSeek-V4.1-Flash, this release offers a direct path to high-performance deployment without relying on the company’s internal cloud. The code is ready to build on supported hardware, though users cannot expect the full production experience of a managed service. The missing KV cache and load balancer mean this is strictly for offline workloads or environments where those systems are already in place.
