Moonshot AI Releases Kimi K3: A 2.8 Trillion Parameter Open MoE Model With Kimi Delta Attention and 1M Context

Moonshot AI has released Kimi K3, an open model containing 2.8 trillion parameters with a native 1-million-token context window. The company describes it as the world’s first open model to reach this scale.

The architecture behind the numbers

K3 is a sparse Mixture-of-Experts system built on two specific updates: Kimi Delta Attention (KDA) and Attention Residuals (AttnRes). Both changes alter how data moves through the model’s depth and sequence length. The team says K3 targets long-horizon coding, knowledge work, and complex reasoning.

Moonshot claims K3 is the first open model to hit 2.8 trillion parameters. For nine of the last twelve months, Kimi models have held the upper bound for open-model sizes.

The company is direct about where K3 sits in the hierarchy. Overall performance still trails the most powerful proprietary models, specifically Claude Fable 5 and GPT 5.6 Sol. Across Moonshot’s own evaluation suite, K3 consistently outperformed other tested models.

How the system works

Kimi Delta Attention (KDA) is a hybrid linear attention mechanism. Moonshot states it enables up to 6.3x faster decoding in million-token contexts.

AttnRes works along the axis of depth. It selectively retrieves representations across depth rather than accumulating them uniformly. Moonshot states AttnRes delivers roughly 25% higher training efficiency at under 2% additional cost.

Sparsity is the third lever. K3 uses Stable LatentMoE, effectively activating 16 of 896 experts. At that sparsity, routing and optimization become first-order challenges. Quantile Balancing derives expert allocation directly from router-score quantiles. That eliminates heuristic updates and a sensitive balancing hyperparameter. Per-Head Muon extends Muon by optimizing attention heads independently. Sigmoid Tanh Unit (SiTU) and Gated MLA improve activation control and attention selectivity respectively.

Refined training and data recipes accompany those structural changes. Together they yield roughly 2.5x better overall scaling efficiency than Kimi K2.

Those choices carry into serving. K3 applies quantization-aware training from the SFT stage onward. It uses MXFP4 weights with MXFP8 activations for broad hardware compatibility. Moonshot team recommends supernode configurations with 64 or more accelerators. Because KDA poses new challenges for prefix caching, Moonshot contributed an implementation to vLLM.

Performance benchmarks

With the mechanics established, the published scores are easier to read. All K3 results use reasoning effort set to max. Harnesses differ per benchmark: KimiCode, Claude Code, or Codex.

Two caveats shape this table. ‘With fallback’ means requests Fable 5 refuses under its usage policy route to Opus 4.8. Also, BrowseComp used context compaction triggered at 300K tokens. Without that context management, K3 scores 90.4.

So K3 leads Program Bench, SWE Marathon, BrowseComp, Automation Bench, and OmniDocBench. It trails Fable 5 on FrontierSWE and HLE-Full, and GPT 5.6 Sol on DeepSWE.

Use cases and examples

Moonshot team states one native multimodal architecture handles text, images, and video together.

Access and a minimal call

K3 is live on Kimi.com, Kimi Work, Kimi Code, and the API. Access runs through the OpenAI SDK against a Moonshot base URL.

from openai import OpenAI
import os

client = OpenAI(api_key=os.environ["MOONSHOT_API_KEY"],
                base_url="https://api.moonshot.ai/v1")

completion = client.chat.completions.create(
    model="kimi-k3",
    reasoning_effort="max",
    messages=[{"role": "user", "content": "Introduce Kimi K3 in one sentence."}],
)
print(completion.choices[0].message.content)