NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a compressed version of Nemotron-3-Super designed to increase server throughput by 2.03x while maintaining the same performance per user. The new model reduces the total parameter count from 120.7B to 75.3B and cuts active parameters from 12.8B to 9.3B.
The architecture was built around fixed targets: doubling server throughput at 100 tokens per second per user and handling eight concurrent 1M-token requests on a single H100 GPU. Three checkpoints are available for download on Hugging Face in BF16, FP8, and NVFP4 formats.
Model structure and compression
Nemotron-3-Super is a hybrid Mamba-Transformer Mixture of Experts model. The new Puzzle-75B-A9B keeps the parent’s 88-block layout exactly the same, consisting of 40 Mamba blocks, 40 MoE blocks, and 8 attention blocks.
The changes occur inside those blocks. The total parameter count drops by 37.6% and active parameters by 26.9%. Mamba SSM state size falls from 128 to 96. MoE routed expert intermediate sizes vary between 1280 and 2688, averaging 59.9% of the original. Activated routed experts per token average 50% of the parent’s 22. Active routed expert capacity averages 30.9% of the original.
The number of routed experts, shared expert size, and MoE latent size remain unchanged. Attention layers were left untouched. The team states that Nemotron-3-Super is already efficient with its KV cache. Mamba layers were pruned uniformly because inference frameworks do not support different SSM state sizes per layer.
The compression is not a uniform scaling. Capacity was preserved in selected middle and late layers while being reduced elsewhere.
Performance results
The following table reports Pareto-optimal total throughput on a single 8xB200 node using single-step decoding.
| Scenario (in/out) | UT floor | Super (tok/s) | Puzzle-75B-A9B (tok/s) | Boost |
|---|---|---|---|---|
| 50K / 2K | >= 100 | 5,128 | 8,210 | 1.60x |
| 50K / 2K | >= 125 | 3,784 | 6,412 | 1.69x |
| 50K / 2K | >= 150 | 2,532 | 4,523 | 1.79x |
| 8K / 64K | >= 100 | 20,939 | 42,601 | 2.03x |
| 8K / 64K | >= 125 | 13,074 | 27,918 | 2.14x |
| 8K / 64K | >= 150 | 8,522 | 18,047 | 2.12x |
Both models were served at matched NVFP4 weights, FP8 KV cache, and FP16 Mamba state. The gap reflects compression rather than a change in numeric format. The prefill-heavy 50K/2K regime gains the least. The decode-heavy 8K/64K regime gains the most.
On a single 8xH100 node at a user throughput of 100, the gains are smaller. They are 1.91x on 50K/2K and 1.82x on 8K/64K. Both models use FP8 weights, FP8 KV cache, and FP32 Mamba state in this configuration.
On a single H100 at 1M context, the binding constraint shifts from compute to memory. Super’s NVFP4 weights occupy about 70 GB of the 80 GB HBM budget. Each 1M-token request adds about 4 GB of KV cache. Effective concurrency is therefore 1.
Puzzle-75B-A9B’s NVFP4 weights occupy about 44.5 GB. The attention layout is unchanged, so per-request KV cost remains the same. Concurrency at 1M rises to 8. Aggregate decode throughput at that concurrency is roughly 4x Super’s single-request throughput. Prefilling a 990K-token prompt is about 1.2x faster.
How Iterative Puzzle works
Puzzle is a decomposed neural architecture search framework implemented as Puzzletron. It defines a discrete search space of alternative layer implementations. Each alternative receives a quality score. A mixed-integer program then selects one alternative per layer under a deployment constraint.
Three pruning techniques form the search space:
- Intermediate channel pruning: Channels inside each routed expert are ranked by contribution to the expert’s output. All experts within one MoE layer are pruned to a uniform size for kernel compatibility.
- Top-k reduction: The number of experts a token is routed to varies per layer, up to the parent’s k=22.
- Mamba SSM pruning: The SSM state size drops from 128 to 96 channels.
The SSM result is measured. Dropping 128 channels to 96 speeds the SSM kernel 1.2x to 1.3x during decode. This holds at batch sizes between 8 and 512. Channels were ranked by estimated contribution to the Mamba layer output. The estimate averaged over 67M tokens of validation data. Appendix A shows this beats random channel selection under aggressive pruning.
The original formulation assumes replacement quality impacts are approximately additive. Each candidate block is scored inside the unmodified parent. That ignores higher-order interactions between replacements.
Iterative Puzzle alternates bounded compression with short knowledge distillation recovery. It builds a sequence M0, M1, … MR instead of jumping to the target. Scores are recomputed against the current compressed model, not the original parent.
Three stages were used:
- MoE weights to 75% of teacher capacity, Mamba SSM state to 75%. Healed for 24B tokens.
- MoE weights to 60% of teacher capacity. Healed for 43.2B tokens.
- Activated routed-expert budget to 50%, allocated heterogeneously. Healed for 52.8B tokens.
The table above compares this against a single-step Puzzle baseline at the same target. The three-step procedure averages 69.05 across ten benchmarks, against 68.48. Gains appear on MMLU-Pro, GPQA, HLE, AA-LCR, LiveCodeBench, SciCode, and RULER-256K. IFBench-Instruction fell 0.2 points and IFBench-Prompt fell 0.5.
Recovery: Distillation, RL, and Verbosity
Knowledge distillation ran on 30% pretraining data and 70% SFT data from Nemotron-3-Nano. During the Puzzle phase, KD used a 32K sequence length. Recovery then trained at 128K, and scaled to 512K. The budget was up to 100B tokens, with a 16M-token global batch, in Megatron-LM.
RL post-training adopted Stage 2 of the Nemotron-3-Super RL pipeline, focused on software engineering. Phase 2.1 did single-step tool-use comparison. Phase 2.2 moved to end-to-end sandbox RL, where agents run up to 200 turns. Both phases used a KL penalty of 0. The team swept learning rates, then averaged the resulting weights.
The figure above shows what each stage contributed. Short-context KD recovers most categories to over 97% of Nemotron-3-Super. Long-context KD then lifts long-input and long-generation benchmarks specifically. The research team states that RL’s impact in these experiments was small.
Verbosity is the quiet detail. After the last Puzzle iteration, the model generated 132% of Super’s token count. That fell to 99% after the full recovery pipeline.
Deployment: Quantization and Multi-Token Prediction
Two post-training quantization recipes were produced: FP8 W8A8 targets Hopper and NVFP4 W4A4 targets Blackwell.
| Component | BF16 baseline | FP8 checkpoint | NVFP4 checkpoint |
|---|---|---|---|
| Sparse and shared MoE GEMMs | BF16 | FP8 | NVFP4 |
| Mamba GEMMs | BF16 | FP8 | FP8 |
| Mamba SSM cache | FP32 | FP32 | FP16+SR |
| KV cache | FP8 | FP8 | FP8 | Source Read original →
