NVIDIA AI has released Nemotron-Labs-3-Puzzle-75B-A9B, a compressed version of the Nemotron-3-Super model designed to increase server throughput. The parent model contains 120.7B total parameters with 12.8B active, while the new variant reduces this to 75.3B total and 9.3B active parameters. The team targeted a 2x increase in server throughput at 100 tokens per second per user and the ability to handle eight concurrent 1M-token requests on a single H100 GPU.
The architecture
The original model is a hybrid Mamba-Transformer mixture of experts. The compressed version keeps the 88-block layout identical: 40 Mamba blocks, 40 MoE blocks, and 8 attention blocks. Changes occur inside the blocks rather than the structure.
| Quantity | Super | Puzzle-75B-A9B | Ratio |
|---|---|---|---|
| Total parameters | 120.7B | 75.3B | 62.4% |
| Active parameters | 12.8B | 9.3B | 73.1% |
| Mamba SSM state size | 128 | 96 | 75% |
| MoE routed expert intermediate size | 2688 | 1280-2688 | Mean 59.9% |
| Activated routed experts per token | 22 | 4-18 | Mean 50% |
| Active routed expert capacity (relative) | 100% | 8.7%-62.3% | Mean 30.9% |
The number of routed experts, the shared expert size, and the MoE latent size remain the same. Attention layers were left untouched. The authors note the parent model is already very efficient with its KV cache. Mamba layers were pruned uniformly because inference frameworks do not support different SSM state sizes per layer. The compression is not a uniform scaling. Capacity was preserved in selected middle and late layers while cuts were made elsewhere.
Performance results
Tests were run on an 8xB200 node using single-step decoding. Results are shown in tokens per second.
| Scenario (in/out) | UT floor | Super (tok/s) | Puzzle-75B-A9B (tok/s) | Boost |
|---|---|---|---|---|
| 50K / 2K | >= 100 | 5,128 | 8,210 | 1.60x |
| 50K / 2K | >= 125 | 3,784 | 6,412 | 1.69x |
| 50K / 2K | >= 150 | 2,532 | 4,523 | 1.79x |
| 8K / 64K | >= 100 | 20,939 | 42,601 | 2.03x |
| 8K / 64K | >= 125 | 13,074 | 27,918 | 2.14x |
| 8K / 64K | >= 150 | 8,522 | 18,047 | 2.12x |
Both models were served with matched NVFP4 weights, FP8 KV cache, and FP16 Mamba state. The gap reflects compression rather than a change in numeric format. The prefill-heavy 50K/2K regime gains the least. The decode-heavy 8K/64K regime gains the most.
On a single 8xH100 node at a user throughput of 100, the gains are smaller. They are 1.91x on 50K/2K and 1.82x on 8K/64K. Both models use FP8 weights, FP8 KV cache, and FP32 Mamba state in this scenario.
On a single H100 at 1M context, the binding constraint flips from compute to memory. Super’s NVFP4 weights occupy about 70 GB of the 80 GB HBM budget. Each 1M-token request adds about 4 GB of KV cache. Effective concurrency is therefore 1.
Puzzle-75B-A9B’s NVFP4 weights occupy about 44.5 GB. Attention layout is unchanged, so per-request KV cost is unchanged. Concurrency at 1M rises to 8. Aggregate decode throughput at that concurrency is roughly 4x Super’s single-request throughput. Prefill of a 990K-token prompt is about 1.2x faster.
How the compression works
Puzzle is a decomposed neural architecture search framework implemented as Puzzletron. It defines a discrete search space of alternative layer implementations. Each alternative gets a quality score. A mixed-integer program selects one alternative per layer under a deployment constraint.
Three pruning techniques form the search space:
- Intermediate channel pruning: Channels inside each routed expert are ranked by contribution to the expert’s output. All experts within one MoE layer are pruned to a uniform size, for kernel compatibility.
- Top-k reduction: The number of experts a token is routed to varies per layer, up to the parent’s k=22.
- Mamba SSM pruning: The SSM state size drops from 128 to 96 channels.
The SSM result is measured. Dropping 128 channels to 96 speeds the SSM kernel 1.2x to 1.3x during decode. This holds at batch sizes between 8 and 512. Channels were ranked by estimated contribution to the Mamba layer output. The estimate averaged over 67M tokens of validation data. Appendix A shows this beats random channel selection under aggressive pruning.
The original formulation assumes replacement quality impacts are approximately additive. Each candidate block is scored inside the unmodified parent. That ignores higher-order interactions between replacements.
Iterative Puzzle alternates bounded compression with short knowledge distillation recovery. It builds a sequence M0, M1, … MR instead of jumping to the target. Scores are recomputed against the current compressed model, not the original parent.
Three stages were used:
- MoE weights to 75% of teacher capacity, Mamba SSM state to 75%. Healed for 24B tokens.
- MoE weights to 60% of teacher capacity. Healed for 43.2B tokens.
- Activated routed-expert budget to 50%, allocated heterogeneously. Healed for 52.8B tokens.
| Model | Average score |
|---|---|
| Single-step Puzzle | 68.48 |
| Iterative Puzzle | 69.05 |
The table compares this against a single-step Puzzle baseline at the same target. The three-step procedure averages 69.05 across ten benchmarks, against 68.48. Gains appear on MMLU-Pro, GPQA, HLE, AA-LCR, LiveCodeBench, SciCode, and RULER-256K. IFBench-Instruction fell 0.2 points and IFBench-Prompt fell 0.5.
Recovery methods
Knowledge distillation ran on 30% pretraining data and 70% SFT data from Nemotron-3-Nano. During the Puzzle phase, KD used a 32K sequence length. Recovery then trained at 128K, and scaled to 512K. The budget was up to 100B tokens, with a 16M-token global batch, in Megatron-LM.
RL post-training adopted Stage 2 of the Nemotron-3-Super RL pipeline, focused on software engineering. Phase 2.1 did single-step tool-use comparison. Phase 2.2 moved to end-to-end sandbox RL, where agents run up to 200 turns. Both phases used a KL penalty of 0. The team swept learning rates, then averaged the resulting weights.
Short-context KD recovers most categories to over 97% of Nemotron-3-Super. Long-context KD then lifts long-input and long-generation benchmarks specifically. The research team states that RL’s impact in these experiments was small.
Verbosity is the quiet detail. After the last Puzzle iteration, the model generated 132% of Super’s token count. That
