NVIDIA has published a technical guide showing how to boost inference speed for the Ling-3.0-flash model on a single DGX Spark workstation. The team achieved a 42.5% increase in decode throughput on the GSM8K benchmark by combining three specific optimisations: a new MoE kernel backend, an FP8 output layer, and built-in speculative decoding.
Why this matters for developers
The DGX Spark is a desktop-class system, not a data centre cluster. This means the performance gains apply to local boxes used for single-user requests. The report also breaks down where time is spent during MoE decoding and shows exactly how to reduce it.
TL;DR
- Model size: Ling-3.0-flash-fp4 is 124B parameters total, with 5.1B active per token. It supports a native context of 256K.
- Hardware: Runs on one NVIDIA DGX Spark (GB10, 128 GB unified memory, 273 GB/s bandwidth) with a batch size of one.
- Performance jump: The three optimisations lift GSM8K decode throughput from 37.81 to 53.89 tokens per second.
- Results: Throughput rises 42.5% and mean latency drops 24.5%, with accuracy remaining flat at 96.89% versus 96.82%.
- Synthetic vs real: Gains of 185.1% seen in synthetic tests shrink on GSM8K as MTP acceptance falls from 3.90 to 2.68.
Why is MoE decoding on the DGX Spark memory-bound?
The DGX Spark sits on the GB10 Grace Blackwell Superchip. It provides 128 GB of unified memory and 273 GB/s of memory bandwidth. Peak compute reaches up to 1 petaFLOP of sparse FP4. NVIDIA rates the system for inference on models up to 200B parameters.
The NVIDIA team profiled Ling-3.0-flash-fp4 at batch size 1. Decoding proved memory-bound, with reading model weights driving execution time. Two operations stood out: MoE computation and the LM head that scores the vocabulary. This points to two levers: move fewer bytes per decode step, and generate more tokens per verification round.
The baseline followed the official W4A8 recipe. It used FlashInfer MXFP4 MoE kernels, CUTLASS dense GEMM and an FP32 LM-head path. On a synthetic 32K-input, 2K-output test, it ran at 27.71 ms per token, equating to 36.09 tokens per second.
What are the three optimisations?
1. Humming MXFP4 MoE kernels: NVIDIA replaced FlashInfer’s MoE backend with Humming, an independent kernel library. Humming uses NVRTC to compile kernels at runtime for the actual matrix shapes, data types and GPU. This includes the narrow 768-wide expert layer. Each kernel compiles in seconds. The FlashInfer CUTLASS path compiles 97 targets with ninja, taking tens of minutes. Throughput rose 9.3% to 39.45 tokens per second.
2. Online FP8 LM head: The LM head is a 157,184 x 2,560 matrix stored in BF16. Every generated token reads all of it. Quantizing only this layer to FP8 at load time halves those reads. Setting SGLANG_ENABLE_FP8_LM_HEAD=1 enables it. It cannot be combined with the baseline –enable-fp32-lm-head option. Combined with Humming, throughput reached 43.94 tokens per second. That is 21.7% over baseline and 11.4% over Humming alone.
3. MTP speculative decoding with NEXTN: Ling-3.0-flash ships with a built-in NEXTN prediction layer, so no extra draft model is needed. It drafts three tokens, and the model verifies four positions at once. Verification costs about 2.26x a normal decode step. The main cost is MoE, not attention. Verifying four tokens can activate up to 32 experts instead of 8. On the synthetic test, an average of 3.90 tokens were accepted per round. Throughput jumped to 102.88 tokens per second, up 185.1%.
The three techniques do not overlap. Humming and the FP8 LM head cut the cost of each step. MTP cuts the number of steps.
The integration trap: Stacking these is not automatic. With FP8 LM head and MTP both on, the draft must share the full LM-head module. Otherwise the framework silently falls back, and the draft interprets FP8 weights as BF16. Accept length drops to 1.00, and throughput falls below the unoptimized baseline. The NVIDIA team advises checking at startup that each optimisation is active. NVIDIA points to the Ling cookbook notebook for enabling both together.
What happens on a real workload?
Synthetic tests flatten speculative decoding, so the NVIDIA team also ran all 1,319 GSM8K questions. It used 5-shot prompting with chain-of-thought, temperature 0.6, top_p 0.95 and top_k 29. With all three optimisations, time per output token fell from 26.45 ms to 18.56 ms. Decode throughput rose 42.5%, from 37.81 to 53.89 tokens per second. Mean latency dropped 24.5%, from 15.21 to 11.49 seconds. Accuracy moved from 96.89% to 96.82%, a change NVIDIA calls statistically undetectable.
Accept length here was 2.68, versus 3.90 on synthetic data. The NVIDIA team gives three reasons for the gap. Synthetic outputs are more repetitive, so drafts are accepted more often. The synthetic test used greedy decoding, while GSM8K used sampling. GSM8K answers are also short, with a median near 310 tokens versus a fixed 2K.
What is Ling 3.0 and why does it suit the DGX Spark?
Ling-3.0-flash belongs to Ant Group inclusionAI’s Ling 3.0 family, released under the MIT license. The open-weight lineup also includes Ling-3.0-tiny, a 7.9B MoE with 1.3B active parameters. Both come with base checkpoints and FP8, INT4 and GGUF builds; flash also has FP4. OpenRouter lists inclusionAI’s vision-language, finance and health variants built on Ling 3.0 Flash.
Ling-3.0-flash is built to keep per-token cost low:
- Sparse MoE: Each MoE layer routes a token to 8 of 512 routed experts, along with 1 shared expert. About 4.1% of the weights are active per token.
- Hybrid linear attention: 35 Kimi Delta Attention (KDA) layers and 7 Gated MLA layers alternate 5:1. KDA, from Moonshot AI’s Kimi Linear paper, keeps a fixed-size state instead of a growing KV cache. The layout is native from the start of pretraining.
- Built-in MTP: The model was trained with multi-token prediction. That NEXTN layer is the draft model NVIDIA used.
- Agent focus: inclusionAI team reports 10,000+ interactive training environments for coding, general and deep-research agents. NVIDIA notes the native 256K context expands to 1M tokens.
How strong and efficient is Ling-3.0-flash?
On its model card, inclusionAI reports 56.6 on SWE-Bench Pro and 72.4 on SWE-Bench Multilingual. It also reports 93.2 on MathArena AIME 2026 and 87.0 on HMMT Feb 2026. Artificial Analysis results shown on OpenRouter list 85.5% on GPQA Diamond and 55.4% on Terminal-Bench 2.1. The hardest reasoning tests remain tough: Humanity’s Last Exam scores 22.7 self-reported and 23.7% independently.
API pricing: Artificial Analysis lists InclusionAI’s API at $0.075 per 1M input tokens and $0.22 per 1M output tokens. That blends to about $0.05 per 1M tokens, using its 7:2:1 cache-hit, input and output mix. It measured output speed at 231.7 tokens per second, third of 26 models in its class. On OpenRouter, Novita and DeepInfra list $0.06 input and $0.18 output.
Against peers: Artificial Analysis lists higher blended prices for comparable models. Step 3.7 Flash is $0.18, Gemini 3.5 Flash-Lite $0.33 and GPT-5 (medium) $1.34. These figures compare price only; they are not quality-adjusted.
Quantization costs little quality: inclusionAI’s quantization table compares BF16, FP8, INT4 and FP4. FP4 scores 83.68 on GPQA-diamond versus 84.97 in BF16. IFBench moves from 74.49 to 73.87, and ArcPrize from 68.75 to 68.06. SciCode shows the largest drop, from 41.24 to 38.43. The FP4 checkpoint is the one NVIDIA ran on DGX Spark.
What it means
Developers with a DGX Spark can now run this 124B parameter model faster locally. The gains come from reducing memory traffic and using the model’s own prediction capabilities. Accuracy remains stable, and the setup requires careful configuration to ensure all optimisations work together.
