Up to 3.2x Faster Inference with LFM2.5-DSpark
The new LFM2.5-DSpark draft models deliver up to 3.18x throughput gains on a GPU and up to 2.87x on-device. For LFM2.5-2.6B, function-calling latency drops by 57% on average. Day-one support is available for llama.cpp and SGLang via open-sourced upstream integration.
How DSpark works
Inference decoding is traditionally memory-bound. Latency comes mostly from streaming weights from DRAM into SRAM, rather than intense computation. Speculative decoding addresses this by using a lightweight draft model to produce candidate tokens, then having the target model verify them all in a single forward pass. This shares the cost of loading weights across all verified tokens.
Multiple approaches have been proposed, with the most prominent being EAGLE-3, DFlash, and most recently DSpark. DSpark combines three components:
- A DFlash-style parallel backbone conditioned on the target model’s context features, producing hidden states for all draft tokens in a single forward pass.
- A lightweight sequential head, modeled as a Markov chain between neighboring tokens, that adds inter-token dependency to raise the acceptance rate at later positions.
- A confidence-scheduled verifier that predicts each token’s survival probability and prunes low-confidence suffixes when verification would cost more than it saves.
Training and Architecture
We follow the DSpark recipe with a larger and more diverse data mix covering SFT, chat, code, and function-calling data. Based on ablations, the first versions of the draft models are simplified attention-only models with 5 layers and a block of 9. For each draft model, we ran 15 epochs on the entire dataset and selected the epoch with the highest acceptance rate rather than the lowest loss.
The resulting draft models are relatively small, with each around ~300M parameters.
| Component | LFM2.5-1.2B-Instruct | LFM2.5-8B-A1B | LFM2.5-2.6B |
|---|---|---|---|
| Decoder stack (5 layers) | 241.2M | 241.2M | 241.2M |
| Hidden-state projection | 21.0M | 21.0M | 21.0M |
| Markov head | 33.6M | 65.5M | 65.5M |
| Norms + confidence head | 27.5k | 27.5k | 27.5k |
| Total | 295.7M | 327.7M | 327.7M |
Quality parity
Under greedy decoding, a draft token is only accepted if it matches the target model’s distribution. On rejection, the target model’s own token takes its place. The emitted sequence is therefore identical to baseline greedy by construction, so benchmark accuracy (pass@1 or exact match) is unchanged.
Inference Speed Up on CPU and GPU
Our DSpark draft models for LFM2.5 ship with day-one support for llama.cpp and SGLang. The llama.cpp implementation builds on the official codebase, which we run with experimental metal kernels. The SGLang implementation builds on the official SGLang implementation of DSpark.
We measure on-device throughput with llama.cpp and Metal on an M4 Max MacBook Pro using FP16 GGUF weights and up to 256 output tokens. We measure GPU throughput with SGLang on a single H100 80 GB in BF16. Both configurations use a DSpark block size of 9, a batch size of 1, and a temperature of 0. We evaluate them on five benchmark datasets.
All three drafter models deliver noticeable throughput improvements on both the large-scale accelerator (H100) and the edge deployment (M4 Max MacBook).
For LFM2.5-2.6B, speedup on the MacBook is especially noticeable, as it pushes the interactivity level a user can enjoy far beyond the throughput offered by most proprietary cloud models (around ~140 tok/s, depending on the dataset).
| Dataset | Acceptance (of 10) | Speedup on H100 | Speedup on M4 Max |
|---|---|---|---|
| MATH500 | 5.42 | 3.06x 326 → 1000 tok/s | 2.25x 61 → 137 tok/s |
| HumanEval | 4.54 | 2.56x 326 → 835 tok/s | 2.63x 61 → 161 tok/s |
| MBPP | 4.71 | 2.64x 326 → 861 tok/s | 2.11x 62 → 132 tok/s |
| GSM8K | 4.32 | 2.22x 312 → 693 tok/s | 2.36x 60 → 143 tok/s |
| MT-Bench | 5.07 | 2.87x 325 → 933 tok/s | 1.99x 62 → 123 tok/s |
| Mean | 4.81 | 2.67x 323 → 864 tok/s | 2.27x 61 → 139 tok/s |
Across various multi-tool scenarios, DSpark reduces the latency by 57% on average for LFM2.5-2.6B.
For LFM2.5-1.2B-Instruct, we see much more variance in dataset acceptance rates, so speedup varies by as much as 52% depending on the underlying text distribution.
| Dataset | Acceptance (of 10) | Speedup on H100 | Speedup on M4 Max |
|---|---|---|---|
| MATH500 | 6.02 | 2.56x 668 → 1712 tok/s | 2.62x 140 → 366 tok/s |
| HumanEval | 5.31 | 2.26x 664 → 1499 tok/s | 2.87x 136 → 389 tok/s |
| MBPP | 5.52 | 2.37x 667 → 1578 tok/s | 2.74x 137 → 375 tok/s |
| GSM8K | 4.34 | 1.67x 624 → 1041 tok/s | 2.73x 140 → 381 tok/s |
| MT-Bench | 3.90 | 1.66x 657 → 1091 tok/s | 1.72x 137 → 237 tok/s |
| Mean | 5.02 | 2.10x 656 → 1384 tok/s | 2.54x 138 → 350 tok/s |
For LFM2.5-8B-A1B, the acceptance rate increases compared to two dense models, yet on-device we get only an 18% improvement on average. This gap is due to the current MoE implementation in llama.cpp’s Metal backend, and to the fact that verifying k tokens activates more experts and thus more weight traffic than a single decode step.
| Dataset | Acceptance (of 10) | Speedup on H100 | Speedup on M4 Max |
|---|---|---|---|
| MATH500 | 8.27 | 3.18x 428 → 1362 tok/s | 1.21x 93 → 112 tok/s |
| HumanEval | 7.02 | 2.58x 426 → 1100 tok/s | 1.12x 91 → 101 tok/s |
| MBPP | 6.93 | 2.64x 426 → 1122 tok/s | 1.09x 89 → 97 tok/s |
| GSM8K | 4.02 | 1.29x 385 → 496 tok/s | 1.44x 90 → 129 tok/s |
| MT-Bench | 8.52 | 3.02x 426 → 1288 tok/s | 1.04x 87 → 90 tok/s |
