Benchmarking vLLM vs SGLang vs llama.cpp on a mixed Blackwell/Ada cluster

Summary of Benchmarks

• vLLM significantly outperforms both SGLang and llama.cpp on mixed multi-GPU setups for long context prefill.
• vLLM handles uneven GPU splits effectively by manually partitioning the layers, leading to substantial speedups even with a 397B model.
• llama.cpp struggles heavily with pipeline parallelism under these conditions, often falling behind vLLM by a factor of 4 to 6. This is due to issues with how the execution graph is handled across multiple devices.

Benchmark Results

I have been running benchmarks on a heterogeneous 7-GPU cluster to evaluate how different inference engines handle long context prefill using pipeline parallelism. My setup includes a mix of Blackwell and Ada cards: one RTX PRO 6000 with 96GB, one PRO 5000 with 48GB, two 5090 with 32GB each, and three modded 4090 with 48GB. All tests were conducted using 4-bit weights (NVFP4 for vLLM and SGLang, MXFP4 for llama.cpp).

The main takeaway is that vLLM significantly outperforms the others on mixed multi-GPU setups for long context prefill. Llama.cpp struggles heavily with pipeline parallelism under these conditions, often falling behind by a factor of 4 to 6 due to issues with how the execution graph is handled across multiple devices, particularly CPU-side embeddings causing graph splits and pipeline bubbles.

SGLang performs wonderfully on a pure Blackwell setup but instantly crashes if an Ada card is introduced into the pipeline because it lacks a software fallback for FP4 weights, strictly requiring Compute Capability 10.0. vLLM handles this seamlessly by emulating FP4 on older cards.

Another interesting finding is how well vLLM handles uneven GPU splits. By manually tweaking the layer distribution using the VLLM_PP_LAYER_PARTITION environment variable, I was able to balance the compute load between fast Blackwells and slower 4090s doing FP4 emulation. This eliminated pipeline bottlenecks and resulted in massive speedups even on a 397B model.

Here is the summary of the benchmark results.

Benchmark Results

Originally published at reddit.com. Curated by AI Maestro.

Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise.

Please enable JavaScript in your browser to complete this form.

Email Name

Name *

First

Last

Email *

Submit

More in AI Research & Science

AI Research & Science Benchmarking the new b9200 update: Optimizing Qwen 3.6 27B mtp for Hermes Agent on a single RTX 3090 May 18, 2026

AI Research & Science Seeking local LLM advice for cybersecurity work. May 17, 2026

AI Research & Science A Coding Implementation to Compress and Benchmark Instruction-Tuned LLMs with FP8, GPTQ, and SmoothQuant Quantization using llmcompressor May 17, 2026