Qwen 3.6 27B on 24GB VRAM setup: backend comparisons, quant choice and settings (llama.cpp, ik_llama.cpp, BeeLlama, vllm)

TL;DR

• best setup I tested on a RTX 3090 24 GB: ik_llama.cpp + Qwen3.6-27B-MTP-IQ4_KS.gguf
• 156k context, q8_0/q8_0 KV, MTP, vision on CPU
• benchmark result on a ~5.9k prompt + 1k output: about 1261 tok/s prefill, 72.9 tok/s decode
• llama.cpp was a good start, BeeLlama worth testing, but ik_llama.cpp performed the best

What was tested

• upstream llama.cpp: easy baseline and a good place to start
• beellama.cpp: promising on paper, but I could not reproduce the expected speed on my setup
• ik_llama.cpp: best decode/prefill, best VRAM fit

I also spent time with vLLM / club-3090, but I am leaving it out of the table because I did not finish a clean apples-to-apples run in this batch. We were seeing about 78 tok/s on responses, but the high-context OOM cliffs were too flaky, so I dropped it until that is fixed. I have not tested it recently, but the repo still flags the single-card long-context issue as unresolved.

The benchmark

One-shot chat-completion task:

• prompt size: about 5.9k tokens
• output size: 1024 tokens
• task shape: a code-review / migration note over local setup files

So it mostly tests:

• prefill speed on a medium-large real prompt
• decode speed on a sustained 1k-token generation

So that is not best-case tok/s, but closer to reality.

The setup I kept

This is the profile I kept as my default:

• backend: ikawrakow/ik_llama.cpp
• current tested build: 4507 (c35189d8)
• model: ubergarm/Qwen3.6-27B-GGUF
• direct model file: Qwen3.6-27B-MTP-IQ4_KS.gguf

High-level launch shape:

• --ctx-size 156000
• --cache-type-k q8_0
• --cache-type-v q8_0
• --flash-attn on
• --multi-token-prediction
• --draft-max 4
• --draft-p-min 0.0
• --merge-qkv
• --merge-up-gate-experts
• --cache-ram 32768
• --ctx-checkpoints 32
• --reasoning on
• --reasoning-format deepseek
• --chat-template-kwargs '{"preserve_thinking":true}'
• --no-mmproj-offload

Notes:

• built-in MTP in ik_llama.cpp worked better for me than the other speculative paths
• q8_0 KV was good quality; you can opt into q4, but there is plenty of VRAM headroom with IQ4_KS

Why IQ4_KS

• much smaller than Unsloth UD-Q4_K_XL
• quality stayed high enough that I did not feel a real penalty
• on a 24 GB card, those saved GiB matter once you start pushing context and sane u-batch sizes
• to be fair, there is probably room for a higher quant, maybe q5; I have not tested that yet
• Qwen-3.6 quants discussion #1663

TLDR:

• Qwen 3.6 quantizes very well in IQ4_KS
• ikawrakow measured IQ4_KS as very close to, or better than, UD_Q4_XL
• Unsloth UD-Q4_K_XL needs about 2.8 GiB more to land in the same neighborhood

If you want the background on the quant family itself:

• New quantization types IQ2_K, IQ3_K, IQ4_K, IQ5_K discussion #8

Vision

• projector on CPU by default: --mmproj ... + --no-mmproj-offload
• move it to GPU if you want faster image processing and are willing to spend roughly 1.5 GiB more VRAM
• if that OOMs, lower context or switch to q4 KV

GPU Stuff

This was on Linux with the desktop on the iGPU and the RTX 3090 used only for LLMs.

• power limit: 330 W
• memory OC: +600
• undervolt: flattened at about 1875 MHz @ 868 mV (LACT now has a curve editor)

Some experiments did not make the default setup better

• --spec-autotune on ik_llama.cpp: no meaningful gain on this workload
• --mtp-requantize-output-tensor q6_K: sometimes faster, but inconsistent and costs about 1 GiB extra VRAM, so I did not keep it
• BeeLlama DFlash precision quickstart: loaded fine, but was much slower here than expected
• upstream llama.cpp MTP paths: good baseline, but slower than ik_llama.cpp in my tests

BeeLlama and vLLM are still worth exploring. I just did not land on a setup there that beat the ik_llama.cpp profile for my workload.

Results

These are the useful comparison points from the same real prompt / 1024-token output benchmark.

Flags tested:

• --spec-autotune did not produce better results on this workload
• --mtp-requantize-output-tensor q6_K had occasional upside, about +5 tok/s decode in the best run, but it was not stable enough to justify the extra ~1 GiB VRAM

Flag comparison

These are the high-level config differences that mattered most.

Links

• ik_llama.cpp: https://github.com/ikawrakow/ik_llama.cpp
• ExLlamaV3: https://github.com/turboderp-org/exllamav3
• BeeLlama: https://github.com/Anbeeld/beellama.cpp
• BeeLlama Qwen 3.6 quickstart: https://github.com/Anbeeld/beellama.cpp/blob/main/docs/quickstart-qwen36-dflash.md
• club-3090: https://github.com/noonghunna/club-3090
• IQ4_KS with MTP: https://huggingface.co/ubergarm/Qwen3.6-27B-GGUF/blob/main/Qwen3.6-27B-MTP-IQ4_KS.gguf
• Qwen-3.6 quants discussion: https://github.com/ikawrakow/ik_llama.cpp/discussions/1663
• IQ4_KS quant family discussion: https://github.com/ikawrakow/ik_llama.cpp/discussions/8

This is the best 24 GB setup I found so far, but things are moving fast and I do not think this is settled yet.

The point of this thread is to compare real single-3090 / 24 GB results: backend choice, quants, flags, and what stays stable under actual use.

I would like this to become a useful reference thread for 24 GB cards: what works, what breaks, and what is actually worth running day to day. I have not tested ExLlamaV3 yet, and there may be other setups that are better.

Also, thanks to everyone building this stuff: backend authors, quant makers, template tinkerers, and the people doing the boring debugging work that makes local LLMs usable.