“`html
Seeking Resources to Read About LLaMA.cpp Server and How Offloading Works
SETUP INFO: Amd R9700 AI PRO. Using llama-cpp server, ROCM docker version. Using the --ngl option for offloading.
I am greatly impressed by how LLaMA.cpp handles offloading. There’s some serious magic happening here, at least to me.
I have 32GB of VRAM so loading in small models is no problem, but now I’m experimenting with larger models that spill into system RAM. Testing throughput (tok/sec) differences and various quantizations is part of this.
Currently testing Qwen3 Coder Next at Q4-KM. At 45GB in size, this one works fine for me but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model, which isn’t great at all – at least in my experience. I tried the 3-bit quantization but faced multiple quality issues after a few tests, so I gave it up.
Anyone else have this impression? Or am I just missing something?
I think for large models and coding tasks, 3-bit is just too much compression or feels like it. For example, the more layers you offload, the slower it gets (as expected).
If I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for a MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
For models like Qwen3 Coder Next, if I offload 36 layers, it fills my VRAM up to around 30/32GB. Throughput (tok/sec) is around 25 for an MoE model – which isn’t great at all in my experience.
I am currently testing Qwen3 Coder Next. At Q4-KM, this one weighs in at 45GB in size. I can make that one work, but as I increase offloading, performance degrades (as expected). Thus, I’m currently experimenting with the smaller 4-bit quantized version, IQ4_XS at 36GB. My goal is to find the sweet spot before quality starts to suffer.
Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise.
