A single 24GB graphics card is the minimum requirement for running serious open-source models locally. Hardware like the RTX 3090 or RTX 4090 falls into this category. The specific card matters less than the choice of model.
Running the largest 70-billion parameter versions previously common is now obsolete. The current approach for 2026 focuses on 20B to 35B parameter models. These fit within the memory limits while leaving space for context and maintaining speed for coding or chat tasks. This guide reviews the specific models that run on this hardware and explains how memory is allocated.
How 24GB of VRAM actually gets spent
Three components consume memory during operation. Balancing these determines whether a model fits.
The first component is model weights. Size depends on the number of parameters and the compression level. At Q4_K_M, a standard setting for home use, each parameter costs about 0.58 bytes. A 32B model requires approximately 18 to 20GB for weights alone. Mixture-of-Experts (MoE) models create a specific issue. Every expert remains in memory even if only a small portion activates per token. Therefore, memory planning must use total parameters, not active ones.
The second component is the KV cache. This grows as context length increases. Longer prompts and sessions consume more VRAM. The third component is runtime overhead from the serving software. A safe estimate adds roughly 1 to 2GB for the cache and system overhead at short context lengths.
Quantization is the method that makes 24GB functional. Q4_K_M offers the best balance of quality and size. Q5_K_M and Q6_K improve quality but increase memory usage. Q8_0 and BF16 formats are usually too large for 30B-class models on a single card.
The six best local LLMs for a 24GB GPU
Each model below uses a permissive license, either Apache 2.0 or MIT. All fit on a single 24GB card at Q4_K_M with room for context. They are grouped by their primary strengths.
Qwen3.6-27B — best all-around and agentic coding
Alibaba’s Qwen3.6-27B is the strongest single default for this hardware. It is a dense 27B model released in April 2026 under Apache 2.0. The release focuses on agentic coding, repository-level reasoning, and frontend workflows. At Q4_K_M it needs roughly 16GB, leaving comfortable headroom for context. Full model cards and the changelog sit in the Qwen3.6 GitHub repository.
Qwen3.6-35B-A3B — fastest general-purpose option
The Qwen3.6-35B-A3B is a Mixture-of-Experts model with 35B total parameters and about 3B active per token. It decodes far faster than a dense 35B model because only a fraction of weights fire each step. The memory footprint still tracks total parameters, so plan for roughly 20GB at Q4_K_M. That makes it a tight but valid fit, best when you want speed for general chat and tool use.
Gemma 4 26B — multimodal and multilingual
Google DeepMind released Gemma 4 on April 2, 2026, under Apache 2.0. It was the first Gemma generation to ship with a fully open license, per the DeepMind Gemma page. The family spans edge models, a 12B unified multimodal model, a 26B MoE with 3.8B active, and a larger dense flagship. The 26B MoE is the natural 24GB pick when you need vision input and 140+ language coverage.
Mistral Small 3.2 24B — polished daily assistant
Mistral’s Small line targets the everyday assistant workload with low latency. Mistral Small 3.1 added multimodal input and a longer context window, and Small 3.2 refined instruction following. It is a 24B dense model under Apache 2.0, and the lightest footprint in this list at roughly 14GB at Q4_K_M. That headroom lets you push context further than the heavier 32B options allow. In 2026 Mistral also shipped larger successors, but those sit above the single-card tier.
gpt-oss-20b — the easy reasoning fallback
OpenAI‘s gpt-oss-20b is an open-weight reasoning model under Apache 2.0. It is a Mixture-of-Experts design with 21B total parameters and 3.6B active per token. It ships in a native MXFP4 4-bit format, loading in roughly 14GB with generous headroom. It is strong at structured reasoning and tool use, and weaker on broad world knowledge.
DeepSeek-R1-Distill-Qwen-32B — deepest reasoning, tightest fit
DeepSeek distilled its R1 reasoning traces into smaller dense models. The DeepSeek-R1-Distill-Qwen-32B is the 32B variant, built on a Qwen2.5 base and released under an MIT license. At Q4_K_M it uses about 18–20GB, which is the tightest fit in this guide. It exposes its chain of thought through visible reasoning tokens, which is useful for slow, deliberate problems. Ready-made GGUF builds are available from bartowski on Hugging Face.
What does not fit on 24GB
The frontier open models of 2026 are large sparse MoE systems. They are quite good in performance and reasoning, but they do not run on one consumer card. GLM-5.2 from Z.ai is a roughly 753B-total MoE. Moonshot’s Kimi K2.7 is around 1T total. DeepSeek shipped V4 as a public preview in April 2026, with a V4-Pro checkpoint near 1.6T parameters. Alibaba’s Qwen3.5-397B and Mistral Large 3 sit in the same server-class range.
Because MoE memory tracks total parameters, all of these need multi-GPU rigs or high-memory unified systems. They are worth knowing as API options. They do not change what runs on a single 24GB card.
How to run these models locally
Three runtimes cover almost every setup. Ollama is the simplest path, with automatic quantization selection and an OpenAI-compatible API. llama.cpp gives fine control over GGUF quantization and offload. vLLM is the throughput-focused server for heavier concurrent workloads.
Start with one model that matches your main job, not the biggest file you can load. Keep context under control, and let the card do what it does best. Run serious local AI without sending a single token to a cloud endpoint.
What it means
The practical shift is away from squeezing massive models into limited space. Users should prioritise 20B to 35B parameter models for better speed and context handling. Quantisation remains essential for fitting these models into consumer hardware.
