You don’t need a paid AI subscription to run a capable language model. Ollama lets you download and run models like Llama 3, Mistral, Gemma 2, and Phi-3 directly on your own computer — no API keys, no monthly fees, no data leaving your machine.
📋 Contents
This guide shows you exactly how to set it up from scratch, which models are worth running, and what you can realistically expect from local AI in 2026.
What Is Ollama?
Ollama is an open-source tool that makes running large language models locally as simple as running a command. It handles model downloads, GPU acceleration, memory management, and exposes a local API that’s compatible with OpenAI’s format — meaning any tool that works with ChatGPT’s API will also work with Ollama.
It’s free, it’s open source, and it runs on Mac (Apple Silicon or Intel), Windows, and Linux.
What Hardware Do You Need?
This is the honest part. Local AI performance scales directly with your hardware:
Apple Silicon Mac (M1/M2/M3/M4): Excellent. Apple Silicon’s unified memory architecture means models use shared RAM between CPU and GPU. An M1 with 16GB RAM runs 7B models smoothly; 32GB runs 13B models well. Best local AI experience for the money.
Windows/Linux with a modern GPU: 8GB VRAM (RTX 3060 or better) handles 7B models. 16GB VRAM (RTX 3080/4080) handles 13B models well. Without a dedicated GPU, models run on CPU — usable but slow (30–60 seconds per response on typical hardware).
Minimum to get started: 8GB RAM, any modern computer made after 2018. Responses will be slow on CPU, but it works for testing and light use.
Installing Ollama
Mac or Linux:
curl -fsSL https://ollama.ai/install.sh | shWindows: Download the installer from ollama.ai. It’s a standard .exe installer.
After installation, Ollama runs as a background service and listens on http://localhost:11434.
Downloading and Running Your First Model
Open a terminal and run:
ollama run llama3.2Ollama downloads the model (3–7GB depending on size) and drops you into an interactive chat. That’s it. You’re running AI locally.
To run a specific size variant:
ollama run llama3.2:3b # 3 billion parameter model, fast on any hardware
ollama run llama3.2:11b # 11 billion parameters, better quality, needs more RAMThe Best Free Models to Run in 2026
Llama 3.2 (Meta): The current benchmark for open-source models. The 3B version runs on nearly any hardware; the 11B version rivals GPT-3.5 for most tasks. Good all-rounder for writing, analysis, coding.
Mistral Nemo (12B): Excellent instruction-following and strong on technical tasks. One of the best models for code generation at the 12B size.
Gemma 2 (Google): Very capable at the 2B and 9B sizes. Fast, efficient, good for conversational use and summarisation.
Phi-3.5 Mini (Microsoft): Only 3.8B parameters but punches significantly above its weight class. Best small model for general use.
Qwen 2.5 (Alibaba): Strong multilingual support and excellent at reasoning tasks. 7B version is particularly good value.
Connecting Ollama to Other Tools
Ollama’s OpenAI-compatible API means you can connect it to almost anything:
Open WebUI: A free, self-hosted ChatGPT-like interface for Ollama. Install with Docker:
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data --name open-webui \
ghcr.io/open-webui/open-webui:mainThen visit http://localhost:3000 for a full chat interface.
Claude Desktop via MCP: You can configure Ollama as a local model provider through various MCP integrations.
n8n, LangChain, LlamaIndex: All support Ollama as a model backend. Point them at http://localhost:11434.
What Local AI Is Good at (and What It Isn’t)
Good at: Writing assistance, summarisation, code generation, Q&A on documents you provide, classification, and any task where privacy matters (medical notes, business plans, personal data).
Not as good as cloud models at: Complex multi-step reasoning, very long context windows, up-to-date knowledge, and highly nuanced creative writing. A local 7B model is capable but not Claude Sonnet.
The key use case for local AI isn’t replacing your cloud subscription — it’s running AI on sensitive data without it leaving your machine, and handling high-volume tasks without running up API bills.
How Much Does It Actually Cost?
The model itself: free. The electricity: minimal — a modern laptop running a 7B model uses roughly 10–30 watts extra. Running it for 2 hours a day costs pennies per month in electricity.
The real cost is hardware. If you don’t already have capable hardware, the upgrade is a consideration. But if you have an M1/M2 Mac or a gaming PC from the last 3–4 years, you likely already have everything you need.
Key Takeaways
- Ollama is free, open source, and runs on Mac, Windows, and Linux with minimal setup
- Apple Silicon Macs are the best local AI hardware for the price — M1 16GB handles most models well
- Llama 3.2, Mistral Nemo, and Phi-3.5 Mini are the best free models to start with
- Local AI excels at tasks involving private data — medical notes, business plans, personal documents
- The API is OpenAI-compatible, so it works with n8n, LangChain, Open WebUI, and dozens of other tools out of the box
AI Maestro covers practical guides to AI tools. Everything on this page is free to implement.
Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise.
[newsletter_form]
