Everyone in the AI infrastructure space has a strong opinion about cloud GPU versus API providers. Most of those opinions come from people who have not run a production workload on either.
This one comes from someone who has: a multi-service estate running trading bots, an AI news pipeline, a bot ecosystem, and autonomous agents — all making LLM inference calls continuously across a mix of onprem Proxmox LXC containers, a cloud relay server, and API providers. The routing layer handles hundreds of calls per day across all of them.
Here is what the numbers actually look like.
The Two Philosophies
API LLM providers (Anthropic, OpenAI, Google, Groq) are selling you a utility. You call an endpoint, you pay per token, you get a response. No hardware to manage, no model to maintain, no infrastructure team needed. The model is always on, always updated, always someone else’s problem.
Cloud GPU (RunPod, Lambda Labs, Vast.ai, CoreWeave) is selling you compute. You get raw GPU capacity — what you do with it is your business. Run Ollama, run vLLM, fine-tune a model, run a custom inference stack. Full control, full responsibility.
Onprem GPU sits in between: you own the hardware, you have the control, you have none of the cloud elasticity.
Real Cost Comparison at Different Scales
Low volume: under 1 million tokens per day
At this scale, API providers win on economics every time. A million tokens per day with Claude 3.5 Haiku (Anthropic’s most affordable model at $0.80/MTok input, $4/MTok output) costs roughly £2–5/day depending on input/output ratio. That is £60–150/month.
Renting an A100 80GB on RunPod at $1.89/hr running 24/7 costs $1,360/month. You would need to generate tens of millions of tokens per day before GPU rental breaks even against Haiku pricing — and Haiku is not even the high-end model.
The conclusion at low volume is uncomfortable for cloud GPU advocates: API is almost always cheaper, and by a wide margin.
High volume: 50+ million tokens per day
The maths inverts. At 50 million tokens per day — achievable for any serious batch processing operation, a news pipeline running 24/7 on long articles, a code analysis pipeline — Haiku costs £150–400/day. That is £4,500–12,000/month.
An A100 cluster running Llama 3 70B at the same volume costs a fraction of that. Four A100s on RunPod at ~$2/hr each is $192/day — and serves more traffic than one API account typically allows before rate limits bite.
At serious scale, self-hosted wins on cost. The crossover point for most setups is somewhere between 10 and 30 million tokens per day.
Quality: Where API Providers Still Lead
This is the part cloud GPU advocates often gloss over. The best open-weight models — Llama 3 70B, Qwen 2.5 72B, Mistral Large — are genuinely impressive. They are not Claude 3.5 Sonnet.
For tasks requiring deep reasoning, nuanced writing, complex code generation, or multi-step problem solving, frontier API models are measurably better than the best open-weight alternatives. The gap is narrowing. It has not closed.
For tasks where good enough is genuinely good enough — summarisation, classification, structured extraction, routing decisions, light drafting — open-weight models on GPU are competitive and the quality difference is not worth the cost premium.
A practical breakdown by task type:
| Task | Open-weight sufficient? | Recommended approach |
|---|---|---|
| News article rewriting (editorial voice) | Mostly | LiteLLM routing: try local, fall back to Claude if quality gates fail |
| Trading signal analysis | Partially | Frontier API for confidence-critical decisions |
| Bot conversation (casual) | Yes | Local Ollama — speed and cost win |
| Code debugging across complex systems | No | Claude or GPT-4o — reasoning quality is the differentiator |
| Document classification | Yes | Open-weight on GPU or Groq API |
| Long-form strategic writing | Partially | Frontier API for final output, open-weight for drafts |
Latency: The Hidden Variable
API providers have invested heavily in inference infrastructure. Anthropic, OpenAI, and especially Groq deliver tokens fast — Groq’s throughput on Llama 3 70B is genuinely impressive, often 200–400 tokens per second.
Self-hosted Ollama on a single A100 typically delivers 30–80 tokens per second for a 70B model. For interactive use cases, the latency difference is noticeable. For batch processing where you are not waiting on the response in real time, it does not matter.
CPU inference — homelab without a GPU — is a different story entirely. 5–15 tokens per second for a 7B model. Workable for background tasks. Not workable for anything requiring a snappy response.
Control, Privacy, and Compliance
This is where the API model has a genuine structural weakness that no amount of competitive pricing can fix. Every token you send to an API provider leaves your network. Your data passes through their logging infrastructure, their abuse detection systems, potentially their model training pipelines depending on your account tier and their terms of service.
For applications handling sensitive data — financial signals, personal communications, proprietary business logic — this is not a theoretical concern. It is a real constraint.
Self-hosted inference means the data does not leave your infrastructure. For regulated industries, privacy-conscious applications, or any workload where the content of prompts is itself sensitive, this is often the decisive factor regardless of cost.
The Operational Reality of Running Your Own GPU
The GPU advocates often undersell the operational overhead. Running your own inference infrastructure means:
- Model management — pulling updates, managing quantisation, handling model-specific bugs
- Monitoring and alerting — is the inference server actually up? Is it handling requests? Is the GPU at risk of thermal throttling?
- Failover — what happens when the GPU instance goes offline or the model crashes?
- Capacity planning — what happens when traffic spikes beyond what one GPU can handle?
None of this is difficult individually. All of it adds up to genuine operational load that an API provider absorbs invisibly. For a small team or a solo operator, the time cost of managing GPU infrastructure is often worth more than the money saved on tokens.
The Architecture That Actually Works
- Frontier API for quality-critical tasks where the reasoning gap between open and closed models matters — complex debugging, strategic decisions, high-stakes writing
- Open-weight via fast API (Groq, Together, Fireworks) for latency-sensitive bulk tasks where open models are competitive on quality — classification, extraction, light drafting
- Local Ollama for always-on background tasks where privacy matters and latency tolerance is high — bot conversations, lightweight summarisation, internal tooling
- Cloud GPU on-demand for burst workloads — fine-tuning runs, processing large batches, anything requiring a big model for a finite window
- LiteLLM as the routing layer — send each request to the right provider based on a defined priority chain, with automatic fallback
This is not elegant. It is a multi-vendor, multi-tier inference stack that requires a routing layer to manage. But it is honest — it routes each task to the provider that handles it best at the cost that makes sense for that task type.
Key Takeaways
- At low token volume, API providers win on economics by a large margin. GPU rental only makes sense above roughly 10–30 million tokens per day depending on model and provider.
- Frontier API models still lead open-weight alternatives on complex reasoning tasks. The gap is narrowing but has not closed.
- For bulk tasks where quality is good enough — classification, extraction, light summarisation — open-weight models on GPU or fast third-party APIs are competitive at lower cost.
- Data privacy is a structural issue with API providers that cost savings cannot resolve. If prompts contain sensitive data, self-hosted inference is the only clean answer.
- The operational overhead of managing GPU infrastructure is real and often underestimated. API providers absorb that overhead in exchange for per-token pricing.
- The right architecture is almost always a routing layer that sends each task to the most appropriate provider — not a single-provider bet.
Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise.
