A coding agent jumped from roughly 30th place to the top five on Terminal-Bench simply by changing its harness while keeping the model identical. That experiment, published by LangChain, suggests the engineering layer matters more than the underlying intelligence. Paul Iusztin’s open-source course, Building a Coding Agent From Scratch, built on this by creating a Python agent named Decode. The course breaks down three distinct ways to run an agent loop, each demanding a different inference provider.
One headless core, three shapes
The system relies on a headless harness with no user interface. Inside, the loop follows a standard pattern: the large language model selects an action, a tool executes it, and the result feeds back into the context window. The agent logic itself is small. In Decode, it is a roughly 20-line definition using Pydantic AI that combines a model, tools, and an output type. By contrast, the core loop in leaked Claude Code source spans about 150 lines. Memory, skills, sandboxing, permissions, and language server feedback sit in the harness.
Interfaces plug into this core to create three modes:
Mode 1: Interactive, online
A terminal UI connects to a single live session running in memory within the same process. Events stream back via async generators as tokens arrive. The main difficulty is steering. Typing while a tool call is in flight corrupts the turn. Decode solves this with a steering queue and a priority gate. Input buffers on arrival and injects only at safe boundaries. The loop exposes two specific points: MODEL_REQUEST, before the next model call, and WOULD_STOP, when the turn ends. Three input modes map to this structure. Plain Enter steers within the turn. Alt+Enter queues a follow-up until the turn stops. Esc triggers a cooperative abort at the next boundary, clearing both queues to keep history intact.
A human reads every token. This mode is latency-bound, so it requires a low-latency hosted API.
Mode 2: Remote, offline
Remote mode keeps the harness headless and runs it on a server through an agent runtime. Decode uses Kitaru, ZenML’s agent runtime, deployed to GCP, with agents executing on Modal. Nobody watches the process. A backlog of tickets fans out to multiple harnesses in parallel, each producing its own pull request. Because the runtime records progress step by step, a sandbox that dies mid-task resumes from its last recorded step instead of restarting. A run that pauses for human input freezes and consumes no compute while waiting.
Tools execute inside Modal Sandboxes remotely or Docker locally. The metric that matters is throughput per dollar, not time-to-first-token.
Mode 3: Async, online
The third shape sits between the two. A live session hands work to a job queue and returns immediately. Background workflows fan out LLM calls and post results back later. The user is online but does not watch each step. The queue owns the work, so the run outlives the client that started it. This pattern drives Slack-triggered agents and background PR reviews, and it bills like batch, not like chat.
Why the provider changes with the mode
The cost model follows the latency requirement, and the gap is large.
Consider 1,000 documents at 30,000 input tokens each, with roughly 500 output tokens per document. At frontier API rates of $3 per million input and $15 per million output, the arithmetic lands near $97. Prompt caching does not rescue this because every document is a different prefix. Batched on a serverless GPU at around 3,000 tokens per second, the same work takes under three hours of GPU time, costing roughly $13.
The reverse case is equally sharp. Decode’s default test model, Qwen3.6 35B, runs on a single H200. Modal’s pricing lists H200 SXM at $0.001261 per second, or about $4.54 per hour. Leave an interactive agent idle overnight waiting for a y confirmation, and ten idle hours add roughly $45 to the bill.
That is the argument. Interactive work pays per token because a human is waiting. Offline and async work pays per GPU-hour because throughput is the objective and idle time is the enemy.
There is a second axis: serverless versus reserved capacity. Modal’s pricing analysis reduces this to one comparison. Reservations charge the peak rate for the whole contract; serverless follows the demand curve. When the peak-to-average ratio exceeds the reservation discount, serverless is cheaper. Modal reports typical discounts of 2–5× against peak-to-average ratios of 5–10× for inference, training, and agentic development. Industry surveys cite reservation utilization below 30%, often under 10%.
What it means for teams building agents
Teams must choose their provider based on the interaction model, not just the desired output. Interactive tools require low-latency APIs where humans wait for every token. Batched processing or background jobs should run on reserved GPU capacity to maximise throughput and minimise cost per hour. The engineering layer dictates the economics as much as the model choice.
