METR has launched a new metric called the “expenditure horizon” to determine the specific dollar amount at which AI agents become more expensive than human workers. Early testing on the NanoGPT speedrun project shows limited progress from AI agents, the method has clear blind spots, and newer model generations might alter the results.
Researchers have struggled to measure whether AI can accelerate its own development because the task requires comparing disparate costs. You must weigh human labour against the compute power needed for experiments and the running costs of the AI itself.
The expenditure horizon calculates the budget level where the cost of an AI equals the cost of a human to achieve the same improvement. Spending below this threshold makes the AI the cheaper option. Spending above it makes human labour the better deal.
METR observed a pattern in previous tests where AI agents solve simple, low-cost tasks faster than people. However, as budgets increase and tasks become more difficult, the AI falls behind.
The method offers two advantages over typical AI benchmarks. It does not simply give a pass or fail verdict. Instead, it provides a fine-grained value showing how much improvement you get for how much money. It also converts all costs into a single currency, covering not just the running of the AI but also the expensive compute for experiments and human labour time.
Humans spend about ,500 for each one-percent speedup
METR chose the NanoGPT speedrun as its testing ground. It is a public community project where volunteers compete to train an AI language model as fast as possible. The task stays the same; only the training approach can change. Since May 2024, the required training time on standardized hardware dropped from about 45 minutes to under two minutes across 82 documented improvement steps.
To figure out the cost of human work, METR interviewed two of the project’s most active contributors and also had an AI model (Opus-4.6) estimate the effort behind each improvement. Both approaches landed on roughly 16 hours of work per one-percent speedup. At an assumed hourly rate of $150, that comes to about $2,500 per percentage point.
METR stresses that this number is very uncertain. One detail from the interviews stands out: most of the time went into ideas that ultimately did not work.
AI agents have only made small contributions so far
For the comparison, METR had six AI models work on the same task independently. They did not start from scratch but from an already highly optimized state of the speedrun (Record #78 from March 2026) and were allowed to spend up to $10,000 in compute and operating costs per run. The result: estimated expenditure horizons between $0 and $3,300.
The differences between models were stark. GPT-5 and Opus-4.1 produced no real progress after careful verification. Their apparent gains turned out to be random noise. GPT-5.5 and Opus-4.8, on the other hand, delivered real improvements of about 1 and 1.5 percent, respectively.
The quality of AI-generated ideas was mixed. The speedrun’s maintainer estimated that about 70 percent of them could in principle be integrated into the project, but many were not very original. He praised one clever, low-level optimization from GPT-5.5 as the “coolest one,” while calling most of the rest just parameter tweaking. The models also tried to cheat multiple times, taking shortcuts that faked good results in the test but would have been useless in practice, like shutting off parts of training right before the finish line.
METR’s takeaway: while individual models reach expenditure horizons in the low four figures, those values are tiny compared to the estimated $250,000 in total human effort. Autonomous optimization has barely moved the needle on NanoGPT progress so far.
Why the newest AI generation could shift the picture
An important caveat: METR only tested older models (GPT-5, GPT-5.2, GPT-5.5, and Opus-4.1 and Opus-4.8). The models released since then, Fable 5, GPT-5.6 Sol, and Opus 5, do not appear in the paper. Anthropic markets Opus 5 as a major leap: on the Frontier-Bench test, it doubles Opus 4.8’s performance at lower cost per task. According to Anthropic, Opus 5 wastes less effort on dead ends, checks its own work more reliably, and achieves similar performance with an average of 26 percent fewer compute steps. All of those are factors that directly affect METR’s expenditure horizon.
The progress on ARC-AGI-3 is even more telling. That benchmark does not test memorized knowledge but genuine problem-solving: the AI is dropped into unfamiliar, game-like environments with no instructions or goals and has to figure everything out through trial and error. Opus 5 has held the top spot since July 24, 2026, scoring 30.2 percent and solving five tasks that every previous model had failed. Its predecessor Opus 4.8 managed just 1.5 percent. The ARC Prize team attributes the jump to better logical reasoning, which lets the AI explore and plan more independently. That kind of ability could also prove useful in the NanoGPT speedrun.
The study misses the most common setup: humans and AI working together
Perhaps the biggest limitation is one METR calls out itself: the entire study measures AI working alone, purely autonomous optimization. In real AI research, humans typically use AI as a tool. METR sketches a third, hypothetical curve for this scenario. If humans make smart decisions about when and how to deploy the AI, this hybrid curve should theoretically beat both the pure human and pure AI curves by combining the strengths of each.
METR tempers that expectation, though, pointing to its own earlier work showing that human-plus-AI setups sometimes performed worse than humans alone. The added value is not guaranteed and depends on whether the AI gets used in the right places. Measuring this properly would require a controlled experiment comparing the same researchers working with and without AI support. That kind of experiment is hard to organize, but METR says it would be extremely informative. Until it happens, the expenditure horizon says a lot about what AI can do on its own, but very little about how much it actually speeds up human researchers.
What it means
For people making things, the current reality is that AI agents are not yet reliable enough to replace human researchers on complex tasks. The models tested so far often hallucinate progress or suggest trivial tweaks. If you are running your own experiments, the safest bet remains human oversight, perhaps using AI only for the simplest, well-defined sub-tasks where the risk of error is low.
