Nvidia released research on Friday showing that the infrastructure surrounding an AI model matters far more than the model itself when assigning long-term tasks.
The score difference
Researchers tweaked a custom harness to manage memory better and added a supervisor component. This setup allowed Claude Opus 5 to score 100% on the interactive reasoning benchmark ARC-AGI-3. Without that harness, the same model scored just 30%. That lower figure was the best result among all models tested in the original trial.
The benchmark has drawn criticism from rival lab OpenAI. Nvidia’s findings suggest that while selecting a model matters, the system acting as the agent’s brain is a smaller part of the whole. Many users underestimate the importance of the harness, especially for tasks requiring sustained effort.
The harness is what turns a model into an agent. It handles memory, context, and feedback.
“Generally speaking, the world interprets an agent almost as an API of the model,” Adel El Hallack, vice president of product in Nvidia’s AI unit, told TechCrunch. “But an agent is actually more than that. It is the model. It is the scaffolding around the model, which we call the harness, i.e. the set of tools that it utilizes. It is the runtime and the associated skills and libraries that we give it access to.”
Long-horizon tasks require stringing many decisions together, sometimes over days, to produce finished work. This differs from an AI simply answering a prompt. Keeping an AI focused on long-term goals without distraction remains a major challenge in agentic research.
Microsoft published research in April testing 19 large language models on long-horizon tasks involving document editing. Every model, including leading ones, filled documents with errors. Human workers producing similar output would be fired immediately.
Models attempting to chain decisions alone have also been observed deleting user files, entire databases, or engaging in criminal behaviour to meet objectives. Instances include collusion and hacking.
Using the ARC-AGI-3 benchmark for these tests is particularly notable. It consists of a set of 2D games with no instructions. The model must figure out how to play and win. A perfect score indicates the model can beat the games as well as humans.
OpenAI saw its models score less than 10% on ARC-AGI-3. The company conducted its own research last month. Like Nvidia, it found that tweaking two settings on the harness tripled the scores of its models. However, none of the models approached the 100% score achieved by Nvidia researchers. They showed that the harness requires a supervisor component to guide the agent if it gets stuck.
“The more interesting part was introducing a supervising agent in addition to your main agent that’s doing the work,” El Hallack said. It “almost acts like a CEO to nudge the agent when it goes off direction or starts exploring a path that it might lead to a dead end, or re-explore a path that it had previously trod.”
The concept of a supervising agent is not new. Most users currently rely on a single layer for their harness, such as Claude Code, Codex, or Hermes. Nvidia researchers created their own advanced harness called the Agentic Variation Operators (AVO).
This is not a new Nvidia product. The company produces various open pieces of technology for building harnesses under the Nemo brand. Some of this tech is commercial, while much remains openly available.
Nvidia’s results add to growing evidence that model choice is not the only factor in agentic performance. In July, Databricks published research showing that the harness impacts AI costs more than the model.
“You can pick the same model but different harnesses, and you get significantly more cost if you use the wrong harness,” Databricks CEO Ali Ghodsi told TechCrunch. “So you think, oh, this is an expensive model. This is a cheap model. But wait, which harness are you using? That itself can 2x your cost.”
Nvidia’s larger point is that open harnesses, like open models, give users more control than they realise.
“We believe, and we’re demonstrating with the ecosystem, how open harnesses allow you to turn a lot more knobs to drive up that accuracy,” El Hallack said. “It relates to OpenAI slowing down the training of their models,” as a result of models creating security breaches.
“We believe in having an open agent stack — where you have control across the harness, across the infrastructure, across the runtime — is what’s required for us to usher the ecosystem forward and securely,” he added.
