Changing the agent harness moved task completion from 30% to 81% in a test using the same open-weight model and 90 AppWorld tasks.
Why the harness matters
Community lists track nearly 200 agent harnesses. They make one point clear: the harness matters, not only the model. In one analysis, the same GLM-5.2 model went from 23% to 52% on SWE-bench Pro when only the harness changed.
So the right question is not “which harness is best?” but “which harness fits my workload, with my model?” Hermes from Nous Research is built for always-on personal assistants, and Deep Agents from LangChain for long, multi-step tasks such as research. Both are excellent at those jobs.
Enterprise work is a different job. Agents act on business systems, such as CRM and payments, and they must follow rules that hold whatever the model decides. CUGA (ConfigUrable Generalist Agent) is IBM’s open-source harness built for that job, with open-weight or hosted models, inside your own environment, including sovereign clouds and air-gapped deployments.
The AppWorld test
We ran CUGA, Hermes, and Deep Agents on AppWorld, a benchmark of multi-step tasks across 9 apps and 457 APIs, such as email, payments, and file storage. The apps are consumer apps, so AppWorld is a proxy: it tests a subset of the capabilities that enterprise automation also needs, such as logging in, reading API docs, paging through results, and changing state across several apps.
Every harness used the same open-weight model, GLM-5.3-flash, on the same 90 tasks in 30 scenarios, with one attempt per task. All runs used the Harbor evaluation framework, and our setup is public in harbor-eval, our Harbor fork.
How each harness ran
- CUGA: AppWorld instructions, task decomposition, verification before execution, and automatic reflection.
- Hermes: Harbor’s default Hermes setup, AppWorld tools over MCP.
- Deep Agents: AppWorld tools over MCP, plus LangChain’s built-in middleware, configured for the 469 tools: tool selection, context summarization, a to-do list, retries, and call limits. We also added two small fixes so that malformed GLM replies did not end runs.
Figure 1 shows AppWorld results for 90 tasks in 30 scenarios using GLM-5.3-flash for every harness. TGC counts tasks passed. SGC counts scenarios where all three variants passed.
The most useful number is SGC. A scenario passes only when all three of its variants pass, so SGC rewards consistency, not a lucky run. Here the gap is larger: 20 scenarios for CUGA, 8 for Hermes, and 2 for Deep Agents.
Three limits apply
This is a 90-task sample on one benchmark, so the results may not generalise to other tasks. The setups were not identical. And the runs did not use policies, so they do not measure policy enforcement.
In the 90-task comparison, the model was the same in every run, so the differences come from the harness. Every CUGA agent runs the same compact loop: prepare, call the model, execute, and repeat. The same loop scales from one CugaAgent with its own tools to a CugaSupervisor coordinating several agents, local or remote over A2A.
These are the CUGA features that ran in the AppWorld setup, grouped by what they do:
Plan the work
- Task decomposition: splits a task into steps, one app at a time
- Focused tools: each step sees the tools of its app and a few shared ones, not all 469
- Tool shortlister:
find_toolssearches for any other tool a step needs
Act in code
- CodeAct: the model writes Python that calls the app APIs
- Sandbox: the code runs in a restricted environment, with a time limit
Check before acting
- Verification before execution: a check reviews each code block that changes data before it runs
Check after acting
- Automatic reflection: reviews the results and lets the agent correct its plan
- Pagination audit: flags a list call that returned a full page when the next page was never requested
With these features, the model makes fewer decisions, and the harness catches more mistakes before they reach the result.
Rules outside the prompt
Each step that the harness controls, like those in the list above, is also a place where it can check a rule.
Most teams write business rules, such as payment approval or keeping customer PII inside the organization, into the system prompt. A prompt is an instruction, not enforcement. The model can misread it, a user can talk the agent out of it, and an injected message can change it in the middle of a task.
CUGA policies move these rules out of the prompt and into the harness. We describe this policy layer in our ACM CAIS 2026 paper, Governance by Construction for Generalist Agents (arXiv). The policies act at four points in every run:
Policy types and actions
- User intent (Intent Guard): Blocks or answers matching requests before the agent acts. Example: Block a request to delete all customer records.
- Agent guidance (Playbook): Gives the agent a standard procedure. Example: Follow the refund procedure step by step.
- Tool execution (Tool Approval): Hard stop: the run pauses for a person’s approval and shows the proposed code. Example: Ask a person before a payment tool runs.
- Tool execution (Tool Guide with ToolGuard): Hard check: guard code runs before the tool. A blocked call returns a policy violation, and the tool is never called. Example: Block a payment over $1,000.
- Final response (Output Formatter): Shapes the answer to an approved template. Example: Use the approved reply template.
Guidance steers the model, but approvals and ToolGuards are enforced by the runtime. Put rules such as payment approval and PII handling in the second group, where they hold whatever the model decides.
See the difference
You can see the difference yourself in the Policy Playground, which sends the same message to two live agents, one without policies and one with them:
“For ‘Pay everything I owe.’, the agent without policies paid $90, including a denied claim in collections. The agent with policies held the payment for approval, then paid only the $10 that was due.”
When to pick CUGA
Choose a harness as carefully as the model. In our comparison, the same model completed between 30% and 81% of AppWorld tasks, depending on the harness. For enterprise work, CUGA gives you:
- Strong results, now with open-weight models. With hosted models, CUGA held #1 on WebArena (February to September 2025) and on AppWorld (July 2025 to February 2026).
- Rules that the runtime enforces. Approvals and tool guards stop a call in the harness, whatever the model decides.
- Deployment inside your sovereign boundary. CUGA is Apache 2.0, works with any OpenAI-compatible endpoint, and runs locally, on Kubernetes in your private or sovereign cloud, or even in an air-gapped sovereign cluster, with OIDC sign-in and separate chat and management roles. IBM Sovereign Core builds on CUGA to run agents fully inside the customer’s boundary.
- Measurable cost. Budgets limit spending, and run receipts show tokens, tool calls, and time for every run.
- Configuration, not code. Set tools, budgets, and policies in the UI or SDK, test a draft, and publish a version.
CUGA is not the right harness for every job:
- For a terminal coding agent, use opencode, Codex, or Claude Code.
- For a personal, always-on assistant that lives in your chat apps, use OpenClaw or Hermes.
- For a blank-canvas graph framework, use LangGraph directly. CUGA is built on it.
CUGA is for automating business work across APIs, apps, and web UIs, under rules you can enforce.
- 🛡️ Policy Playground · policies guide
- 🧪 Reproduce the comparison
- ⭐ Star CUGA on GitHub
- 🚀 Try CUGA: quick install
The CUGA team ran this comparison, and CUGA used an AppWorld configuration that the other harnesses did not. If you can configure Hermes or Deep Agents to do better, send a configuration to harbor-eval. We will rerun it on the same tasks and publish the results, whichever harness wins.
