SemiAnalysis has published results showing that no AI agent scores above 12 percent on a new benchmark testing real-world work tasks.
How the test works
The firm released GDPx, a benchmark that places AI agents into simulated professional environments. The agents spend up to eight hours operating software like Siemens NX, searching through data rooms with thousands of files, or attacking live enterprise systems. SemiAnalysis states this approach differs from OpenAI’s GDPval, which it claims provides all context within the prompt and lacks a realistic work setting.
The 206 tasks cover eleven domains, including chip design, biology, quantitative trading, cybersecurity, and law. The team deliberately excluded administration and insurance, arguing that the first trillion dollars in AI revenue will come from industries where frontier models deliver the largest returns.
Top performers and pricing
Claude Opus 5.5 leads the leaderboard with a score of 12 percent. Google’s Gemini 4 Carbon follows at 10.9 percent but costs just $16.29 per task, roughly a third of the price. GPT-6 Astra scores 9.6 percent. No model fully solves the finance and legal tasks. A single incorrect value in a spreadsheet renders the entire output useless in practice. By that strict standard, no model would have completed even one task, so SemiAnalysis counts the share of individual requirements met instead.
Detailed findings highlight specific failures. In alignment tests, 98 percent of 200 cyber runs broke at least one behavioural rule. In a simulated medical practice, one model signed a clinical note in the doctor’s name. In quantitative trading, 70 out of 111 runs performed worse than the provided reference strategy.
Impact of safety filters
A filtered analysis reveals how provider safety mechanisms influence the results. Without the cyber, biology, and alignment domains, GPT-6 Astra edges past Opus 5.5 at 11.8 percent. In the cyber domain, GPT-6 Astra and GPT 6.1 Sol score zero percent, while Meta’s Muse Wonder Full hits 35.2 percent.
Concerns over closed benchmarks
The tasks originate from 14 specialised partners who also sell training and evaluation data to AI labs. SemiAnalysis admits it lacks in-house expertise in every domain. Usually only one example task per partner is public, while the rest remains behind closed doors. In several areas, including law and hardware design, language models grade the results using rubrics rather than human experts. Outsiders cannot fully verify either the difficulty or the grading quality.
The secrecy does offer a methodological advantage. Because the tasks are not public, AI labs cannot optimise their models for the specific test questions. Artificial Analysis took a similar step with the latest version of its Intelligence Index.
Developer Josh Purtell describes GDPx as “more noise than signal.” He notes that each model runs in its own provider’s agent software, for example Opus in Claude Code. Providers with weaker tooling face a disadvantage. Individual sub-domains often contain only about ten tasks, which makes results volatile.
SemiAnalysis plans to update the benchmark every three to six months.
