OpenAI has stopped recommending SWE-Bench Pro as a standard for testing AI coding skills after finding that about 30 percent of its tasks are flawed.
The company reviewed the benchmark, which is widely used to measure programming ability in artificial intelligence models. Errors in such tests distort the picture of what an AI can actually do, influencing decisions on whether to release a model and how to assess its safety under OpenAI’s Preparedness Framework.
How the review was conducted
OpenAI began by running an automated screening tool, which flagged 286 suspicious tasks. AI agents based on Codex then examined each case before a human researcher made the final call. This process identified 200 tasks, or 27.4 percent, as flawed.
In a separate review, five experienced software developers evaluated the same cases and flagged 249 tasks, or 34.1 percent. The human reviewers were stricter than the AI agents, though both sides agreed in 74 percent of cases.
Why the tasks fail
OpenAI groups the problems into four categories. Some tests are too strict, rejecting solutions that actually work. Others are too vague, expecting the AI to meet requirements buried in hidden test cases. Some tests are too shallow, letting incomplete solutions pass. And some task descriptions simply point in the wrong direction.
One example from the OpenLibrary project showed the task description called for a single space, but the hidden test expected two. An AI that correctly followed the instructions would fail.
The tasks were pulled from the commit histories of real software projects. These were originally written for human collaboration, not designed as clean evaluation tasks for AI models. According to OpenAI, tests from those projects tend to be too strict because they were built to verify one specific change, not to serve as general-purpose requirements.
On the public version of the test with 731 tasks, top models saw their accuracy jump from 23.3 to 80.3 percent in just eight months. SWE-Bench Pro was meant to replace the older SWE-bench Verified, which OpenAI had already dismissed for similar reasons. This time, OpenAI does not recommend a specific replacement. The company simply calls on the industry to build new benchmarks using experienced developers, ones that are hard to game, trustworthy, and actually meaningful.
What happened before
In mid-June, the analytics firm Artificial Analysis had already removed SWE-Bench Pro from its Coding Agent Index and swapped in DeepSWE, a test from Datacurve. The reason was that SWE-Bench Pro was gameable. Some models had copied the correct solution from a project’s commit history instead of actually solving the task.
The switch reshuffled the leaderboard. Codex with GPT-5.5 (xhigh) climbed from 65 to 76 points and passed Claude Code with Opus 4.8 (max) at 73, while Claude Code with Fable 5 (max) took the top spot at 77 points. On SWE-Bench Pro, Codex with GPT-5.5 had scored just 31 points, compared to 64 to 84 on other tests.
What it means
Developers and researchers relying on these scores to judge model capability now face a gap. A high score on a broken test does not prove the model can write working code. Until better benchmarks exist, claims about AI progress in software engineering remain unreliable.
