The Benchmark of Benchmarks: When the Auditor Is Also the Accused
On July 8, OpenAI published a detailed audit of SWE-bench Pro — the benchmark they had championed just months earlier as the successor to SWE-bench Verified — and found that roughly 30% of its 731 public tasks are broken. The issues fall into four categories: tests so strict they reject correct solutions, prompts so underspecified that hidden requirements are unguessable, tests with coverage so thin that incomplete fixes pass, and prompts that actively mislead. This from a benchmark Scale AI built using SETUPAGENT, an automated pipeline that reconstructs environments and iteratively repairs test commands with LLM-driven fixes. As Mat notes, it’s something of an open secret that no human has actually verified each test makes sense. The benchmark was built by AI, graded by AI, and now the audit finding it broken is — by OpenAI’s own description — also an AI-powered pipeline, with Codex-based investigator agents doing the heavy lifting before five experienced engineers review the flagged subset.
The timing is impossible to ignore. Grok 4.5 launched the same day, and SpaceXAI’s own numbers put it at 64.7% on SWE-bench Pro — ahead of GPT-5.5’s 58.6%. OpenAI previously pivoted to SWE-bench Pro precisely because SWE-bench Verified had become saturated and contaminated, and because their models weren’t closing the gap to Claude Opus on it. Now that a competitor has posted a strong score on their benchmark of choice, they’ve revealed that a third of it is noise. The cynicism writes itself. But even if the timing is convenient, the findings are real, and independently corroborated: Datacurve’s DeepSWE audit in May found a 32% verifier error rate, and a LessWrong post by Jonathan Gabor documented specific broken tasks — including one where SWE-bench Pro scraped incorrect test cases from a flipt repository, so that a correct implementation would fail the benchmark.
The deeper problem Mat identifies is the recursion. We have benchmarks evaluating AIs where the benchmarks were written by AIs, and the benchmark validity investigation is also AI-powered. The human layer — five engineers per flagged task, in OpenAI’s case — is thin, applied only to the subset an automated filter already flagged, and the humans were more likely than the agents to find tasks broken. Which means the agents undercount. The entire apparatus is a closed loop: the same class of model that fails the benchmark is also the class of model used to build and validate it. When the error bars from broken tasks (~30 percentage points of uncertainty) exceed the spread between most frontier models on the leaderboard, the leaderboard isn’t measuring model capability anymore — it’s measuring scaffolding luck and which broken tasks happened to favour your model’s failure modes.
And yet. Mat is right that the alternative is worse. Strip away the numbers and you’re left with subjective feels — and the internet is already awash in cursory anecdotal claims that one model is better than another, usually from someone who tried it for twenty minutes. The Chinese models illustrate the trap in the opposite direction: they benchmark exceptionally well but cursory use shows they often don’t hold up. Benchmarks are necessary but insufficient, and pretending they’re either authoritative or worthless misses the point. They’re a noisy signal in a regime where every signal is noisy. The honest position is that none of these tests — SWE-bench Pro, DeepSWE, even the model-versus-model exercises on this site — can give you the full picture of how a multi-trillion-parameter model compares to another. We’re not putting numbers to things here precisely because we know that. But the industry writ large still does, and capital still flows on the basis of those numbers, and safety frameworks — OpenAI’s own Preparedness Framework cites benchmark results for deployment decisions — depend on them.
If 30% of the tasks on the benchmark you recommended are broken, and your competitor just beat you on it the day you disclosed that, the lesson isn’t that benchmarks are useless. It’s that the industry has been treating measurements it can’t validate as ground truth, and the gap between the leaderboard and reality is now wide enough to drive a Grok through.