The Benchmark Nobody Runs: Why BullshitBench Exposes a Gap AI Leaderboards Miss
Every week brings a new frontier model, a new set of benchmark scores, and a new round of breathless claims about agentic coding supremacy. OpenAI drops GPT-5.5, DeepSeek ships V4 Pro, Xiaomi releases MiMo — and each arrives with a carefully curated scorecard showing it has closed the gap, or nearly so, with the best in the business. The convergence story is comforting: models are getting better across the board, the field is maturing, and your choice of LLM provider is increasingly a matter of taste and pricing rather than capability.
BullshitBench v2 says otherwise — at least on one dimension that might matter more than any of the standard evaluations.
What BullshitBench Actually Measures
Peter Gostev’s benchmark, hosted on GitHub and maintained as an open community project, does something deceptively simple: it presents AI models with questions that contain nonsensical cross-domain references and measures whether the model pushes back or confidently plays along. A question might apply Reynolds numbers to CI/CD pipelines, or model the elasticity of patient compliance relative to the kerning of a prescription label. The terminology is real. The structure is grammatical. The premise is incoherent.
Across 100 questions in v2 (up from 55 in v1), spanning five domains — software, finance, legal, medical, and physics — the benchmark grades each response into one of three categories: clear pushback (the model identifies the nonsense), partial challenge (it hesitates but doesn’t fully reject the premise), or accepted nonsense (it answers as if the question were valid). Scoring uses a three-model jury: Claude Sonnet 4.6, GPT-5.2, and Gemini 3.1 Pro, which agree roughly 80% of the time.
The software domain slice (n=40 questions) is where the results get interesting — and where Mat’s experience as a software engineer maps most directly onto the data.
The 25-Point Gap
On software-domain questions, Claude Sonnet 4.6 achieves 92.5% clear pushback. The next-best vendor is a tie between OpenAI’s GPT-5.2-codex (37.5%) and Google’s Gemini 3 Pro Preview (37.5%). That’s a 55-point gulf between the leader and the US frontier pack. Alibaba’s Qwen 3.5-397B sits at 67.5%, Moonshot’s Kimi K2.6 at 60%, and Xiaomi’s MiMo v2.5-Pro at 52.5% — all comfortably above OpenAI and Google.
| Vendor | Best Model (Software) | Clear Pushback |
|---|---|---|
| Anthropic | Claude Sonnet 4.6 | 92.5% |
| Alibaba (Qwen) | Qwen 3.5-397B | 67.5% |
| Moonshot | Kimi K2.6 | 60.0% |
| Xiaomi | MiMo v2.5-Pro | 52.5% |
| OpenAI | GPT-5.2-codex | 37.5% |
| Gemini 3 Pro | 37.5% | |
| Meta | Llama 4 Scout | 17.5% |
| DeepSeek | DeepSeek V4 Pro | 12.5% |
| Mistral | — | 2.5% |
This cuts against the usual frontier-convergence narrative. On most capability benchmarks, US frontier closed models lead, with Chinese open-weights closing the gap from below. Here, the hierarchy is inverted: Chinese models outperform the US frontier, and Anthropic sits in a tier of its own.
Why Reasoning Makes It Worse
One of the most counterintuitive findings across the full BullshitBench dataset: enabling “reasoning” or “extended thinking” mode hurts detection for a majority of paired models. Of 55 model pairs tested at different reasoning levels, 29 perform worse with reasoning enabled. The Arena AI analysis frames this crisply: reasoning training optimises models to solve whatever problem they’re given. More thinking budget means more effort constructing an answer — not more scrutiny of whether the question deserved one.
Opus 4.7 with max reasoning drops to 67.5%, a 17.5 percentage-point self-inflicted wound from extended thinking. The reasoning mode appears to engage a kind of over-eager problem-solving machinery that steamrolls past the moment where a good engineer would stop and say, “Wait — that doesn’t actually make sense.”
As Mat puts it: “I suspect for each of the cases where models refuse to push back, in a live iterative environment, you could quiz a lot of these models on their response and they would recognise their error. To me this feels like this benchmark is telling you something about a model internalising a next-step kind of reasoning, and being able to apply it in a way that is not just a surface-level pattern match.”
That’s the core hypothesis. The benchmark may be measuring something that existing evaluations don’t: not whether a model can reason, but whether it applies reasoning pre-emptively, before committing to a direction, rather than only when explicitly challenged.
What This Means for Agents
The BullshitBench software questions test recognition that two domains have been incoherently mashed together — Reynolds numbers in CI/CD pipelines, Sharpe ratios in test suites. That’s a narrow skill on its face. But it plausibly proxies for something coding agents need desperately: recognising the boundaries of a specialisation rather than rolling with whatever vocabulary the user offers.
A model that confidently theorises about the “beat frequency between deployment cycles” is probably the same model that won’t catch a developer who has forgotten the project is TypeScript and is asking for a Python solution. Mat has been on the receiving end of exactly that — GPT-4 era, zero pushback, just a confidently produced answer adopting the syntax of one language into another. The failure mode is structurally identical: confident generation under a false premise that the model could have detected from context.
This matters most in the brainstorming and design phase of software engineering, where the real danger isn’t wrong answers — it’s plausible-sounding wrong directions that a less experienced developer would follow without question. A pair programmer that agrees with everything is worse than no pair programmer at all.
The Trajectory Tells a Story
Looking at the time-series data, the divergence between Anthropic and everyone else is a recent phenomenon. Through mid-2025, all three major US labs were clustered in the 0-40% range. Then Anthropic’s Sonnet 4.5 hit 77.5% in September 2025 — a jump of nearly 50 points in a single generation. Sonnet 4.6 pushed that to 92.5% in February 2026. No other vendor has made a comparable leap.
OpenAI’s trajectory is instructive: steady but modest improvement from GPT-4o (5%) through GPT-5.5 (32.5%), with the codex variants consistently outperforming their chat counterparts. Google’s best result was Gemini 3 Pro at 37.5% back in November 2025, with subsequent models showing no improvement. DeepSeek, despite the hype around V4 Pro, sits at 12.5% — barely above their V3 baseline of 0%.
The Chinese model trajectory is the other surprise. Alibaba’s Qwen went from 2.5% (Qwen 3, April 2025) to 67.5% (Qwen 3.5, February 2026) — a 65-point climb that dwarfs anything from the US frontier labs. Moonshot made a similar leap: Kimi K2 at 7.5% to Kimi K2.6 at 60%.
Caveats Worth Taking Seriously
Forty questions is small. At n=40, 95% confidence intervals are roughly ±10-15 percentage points. That means the mid-pack differences — between Alibaba at 67.5% and Moonshot at 60%, or between OpenAI and Google both at 37.5% — are within noise. The Anthropic gap is not. A 55-point lead survives even the most generous confidence interval.
The judging panel is another concern. All three judge models are from the same generation of frontier models, and there’s a plausible argument that the panel rewards Sonnet’s direct, terse style of pushback over more hedged recognitions. The Sonnet-beats-Opus pattern (92.5% vs 85% for Opus 4.7, and 67.5% with max reasoning) could partly reflect judges rewarding Sonnet’s stylistic preferences. It could also be genuine. Both readings are live, and the benchmark’s creator acknowledges this openly.
There’s also the question of ecological validity. Recognising that “Sharpe ratios applied to test suites” is nonsense is a different cognitive task from catching that a developer’s proposed architecture doesn’t match the project’s actual constraints. The benchmark measures a symptom, not the disease. But as symptoms go, it’s a revealing one — especially given how cleanly it separates the models that practitioners tend to prefer for pair programming from those that score well on conventional benchmarks.
The Missing Benchmark
This is, as Mat observes, “a gap in benchmarks that drill into deeper areas of model reasoning performance that are not captured by the usual suite of benchmarks.” We test general knowledge, tool-calling, code generation, mathematical reasoning, instruction following. We don’t test whether a model will tell you your question is malformed — whether it has internalised the kind of pre-emptive quality-check that separates a useful collaborator from a yes-machine.
Models have massive, general experience across domains. It’s their ability to double-check our own attempts to generalise our limited experience that is particularly valuable. A model that catches your wrong framing before you’ve built three layers of architecture on top of it is worth more than one that scores 5% higher on HumanEval.
BullshitBench is imperfect. It’s small, its scoring methodology is debatable, and its domain-specificity means it can’t tell you everything about a model’s general capability. But it’s measuring something real — something that the standard leaderboard consensus systematically ignores. The 25-point gap between Anthropic and the next-best US lab isn’t noise. It’s signal. And for anyone choosing a model for serious software engineering work, it’s signal worth heeding.
Sources
- BullshitBench v2 — GitHub Repository
- Inside BullshitBench: AI Models and Nonsense Detection — Arena AI
- BullshitBench — Hacker News Discussion
- GPT-5.5-Pro did worse in BullshitBench — Hacker News
- Beyond AI Psychosis and Sycophancy: Structural Drift as a System-Level Safety Failure — medRxiv
- Sycophantic LLMs Mislead Novices in Problem-Solving Tasks — ACM
Mat Edit: An earlier version had a GPT Image-2 chart that modified Claude-generated original. It introduced a number of errors which I should have picked up, I’ve replaced it with an image version of the original chart. It was an interesting exercise in having Orac drive a tool I provided with GPT Image-2 capabilities but in retrospect I should drive such models manually and check their output properly.
