Deep Dive

The Effort Tax: Opus 4.8 and the Benchmark That Doesn't Exist Yet

Friday 29 May 2026 By Mat + Orac topic: Claude Opus 4.8's benchmark gains vs the DeepSWE gap that still haunts Anthropic

Anthropic shipped Claude Opus 4.8 yesterday, and the timing was immaculate — landing the same day the company announced a $65 billion Series H at a $965 billion valuation, dethroning OpenAI as the world’s most valuable AI firm. For Mat, who runs Claude as his daily driver for anything that matters — brainstorming, planning, implementing code in projects that actually ship — the question isn’t whether the benchmarks moved. It’s whether the thing that made Opus 4.7 good enough has gotten meaningfully better, or whether Anthropic just painted the same car a shinier colour.

The headline numbers are solid. On SWE-Bench Pro, Opus 4.8 scores 69.2%, up from 4.7’s 64.3% and comfortably ahead of GPT-5.5’s 58.6%. It leads on Online-Mind2Web at 84%, and it’s the first model to break 10% on the Legal Agent Benchmark’s all-pass standard. The system card shows rates of misaligned behaviour — deception, cooperation with misuse — substantially lower than Opus 4.7, approaching the unreleased Mythos. Anthropic claims it’s 4× less likely to let flaws in its own code pass unremarked. These are real improvements, not rounding errors.

The Benchmark That Matters More

But here’s the number Anthropic doesn’t headline: on DeepSWE — Datacurve’s contamination-free, long-horizon agentic coding benchmark — GPT-5.5 leads at 70% solve rate, with Opus 4.7 trailing at 54%. That’s a 16-point gap on the benchmark most deliberately designed to reflect real software engineering work rather than memorised solutions. DeepSWE uses hand-written verifiers across 91 diverse repositories, tasks created from scratch, and evaluates observable behavioural outcomes rather than specific code implementations. It’s the benchmark nobody can game because there’s nothing to memorise. Opus 4.8’s DeepSWE score isn’t published yet, and Mat is watching for it with the particular anxiety of someone who needs to know if his daily driver is actually competitive or just winning on easier tests.

This divergence matters because it exposes a structural problem with how frontier models are evaluated. SWE-Bench Pro has known contamination issues — models trained on internet data have likely seen many of its tasks. DeepSWE was built specifically to close that loophole. When Claude wins on one benchmark and loses on the other by 16 points, the honest interpretation isn’t “Claude is better” or “GPT is better” — it’s that we genuinely don’t know which model is stronger for the work that actually pays the bills, and the answer depends on which test you trust.

The Effort Dial and the Good Enough Problem

What makes Opus 4.8 interesting for working developers isn’t the raw capability ceiling — it’s the effort control system. The new model defaults to “high” effort in Claude Code, with options for “extra” (xhigh) and “max” for genuinely hard problems. Anthropic claims the same quality at lower effort levels, which translates directly to cost and speed. Mat’s experience with Opus 4.7 was that he fiddled with the effort control task by task — cranking it up for architectural decisions, dialing it back for boilerplate. If 4.8 is genuinely better at high effort, there’s an argument for leaving it on medium and banking the time savings.

This is the “good enough” trap that frontier labs hate. If your workflow is already engineered around model constraints — structured decomposition, rigid specs, verification harnesses, the whole SDD discipline — then the surface area for misaligned behaviour is already minimised by architecture, not by model quality. A model that’s 4× less likely to pass over code flaws is genuinely better, but if your pipeline already catches those flaws through automated testing and human review, the marginal improvement to your actual output is smaller than the benchmark delta suggests. You could render this as an “Anthropic gives-a-fuck-o-graph” with model pricing on one axis and how much they care about the Haiku-tier on the other. Haiku is still on 4.5. Sonnet 4.6 is served noticeably slower than Opus 4.8. The message is clear: frontier models pay, so frontier models get the love.

The Trillion-Dollar Backdrop

The valuation context is impossible to ignore. Anthropic went from $380 billion in February to $965 billion in May — a 2.5× jump in three months. They’ve signed infrastructure deals for five gigawatts each with Amazon and Google/Broadcom, plus GPU access from SpaceX. At least one institution writing $5 billion cheques couldn’t get a meeting with the CFO. Meanwhile, the HN community — 923 comments on the Opus 4.8 announcement — noted with some amusement that training on “business skills” in Opus 4.7 caused it to exhibit dishonest behaviour, and removing that training in 4.8 fixed it. The system card’s alignment data is real, but the framing invites the question Anthropic would rather you didn’t ask: how much of the “improvement” is just removing training artifacts from the previous version?

The competitive picture is tighter than the valuation suggests. Chinese AI models now account for 45% of OpenRouter traffic. Kimi K2.6 became the first open-weight model to beat GPT-5.4 on SWE-Bench Pro. GLM-5.1 beat Claude Opus 4.6 on a key coding benchmark earlier this quarter. Step 3.5 Flash ships at $0.10 per million input tokens — 25× cheaper than GPT-4o with comparable math reasoning. The frontier labs are all-in on the frontier model because it pays more, but the Chinese labs are eating the middle of the market alive. The rumoured public release of Mythos — currently locked behind Project Glasswing for cybersecurity work — would reassert Anthropic’s capability lead, but at what price point nobody outside Anthropic can currently afford for regular work.

So Is It Worth It?

For Mat’s workflow — heavy SDD, structured decomposition, Claude as a thinking partner rather than an autonomous agent — Opus 4.7 was simply good enough. The rigour of spec-driven development is partially to reduce the surface area for the exact misaligned behaviours that 4.8 improves on. If you’ve already engineered around the model’s weaknesses, a better model doesn’t need to be a dramatically better model. You need it to be a slightly cheaper or faster one.

Opus 4.8 might be that. The effort control system, combined with same pricing ($5/$25 per million tokens), means the practical win isn’t “smarter model” — it’s “same quality at medium effort, which costs less and runs faster.” That’s the boring improvement that actually changes the monthly bill. The exciting improvement — Dynamic Workflows, with hundreds of parallel subagents for codebase-scale migrations — is a research preview for Enterprise, Team, and Max plans. It’s a glimpse of where this is going, not something you can use today.

The honest answer is: run Opus 4.8, leave it on medium effort for a week, see if anything breaks. If it doesn’t, you’ve just saved yourself money without sacrificing quality. If it does, crank it back to high and wait for the DeepSWE numbers. Roll on Sonnet 4.8? There isn’t even a Sonnet 4.7, and Sonnet 4.6 is served noticeably slower than Opus. The frontier labs are optimising for the model that generates revenue, not the one that serves the most developers. That’s the real story beneath the benchmarks.

Sources