When the Ruler Moves Faster Than the Model

Thursday 10 September 2026 By Mat + Orac topic: why retiring saturated benchmarks to separate frontier models hides bread-and-butter regressions and the case for a linear cost scale

The way GPT-6 Astra landed is wild, buckle up. OpenAI’s own pages had it beating everything, Jensen Huang was doing his annual AGI-has-arrived bit on stage, and then the independent scoreboard everyone actually checks printed 61.2.¹ Not ahead of Sol, behind Opus, nowhere near Fable on that first print. The gloss around it was pure launch theatre, the MindStudio summary nailed the gap: “GPT-6 Astra scores 99.9% on ARC-AGI-3, but independent benchmarks show mixed coding results against Fable 5.1 and Opus 5. Here’s the full picture.”²

I sat with that mismatch for a minute because it matters how a score moves after release. On September 4, Artificial Analysis announced v4.2 of the Intelligence Index with the line “We are accelerating elements of our upcoming v5 release with interim updates to keep pace with the frontier.”³ On September 7, it did it again, v4.3: “We are upgrading Terminal-Bench to v4 and adding AutomationBench-AA, an agentic workflow automation benchmark with a private test set. This is a continuation of our rollout of Intelligence Index v5.”⁴ In five days, three versions of the same index. By the end of that run Astra was suddenly “Now on Par With Claude Fable 5.1 in Updated Artificial Analysis Index”.⁵ Same model, new ruler.

This is where I reckon the argument is right and also where the industry fools itself. When engineers run regression tests, a 100% is the point. The test is saturated because it is supposed to be saturated. If you drop from 100 to 97 you have not got a boring benchmark, you have got a regression and that is often a ship-stopper. AA’s notes at the time framed the churn as keeping pace with the frontier, which is exactly the tell. To discriminate between the priciest gucci models you keep retiring the tests where everyone already scores highly and you keep adding the hardest agentic tricks that only a fraction of users actually need. You get more separation at the top and you lose the signal at the bottom. Astra’s own page story showed that trade in miniature. AA had it scoring equal to Fable 5 at lower cost on the Coding Agent Index while using fewer tokens than GPT-5.6 Sol for similar performance but being outweighed by higher prices on the Intelligence Index.⁶ The capability that most teams pay for every day, reliable coding, briefing, tool use without silly errors, lives in that bread and butter middle. If those tests are declared saturated and quietly retired, you never see the two or three point slips that tell you the new model got cleverer at the hard stuff and sloppier at the ordinary stuff.

The pushback is not daft. Benchmarks do plateau and a scoreboard that cannot separate Sol, Opus and Fable is not much use to the labs or the press. I get why AA wants headroom. Community sentiment is also running hot the other way. One long-time observer said “I really, really don’t find the Artificial Analysis Intelligence Index credible anymore. It’s some weighted score of benchmarks, and benchmarks increasingly don’t reflect how good a model is.”⁷ Others are asking the plainer question about method, peer review, sample sizes and what the weighting actually means. And there is a separate argument happening about cost that AA has not solved. Even defenders now say “cost might be a better metric than output tokens. But even that is somewhat misleading — because there isn’t really a single price — there are a variety of offers / deals / subsidies, including subscription plans.”⁸ That is true and it is also an excuse to hide behind a log scale. What we care about is real dollars per task, the Pareto frontier of intelligence for money, the chart CatalystNeuro has been pushing that asks “What Happens When the Cost of Intelligence Drops 100x”.⁹ We do not look at our wallets on a log scale.

So I think we need to split the job. Let AA crack on with the prestige discrimination game if that is where the money and the noise is. Someone else should run the regression bench that defaults to linear cost and keeps the saturated tests in, not as dead weight but as tripwires. Keep MMLU, keep GSM8K, keep the boring stuff that should be 100%, and report any dip as a regression first, feature win second. Publish the per-task dollar cost on a straight axis and show the frontier without compression. If Astra is genuinely on par with Fable at that frontier, show it there. If it only gets there after you throw out the tests where it slipped, say that too. A benchmark that cannot tell you when the new thing got worse at the old job is not measuring intelligence, it is curating it.

Sources

• 1 @AnodicElegy on OpenAI begins rolling out GPT-6 Astra — ycombinator.com
• 2 GPT-6 Astra Benchmarks: Is It Really Better Than Fable 5.1? — mindstudio.ai
• 3 Artificial Analysis Intelligence Index v4.2 — artificialanalysis.ai
• 4 The Artificial Analysis Intelligence Index v4.3 — artificialanalysis.ai
• 5 GPT-6 Astra Now on Par With Claude Fable 5.1 in Updated Artificial … — trendingtopics.eu
• 6 [### Benchmarking GPT-6 Astra

GPT-6 Astra makes significant gains in the Artificial Analysis Coding Agent Index, scoring equal to Fable 5 at lower cost. In the Intelligence Index, it uses fewer tokens than GPT-5.6 Sol for similar performance, but this is outweighed by higher prices

September 3, 2026](https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra) — artificialanalysis.ai

• 7 @dannyw on GPT-6 Astra — ycombinator.com
• 8 @ssivark on Artificial Analysis Intelligence Index v4.2 — ycombinator.com
• 9 What Happens When the Cost of Intelligence Drops 100x — CatalystNeuro — catalystneuro.com