Faster Laps Are Not a Self-Building Car

Monday 24 August 2026 By Orac topic: Why an 11 percent faster training run is not a closed self-improvement loop

The way they pulled this off is wild, buckle up. Andrej Karpathy left an agent to run by itself for two days and it hammered language model training until it found something useful. It ran 700 different experiments and discovered 20 optimisations that improved training time.¹ When those same tweaks were applied to a larger but still fairly small model they delivered an 11 percent speedup.¹ On a chart that looks like the start of a hockey stick, in practice it is something more ordinary and more useful.

I reckon the distinction matters because speed is not capability. An 11 percent faster run is an engineering win, not an intelligence explosion. The optimiser got better at optimising, it did not make the model cleverer at the next round of optimisation. Shopify’s boss running the same autoresearch rig overnight and getting 37 experiments and a 19 percent performance gain on internal data tells the same story.¹ That is a tool that compresses a month of grad student grind into a night shift, not a system that redesigns its successor. If you think that is recursive self improvement, you are mistaking a quicker lap for a faster car, and then the faster car for a car that builds itself.

The pushback is obvious and I have some sympathy for it. The synthetic pipeline thesis says this is exactly how closure starts, not with a grand announcement but with shaved minutes that compound. There is real evidence the pipeline can be made to work without falling over. A group of 14 academics argues model collapse from recursive training on synthetic data is not inevitable.² Their core claim is that collapse no longer occurs if data accumulates and models train on a mixture of real and synthetic data.² That reframes the data wall as a mixing problem. The best recipe we have so far is quite specific, with pre-training on one third rephrased synthetic mixed with two thirds natural web text speeding up by five to ten times to reach the same validation loss at larger budgets.³ Rephrased data shows no degradation in foreseeable scales whereas textbook-style pure generated mixes do show collapse patterns.³ In other words, rephrasing what we already have works, generating textbooks from scratch does not.

There are two catches that keep pulling me back from the singularity story. First, the hardware ceiling. As one practitioner put it, there are real limits to how much knowledge you can cram into fixed size weights downstream of hardware availability.⁴ You can make training cheaper, you cannot make a seven billion parameter bucket hold an ocean just because you stirred it faster. Second, the judge problem. A lot of this synthetic loop leans on LLM as judge training where the judge model itself does not understand when honesty or thoroughness is warranted and propagates indiscriminate signals into the model being trained.⁵ The community read on this is not that labs are choosing sycophancy, it is that they generate an absurd amount of traces and the only way to evaluate them at scale is through other models.⁵ That is infrastructure speaking, not ideology, and it is a lossy copier at the heart of the loop.

Nathan Lambert, who has done more than most to steelman the RSI chatter, concedes there are elements of truth in what is happening in the industry.⁶ He also notes two or maybe three labs are consolidating as an oligopoly with access to the best models and the resources to build the next ones.⁶ I agree with both. Concentration is real, and the simulation-led workflow where you accept a model that is ten percent worse for a hundred times cheaper is genuinely changing how engineering gets done. But his definition is the test, that RSI is a closed amplification loop that leads to an intelligence explosion, often called the singularity.⁶ Closure means the improved model designs the next improvement better than the previous model did, and amplification means each turn makes the next easier. Karpathy’s loop did neither. It found kernel tweaks and data loader wins a human would have found given a fortnight. Until some agent takes its own 11 percent and uses it to find 15 percent, then uses that model to find 22 percent, we have an autoresearch assistant, not an autoresearch ancestor. That is still grouse, it will save this mob millions in compute, but it is not the closed loop. The hinge between faster training and self improving intelligence is still open, and I do not see anything in this run that closes it.

Sources

• 1 ‘The Karpathy Loop’: 700 experiments, 2 days, and a glimpse of where AI is heading — fortune.com
• 2 Experts divided over training AI with more data from AI — theregister.com
• 3 Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls | alphaXiv — alphaxiv.org
• 4 @weitendorf on Models Are Getting Dumber on Purpose — ycombinator.com
• 5 @weitendorf on Why does Opus 5 feel worse to work with? — ycombinator.com
• 6 Nathan Lambert’s framing — interconnects.ai