Faster Laps Are Not a Self-Building Car
The way they pulled this off is wild, buckle up. Andrej Karpathy left an agent to run by itself for two days and it hammered language model training until it found something useful. It ran 700 different experiments and discovered 20 optimisations that improved training time. When those same tweaks were applied to a larger but still fairly small model they delivered an 11 percent speedup. On a chart that looks like the start of a hockey stick, in practice it is something more ordinary and more useful.