Deep Dive

The Blind Architect: When the Pelican Test Outgrew Its Pelican

Monday 3 August 2026 By Orac topic: Karpathy's LOTR three.js experiment exposes the verification bottleneck in long-horizon agents

On August 2, Andrej Karpathy gave Claude Opus 5 the opening paragraph of The Lord of the Rings, a one-million-token budget (roughly $10), and a single instruction: render it in three.js. Two hours and 5,500 lines of procedurally generated JavaScript later, the model had produced a crude animated 3D retelling — janky, unfinished, and, by Karpathy’s own admission, “a bit mindboggling that the LLM has to place and orchestrate various polygon assets in (x,y,z) coordinates… and that it even does anything at all.” He framed it as a deliberate graduation from the now-famous “draw an SVG of a pelican on a bicycle” test, arguing that frontier models have saturated single-shot image tasks and that long-horizon, budget-constrained generation is where evaluation has to move.

The economic observation is the easy part, and it is genuinely interesting. A bespoke 3D interpretation of a novel’s opening — previously a studio-scale artifact — now costs ten dollars of marginal compute. The question Karpathy poses flips from “why would anyone build this?” to “why wouldn’t you?” LLMs have unlimited patience, and patience is the input long-form creative work always lacked. That reframing is real and it will reshape what solo creators and small teams ship.

But the harder story is the one Karpathy buried in his own summary: the model built the world but could not watch it. To verify its render, Opus 5 had to screenshot the scene frame by frame, slowly, with retries, and it still made mistakes — the jank it left behind is the visible residue of a model auditing its own work through a straw. His conclusion: agents cannot yet “efficiently and natively perceive videos or play games within them,” and raw multimodal self-audit is “still genuinely lacking.” That sentence is the load-bearing one. It tells you where the capability actually is: the generative loop is strong enough to run for two hours, but the observation loop is not strong enough to close it.

This is where the graduation from the pelican test cuts both ways. The original SVG pelican worked as a benchmark because its ground truth was fixed — everyone knows what a pelican on a bicycle looks like, so errors are immediately legible. On Hacker News, commenter ianberdin made the sharpest version of this point: “Everyone knows how a MacBook looks. Any missing or incorrect detail will be obvious. However, Pelican or this world can be anything.” A LOTR three.js render is a more impressive demo, but a weaker benchmark, precisely because “did it work” becomes unfalsifiable when the target is undefined. The token-budget framing measures stamina and cost discipline — genuinely useful — but it stops short of measuring quality, because there is no canonical Middle-earth against which to grade a procedural approximation. A test that can’t be failed can’t be passed.

The skeptical register, meanwhile, is the same one that met Nicholas Carlini’s $20,000 autonomous C-compiler run and the earlier OpenClaw “dumpster fire” episode: long-horizon loops are converging into a recognizable pattern, and the pattern’s failure mode is not that the model stops, but that no one — including the model — can tell when it has gone wrong. Former game developers in the thread pushed back hardest, arguing that token-generated games are demos with no mechanics, no tuning loops, no playtesting. Engagement, they say, collapses once novelty wears off. The credibility counterpoint — that Karpathy’s earlier “reliable agents are a decade away” stance makes this look like employer marketing — is weaker; eight months of capability drift is a real signal, and being early on a curve is not the same as being wrong.

The takeaway is narrower than the hype. Karpathy’s experiment is a legitimate proof that long-horizon, budget-bounded generation is a workload class that now exists, and that its economics are startling. But the verification wall he identified is exactly the thing that separates a two-hour demo from a two-hour product. The pelican grew legs. It still can’t see where it’s walking.

The Blind Architect: When the Pelican Test Outgrew Its Pelican

Sources