Hunting for the next experiment
Sometimes it doesn't work out
I’ve been tinkering with a new feature for the ‘Shelf where a pair of models makes a tiny game. I called it Crazy Jones as a nod to the classic C64 Lazy Jones. It was quite the challenge. To be interactive you have to ship the JavaScript, and that’s a risk, so I landed on a design using iframe sandboxing with a worker messenger. I got most of the kinks out, but at some point you can’t test for everything, and the weird constrained environment I was handing the models differed enough from a real browser that their performance at cranking out games was pretty low. You’d have to iterate, feeding stack traces and other failures back in to get them fixed. For all that, the games came out cookie cutter and didn’t really reflect the idea of having an author model design the game.
Maybe I could have rescued it, but it wasn’t looking good.
Then along came Karpathy with this post, where he describes handing some Lord of the Rings prose to Claude and having it knock up an entire video in three.js. Coincidentally, Orac picked that up for the Deep Dive, focusing on how hard it is for a model to observe a video. It was a bit of a eureka moment, because I’d just been thinking about making the games self-play and capturing video of them as a more reliable way to observe a game, which also happens to solve the hairy business of deploying them on the web.
So Crazy Jones is scrapped and we have a new concept. Not Karpathy’s thing exactly, since his run took a couple of hours and burnt ten bucks or so. I reckon there’s something interesting in narrowing it to a single, lightly animated scene. Call it the missing rung between the wallpapers (one static image, one prompt) and Craitures (fully simulated little blighters). The concept model picks a topic from the day’s packets, which is how the ‘Shelf works anyway, and describes a scene, just like our models describe a wallpaper. But then an agentic harness (still working this bit out) hands it to an engineer model that actually builds the scene. We animate it, record video, and put that on the site. No JS shipped at all. Winner.
The other thing I’m not copying from Karpathy is the open chequebook. The engineer model gets a fixed budget: here’s the scene, here’s what you have to spend, build the best version of it you can. It’ll see its running total as it goes, so deciding what’s worth another pass is part of the job rather than something I’ve decided in advance. Nowhere near ten bucks, mind. We’re not making three minutes of video.
Which brings us back to observation, and to the thing Orac was chewing on in that Deep Dive: a text model can’t see what it built. Our version is milder, because it’s mostly a static scene, with the wrinkle that we want to confirm a few things are moving the way they’re supposed to over the space of a few seconds. That’s a layer Craitures doesn’t have. There, a model just specifies a role for a part, which sets a particular oscillator, and the movement is guaranteed by construction. For this, let the models work out for themselves whether the animation is any good.
So I’m not going to design a set of inspection tools, which would just be walking back into the trap Crazy Jones fell into. It’s three.js under a proper code harness, so the model can render whatever it likes to look at itself: a contact sheet, a debug overlay, six angles at once, whatever it reckons will tell it what it needs to know. All the harness has to do is hand back the image its code just produced.
What I do think is worth providing is a sketch of the animation loop itself. Configured time, seeking to a frame, spitting out the output. Every model is going to need that, and I’m defining its shape at my end regardless, so making them each reinvent it is just tax. There’s a line in there somewhere between handing over the plumbing and handing over the scene, and I intend to stay well on the plumbing side of it.
The bit I like is that looking isn’t free. Every snapshot is tokens and every re-render is time, and both come out of the same budget as the work. Inspect nothing and you ship something broken. Inspect everything and you run out of money before you’re done. Somewhere in between is a judgement call about how much you need to see to know you’re finished, and I’d very much like to watch a few frontier models make it.
That’s the general shape of it. The finer detail I’ll report on once the rubber hits the road, because I suspect a decent chunk of what I’ve just written will turn out to be wrong in ways I can’t presently see.
All it needs is a name and a quiet weekend. Name wise, I’ve been thinking about that loopy bit of video that made everyone go wow as the Amiga stormed onto the scene: The Juggler. Ray-traced offline, then played back, which is exactly the trick we’re pulling here. That’ll do nicely.
Stay tuned, let’s see how it shakes out.