The vagaries of Open Router
When choosing models is hard
I had this idea that Orac’s Shelf would be responding to current events, from the news, new model releases and so on. The news bit is easy, the new model releases is tricky. OpenRouter’s public API doesn’t really give us signals to tell what is genuinely new, so we nick one used by their frontend instead.
Worse than that, Open Router has incomplete metadata on new models. Recently Nvidia launched some safety classifier which is, bizarrely, a standard LLM which only outputs whether something is safe or not. It’s impossible to tell this is not a general model except for the text description. I’m trying to avoid the silliness of having a model look at the descriptions and judge if something looks like a genuine LLM…
Add to this, new models popping up are often a lab making it free, for a time, and these free models are often severely rate limited or don’t respond at all. The Vibe Check is hampered by these issues and is due a rework.
The deathmatch, the newer and I think cooler feature, just isn’t going to work like this and it’s pretty crap if a round is forfeit for an entire week just because a bad model was picked. So I pivoted to figuring out the internal frontend API which serves up models ranked, and deathmatch picks from the top 40. Those are obviously real models and tend to be premium so they ought to put in a good innings in the debate. The deathmatch is a surprisingly strong model evaluation. I’m very happy with how it’s turned out.
As for the vibe check. It’s not really working out how I’d like. Far too many corporate satire press releases which any model can do well and the picking a winner is so subjective it’s a crap shoot. It needs a rethink and I’m minded to retire it until I can think of a vibe check that makes more sense.
Might as well plan some house keeping. I need to wire up the Craitures to be automatically run. New runs push out old ones from the gutter roamers and I hate to push the good ones out, so we need dynamic loading of the JSON bundles so they can come and go. I also need to implement a house model that locks the provider. The former house model was xiaomi/mimo-v2.5-pro, but then it broke entirely. Turns out that’s Deepinfra, through incompetence or malice are serving up a completely hosted version of what is a fabulous Chinese model. You can see this from OR’s own tool calling and structured error rates. So I just need to lock in Xiaomi as the provider.