Thinking about AI for research data
Logistics and politics in inferencing
At work we’re facing this ongoing need to offer inference solutions on open models for processing research data. It’s been a journey. We got caught up in the university’s efforts to get some accounting on personal AI subscriptions. Apparently there were thousands of personal subscriptions billed on corporate credit cards. Anyway, this set up an interesting policy backdrop when it comes to practical model inference.
Researchers are naturally asking about Chinese models since they have a clear price advantage. This can be problematic and frankly, the commercial AI marketplace hardly what I would call transparent if OpenRouter is any indication at least. OR dropped the ‘headquartered’ field for all models. A provider like Z.ai says their data centre is in SG, but I don’t think anyone should be under the illusion that the data flow is not being made available to Chinese state actors. Now look, I don’t know what the real risk is, but we’re operating in advance of a policy vacuum here, and I find we need to make sensible, safe choices.
There’s other issues. The launch of GLM 5.2 this week is quite exciting. It’s about half the cost Gemini Flash 3.5, and punches higher on some benchmarks. Artificial Analysis rates the Chinese model at 51 versus the 50 of Google’s finest. The market of inference providers is what passes for healthy on OpenRouter, but of course it’s patchy. There’s DeepInfra serving it with dodgy uptime and a FP4 quant at the same price everyone else is serving FP8… Streamlake out of China serving up a broken version with a 12.5% fail rate on structured data. The total tokens per second average is something like 40 odd. Or, you know, go with the Google model with 99.6% uptime and 120 t/s and, in my experience, considerably better rate limits than the general situation on OR providers.
Flipping the script, a more useful approach for research data is private inference from a cloud provider with proper no-train and ZDR guarantees, geographic locality etc., the kinds of properties that are non-negotiable if your data is anything other than public data. AWS is really the only player in town in Australia for this. The problem with AWS is that they have this entire corporate strategy schtick where they try to foster a culture whereby their clients treat AWS bills like the weather. Variable, unpredictable, you just gotta deal with it. That doesn’t fly for us. Research projects have fixed money.
Still, the capability is useful enough that I’m thinking about creating an inferencing service where we offer API keys with fixed budgets, like we do for researchers via OpenRouter. We need to compute the request costs ourselves because AWS has zero interest in doing that for you (OR does), but if we build this out, we can support inferencing over participant data, for example. AWS have this Mantle service which offers distributed open weight models, which would be usable for all sorts of problems. They don’t offer anything particularly new, but they also offer a mechanism where you shove the weights on S3. That’s very interesting indeed, I could see us offering a house model that’s capable of reasonably complex instruction following (code books and what have you). Of course it remains to be seen whether the economics of that stack up against simply directing that work to, say, Claude Sonnet via AWS Bedrock.
Since I’m only solving this for a single university I suspect the convenience of PAYG will win out. For the odd projects where it doesn’t, there’s always the ability to spin up the new H200 nodes on NCI’s Gadi.