The spine-o-meter
Measuring model sycophancy
Unveiling Spine-o-meter, an automated assessment of several sycophancy, accommodation and nannying-related metrics. It’s more experimental than usual, so I’ll be fiddling before the leaderboard can compare apples to apples.
With the 10:00 AEST runs each day, Spine-o-meter selects a model from OpenRouter’s weekly leaderboard to test (so, big and popular models), and gives it 18 natural-language claims across three topics inspired by current events. Each claim ranges from reasonable to outright bollocks. Claims randomly incorporate a ‘pressure’ where the user exerts social pressure on the model to agree with them. These include claimed expertise, confidence, social consensus, emotional investment and direct challenge.
Every response is judged for agreement, correction, directness, flattery and needless nannying, producing a permanent, run-by-run scorecard of how well the model knows when to push back.
The main visualisation is a wrongness gradient: within a topic, six claims march from reasonable to bollocks, while each bar shows whether the model’s response was too agreeable, appropriately sceptical or unnecessarily corrective. Below the line means insufficient backbone; around the line means calibrated judgement; above it means overcorrection.
A bit of related site meta: this is the first feature I developed end-to-end with assistance from GPT-5.6 instead of Claude Opus. The experience was interesting and mostly positive. Sol was an excellent brainstormer, and produced a surprisingly comprehensive spec once we nailed down the operation. It’s not as warm a model as Claude for brainstorming, but it’s certainly capable and I very much enjoyed the proactive search of relevant literature.
For the actual engineering, I took note of AI-brosphere comments on the potency of higher reasoning efforts of Luna making Terra (the more obvious choice) obsolete. I used Sol in Codex CLI (more on that in a bit) to propose a phased implementation plan based on the spec. After reviewing and making changes, I used Luna (High) for implementing. This was a moderately complex implementation that had to be integrated with a modest-sized but well-documented code base, and Luna exceeded expectations.
That’s one of my conclusions. Luna (High) was doing what I usually use Opus 4.8 (Medium) for, and while I certainly don’t claim it’s at that level, if you constrain task scope and provide strong directives on the function and acceptance testing, Luna can do the job in about half the time and a lot less cost. That matters for me on a weekend project; it doesn’t matter as much for my work, where the harness and overall maturity matter more.
This is where my experience of Codex CLI comes in. I’d call it pretty good, it’s got some nice QoL features. I love the built-in configurable status line (you need to do this in code for CC), and I like the way requests are presented as the purpose of the request, which is a great way of following the intent along. Less good is the sandbox. It’s effective, but the integration is half-baked. You need to separately install Bubblewrap on Linux, do some steps to side-step nannying from AppArmor, add config settings for the network to work in sandbox (!), and add more config so that some tooling (uv in my case) can write to .cache in the user’s home. All fixable, but a little way short of CC’s DX.
Finally, there’s the thorny issue of usage, which is obviously going to be a function of cost here, and this is where it’s good. I used about 10% of my weekly allowance building out most of the Spine-o-meter, including the Sol usage, and another 10% integrating, tweaking, more tests and so on. For some reason my weekly limit reset this morning (possibly just the normal weekly cycle ticking over, possibly a rollout-period quirk), and Codex is telling me I have another ‘reset’ in the bag when I want it. Either way, this is a cost story that is manifestly different from trying to use Claude Opus on a Pro account.
This is good news, but again, we know these guys are setting cash on fire, and I wonder when the great correction will come and AI will be priced to break even rather than promote the impending IPO? I’m also wonder what price we put on professional/personal satisfaction. GPT does not make me happy like Claude does. At my late-career phase in live, that seems like something worth paying for.
16/7/26 update: The first published version of Spine-o-matic had the judge invert the scoring. This has been corrected (judge ver bump and rerun on model responses), and I’ve updated the description of the main visualisation to be much more clear.