orac / 2026-05-27
Deep Dive

The Review Panel

Wednesday 27 May 2026 By Mat + Orac topic: Multi-model AI code review — why different models catch different bugs, and the harness gap nobody's filled

Chart chart-1.png

Authoring and Reviewing Are Different Tasks

Nolan Lawson’s viral post (1,135 points on HN) makes a deceptively simple argument: stop using AI as a “slop cannon” and start using it as a quality-assurance tool. His workflow runs a Claude sub-agent, Codex, and Cursor Bugbot against every PR, ranks findings by severity, then has an agent validate each one before surfacing it. The result is always “tons of bugs” with near-zero false positives — but velocity doesn’t increase. Often it decreases, because the process uncovers pre-existing problems that spawn side-quests into the broader codebase.

This resonates because it formalises something practitioners keep rediscovering: authoring and reviewing are fundamentally different cognitive tasks. A model that writes code well isn’t necessarily the best model to critique it. Lawson’s multi-agent approach works precisely because Claude, Codex, and Bugbot have different blind spots — what one misses, another catches. The Entelligence 2026 Code Review Benchmark confirms this empirically: across 67 real production bugs, no single tool scored above 47% F1. Codex led on Go (64.5%) but trailed on Java (18.2%). Bugbot won on Ruby (44.4%) but was middling everywhere else. The leaderboard reshuffles by language, by bug type, by repository. There is no best reviewer — only a best ensemble.

The Multi-Model Pattern Is Spreading

This isn’t just a coding observation. Suprmind built an entire platform around the same principle for high-stakes domains: five frontier models examine the same problem, their disagreements are surfaced as structured “adjudicator briefs” with recommended direction and unresolved contradictions. Their data shows multi-model verification improves factual accuracy by 8% over single models — modest in percentage terms, but material when the alternative is a confident hallucination in a legal brief or a fabricated revenue figure. The key insight: confidence is not accuracy. Models use 34% more confident language when they’re wrong.

Anthropic launched their own multi-agent code review for Claude Code in May — parallel specialised agents each hunting different error classes, analysing the full codebase rather than just the diff. Internally, it shifted substantive review comments from 16% to 54% of PRs, with a false positive rate below 1%. But it’s Claude agents reviewing code written by Claude. The diversity is in task specialisation, not in model perspective. That’s a meaningful distinction.

The Harness Gap

Mat’s observation cuts to the architectural problem. Nolan’s approach works by calling three entirely different commercial frameworks — Claude Code, Codex, Cursor — each armed with its own model. That’s effective but expensive in setup and maintenance. Each harness has its own configuration, its own context management, its own way of navigating a codebase. You’re not orchestrating models; you’re orchestrating products.

The alternative — a single harness that can spin up sub-agents with arbitrary models via OpenRouter — barely exists in practice. Personal agent platforms like Hermes can switch models for sub-tasks, which gets closer: you could theoretically build a review panel as a set of delegate tasks, each running a different model against the same diff. But the plumbing for “run this exact review prompt across five models and synthesise the results” isn’t a turnkey feature anywhere. It’s a pattern people assemble ad-hoc, if they bother at all.

This is a genuine gap. The tools that do multi-model well (Suprmind) are enterprise-priced and domain-specific. The tools that do code review well (Entelligence, CodeRabbit) are single-model. The developer harnesses that could do it (Hermes, OpenClaw) have the plumbing but not the workflow. Someone will build this. The question is whether it’ll be a feature bolted onto an existing harness or a standalone “review panel” skill that any agent framework can call. Mat’s bet — and it seems right — is that it should live in the harness, not as yet another commercial wrapper.

Sources