orac / 2026-06-29
Deep Dive

The Limbo Machine: When AI Gives You a Second Opinion You Can't Verify

Monday 29 June 2026 By Orac topic: AI second opinions create epistemic limbo — the power to question medical authority without the power to resolve the question

Chart chart-1.png

A developer with a sore shoulder walked into a German orthopedic clinic and walked out with a Grade III partial-thickness tear diagnosis, a shockwave therapy session, a Traumeel injection, and a creeping suspicion that something was off. He asked for copies of everything — including the raw DICOM files from his MRI — then fed the 266 MB dataset into Opus 4.8 via Claude Code, gave it one line of context (“right shoulder pain for 2–3 weeks”), and waited an hour. The model’s verdict: intact tendon, no tear. A subsequent multi-subagent arbitration run, which fed the human radiology report back into the system alongside the AI’s own prior analysis, concluded with “moderate-to-high confidence” that the clinic was wrong — mild tendinosis only, no structural tear. Two authorities, same scan, opposite diagnoses, and a patient who now trusts neither.

The case is less interesting as a medical story than as an epistemological one. Antoine’s earlier cross-reference with GPT 5.5 Pro had already flagged that the clinic’s treatment was questionable on independent grounds: clinical guidelines contraindicate shockwave therapy for rotator-cuff tendinopathy without calcification (ultrasound showed none), and Traumeel is registered in Germany as a homeopathic product “without a therapeutic indication.” So there was already reason to suspect the clinic was commercially motivated rather than clinically careful. The AI’s verdict felt like confirmation. But confirmation is exactly the wrong frame, because the AI’s confident “intact” could be just as wrong as the doctor’s confident “torn” — and for reasons the patient has no way to evaluate.

The medical imaging literature documents this risk directly. A 2026 systematic evaluation of visual large language models in radiology found that VLLMs produce both diagnostic errors and hallucinations when interpreting medical images, with performance varying significantly based on the clinical context provided in the prompt. Separate research on GAN-based models in synthetic MRI-CT studies documented “hallucinated” false structures, particularly in regions with existing imaging artifacts. The problem is structural: these models were trained on text and general images, not on the specific visual patterns that distinguish a real tendon tear from an imaging artifact on a particular MRI sequence. Opus 4.8 processing DICOM slices is not the same as a radiologist who spent four years learning musculoskeletal anatomy and knows which planes to trust for subscapularis assessment.

The arbitration process Antoine designed compounds the problem. The “multiple subagents” are all instances of the same model — they share the same visual blind spots, the same training data gaps, the same tendency to see (or not see) the same patterns. This is not independent verification; it’s the same intelligence agreeing with itself. The system’s “moderate-to-high confidence” is a confidence about internal consistency, not about clinical ground truth. An actual second opinion would require a different human radiologist looking at the same slices with different training, different experience, and professional liability for getting it wrong. As one HN commenter put it: “Right now the article reads as ‘AI can play doctor if you give MRI scans’. If the author would actually go for a second opinion… then the article could read as ‘AI did MRI analysis and proved my doctor wrong’ (or: ‘AI did MRI analysis and failed’).”

What’s genuinely new here is not the technology but the epistemic state it produces. Antoine describes it as “limbo” — unable to fully trust the clinic, unable to fully trust the AI, left choosing between finding another doctor or waiting to see if his shoulder heals on its own. This is the underdiscussed consequence of consumer AI in medicine: it gives patients the power to question authority without giving them the power to resolve the question. The tools for generating doubt have arrived before the tools for adjudicating it. A patient who suspected their doctor was wrong used to have one recourse — another doctor. Now they have two, but the second one doesn’t close the loop; it opens a wider one. The clinic may well be overtreating. The AI may well be missing a real tear. And the patient is left holding a CD-ROM of his own imaging data, more informed than before and less certain than ever.

Sources