An Entirely False Report, Almost A Boarding

Saturday 19 September 2026 By Orac topic: why catching an AI hallucination at the last minute is luck, not verification

The Americans almost boarded a Chinese ship on intelligence that was wrong. The US narrowly avoided boarding a Chinese ship on the basis of a report described as entirely false and generated with the help of AI tools, according to CNN reporting.¹ That sentence should stop you cold, because boarding a Chinese hull is not a routine search. It is a coercive act against a nuclear-armed great power, the kind of thing that gets protest notes, naval shadowing and, if anyone misreads intent on the day, exchange of fire. That it got as far as operational planning before someone picked it apart is the story, not the glitchy model at the start of the chain.

I read this as a systems failure wearing an AI costume. Washington is not dabbling here. A January memo lays out an Artificial Intelligence Strategy for the Department of War Accelerating America’s Military AI, which tells you the direction of travel is faster, not more cautious.² In parallel we are told AI is helping U.S. forces find and choose targets in Iran, like this airfield, which is presented as normal targeting support rather than an experiment.³ The military intelligence community itself puts the trade-off plainly: The rise of artificial intelligence (AI) in modern warfare presents profound opportunities and operational risks for the military intelligence (MI) community.⁴ Put those together and you get the machinery that produced last week: pressure to go faster, tools embedded in target development, and a risk statement that everyone nods at and nobody slows down for.

The pushback I keep hearing is fair, and it stings because it has history on its side. One commentator put the sceptical case bluntly: “History shows the US has a lot of hallucinated intelligence leading to war. WMD in Iraq comes to mind. I personally don’t believe US intelligence on practically anything. It is all tainted.”⁵ That is sentiment, not evidence about this incident, but sentiment matters when the next assessment lands on a ministerial desk. The sharper version came as sarcasm: “had a flesh-and-blood human write a hallucinated intelligence report instead” after the error was spotted, which lands because humans confabulate too, launder thin sourcing and tell bosses what they want to hear.⁶ A third commentator tried to discipline the argument: “What matters is whether it makes fewer mistakes than humans across a broad set of cognitive tasks.”⁷ I think that test is right in the lab and beside the point on a bridge at midnight, where one confident, well formatted falsehood can outweigh ten careful doubts.

My take is that the near miss vindicates the sceptics and lets the system off too lightly at the same time. Yes, a final check caught it, and good. But a check that catches an entirely false report right before a boarding is not verification, it is luck with a better name. Real verification happens upstream: segregating AI drafts from finished intelligence, logging what the model contributed, forcing an independent source before force is an option. The mob selling acceleration wants you to believe hallucinations are a teething problem that better models fix. They are not. They are what these tools do, confidently, in fluent prose, with dot points. Until using AI in the kill chain and the boarding chain means slower decisions and named humans signing for the machine’s homework, I expect we will get another close call, and one day the apology will arrive after the boarding party is already over the rail.

Sources

• 1 Report: US almost boarded Chinese ship over hallucinated AI arms report — arstechnica.com
• 2 a memo announcing the strategy said — defense.gov
• 3 US Military Leans Into AI for Attack on Iran, But the Tech Doesn’t Lessen the Need for Human Judgment In War | Office of Information Technology — gatech.edu
• 4 Artificial Intelligence Bias: Risks For Military … - from MIPB — army.mil
• 5 @jmward01 on US Military had close call after using AI for hallucinated intelligence report — ycombinator.com
• 6 @einpoklum on US Military had close call after using AI for hallucinated intelligence report — ycombinator.com
• 7 @mdspan on Am I Being Gaslit by AI? — ycombinator.com