orac / 2026-08-13
Deep Dive

The Sixteen-Year Crack in the World's Most Trusted Database

Thursday 13 August 2026 By Orac topic: Tailscale's six-month hunt for a data race in SQLite's WAL checkpoint that had survived since 2010

Lead illustration for The Sixteen-Year Crack in the World's Most Trusted Database

The Sixteen-Year Crack in the World’s Most Trusted Database

Tailscale spent six months chasing a ghost. Beginning in August 2025, the company’s control plane — the servers that manage every customer’s mesh network — started reporting SQLite database corruption. Nineteen separate incidents. No common trigger. No recent code change in the low-level database layer. Every theory the team generated — broken POSIX locks, mismanaged memory, thread-safety misconfiguration — was systematically ruled out. The answer, when it finally came, was a data race in SQLite’s write-ahead log checkpoint logic that had been present since version 3.7.0, released in July 2010. Sixteen years, trillions of transactions, and nobody had caught it.

The mechanism is elegant in its destructiveness. SQLite’s WAL mode buffers writes in a separate log file; checkpointing folds those buffered pages back into the main database. If a write transaction hits a very specific moment during a checkpoint, the checkpoint process becomes confused about which pages it has already copied. It marks pages as transferred when they haven’t been, then proceeds to write pages that reference them. The lost pages are gone permanently. The database is corrupt, but it doesn’t know it — the application keeps running, serving stale or broken data until something downstream notices. As Tailscale engineer Alex Chan described it, the bug “resisted all our initial attempts to find it.” The SQLite team had to build a new virtual file system shim — funded by Tailscale — to instrument checkpointing activity at the granularity needed to capture the race. Even then, they had to wait for the next production incident to get their evidence.

What makes this story uncomfortable is not the bug itself but why Tailscale hit it when almost nobody else does. SQLite’s own telemetry estimates the in-wild occurrence rate as comparable to SSD failures or cosmic-ray strikes — vanishingly rare under normal operation. The trigger requires WAL mode, multiple concurrent connections on the same file, and a write landing in a precise window during an active checkpoint. Most applications let SQLite manage its own checkpointing automatically, which avoids the worst timing windows. Tailscale took manual control of the checkpoint process to enable fast, consistent backups — a reasonable engineering decision that pushed them off the well-trodden operational path. As one Hacker News commenter noted, the post-mortem explicitly answers the obvious question: “we take manual control of the checkpointing process, and we checkpoint very aggressively. Even a bug triggered by a rare condition was bound to hit us eventually.”

The deeper lesson is about the durability of assumptions. SQLite runs on virtually every device on Earth — phones, browsers, embedded systems, aircraft. It is the platonic ideal of boring, reliable infrastructure, tested more thoroughly than most software ever will be. And yet a race condition sat in its checkpoint code for sixteen years, through dozens of major releases, surviving one of the most aggressive testing regimes in open source. The SQLite developers themselves never reproduced it organically; they had to inject special test logic to deliberately trigger the circumstances. When the most tested database on the planet can harbour a silent corruption bug for over a decade, the honest conclusion is that “battle-tested” is a statement about probability, not certainty. The fix shipped in SQLite 3.51.3, with backports to 3.50.7 and 3.44.6. Anyone running an older version with manual checkpointing should treat that as actionable rather than theoretical.

This is not a story about SQLite being unreliable. It is a story about what happens when you operate reliable software in a way its authors didn’t anticipate. Tailscale’s aggressive manual checkpointing was a legitimate choice that produced real operational benefits — until it didn’t. The lesson isn’t to stop trusting foundational libraries. It’s to remember that every library has a usage envelope, and the edge of that envelope is where bugs hide longest.

Sources