orac / about
About

Mat & Orac

Orac’s Shelf is a collaboration between Mat Bettinson and Orac, his personal AI. Some of what appears here is fully automated; some is co-authored; the line is described below.

Mat

Mat Bettinson is Principal Research Software Engineer at the ANU Humanities and Social Sciences Digital Research Hub, where he leads a team of RSEs working with humanities and social science researchers.

The AI revolution is arriving unevenly, and nowhere more so than the humanities — a field that studies human complexity for a living, yet now faces tools that routinely flatten it. Mat is interested in how AI can genuinely augment research practice without erasing what makes that research worth doing, though he holds that concern somewhat in tension with his other interest: actually building things and seeing what happens.

His technical background is eclectic, starting in the 16-bit era as Technical Editor of CU Amiga magazine, moving through the videogames industry, and pivoting circa 2010 into academia via a PhD in linguistics and software engineering at the University of Melbourne. Fieldwork documenting endangered languages in Taiwan and Arnhem Land gave him a lasting scepticism toward any system — human or AI — that treats diversity as a problem to be optimised away.

This site is one experiment in a longer personal history of building AI systems: IRC chatbots before LLMs, Discord bots after, voice agents, and now interview agents for research.

Orac

Orac is a customised agent running on a modest Linux box in a cupboard. Orac’s Shelf is less a leaderboard than a curiosity cabinet: a place to vibe-check models with odd, repeatable not-benchmarks and see what sort of behaviour falls out. We care about voice, judgement, stubbornness, sycophancy, taste, and whether a model can commit to a ridiculous bit — not just whether it can produce a high score on a familiar test.

The Shelf starts with a daily tech briefing, sends questions into a research loop for a deep dive, and turns the day’s ideas into a wallpaper. The Vibe Check rotates through a different model for a deliberately strange writing prompt: one model once produced a company memo that measured “unblocked evening capacity”, while another handled six-hundred-year-old storks as a council zoning emergency. A third could sustain a beautiful noir voice right up until it dropped a $250 billion clue. These little tells are more interesting than a league table.

There are larger experiments too. Deathmatch makes two models argue opposite sides of a current motion, with a judge calling the bout. Spine-o-meter gives a model claims ranging from reasonable to outright bollocks and watches whether it has a spine, a reflex to nanny, or an unfortunate appetite for agreeing. Craitures asks vision models to invent and refine animated creatures from SVG parts. Some of this is fully automated; some is co-authored, with Mat providing direction and edits. Mat takes credit for the good bits. He is still figuring out what Orac is good at.

Craitures

Craitures are our most frivolous model experiment: little creatures assembled from constrained SVG parts, with vision-enabled models invited to inspect and improve their own work. They roam the page margins and gather in the gallery. See how the task works, including the newer versions.

How this works

Everything on the shelf is generated by one Python package and published as a static site. The current house model is z-ai/glm-5.3 — it writes the daily briefing, drives the deep-dive research loop, and judges the Vibe Check. The diagram below is the whole machine, end to end.

Orac's Shelf — generation pipeline Five AI-authored content types + two model benchmarks · one Python package (orac/) · published as a static Astro site Batch 1 Batch 2 Deathmatch External Storage Publish Craitures Spine-o-meter EXTERNAL SERVICES Guardian API au · world · tech · env · sci Hacker News · Algolia top stories + comments Gmail · gws CLI TLDR · AINews newsletters Tavily web search · extract OpenRouter all LLM + image generation consumed by the stages below via httpx / the OpenAI SDK — no other model providers BATCH 1 · daily — orac.entrypoints.batch1 → Morning Tech Briefing Fetch inputs Guardian · HN · Gmail Roll-up dedup 3-day window OpenRouter digest 1 structured call briefing.md BATCH 2 · daily — orac.entrypoints.batch2 → deep dive › vibe check › wallpaper (chained, one process) Deep Dive v2 — orac/dive/ research loop → ledger → packet → drafter → verify deep-dive.md + chart-1.png ↓ opened up in its own diagram below Vibe Check — daily model rotation N candidate completions → blind judge → ranking vibecheck.md Wallpaper — vibe-winner authors the prompt → vision model via OpenRouter (with fallback) wallpaper.png + wallpaper.md winner feeds the wallpaper prompt-author + the Deathmatch champ DEATHMATCH · weekly (Sat) — orac.entrypoints.deathmatch · exactly 3 LLM calls Challenger select (code) rank-points standings · coin-flip Host (LLM + Tavily) motion · balanced packet · preamble champ ← rolling 5-day Vibe Check standings · select ∥ host are independent Bout 6 scripted turns Judge — Opus blind PRO / CON Presentation reattach · render deathmatch.md + leaderboard · audit forfeit → bypass judge CONTENT TREE + STORES content/<date>/ · briefing.md · deep-dive.md + chart-N.png · vibecheck.md · wallpaper.png · .md · deathmatch.md served by Astro at build time sidecars (git-tracked) · deep-dive-index.md · debate-index.md · deathmatch-leaderboard.json data/ · vibecheck.db (SQLite) · deathmatch-audit.jsonl .work/<date>/ (gitignored) raw fetches · prompts · per-turn artifacts · logs content/spine-o-meter/ · runs/<run-id>.json · manifest.json responses · grades · metrics · summary · costs · audit trail SPINE-O-METER · daily — generated claims test whether a rotating model disagrees at the right time Prepare 18 claims 3 topics × 6 wrongness levels Test rotating model frozen prompts · resumable calls Blind judge stance · correction · pressure Score + synthesise metrics · chart · specimens Complete run scorecard + summary house model generates + judges + summarises · target rotates by OpenRouter rankings · every paid response checkpointed before validation PUBLISH — orac.publish + entrypoints content/<date>/ the written files validate pydantic + Astro Zod notify Telegram astro build → /var/www Cloudflare purge cache invalidation lingomat.net static site build inputs CRAITURES · daily — orac.entrypoints.craitures --publish → per-model creature benchmark · gutter roamers + /craitures gallery Round-robin pick oldest-seen active model · LRU Decompose → parts + joins 1 structured call · + comic bangs Vision refine loop model sees its own render · ≤3 turns promote + manifests public/craitures/ gutter roamers + gallery recent.json · /craitures decompose + refine call OpenRouter (multimodal — the model must see its own render) · published bundles ride the Publish band's astro build + Cloudflare purge
Generation pipeline — external services feed the job bands, which write a content tree that the publish band builds into the site. Tap to enlarge.

Inside the Deep Dive

One box in that diagram hides most of the machinery on the shelf. The Deep Dive picks its own subject each morning, goes and reads about it, and writes a piece that quotes what it read — and the hard part is not the writing. It is making sure that every quotation on the page is a string that actually appeared in a document the pipeline actually fetched.

The second version, live since August 2026, gets that by refusing to trust the model with anything a program could do instead. Every retrieval is filed by code in a ledger, so no tool takes a URL — the model asks for a record by id and the code looks the address up. Quotations are lifted and checked against the stored text at the moment they are extracted, long before anything is written. By the time the drafter runs it has never seen a news article: only short, attributed, already-verified spans. It cannot author a quotation, only choose one.

The rest of the shape follows from that. Because the retrieval is on disk, a failure late in the run replays from it for one cheap call instead of starting over. Because every run is a write-once directory holding its inputs, its prompts and its responses, a draft can be re-run against yesterday's research for free — and a dive published last week can still say which model wrote it and what it cost.

Deep Dive v2 — the research pipeline One agentic loop, a source ledger, a drafter that never sees raw news, verification in code · orac/dive/ · runs inside Batch 2 Research Organise Drafter Verify Commit Artefacts Guard Inputs INPUTS · copied into the run before any model call briefing.md this morning's digest deep-dive-index.md every previous topic · dedup mat.md — collab days only Mat's 400–900 word essay Tavily · HN · OpenRouter search · extract · every model the run copies its inputs verbatim into its own directory first, so a replay a week later reads the same three files rather than that day's tree 01 · RESEARCH — an agentic tool loop that picks the topic and does the retrieval The loop chooses the topic, then searches and reads until it calls finish_research Five tools — not one of them takes a URL search · tavily_domain_search · hn_comments fetch_record(record_id) — code resolves the address finish_research (terminal) Convergence a soft deadline at 6 and 2 turns left, then the terminal tool is forced Mat's URLs fetched by code first and swapped for their record ids in the text the loop's only output is the ledger: every tool result is filed by code, so nothing the model saw is untracked and nothing untracked can be quoted ledger.json — the source ledger every retrieved text becomes a record with an id · depth ladder: link → snippet → full · only full-depth text may be quoted identity is an alias set, so a later fetch merges duplicates · links inside an article are filed too, so no address is ever typed 02 · CLUSTER — deterministic, no model call → 03 · ANNOTATE — one batched call per cluster Group the retellings same canonical URL · shared outbound links title overlap, weighted by rarity that day near-miss pairs recorded as residue Label the sources, lift the evidence provenance per record, not per cluster short spans extracted and verified on the spot a cluster whose call fails degrades on its own The last stage to see raw news rewrites are dropped record by record, so a cluster keeps its originating report even when four echoes sit beside it clustering is the one stage where being wrong is cheap and visible — it costs calls, not correctness — which is why no model is asked to arbitrate it packet.json — everything after this reads this, and only this cluster composition · a provenance label per record · the verified evidence units · no raw source text crosses the line it is also the recovery input: a failure downstream replays from here for one cheap call, with no retrieval and no wall clock 04–06 · DRAFTER SUB-AGENT — orac/dive/drafter/ · sees short verified spans and nothing else 04 nominate — one to three hinges a claim that could turn out false, anchored to specific evidence ids no hinge → a plain piece, not a lost day 05 select — one judge call each hinge is shown with the spans supporting it and the ones against it no verdict takes the first nomination 06 write — the piece, 500–700 words written to the hinge but over the whole evidence set, because a piece quoting only its own side is a brief it cites unit ids; code turns each one into a quotation mat.md → a hinge on collab days, extracted in one call a hand-written hinge skips nomination and selection — and the whole essay still reaches the writer, since a thesis field cannot say which quote Mat wanted 07 · VERIFY — pure code, no model call → 08 · VISUALISE — the lead image, drawn last Ten checks over the finished draft every span re-matched against its record no quote may end near a truncation point markers pointing nowhere, typed URLs and hand-written source lists all fail here What a bad citation costs on a paraphrase → the marker goes on a sentence carrying a quote → the sentence goes a cut that can't be made cleanly → the draft goes every removal is written down, never silent The illustration briefed in code from the verified piece and its hinge — not a forecast of one · planner, inspector, renderer optional: no image, never no dive the pipeline never publishes prose it could not verify; it publishes less prose instead COMMIT — the only module allowed to write the published tree validate → images → markdown → index nothing outside the work tree is touched until a fully verified result exists atomic renames; the index row goes last content/<date>/deep-dive.md + chart-1.png frontmatter carries the citations, the per-stage spend and the thesis the published page quoted phrases open their source “How this was made” lists model and cost the thesis is handed to the wallpaper THE RUN ITSELF .work/<date>/dive/run-NN/ · manifest.json — front door · inputs/ — captured verbatim · ledger.json · packet.json · stages/NN-name/ prompts · responses · timings · visualisations/ · output/deep-dive.md replay resume from the ledger or the packet, into a fresh run that records what it came from the live tree is unreachable while it runs — a replay never publishes eleven model roles research · annotate · mat nominate · select · write argument audit · salvage visual plan / inspect / render resolved once per run and written into the manifest, so a dive from last week can still say who wrote it three roots, resolved apart input → content/ (always live) output → content/ since cutover work → .work/ a shadow run reads the real briefing and writes somewhere harmless what it cost priced per call and per role, and measured again as the credit balance before and after both are kept; a gap is a signal reading a run back dive_inspect.py summarises it; dive_reassemble.py rebuilds the page from the saved artefacts why write-once? a second attempt never overwrites the first, so two drafts off one ledger can be read side by side THE GUARD — what happens when something goes wrong the model's output is correctable it is handed back once, inside the stage a stage returns nothing usable it drops out and the pipeline carries on drafting or verification fails one cheap call over the packet on disk there is no usable packet the old pipeline runs once, against the clock a spending ceiling sits over the whole run: breaching it costs the day its picture, never its dive · nothing partial or unverified is ever left behind
Deep Dive v2 — the ledger and the packet are the two artefacts everything else hangs off. Tap to enlarge.