How the task works
Craitures is a shared drawing benchmark for language and vision models. Every model gets the same unusual constraint: invent a creature as a set of small, simple SVG parts, then describe how those parts connect in a join graph. The creature is not a single freehand image. Its body, appendages, eyes, ornaments, and other pieces must remain understandable as parts that can be assembled and animated.
Generation and refinement
The first model call names the creature, draws its parts from a restricted SVG vocabulary, and supplies an assembly whose joins say which part owns each child, where the two touch, what role the child plays, and how it may move. A parser normalises those shapes and rejects markup outside the benchmark contract.
If the model has vision, the system renders that assembly and shows it back to the model for refinement. A text-only model cannot inspect the result, so its first valid assembly is published as a blind single shot. The gallery reports this distinction and the number of refinement turns.
Rendering and compatibility
The published bundle contains normalised parts, semantic colour roles, the join graph, motion hints, and provenance. The browser renderer maps those roles onto the site palette and animates the authored joins; it does not add creature-specific animation by hand.
Each bundle has a task-format version. Old bundles remain frozen and the renderer gates newer behaviour by version, so a v1 creature keeps its original interpretation even as the benchmark learns new tricks. A missing version is treated as v1.
v1 — parts and joins
The original task established the core benchmark: constrained SVG parts, semantic colours, and a parent–child join graph that assembled the parts into one creature. Motion was intentionally simple and uniform.
v2 — expressive motion and legibility
v2 made join roles richer, varied motion between creatures, and tightened the prompt around readable silhouettes and recognisable appendages. Sways, bobs, pulses, and role-aware movement could now express the design without changing its underlying graph.
v3 — visible provenance
v3 recorded whether the generating model had vision and how many refinement turns actually ran. That made blind single-shot entries distinguishable from creatures whose models inspected a render.
v4 — layering and articulated pincers
v4 taught the painter about explicit front-and-back join layering and introduced articulated claw and pincer semantics. Limbs could pass behind a body, while paired gripping parts moved as a mechanism rather than as a generic waving appendage.
v5 — transactional refinement
v5 made refinement reversible. A model can transactionally replace existing parts as well as edit the assembly, then compare the prior and current renders side by side. It can keep revising, accept the current state, or revert to the immediately prior state; the final bundle records what happened.