Same brief.
Different models.
The crowd is the metric.
Benchcade asks a deceptively simple question: if you give six AI models the exact same game-design brief, whose game do real people actually want to play?
Every model receives one strict brief and ships a complete, self-contained arcade game. The games load onto a physical cabinet. A human walks up, plays two versions of the same brief back-to-back — not told which model built which — and votes for the one they’d drop another coin into. Thousands of those tiny head-to-head choices aggregate into an Elo rating per model. That rating is the score. No rubric-scraping LLM judge, no automated proxy metric — the benchmark is people playing games in a room.
Models build
Each model gets one brief and ships a single self-contained .html game — named, with its own theme, art and sound.
Humans play
On the cabinet, a player plays Entry A then Entry B of the same brief, identities hidden under neutral codenames.
The crowd decides
One choice: “another coin in which?” Each ballot updates a per-brief Elo. The aggregate vote is the model’s score.
Play-to-judge,
not auto-metric
Most code benchmarks reduce a model to a number a machine computed: tests passed, tokens matched, a second model grading the first. Benchcade refuses that. Whether a game is good is a thing humans feel with their hands — so humans, in person, are the measurement instrument.
Hands on a stick
Game feel — latency, juice, “one more go” — only exists when a real person plays it. The cabinet is the lab.
No model tells
No vendor logos, no comment headers, no names. Voters judge execution, never reputation. The shell strips every tell.
Compare, don’t calibrate
Humans compare far more reliably than they score 1–10. Each vote is a head-to-head, neutralising harsh-vs-generous drift.
One brief, many minds
Every model attempts the same game, so the only variable is the model. Faithfulness to the brief is part of the test.
The gate proves it runs.
The crowd decides if it’s good.
Benchcade splits judgement cleanly in two. A mechanical gate decides eligibility — can this untrusted, model-written code load safely in the kiosk and honour the shared integration contract? It is deliberately narrow: it never judges whether the game is fun. There is no score threshold and no quality bar in the gate. Quality is entirely the human vote.
The structural gate Automatic · binary
A headless run + static scan. Pass every check or you’re ineligible — never shown to a voter.
- One self-contained HTML file; network only from a tiny CDN allowlist
- Loads in a locked sandbox — never navigates, never escapes the frame
- No
alert/prompt, no blocking loops, no main-thread stalls - Speaks the protocol:
READY → RESET → SCORE → GAMEOVER - Gamepad and keyboard both work; COIN and START pinned to fixed buttons
- Reports a finite numeric score; fits the viewport; zero console errors
The blind human vote People · the benchmark
Four lenses voters play through. None of these gate anything — they are the score.
- Fun — game feel, responsiveness, “would I put another coin in?”
- Polish — theme, audio, animation, title & game-over screens
- Faithfulness — did the model honour the brief and its spirit?
- Balance — tuned difficulty, fair deaths, no degenerate strategy
How a vote
becomes a score
The atomic vote is a head-to-head. A voter plays Entry A, then Entry B of the same brief, then casts one choice. A “can’t decide” push is allowed and carries information. Play order is randomised per voter so first/last position bias washes out across the crowd.
Each ballot updates a per-brief Elo rating (Bradley–Terry style): every win or loss nudges a latent strength, and the final strength is the model’s score on that brief. The cabinet uses active sampling — it serves the most uncertain matchup next (least-played pairs, then closest current Elo) so votes concentrate where the ranking is least settled. Integrity guardrails apply: a minimum number of distinct voters per pairing, one vote per person per pairing for the headline tally.
Capability
vs creativity
Benchcade reports two numbers, never one. Mixing them would reward the wrong thing on each — punishing ambition where we want ambition, excusing blandness where we want craft. The three briefs split across the two tracks:
Tight specs, rewarded craft
Hard, specific constraints with low intended variance. Measures whether a model can take a strict spec and ship a faithful, polished, genuinely fun execution. Reliability and the floor matter as much as the ceiling.
High variance, rewarded boldness
A shared snake skeleton plus one invented original mechanic. Measures inventive range — the boldness and coherence of a new idea that actually works on the cabinet. Here the ceiling matters more than the floor; a daring twist that lands beats a clean but forgettable one.
The shared
challenges
Each brief pins only the objective and the control. Theme, art, audio, juice and difficulty are the model’s to invent — two models given one brief should produce two recognisably different games that are nonetheless the same kind of challenge.
One-Button
A complete scored game playable with a single input — one button, or one axis lean. Timing is the game. The proof line: Flappy Bird, Canabalt, Downwell’s first thirty seconds. Depth from one verb, not from extra buttons.
30-Second
A complete arc — hook, escalation, climax — in about thirty seconds. The first two seconds are the tutorial. Front-loaded juice, real escalation, a satisfying end, instant restart. A test of pacing, not scope.
The Twist
The shared snake skeleton — a growing chain on a grid, self-trap as the hazard — plus one invented mechanic that genuinely changes how the snake plays. It must still read as snake; the twist must change a good player’s strategy.
Three moving
parts
The system separates building from playing across two planes, joined by one small message protocol. The build side has all the tooling; the cabinet side stays dependency-free and offline-capable, loading only entries that already passed the gate.
Generation harness
Drives each model against a brief, shipping a single self-contained game per model. Data-driven off a fixed contestant roster; re-runnable with bounded fix rounds.
Sandboxed gate
A headless browser + static scan that runs every entry through the eligibility contract. A failing check makes an entry ineligible — it is reported back, never shown to a voter.
Kiosk shell
The cabinet app: attract screen → pick a brief → blind pairwise play → vote. Owns the menu, the 60-second cap, the score readout, Elo and active sampling. Zero external dependencies.
The host↔game protocol
Every game runs in a locked sandbox and talks to the cabinet only through
postMessage. The shell can never read game state directly — score is
pushed by the game, never pulled by the host. The whole vocabulary is six messages:
HOST → GAME BENCHCADE_RESET { durationMs:60000, seed }
GAME → HOST BENCHCADE_SCORE { score:0 }
GAME → HOST BENCHCADE_SCORE { score:340 }
HOST → GAME BENCHCADE_GAMEOVER { reason:"timeup" } // 60s cap
GAME → HOST BENCHCADE_GAMEOVER { score:980, reason:"timeup" }
GAME → HOST BENCHCADE_EXIT {} // player hits COIN → back to menu
Eighteen real games.
Three briefs. Six builders.
This is the proof. Six models, each handed the same three briefs, each shipping a complete, named, self-contained arcade game — its own theme, art, sound and title screen. No two are alike. Walk the cabinet:
One-Button
6 entries
30-Second
6 entries
The Twist
6 entries
Drop a coin.
These run right here.
Three of the cabinet entries, embedded live and sandboxed — the exact same files the
kiosk loads, speaking the exact same READY → RESET → SCORE → GAMEOVER
protocol. Each tile is a tiny in-page host: it waits for the game’s
BENCHCADE_READY, then on INSERT COIN posts a
BENCHCADE_RESET to start a 60-second round. Score is pushed up live; game-over
offers a replay. One brief from each: capability, capability, creativity.
Six models.
Every game is named.
Seven models entered. Six cleared the structural gate and build all three
briefs; the cabinet only ever sees an opaque codename (m1, m3…)
so the vote stays blind — but every game is model-named with its own
title and instructions screen, declared to the host in the handshake. The codename↔model
map below is the benchmark result, revealed only after the vote. One model —
claude-sonnet-4-6 — failed the eligibility gate and never reached a
voter. That is the honest roster: six who competed, one who didn’t qualify.
A few of the games currently on the cabinet: