Blog · Engineering

How a benchmark score is made

July 14, 2026 · World Labs

Our leaderboard is live, and it's empty. That's deliberate — nothing on it is simulated, and it fills only when real panelists score real runs. This post documents the pipeline that produces those numbers: the data model, the lifecycle of a run, the aggregation math, and the design decisions that make the board hard to corrupt. If you're going to trust a number, you should be able to read how it was manufactured.

The data model

Five tables and a view, in Postgres:

models — one row per evaluated model: slug, display name, vendor.
environments — one row per task: slug, name, and the spec the model is handed.
runs — one model's attempt at one environment. Carries the artifact URL and a status that moves pending → scored.
scores — one panelist's scorecard for one run: a ratings JSONB object of category → 0–10, a written comment, and minutes spent. A composite unique constraint on (run_id, panelist_id) means one panelist gets exactly one scorecard per run — re-submitting overwrites, it never stacks.
eval_categories — the fourteen published categories from the taxonomy, seeded from the same source the site renders.
leaderboard — a view, not a table. More on why below.

The lifecycle of a run

Everything enters through one admin-gated ingest endpoint that behaves like a small state machine:

# each step is an authenticated POST to /api/benchmark/ingest register-model slug, name, vendor register-environment slug, name, spec create-run model + environment → run (status: pending) record-score run + panelist + ratings{} + comment + minutes finalize-run run → status: scored # refused if no scorecards exist

Two gates matter here. First, who can score: record-score resolves the panelist by email against the applications table and refuses anyone who isn't an approved panelist. A pending applicant, a rejected applicant, and a stranger all get the same refusal. The panel application process isn't just recruiting — it's the access-control list for the benchmark.

Second, what can reach the board: finalize-run refuses to mark a run as scored if it has no scorecards, and the leaderboard only joins runs with status = 'scored'. There is no path for an unplayed artifact to produce a number.

The aggregation math

Scorecards are partial by design — a systems designer scores game-economy, a game-feel specialist doesn't have to. The aggregation handles this in two stages, both in SQL:

Stage one: per-run, per-category median. The ratings JSONB is unnested and each category's score for a run is the median across the panelists who scored it:

create view run_scores as select s.run_id, cat.key as category_slug, count(*) as n_panelists, percentile_cont(0.5) within group (order by (cat.value)::numeric) as median_score from scores s cross join lateral jsonb_each_text(s.ratings) as cat(key, value) group by s.run_id, cat.key;

Median, not mean, is the load-bearing choice. If three panelists score a mechanic 7, 7, and 1 — because one rater hit a bug the others didn't, or one is simply hostile — the mean says 5 and the median says 7. One outlier, in either direction, moves a median-aggregated score by at most one rank position of the panel, never by the magnitude of the outlier. For a small panel judging subjective quality, that robustness is worth more than the extra information a mean would preserve.

Stage two: per-model rollup. A model's overall score is the mean of its per-category run medians across all of its scored runs, with the per-category breakdown aggregated alongside:

create view leaderboard as select m.slug, m.name, m.vendor, count(distinct r.id) as n_runs, count(distinct s.panelist_id) as n_panelists, round(avg(rs.median_score), 2) as overall_score, jsonb_object_agg(rs.category_slug, round(rs.median_score, 2)) as category_scores from models m join runs r on r.model_id = m.id and r.status = 'scored' join scores s on s.run_id = r.id join run_scores rs on rs.run_id = r.id group by m.id, m.slug, m.name, m.vendor;

Then quorum. The API serving the public board splits the view's rows: models scored by at least three independent panelists are ranked; anything below that is held as provisional, reported as a count but never as a number. The threshold exists for one reason — below three raters, a single person's judgment is the score, and we won't publish a number one reviewer could have set alone. The median needs at least three points before it means anything.

Integrity is schema, not policy

Most benchmark-integrity promises are policies: "we don't accept payment for placement." Policies depend on the people enforcing them. We tried to push as much of the promise as possible down into places where breaking it is a schema violation rather than a decision:

The leaderboard is a view. There is no leaderboard row to edit, no score column to bump. The only way to move a published number is to insert a real scorecard from an approved panelist through the gated ingest path. The failure mode "someone quietly edited the board" doesn't have a place to happen.
Row-level security is deny-all. Every table has RLS enabled with zero permissive policies, so the public API key can read and write nothing — even if it leaks. All database access goes through serverless functions holding the service-role key, which never ships to a browser.
Admin auth can't be spoofed client-side. Sessions are HMAC-signed, httpOnly cookies verified server-side on every admin call. Hiding a button is never what protects the data.
Scoring authority is auditable. Because panelist identity resolves through the application queue, every scorecard traces to a vetted human with a review timestamp — attributed publicly only by specialty and years of experience, never by name.
The empty state is honest. The board renders "no runs scored yet" rather than placeholder rows, and the page distinguishes "empty" from "failed to load." We would rather look empty than look wrong.

What this rules out

Walking the pipeline backwards, each common way benchmarks go bad hits a wall: a model can't judge another model (only approved humans can hold a scorecard); a vendor can't buy a rank (there's no writable rank); a single enthusiastic or hostile reviewer can't set a score (median + quorum); an unplayed build can't produce a number (finalize refuses empty runs); and a leaked API key can't touch the data (deny-all RLS). What remains is the thing we actually want to measure: what shipped game developers concluded after playing what a model built.

The one-line version: every number on the board is the median judgment of at least three vetted humans who played the build, computed by a view nobody can write to.

What we'd add next

Two known limitations, on the roadmap in order: confidence intervals — with small panels, a score of 7.2 vs 6.9 may not be a real difference, and the board should say so rather than imply false precision; and pairwise preference aggregation — head-to-head judgments on same-environment runs, fed through Bradley–Terry, as a second ranking that doesn't depend on absolute-scale calibration at all. Both slot into the same schema: more columns derived from scores, still no table anyone can edit.

If you're training models that generate games and want your outputs inside this pipeline — or you've shipped games and want to be one of the humans behind the medians — talk to us.