Introducing World Labs

Models can generate playable games now. Prompt-to-game tools ship browser builds in minutes, world models render interactive scenes frame by frame, and coding agents will happily scaffold a Unity project from a paragraph. What none of them can do is tell you whether the thing they made is any good, and that gap is now the binding constraint on the whole field.

World Labs exists to close it. We build RL environments: instrumented game worlds in which models are given generation tasks. And we run expert evals inside them: structured judgment from game developers with shipped titles, captured in a form that models can train on. This post explains the architecture in some detail.

The verification gap

Game generation is a textbook case of an unverifiable-reward domain. The properties that matter are exactly the ones that resist automation:

• Compilable is not correct. A build that runs at 60fps with zero exceptions can still have contradictory rules, a dead difficulty curve, or a mechanic that silently ignores the one constraint the prompt cared about.
• Proxy metrics get Goodharted immediately. Optimize for session length and you get treadmills. Optimize for crash-free minutes and you get empty rooms. Any cheap scalar you attach to "game quality" becomes the thing the model learns to exploit instead of the thing you meant.
• The reward is sparse, delayed, and perceptual. Whether a gravity-flip mechanic "feels right" is a judgment formed over minutes of play, integrating input latency, animation timing, level geometry, and learned genre priors. There is no unit test for it.

The standard answer in unverifiable domains is human feedback. But generic crowd feedback fails here for a specific reason: naive raters can't separate novel from broken, or hard from unfair. The judgment that discriminates quality in games is a trained skill. People acquire it by shipping games and watching thousands of players bounce off them. That is the resource we're organizing.

RL environments, concretely

An environment, for us, is not just a game. It's a contract that makes generation tasks measurable and episodes reproducible. Each environment specifies:

• A task spec. A prompt decomposed into checkable clauses: genre, required mechanics, and at least one deliberate twist or constraint ("a 2D puzzle-platformer where gravity flips on jump"). The clause structure matters because it turns "prompt adherence" from a vibe into a per-clause verdict.
• Deterministic reset and seeding. Every episode starts from a known state. The same artifact plays the same way for every rater, and regenerating from the same prompt isolates model variance from environment variance.
• Dual builds from one artifact. Each generated artifact produces a playable browser build for human sessions and a headless build for automated probes (does it boot, does the input map respond, does the win state reach).
• Event instrumentation. The environment logs input events, state deltas, progression markers, deaths, and idle time. When an expert says "level 3 softlocks," the trace shows exactly where, and the critique becomes a labeled pointer into telemetry rather than free-floating prose.
• Artifact diffing. For iteration tasks ("make the boss easier and change nothing else"), we retain both builds and diff them at the system level. The eval question becomes precise: did the requested change land, and what else moved?

Task families map onto this contract directly. Mechanic synthesis: given a clause-structured prompt, produce a build; experts probe the specific mechanic for rule fidelity, edge cases, and interactions with other systems. Constrained edit: given a build and a change request, produce a minimal-diff successor; regressions get cataloged. System generation: produce an economy or progression system; the environment instruments sources and sinks so an economy designer can read the ledger, not just play the loop.

The eval harness

On top of the environments sits the harness that turns expert play into data. Its design goal is simple to state and hard to do: make subjective judgment reliable enough to train on. The mechanisms:

Blind mode

Tool and model identities are stripped from every rater-facing surface for the entire round: pages, URLs, and API responses included. Raters judge the artifact in front of them, not the logo behind it. Identities unmask only after a round locks.

Enforced play time

The scoring form stays locked until a minimum session has elapsed, tracked server-side so a refresh can't reset the clock. Nobody scores a game they bounced off in ninety seconds, and time-played ships with every record as a confidence covariate.

Anchored rubrics

Dimensions are scored 0 to 10 against written anchors ("0 = contradictory, broken rules; 10 = a coherent system with meaningful interactions"), so a 5 means the same thing across raters. Every rubric score requires a written critique and a ranked fix list, which forces the rater to commit to a causal account, not just a number. Scorecards are partial by design: a panelist scores only the categories they're qualified to judge.

Median aggregation and quorum

A run's score per category is the median across its panelists, so one outlier — generous or hostile — can't swing it. A model isn't ranked at all until at least three independent panelists have scored its runs; below quorum it's held as provisional rather than published with a number one reviewer could move. We report panelist counts alongside every score, and the disagreement cases are themselves valuable, because they mark the genuinely contested regions of the quality landscape where a reward model will be least trustworthy. The full aggregation pipeline is documented here.

Pairwise preferences

Alongside absolute scores, same-prompt artifacts are judged head-to-head in randomized but per-assignment-stable order: a winner, a confidence grade, and a required written reason. Pairwise data sidesteps calibration drift in absolute scales and drops directly into Bradley-Terry-style reward modeling.

Broken is data

Artifacts that can't be meaningfully played are excluded from score averages but never from the record. Broken rate per model is a first-class metric, because fragility under generation is part of what labs are trying to train away.

From judgments to training signal

Every episode resolves to a stable schema:

The same records serve three consumers. For benchmarking, per-dimension means, win rates, and broken rates, sliced by task family. For reward modeling, pairwise preferences with rationales, plus multi-dimensional rubric scores that support shaped rather than scalar reward. For critique training, the written critiques and fix lists pair naturally with artifacts as inputs for models that learn to judge, in the RLAIF pattern, with expert data as the ground truth the judge distills from.

Provenance travels with every record: rater specialty and years of experience, never names. Specialty matters more than it looks; a game-feel specialist and an economy designer disagree about a build in informative, structured ways, and routing tasks by specialty is one of the quieter advantages of a curated panel over an anonymous crowd.

Why experts, and why now

The panel is the part that can't be replicated quickly. Every rater is vetted on shipped titles and calibrated on a sample eval before their judgments enter the pool. The games industry has laid off tens of thousands of exactly these people since 2023. The judgment they spent careers building is the scarcest input in game-AI training right now, and paying fairly for it is both the right thing and a durable moat.

What's next

Our taxonomy spans fourteen eval categories, from mechanic fidelity and spec adherence through level design, NPC interaction, and game economies, down to universal pairwise preference. We're starting where the demand is sharpest: whole generated games and mechanic fidelity, running inside the first environments now. If you're training world models, game agents, or generative game systems and want your outputs inside the harness, or you've shipped games and want to be one of the people whose judgment defines the benchmark, we want to hear from you.