World Labs: expert evals for AI-made games, by game developers who ship.

Research

The eval space

Games are the richest testbed AI has: strategy, perception, planning, taste, and economics in one loop. Our first focus is whole generated games, one cell of a much larger grid. This is the full taxonomy of expert human evals we're building toward, each one a distinct question, a distinct rater pool, and a distinct dataset.

Playing intelligence

Gameplay Decision Evals

Did the AI make the right decision? Strategic intelligence, judged move by move.

Macro (LoL, Dota, StarCraft) Micro & mechanics Resource allocation Objective priority Risk / reward Timing Positioning Team coordination Win-condition ID

decisionQuality: 8/10 alternativeAction: "Take Baron" reason: "Enemy had no smite" mistakeSeverity: MEDIUM

Judged by: Challenger-tier players, coaches, esports analysts

Gameplay Understanding Evals

Does the AI actually understand what's happening? Game sense, not just action quality.

Identify player intentions Explain success / failure Predict next action Threat identification Game-state reads Strategy comprehension

Judged by: High-ranked players, game analysts

Agent Trajectory Evals

Is this agent behaving intelligently over time? Whole runs, not single moves.

Full playthroughs Multi-step objectives Long-horizon planning Adaptation after mistakes Learning from opponents

Matters for: game-playing agents, autonomous agents, world models

Esports Performance Evals

How close is the AI to professional play? The hardest human baseline there is.

Mechanical execution Decision making Team play Draft strategy Adaptation

Judged by: Pros, coaches

Design quality

Balance Evals

Is the game or system balanced?

Characters Weapons Items Economy Cooldowns Progression speed Matchmaking fairness Difficulty spikes

Judged by: Competitive players, game designers

Level Design Evals

Is this environment well designed?

Map flow Navigation Exploration Difficulty curve Player guidance Reward placement Enemy encounters Puzzle quality

Judged by: Level designers, experienced players

Game Mechanic Evals

Is this mechanic fun and understandable?

Combat systems Movement Abilities Crafting Progression New features

Labeled on: fun factor · learning curve · depth · replayability · frustration

Narrative Evals

Does the story work?

Dialogue quality Character consistency Quest structure Pacing Emotional impact Lore consistency

Judged by: Narrative designers, writers

Player experience

Player Experience Evals

Will players enjoy this? The closest category to pure taste.

Fun rating Engagement Frustration Confusion Emotional impact Would you keep playing? Would you recommend?

Note: Our playtest rubric lives here and in design quality

UX / UI Evals

Can players understand and use the game?

Menus HUD Inventory Controls Tutorials Accessibility Information clarity

Judged by: UX designers, onboarding specialists

AI NPC / Agent Interaction Evals

Huge future category

Does this AI feel intelligent to play with?

NPC conversations Memory consistency Personality Improvisation Realism Safety Player immersion

Judged by: Players in extended sessions, narrative + systems designers

Generated artifacts

Generated Content Evals

Is AI-generated content good? Humans rank quality, creativity, usefulness, and consistency.

Quests Characters Weapons Levels Dialogue Textures Animations

Note: Whole generated games, the top of this stack, are where we start

Code / Technical Evals

Is AI-generated game code good?

Unity scripts Unreal Blueprints Shaders Gameplay systems Networking Optimization

Judged by: Unity/Unreal developers · Labels: does it work? performant? maintainable? would you ship it?

QA / Bug Evals

Did the AI correctly identify issues?

Bug detection Severity ranking Reproduction steps Regression testing Edge cases

Judged by: QA engineers

Economy & live operations

Game Economy Evals

Is the economy healthy? Especially valuable for live-service games.

Currency inflation Progression pacing Monetization balance Reward systems Battle pass design Retention loops

Judged by: Economy designers, live-ops specialists

Monetization Evals

Will this convert without hurting players?

Store layouts Pricing Bundles Battle passes Ads Offers

Judged by: Mobile game experts, growth teams

Preference data

Human Preference Evals

The universal one

A or B: which is better, why, and by how much? The AI produces two outputs (two levels, two quests, two whole games); an expert picks the winner, states a confidence, and writes the reason. Every other category in this taxonomy can be run in this format.

Pairwise choice Written rationale Confidence: slight / clear / strong

Used for: RLHF · reward models · benchmarking

Run one of these with us Join the rater pool →

Seventeen questions. One harness.

Talk to Us Join the Community