Learning to act in complex worlds.

World Labs is developing training data and executable environments for AI agents that work across interactive worlds. Our research question is whether practice in these systems can produce better planning, spatial reasoning, tool use, and recovery on unfamiliar tasks. The immediate applications are agents that build game systems, operate inside existing games, and complete workflows across creative software. The broader opportunity is training infrastructure for agents working with complex digital and simulated physical systems.

A useful training example should connect an intention to an action and its consequences. An agent might write code, edit geometry, change a material, move a camera, or control a character. What matters is whether the resulting world satisfies the objective. This gives us a common research direction across software development, content creation, and interactive decision-making.

Why game engines are a practical starting point

Game engines bring together real-time simulation, rendering, physics, interaction, networking, and asset pipelines. That combination supports applications in healthcare, robotics, architecture, manufacturing, and other domains that need interactive 3D systems. Game development supplies mature engines, executable projects, extensive tooling, and publicly available examples. As GameEngineBench argues, this makes it a practical research setting for a broader class of stateful, real-time software. [3]

The relevant state extends beyond source files. It includes scene hierarchies, object transforms, collision geometry, animation state, imported assets, network authority, and the sequence of events already executed. An agent only observes part of that state through tools, logs, images, and playtests. Technically, this is a partially observed decision problem: the agent must gather information, maintain a working model of the system, and choose actions whose effects may appear many steps later. Running an experiment can be as important as making an edit.

Two complementary kinds of environment

In a development environment, the agent changes the world itself. Tasks can require coordinating scripts, scenes, assets, and configuration while preserving existing behavior. A successful outcome is an executable artifact that survives testing under the specified conditions.

In an interaction environment, the world’s rules remain fixed and the agent learns to act within them: navigate, collect resources, use tools, or coordinate a sequence of actions under uncertainty. SIMA 2 demonstrates a relevant training recipe: human demonstrations followed by online reinforcement learning on tasks with starting states, instructions, and verifiers, using visual observations and keyboard-and-mouse actions. Its evaluation includes games excluded from RL training. [5] These two environment families can share execution infrastructure while retaining different action spaces, rewards, and evaluation protocols.

Connect creative software into complete tasks

The next step is to connect modeling, procedural geometry, materials, animation, CAD, and engine tools into a single assignment. Consider turning a warehouse workstation model into an interactive training scene. The agent could repair geometry in Blender, generate variants in Houdini, prepare materials, import the asset into an engine, configure joints and collisions, add interaction logic, and verify the result from several viewpoints. Success depends on dimensions, appearance, behavior, and performance together.

Existing software provides useful building blocks. Blender exposes scene data and operations through Python. OpenUSD supports assembling and layering scene descriptions across applications. Unreal’s Datasmith imports scenes and assemblies from design tools for real-time use. [9][10][11] These interfaces make integration feasible, but file interchange alone does not preserve every material, rig, constraint, or simulation behavior.

Our proposed adapters would track asset identities, units, coordinate systems, dependencies, and application versions. An episode would include the documents and relevant application state, not just a repository checkout. Each tool call would return observable results and changed artifacts; checkpoints would support rollback across the workflow. Verification would check that an upstream edit survives export, import, and runtime use. The resulting training signal concerns a complete production task, even when the agent chooses to perform every operation through code.

One task across creative tools

Turn executions into training data

In reinforcement learning, a model attempts tasks, receives rewards, and a training algorithm updates it to favor successful behavior. For our setting, the basic unit would be a versioned task package: a starting project or checkpoint, an objective, available observations and actions, execution limits, termination rules, and a verifier. A recorded attempt, or rollout, would align observations, actions, tool results, timestamps, artifact changes, and scores. This makes failures inspectable and training runs comparable.

Different records serve different learning stages. Original code, scenes, and assets can support pretraining; expert demonstrations can teach initial tool competence; scored rollouts can support multi-turn RL. Failed attempts and successful repairs provide examples of recovery, provided their outcomes are verified. Our intended datasets would retain provenance, usage rights, and task versions. Evaluation projects would be kept separate from training releases.

Curricula should increase the coordination required: edit one object, preserve surrounding behavior, integrate several systems, then complete an unfamiliar workflow. Difficulty can vary through layout, dependencies, resource limits, or missing information. Diversity must change the decisions an agent needs to make, rather than merely reskinning the same solution.

A reliable reward is part of the product

We would separate agent-accessible tools from protected verification. Checks should combine artifact structure, runtime behavior, preservation of existing features, and task-specific resource limits. For a scene, these might include valid references, collision clearance, successful interaction replays, and a frame-time budget. Hard requirements should act as gates: an attractive render cannot compensate for a broken interaction. Intermediate rewards can help with long assignments, but they must not make an incomplete shortcut more profitable than finishing the task.

Visual and subjective requirements need a different treatment. Human preferences or calibrated model judgments can assess presentation, but should remain distinguishable from executable checks. We would audit disagreements, test obvious ways to exploit the scoring rules, and run repeated trials when physics or networking introduces variation. Reset reproducibility must also be measured: a random seed alone does not guarantee identical execution. The verifier should preserve enough evidence to distinguish an agent error, an invalid reward, and an infrastructure failure.

From an attempt to a learning signal

Reuse existing worlds instead of rebuilding every one

World Labs would not need to create every RL environment from scratch. Existing games already contain mechanics, assets, progression systems, and interacting constraints developed over years. With developer partnerships, we could adapt suitable games into collections of training tasks. The same world could support navigation, resource planning, tool use, and recovery tasks with different starting conditions. Integration work could then be amortized across many curricula and model-training runs.

Adaptation still requires engineering: a controllable agent interface, repeatable starts, observable outcomes, reliable resets, and access to run many isolated sessions. We would assess those properties before selecting a game, agree on permitted hosting and training uses, pin a supported build, and build task verifiers alongside the developer. Games that depend on fragile external services or cannot reset reliably may be poor candidates.

This approach could substantially reduce environment-development time and cost while offering AI labs greater variety. The amount saved is a pilot measurement, not an established result. We would compare the cost of adapting an existing title with building an equivalent task family, including licensing, maintenance, verification, and runtime expenses. Original environments would remain useful where existing games cannot isolate a capability or supply the required controls.

A hosted environment service for AI labs

The proposed service would separate environment execution from model training. An AI lab could request a task and seed, start a session, submit actions, and receive observations, rewards, and termination signals through a versioned API. The lab could keep its model and optimizer on its own infrastructure. World Labs would manage the engine processes, task catalog, resets, verification, and execution records.

Operationally, this requires isolated workers, asset caching, reproducible build images, session scheduling, timeouts, and observability. Lightweight tests could run without rendering where valid; visual tasks would retain the required renderer and sensors. Throughput should be measured as useful, verified episodes per unit cost, including startup, reset, rendering, and failed runs. A larger catalog only helps if customers can use it reliably at training scale.

A possible commercial model combines recurring fees for reserved capacity or private hosting with usage fees for metered environment execution. Developer revenue sharing and separately licensed datasets could complement it. The economic case depends on utilization, support burden, infrastructure cost, and measurable training value. These are proposed offerings and pricing mechanisms, not claims of existing customers or revenue.

Expand through shared workflows and simulation

The nearest expansion is into work that already uses the same 3D tools and engine infrastructure. In architecture, an agent could update a building model, preserve object metadata, and test an interactive walkthrough. In manufacturing, it could assemble a virtual workstation and check clearances or maintenance sequences. In healthcare, a prospective use is constructing and testing simulated training scenarios. These tasks involve assets, spatial relationships, interfaces, and behavior as well as code.

Robotics adds another direction: agents that perceive and act in simulated environments. Isaac Lab provides an existing framework for robot learning, demonstrations, and motion planning, with simulation and sensor tooling. [12] We could connect environment authoring to such specialized simulators rather than requiring a game engine to reproduce every physical process. A creative workflow might generate a warehouse layout; a robotics backend could evaluate navigation or manipulation within it.

There are two distinct transfer claims to test: whether experience improves agents that build these applications, and whether policies trained in simulation perform well on the target task. Physical transfer requires appropriate dynamics, sensor models, control timing, and validation outside the simulator. Varying conditions during training can probe robustness, but neither visual realism nor success in a game establishes real-world performance. Expansion should follow measured capabilities and domain-specific validation.

What the research establishes

GameDevBench reports GPT-5.4 task success increasing from 41.1% to 52.0% with screenshot and runtime-video feedback. This demonstrates the value of observations during execution; it is not an RL training result. [1] OpenGameEval checks both edited Roblox scenes and simulated play, making exploration and state changes measurable. [2]

GameEngineBench evaluates 110 C++ tasks across nine Unreal projects, exposing failures in integration and runtime behavior. GameCraft-Bench evaluates complete Godot games through replayed interaction and multimodal judging. [3][4] These benchmarks motivate richer tasks, but their different scoring protocols are not interchangeable. Model-based evaluation also needs additional auditing before becoming a reward used repeatedly in training.

There is evidence for learning beyond the trained games. Play to Generalize reports that RL on visual Snake and Rotation tasks improves Qwen2.5-VL-7B’s MathVista score from 68.0% to 71.9%. SPIRAL reports Qwen3-4B-Base improving from 34.0% to 44.5% on an average of eight reasoning benchmarks after strategic self-play. [6][7] These specific results support further experiments; they also show why simpler games must remain serious cost and capability baselines.

Two different kinds of evidence

A. Redrawn from Chi et al., GameDevBench, Table 2. The comparison changes inference tools, and inference costs differ; it is not a matched-compute RL experiment. These rows do not report uncertainty intervals.

B. Redrawn from Xie et al., Play to Generalize, Table 2. Error bars show the reported standard deviation over three RL runs, not confidence intervals. Baseline variability is not reported. This selects the paper’s three math benchmarks; it does not establish transfer to game-engine work or physical systems.

Procgen emphasizes diversity and evaluation on unseen levels. [8] Our extension of that principle is to hold out projects, task families, asset collections, and workflows. Generalizing to a new random seed inside a familiar project is a weaker result than succeeding in a new application context.

The experiment that would justify the platform

For development agents, we would compare training on generic coding tasks, engine tasks, and a mixture that includes connected creative workflows. Runs would start from the same model checkpoint and use matched training budgets. Evaluation would hold tools and execution budgets constant. Separate ablations would vary visual feedback and access to structured scene data, so better observations are not mistaken for learned capability.

For agents acting inside worlds, we would compare simpler game curricula with adapted commercial or partner environments, accounting for both training and simulation cost. The key outcomes are verified completion, repeatability, recovery after errors, preservation of existing behavior, and cost per successful task on unseen environments. Multiple training seeds, uncertainty estimates, and independently reviewed verifier samples would make the conclusions more credible. Public-project exposure and possible training-data overlap should be documented rather than assumed away.

The first milestone should be a small set of reliable environments, one complete workflow across creative tools, and a controlled demonstration of learning. Only then should we expand the catalog around observed capability gaps. World Labs’ opportunity is to turn existing digital worlds and production tools into dependable sources of training experience, then show which experiences improve useful behavior at a sustainable cost.