World Models and Embodiment.
Predicting the next token of text is not predicting the next frame of the world — but the methods converge.
This volume describes the field as of October 2026. Claims are marked established (replicated or long-standing), reported (from a single lab or report, not independently reproduced) or contested (actively disputed). A world model predicts how a situation evolves under actions. Two lines of work bring language-model methods to it. Video world models generate an environment frame by frame in response to actions: DeepMind’s Genie 3 (August 2025) produced navigable worlds from text prompts at 720p and 24 frames per second, consistent for minutes (reported, research preview). Vision-language-action models for robots — RT-2 (2023), π0 (2024) and successors — put camera images and instructions into a Transformer and output motor actions, transferring web-scale knowledge to physical tasks (established as a working approach; general-purpose reliability open).
Using a learned model to plan has a characteristic weakness: its small error each step compounds over a long rollout, and a planner exploits the model’s mistakes as readily as the world’s real opportunities. The workshop shows how the gap between a learned simulator’s rollout and reality grows with the planning horizon for the chosen per-step error.
Below: the expected drift of a learned model’s rollout from reality against planning horizon, for the chosen per-step error and its correlation.
- Define a world model.
- Describe video world models and VLA robots.
- See errors compound over a rollout.
- Explain why learned planners use short horizons.
Errors that compound along a plan
With 2% error per step, half of it systematic, a learned simulator stays within the usable limit for 43 steps; if the errors were all independent it would last 200, and all systematic only 24. Systematic errors are the dangerous kind, because they push every rollout the same way, and a planner optimising against the model will find and exploit them — the same over-optimisation as against a reward model (plate XVIII.6). Model-based agents therefore replan often, keep horizons short, and correct their model against real observations.
Language models bring two things to embodiment: broad knowledge of objects, instructions and goals learned from text and images, and the Transformer machinery for mapping sequences to sequences, now applied to frames and motor commands. What remains open is reliability: robots must succeed far more often than chat answers, and data for physical skills is scarce compared with text — which is why simulated worlds, including generated ones, are a research bet for producing it.
Terms
- World model
- A model predicting how a situation evolves under actions.
- Vision-language-action model
- A Transformer mapping images and instructions to robot actions.
- Model-based planning
- Choosing actions by simulating outcomes in a learned model.
- Compounding model error
- Drift of a simulated rollout from reality as steps accumulate.
Sources
- Ha and Schmidhuber, “World Models”, arXiv:1803.10122, 2018.
- Google DeepMind, “Genie 3: A new frontier for world models”, August 2025.
- Brohan et al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control”, CoRL 2023.
- Black et al., “π0: A Vision-Language-Action Flow Model for General Robot Control”, arXiv:2410.24164, 2024.
- Hafner et al., “Mastering Diverse Domains through World Models” (DreamerV3), arXiv:2301.04104, 2023.