World models, from Ha and Schmidhuber to today

“World model” is one of those phrases that means three different things depending on who is saying it. A reinforcement learning researcher means a network that predicts what happens next so an agent can practice in its head. A video company means a model that generates realistic footage. An interpretability researcher means “does this language model have some internal picture of the thing it is talking about, or is it just matching patterns.”

Following the history from Ha and Schmidhuber through Dreamer, video generation, and Othello-GPT helped me understand what those different uses share, and where they come apart.

Start with a car

The setup that kicked things off is a top-down racing game, CarRacing in OpenAI Gym. The agent sees 64 by 64 pixel frames and has to steer, accelerate, and brake around a randomly generated track. The task counts as solved if you average a score of 900 over 100 random tracks.¹

The obvious approach is to feed pixels into a big network and train the whole thing with reinforcement learning. That works badly when the reward signal is sparse and the network is huge, because RL is a weak, noisy training signal for millions of parameters.

David Ha and Jürgen Schmidhuber split the problem into three parts instead, in a paper called World Models, first posted to arXiv on March 27, 2018.²

• V (vision) is a variational autoencoder. It squeezes each frame down to a small vector z. For the racing game, z has 32 numbers.
• M (memory) is an LSTM with a mixture density output. Given the current z and the action, it predicts a probability distribution over the next z. It had 256 hidden units for the car task.
• C (controller) is a single linear layer. It takes z and M’s hidden state h and outputs an action. For CarRacing, that is 867 parameters total.¹

The trick is that V and M are trained without any reward at all. You just collect random rollouts and train V to reconstruct frames and M to predict the next latent. That is plain unsupervised learning, which is a strong signal. Only C, the tiny part, gets trained against the reward, and with 867 parameters they could use an evolution strategy (CMA-ES) instead of gradient-based RL.

The result was 906 ± 21 on CarRacing, which cleared the 900 bar.¹ Here is the loop in rough Python, in my own words:

def rollout(env, V, M, C):
    obs = env.reset()
    h = M.initial_state()
    total = 0
    done = False
    while not done:
        z = V.encode(obs)          # 32 numbers summarizing the frame
        a = C.act(z, h)            # linear layer on [z, h]
        obs, reward, done = env.step(a)
        h = M.step(z, a, h)        # update memory with what just happened
        total += reward
    return total

The thing I like about this is how small C is. All the hard work of “understanding the scene” lives in V and M. The policy just reads off a good summary.

Learning inside the dream

The part people remember is the second experiment. In a VizDoom task where you dodge fireballs, they did not train the controller in the real game at all. They used M as the environment. M predicts the next z, V’s decoder can turn that back into a frame if you want to look at it, and the agent plays inside that generated world. Then they dropped the trained controller into the real game.

It worked. The agent trained in the dream survived an average of 1092 ± 556 time steps in the actual environment, where the bar for solving was 750.¹

It also cheated, which is the more interesting part. When the dream was too predictable, the agent found a policy that stopped the dreamed monsters from firing at all. That is an exploit of M’s mistakes, and it falls apart in the real game. Ha and Schmidhuber fixed this with a temperature knob τ that makes M’s predictions noisier. At τ = 1.15 the dream was hard enough that the agent could not exploit it, and the policy transferred.¹

To me this is the central problem with world models in general. An agent optimizing against a learned model can discover and exploit its mistakes. A model that is 99 percent right is not good enough if the agent goes looking for the 1 percent.

The idea was not new in 2018, and the paper says so, pointing back to work from 1990, including Schmidhuber’s “Making the World Differentiable,” on agents that learn a model of their environment and use it to plan, and the interactive version of the article spends a section on that history.¹ What the 2018 paper added was a clean, modern recipe and a demo people could watch.

Dreamer: do the learning in latent space

The next big thread is Danijar Hafner’s Dreamer agents. These take the same spirit and push it much further.

DreamerV1, from the paper “Dream to Control” (arXiv December 2019, ICLR 2020), learns a world model from pixels and then trains the policy entirely on imagined trajectories in the model’s compact latent space.³ The key change from Ha and Schmidhuber is how the policy learns. Instead of evolution, Dreamer backpropagates value gradients straight through the imagined trajectories, which is possible because the whole world model is differentiable. On 20 visual control tasks it beat prior methods on data efficiency, compute time, and final score.³

DreamerV2 (arXiv October 2020, ICLR 2021) switched to discrete latent variables and took on Atari. The authors describe it as the first agent to reach human-level performance on the 55-game Atari benchmark by learning behaviors inside a separately trained world model.⁴

DreamerV3 came out on arXiv in January 2023 and was published in Nature in April 2025.⁵ The headline is generality. One fixed set of hyperparameters worked across more than 150 tasks, and it was the first algorithm to collect diamonds in Minecraft from scratch with no human data and no curriculum.⁵ If you have played Minecraft, you know diamonds take a long chain of steps: wood, a crafting table, a wooden pickaxe, stone, a stone pickaxe, iron, a furnace, an iron pickaxe, then digging deep enough. With sparse rewards, that is a brutal exploration problem.

Then video models borrowed the name

On February 15, 2024, OpenAI released a technical report about Sora titled “Video generation models as world simulators.” Its conclusion was that scaling video generation is “a promising path towards building general purpose simulators of the physical world.”⁶

Sora is a diffusion transformer trained on video and image patches, and it could generate about a minute of high-quality video.⁶ It is not an RL agent. It has no actions and no reward. So calling it a world model is a different use of the term: the claim is that to generate realistic video, the model has to have learned something about how the physical world behaves.

The report itself listed limitations. It said Sora does not accurately model the physics of many basic interactions, with glass shattering as an example, and that eating food does not always leave the right change in the object.⁶

The pushback came fast. Yann LeCun argued on X shortly after the release that generating pixels is the wrong way to build a world model for action, because predicting high-dimensional, continuous sensory data at the pixel level runs straight into uncertainty, and that detail is not what you need for planning. His alternative is to predict in an abstract representation space instead, which is the idea behind his JEPA architectures.⁷

A more careful critique came from a research group that tested video models directly. Kang and colleagues built a simple 2D physics simulator with balls moving and colliding, trained diffusion video models on it, and asked whether the models learned the physics.⁸ In distribution, they did well. Out of distribution, they failed. The models seemed to do “case-based” generalization, copying the closest training example rather than applying a rule. When matching to training examples, they prioritized color first, then size, then velocity, then shape.⁸ So a ball that is the wrong color can confuse the model more than a ball moving at the wrong speed, which is backwards from what physics cares about. Their conclusion was that scaling alone is not enough for video models to discover physical laws.

I think both sides have a point here. My view is that “looks realistic for ten seconds” and “can be used to make decisions” are different bars. Ha and Schmidhuber’s VizDoom model looked terrible, blurry 64 by 64 frames, and it was a useful world model because an agent could learn from it. A gorgeous video model that cannot answer “what happens if I push this” is not a world model in the RL sense, whatever it is called.

Genie: video you can act in

DeepMind’s Genie line is the interesting bridge between the two meanings, because it adds actions back in.

The first Genie paper was posted February 23, 2024, a week after Sora.⁹ It is an 11 billion parameter model trained on about 30,000 hours of 2D platformer gameplay filtered from public internet videos. The clever part is that those videos have no action labels. Nobody recorded which button the player pressed. Genie learns a set of 8 latent actions on its own, discrete codes that explain the difference between consecutive frames. At play time, you pick one of those latent actions each frame, and the dynamics model generates the next frame.⁹ The paper also showed those latent actions could help train agents to imitate behavior from videos they had not seen.

Genie 2, announced December 4, 2024, went to 3D. It takes a single prompt image and generates a playable environment that responds to keyboard and mouse.¹⁰ DeepMind said worlds stayed consistent for up to a minute, though most examples ran 10 to 20 seconds, and that the model could remember parts of the scene that went out of view and render them correctly when you turned back.¹⁰

Genie 3, announced August 5, 2025, runs in real time at 720p and 24 frames per second, with worlds that stay consistent for a few minutes and visual memory reaching back about a minute.¹¹ DeepMind said it does not use an explicit 3D representation like NeRFs or Gaussian splatting. It generates frame by frame from the description and the user’s actions. Their own list of limitations includes a limited action space, trouble with multiple agents interacting, and short interaction length. It was released as a limited research preview, not a public product.¹¹

Look at that last limitation next to the VizDoom cheat. A world that holds together for a few minutes is great for a human exploring it. For an RL agent that will run millions of steps looking for exploits, a few minutes of consistency is a starting point. DeepMind frames Genie as a way to train embodied agents, and I am curious whether agents trained in it transfer, the way Ha’s Doom agent did. I have not seen a clean answer to that yet.

Do language models have world models?

The third meaning is about language models, and it is the one I find most fun, because you can actually open the model up and check.

Othello-GPT

In 2022, Kenneth Li and colleagues trained a GPT-style model on sequences of Othello moves, written as strings like “C4, D3, E6, …” The model was never told the rules and never saw a board. Its only job was to predict a legal next move.¹²

It got very good at that. Then they asked whether it was tracking the board internally. They trained probes, small classifiers that read the model’s hidden activations and try to predict the state of each square. Nonlinear probes could recover the board state. And when they edited the activations to change what the probe read, say flipping one square, the model’s predicted moves changed to match the edited board.¹² That intervention is the important part. A probe alone can find information that the model never uses. Changing the representation and watching the behavior follow is much stronger evidence. The paper was an oral at ICLR 2023.¹²

There was a catch. Linear probes did not work well in the original paper, only nonlinear ones. That matters because a strong enough nonlinear probe can sometimes compute the answer itself from scattered information.

Neel Nanda found the fix in 2023. The model does not represent “this square is black” or “this square is white.” Since it plays both sides, it represents “this square is mine” or “this square is theirs,” relative to whoever is about to move.¹³ With that framing, linear probes work, and you can steer the model by adding and subtracting those directions.¹⁴ I wrote earlier about how features inside models tend to be directions in a space, in how a word gets meaning inside a model. Othello-GPT is one of the cleanest examples of that I know of, because the “ground truth” world is a tiny 8 by 8 board you can check exactly.

Passing the test while getting the map wrong

So that is a yes for Othello. In 2024, Keyon Vafa and coauthors asked a harder question: even when a model passes the usual checks, is its world model actually coherent?¹⁵

Their main testbed is Manhattan. They took millions of real NYC taxi rides from 2014 and turned them into turn-by-turn directions on the street graph, which has 4,580 intersections and 9,846 street segments. Then they trained transformers from scratch to predict the next turn.¹⁵

By the usual measures, the models were excellent. Between 96 and 99 percent of generated routes were valid, and for the model trained on shortest paths, 97 percent of its routes were the true shortest path.¹⁵ A probe could read off the current intersection with high accuracy too.

Then they introduced detours. If you randomly replace just 1 percent of the model’s turns with some other valid turn, the shortest-path model’s success rate drops to 0.69. At 10 percent detours it is 0.08. At 50 percent, zero.¹⁵ A person who actually knows the map does not fall apart because they took one wrong turn. They reroute.

Their new metrics are based on a classic idea from automata theory, the Myhill-Nerode theorem. Roughly: two routes that end at the same intersection should allow exactly the same continuations, and two routes ending at different intersections should not. A model with a real map should treat the first pair as the same state and the second as different. The shortest-path models scored badly on both checks, even though the next-token tests looked nearly perfect.¹⁵

The picture is the part that stuck with me. They reconstructed the street map each model implicitly believes in, and it is a mess, with streets in physically impossible orientations and flyovers across the island.¹⁵ Strong performance on the original route distribution can coexist with a distorted implicit map.

One more detail I think is underrated: the model trained on random walks, which is worse at shortest paths, had a much more coherent world model and held up under detours.¹⁵ Seeing more varied data, including “bad” routes, forced it to learn the actual structure. That lines up with the Ha and Schmidhuber trick of training M on random rollouts. A world model built only from expert behavior has never seen what happens off the expert’s path.

So what is a world model, really

The examples in this post make me think about what we want a model’s imagined world to be good for. A convincing scene is exciting to look at. Using it to learn what happens after an action asks something more: the consequences need to be reliable enough to guide a decision.

I’d be curious to try actions that are unusual or inconvenient for the model and watch where its predictions break. Does a change persist? Can an object do something unexpected without the scene losing its rules? If we could understand those failures better, we’d have a clearer idea of when an imagined environment could help an agent learn something useful about the real one.

Sources