World models vs. LLMs: a division of labor, not a choice

There is an argument that keeps coming up in AI circles, and it usually gets framed as a fight. On one side, large language models: predict the next token, scale it up, and understanding falls out. On the other, world models: systems that learn how the world changes when you act on it, so they can plan. People pick a team.

I think the framing is wrong. When you look at how agents actually break, the useful question is less “which one wins” and more “which job goes to which part.” I’d start with a division of labor. Let the LLM do language. Keep the state of the world in something explicit that you can read, check, and replay. Put boring code in between to control what changes.

What people mean by “world model”

The term gets used loosely. I wrote a longer history of it last week, from Ha and Schmidhuber’s car-racing agent through Dreamer, Genie, and Othello-GPT, so here is only the short version.

The first meaning is the reinforcement learning one: a learned network that predicts how the world changes when you act, so an agent can practice inside it. Ha and Schmidhuber’s 2018 “World Models” agent is the classic example, trained partly inside its own “dream” and then dropped into the real game.¹

The second version is Yann LeCun’s. In his 2022 position paper “A Path Towards Autonomous Machine Intelligence,” he lays out an architecture with separate modules: perception, a world model, a cost module, an actor, short-term memory, and a configurator that sets the others up for the task at hand.² The world model is the center of it. It predicts future states of the world, in an abstract representation space rather than raw pixels, so the actor can plan by imagining what its actions would do. His bet for training it is a family of architectures he calls JEPA. He is also openly skeptical that autoregressive LLMs alone get you there.²

A predictive world model should be distinguished from a state store. The design I propose below starts with the store and explicit transition rules; it does not learn a simulator.

Do LLMs already have one? Sort of, and this is where the debate gets interesting. My favorite piece of evidence is Othello-GPT, a GPT-style model trained only to predict legal Othello moves. Probes found a board state inside it, and editing that internal board changed its move predictions to match.³ Nobody told it there was a board. It built one because predicting moves was easier with it.

So “LLMs are just autocomplete with no model of anything” is too strong. Something world-like can form inside.

But look at what that representation is. It is implicit, spread across activations, and you need a trained probe to even see it. You cannot open it and ask “where is the black piece on D4?” Probes and activation edits make parts of it inspectable, but it is not a ready-made state store you can query or roll back like a database. For Othello that is a fun research finding. For an agent booking your flights, editing a codebase, or tracking a debate round, it is a problem.

Where agents actually fall over

When an LLM-based agent works on something long, the default way to give it “state” is to put everything in the prompt. The conversation so far, the files, the tool outputs, the plan, the notes. Context windows are big now, so this feels fine.

It mostly isn’t, for a few reasons.

Long contexts are not used evenly. The “Lost in the Middle” paper tested models on finding an answer in a pile of documents and on retrieving a value from a long list of key-value pairs. Accuracy was best when the relevant piece sat near the start or the end of the input and dropped when it sat in the middle, including for models built for long contexts.⁴ RULER, a benchmark from NVIDIA researchers, tested 17 long-context models and found that although they all claimed 32K tokens or more, only about half held up at 32K on its tasks.⁵ The number on the box and the number that works are different.

Nothing is authoritative. If the plan changed three times, all three versions are in the context. The model has to work out which one is current by reading. Sometimes it picks the stale one, and you have no record of why.

State changes are invisible. When the model “updates” its belief about the world, that update lives in the next token it writes, and maybe in a summary that replaces old messages. You cannot point at the moment a fact changed.

It costs a lot. Every turn sends the whole history back in, and you pay for those input tokens again. I wrote about how tokens are the unit you pay for, and attention is a weighted average over every position you feed it. Feeding it 100,000 tokens of history to answer a question that needs 300 tokens of current state is wasteful even when it works.

None of these are about the model being dumb. They are about asking one component to be the language processor, the database, and the audit trail at the same time.

The idea: split the jobs

Here is the setup I would use.

The LLM does language. It reads what you said, figures out intent, writes replies, and proposes actions in a structured form. This is the thing it is genuinely good at.

An explicit world state holds what is true right now. For an agent doing work, that might be a task graph: tasks, their status, who owns them, what depends on what. Plus whatever entities the domain needs, like files, people, dates, bills. It is a data structure, not prose.

An event log records how the state got that way. Every change is an appended event, never an overwrite. This is an old idea from software engineering called event sourcing: store every change to application state as an ordered log of events, so you can rebuild any past state and query the history.⁶ Martin Fowler wrote it up in 2005, long before anyone was building LLM agents.

A controller sits in between. Plain code. It takes the LLM’s proposed action, checks it against the current state (does this task exist, is it already done, does this violate a dependency), applies it if it is valid, logs it, and hands the LLM back only the slice of state it needs for the next step.

In code, the core loop is small:

def step(user_msg, state, log, llm):
    view = state.relevant_slice(user_msg)      # a few hundred tokens, not the whole history
    proposal = llm.propose(user_msg, view)     # e.g. {"op": "complete", "task": "T7"}
    ok, reason = state.validate(proposal)      # plain checks, no model involved
    if not ok:
        return llm.explain_rejection(proposal, reason)
    event = log.append(proposal, why=proposal.get("why"))
    state.apply(event)
    return llm.reply(user_msg, state.relevant_slice(user_msg))

The model never touches state directly. It suggests, and code decides.

Why explicit beats stuffed

The case for this, in my view, comes down to a few properties you just do not get from a long prompt.

You can audit it. If the agent marked a task done that was not done, you open the log and find the event, the input that produced it, and the model’s stated reason. With a stuffed context, you are rereading a transcript and guessing.

You can replay it. Because state is a fold over events, you can rebuild the world at any point. Found a bad event? Remove it and replay. Want to test a new model? Run it against the same event history and compare decisions.

You can validate it. A task graph can refuse a cycle. A schedule can refuse two meetings in one slot. These are cheap checks in code and unreliable checks in a prompt.

The context stays small and current. The model sees the present, not every past version of the present. That sidesteps the lost-in-the-middle problem instead of fighting it.

The model can be swapped. If the world lives outside the model, switching models does not lose the world.

The Ha and Schmidhuber paper has a great warning about what happens when the world model is not auditable. Their VizDoom controller learned to exploit the places where the learned model was wrong, finding a policy that stopped the dreamed monsters from firing, which did nothing in the real game.¹ I read that as a general lesson. The example makes me wary of optimizing against an unchecked picture of the world, even though a learned simulator and a task database can fail in different ways.

This is not a new idea, and that’s good

A lot of agent research has been drifting toward this split already.

ReAct had models interleave reasoning with explicit actions and observations from tools, so facts came from the environment instead of the model’s memory.⁷ MemGPT took the operating system as its model: a fixed-size main context, like RAM, plus external storage the model pages in and out through function calls.⁸

The Stanford “Generative Agents” paper gave 25 simulated characters a memory stream: a full natural-language record of what each one experienced, plus reflections built on top of it and a retrieval step to pull relevant memories when planning.⁹ In their ablations, removing observation, planning, or reflection made the agents less believable.⁹

What I want to push on is one step further than these. MemGPT and Generative Agents mostly store memories as text and let the model decide what to write. That is better than one giant prompt, but the store is still prose that the model curates. I would rather the important stuff be typed and checked. “Task T7 is complete, completed by Arnav, depends on T3” is a row. It should not be a sentence the model wrote and might rephrase later.

This is close to something I felt at a hackathon earlier this year. At BISV Hacks with Trajectory, the best part of the project was drawing concepts as a knowledge graph you could actually see and drag around. Once the links between subjects were explicit, we noticed things we would never have seen in a list. Explicit structure makes a system easier for people to reason about, and I think it does the same for agents.

Where LeCun and I differ, a bit

I agree with LeCun’s main point that next-token prediction on text is not the same as having a model of how the world works, and that planning needs something that predicts the consequences of actions.² Othello-GPT shows a model can build some of that internally, but you cannot inspect it or trust it without a lot of extra work.³

Where I differ is on what to do this year. His architecture asks for a learned world model that is very good, trained mostly from observation, predicting in abstract representation space.² That is a research program, and it might be the right long-term one. But for most of the agents people are building right now, the “world” is not physics. It is a task list, a repo, a calendar, a set of bills. Those worlds are already symbolic. We do not need to learn their dynamics from video. We can just write them down.

For software agents, I would borrow the separation of roles while starting with explicit state. A database and controller are not a substitute for LeCun’s learned predictive world model. The configurator, the cost module, the short-term memory: a lot of that maps onto plain code and a database in an agent you can ship today. The learned world model matters most where the world cannot be written down, like robots and video.

What would break

I don’t fully know yet whether this holds up, and there are real weak spots.

Schema design is hard. Someone has to decide what the state looks like. Pick the wrong schema and the model keeps trying to express things the state cannot hold. Then you get the worst of both: a rigid structure plus a pile of free text in a “notes” field.

The translation step can fail. The LLM still has to turn “move my thing with Sam to after lunch” into a correct structured action. Validation catches impossible actions, not wrong-but-possible ones. If it picks the wrong meeting, the log will faithfully record a mistake.

Some state really is fuzzy. User preferences, tone, half-formed goals. Those probably stay as text, maybe retrieved the way Generative Agents does it.⁹ The split is about the parts that can be made explicit.

More moving parts. One prompt is simple. A model, a controller, a store, and a log is a system. For a quick chatbot it is overkill.

What I would build

I’d start with an assistant that has to handle a changing plan. Someone adds a task, moves a deadline, and then takes back an earlier instruction. Can it keep up, explain the current plan, and undo the right change? That seems like a useful test before giving it a much larger job.

I’d compare a plain conversation history with a system that also keeps explicit records of those updates. The model would stay the same, so we could see whether the structure helps and where it creates new mistakes. A wrong record can be just as troublesome as a forgotten sentence.

What excites me about this approach is the possibility of making an assistant easier to understand as well as more capable. I want to be able to ask why it thinks something is true, correct that belief, and see the correction affect what it does next.

Sources