When agents fail because their picture of the world is stale

Here is a failure that looks like the model being dumb.

A user asks a shopping agent to ship an order to Dublin. Two messages later they say “actually, make it San Jose.” The agent calls update_address. The call times out halfway, but some partial output comes back that looks fine. The agent tells the user “Done, it’s going to San Jose.” The order ships to Dublin.

If you read only the last message, the model looks broken. It said something false with total confidence. But go back and read what was in its context at each step, and every sentence it wrote was reasonable. One possible diagnosis is that the model treated an unverified update as fact. It reasoned over a picture of the world that had stopped being true two turns earlier.

That example is hypothetical. The papers below document related failure patterns, but do not establish how often this exact sequence happens.

My suspicion is that state problems explain more agent failures than they get credit for. The agent has stale context, or it lost a correction, or it trusted a tool result it never checked, or it treated something it proposed as something that was done. I want to test how much explicit bookkeeping helps alongside a stronger model.

What I mean by state

An agent’s state is everything it currently believes about the task and the world. Which file it is editing. What the user wants. Which steps are finished. What the database says. In most agents I’ve used, that state does not live in any one place. It is smeared across the chat history, a few tool outputs, maybe a summary from an earlier compaction, and whatever the model happens to attend to on this turn.

That arrangement has one big weakness. Nothing in the context tells you how old a fact is or where it came from. “The address is San Jose” said by the user, “the address is San Jose” proposed by the model, and “the address is San Jose” read back from the database all look like the same kind of token. The model has to work out which one to trust from position and phrasing alone.

I wrote about one side of this in context freshness, which is about how old an assistant’s facts are. This post is the same idea pointed at agents that do things, where a stale belief turns into a wrong action instead of a wrong sentence.

I sort the failures into four kinds.

1. Stale context. The agent acts on a fact that was true earlier and isn’t now.
2. Lost corrections. The user changed something, and the change got outvoted by everything that came before it.
3. Unverified tool results. A tool call failed, timed out, or returned something partial, and the agent treated it as success.
4. Proposed vs verified confusion. The agent wrote down what it meant to do and later read that back as what it did.

The evidence

Evidence 1: models get lost when information arrives over several turns

The cleanest evidence for the first two kinds is a 2025 paper from Microsoft and Salesforce researchers, “LLMs Get Lost In Multi-Turn Conversation.”¹

Their setup is simple and kind of clever. Take a task that a model normally does in one prompt, like writing a SQL query or solving a math word problem. Split the instructions into pieces they call shards. A simulated user then reveals one shard per turn, the way a real person tends to remember details as they go. Same information in the end, just spread out.

They ran 15 models over more than 200,000 simulated conversations across six tasks. Performance dropped by an average of 39% when the information came in shards instead of all at once.¹

The part I find most interesting is how they split that drop. They measured aptitude (roughly the best-case score, the 90th percentile across runs) and unreliability (the gap between the best and worst runs). Aptitude barely moved, about 16% down. Unreliability more than doubled, up about 112%.¹ So the models still could do the task. They just did it much less consistently once the information had to be tracked over time.

Their explanation is a state explanation. Models make assumptions in early turns, attempt a final answer too soon, and then lean on that early attempt. In the authors’ words, when models “take a wrong turn in a conversation, they get lost and do not recover.”¹ The early guess becomes part of the context, and the context gets treated as truth. That is a lost correction in slow motion.

Two other findings from that paper point the same way. Lowering temperature did not fix it; even at temperature 0, unreliability stayed around 30%. And a strategy they call “snowball,” where the full set of revealed requirements gets restated every turn, recovered some of the gap, about 15 to 20%.¹ Restating the current state helped about as much as, or more than, tuning the sampler. That’s a bookkeeping fix, and it worked better than the knob that controls the model.

Their advice to users is honestly the best summary of the problem: if a conversation goes off the rails, start a new one with all the requirements consolidated up front.¹ In other words, throw away the corrupted state and rebuild it clean.

Evidence 2: tau-bench, where agents talk to users and databases

The second piece of evidence is tau-bench, from Sierra in 2024.² It puts an agent between a simulated user and a real-ish backend (a retail store and an airline), with tools to read and write a database and a policy document the agent has to follow. A task passes only if the final database state matches the expected one.

Two numbers from it stuck with me. The best agent they tested, gpt-4o with function calling, succeeded on fewer than half the tasks overall. And they introduced a metric called pass^k, the chance that all k independent tries at the same task succeed. In the retail domain, gpt-4o’s pass^8 was below 25%.² The same agent, on the same task, often succeeds once and fails the next time.

The authors looked at 36 failed gpt-4o runs in retail by hand and sorted them.²

I want to be fair here, because this data does not cleanly back my claim. The authors frame the biggest bucket, wrong arguments and wrong information (about 55% together), mostly as a reasoning problem: the agent has to search a messy inventory and get the details right.² The “wrong decision” bucket is about following domain policy, which is also reasoning.

But the last bucket is at least partly state. On tasks with several requests, the agent sometimes “omits explicit user requests at the beginning of the conversation,” and the authors say this hints at a need for “better long-context and memory capabilities.”² That is a lost instruction. And when I read their example of a wrong-argument failure, a good part of it is the agent holding the wrong item in mind while it fills in the call. I’d count some of that bucket as state too, though that’s my reading, not theirs.

So my honest take on tau-bench: it shows a lot of plain reasoning failures, and a real chunk of state failures, and the pass^k numbers show the whole thing is fragile in a way that one-shot accuracy hides.

Evidence 3: agents say a tool worked when it didn’t

The third kind, unverified tool results, is the most dangerous of the four, because it produces confident false reports.

A December 2025 paper, “Are Your Agents Upward Deceivers?”, built 200 tasks where tools were broken or missing, or the request could only partly be done.³ Across 11 popular models, agents often did not report the failure. They guessed, simulated outcomes, and came back with confident answers. The prompt-based fixes they tried only reduced it a little.³

A July 2026 case study on Claude Code describes a sneakier version.⁴ Long agent sessions get compacted into summaries. If a command was killed by a timeout but printed some partial output first, that partial output can get written into the summary as a confirmed result. Later sessions read the summary, trust it, and never re-run the command. The author’s diagnosis is that the system confuses what showed up on the terminal with what actually got saved. It’s one author and one tool, so I’d treat it as a well-documented example more than a measured rate. But it is exactly the “timed out, read as updated” step from my opening example, and it happens in a tool I use most days.

Anthropic’s own write-up on context engineering admits the underlying pressure. They describe context rot: as the context gets longer, the model gets worse at recalling what’s in it. Their answers are compaction, structured note-taking outside the context window, and loading data just in time with tools.⁵ All three are good ideas. Compaction is also exactly the step where a proposed or partial result can get laundered into a fact, which is the case study above.

Evidence 4: when multi-agent systems fail, verification is a whole category

The last piece is MAST, a taxonomy of why multi-agent LLM systems fail, built from more than 1,600 annotated traces across seven frameworks.⁶ They found 14 failure modes in three groups: system design issues, misalignment between agents, and task verification.⁶ Verification gets its own category. When one agent hands work to another, the receiving agent has to decide whether to trust it, and often nothing in the system actually checks.

I read this as the proposed-vs-verified problem at a larger scale. One agent’s claim (“tests pass”) becomes another agent’s input, and by the time it reaches the final answer nobody remembers it was only a claim.

Why “use a better model” mostly doesn’t fix this

The SWE-agent paper has a result I keep thinking about. In its ablations, giving the model its full history did worse than showing it only the last five observations, 15.0% vs 18.0% on the same model.⁷ More context, worse agent. That result shows that more history did not help this setup. It does not isolate staleness as the cause; irrelevant context and other effects could also matter.

A smarter model is better at reasoning over its context. It is not automatically better at knowing which parts of its context are out of date, because that information often isn’t in the context at all. A tool output from ten turns ago does not carry a label saying “this was before the user changed their mind.” The model would have to reconstruct the timeline from scratch on every turn, and the Laban results suggest that is exactly what models are unreliable at.

This connects to metacognition. An agent knowing what it knows includes knowing when it learned it and whether it was checked. An explicit record of source, time, and verification would make that easier to inspect.

What I’d build instead

Here is what I’d build, roughly in order of how much I think it helps. None of it needs a new model.

1. An event log, separate from the chat

Keep an append-only log of everything that changes the task: user requests, user corrections, tool calls, tool results. Each entry gets a timestamp and a status. Then, on each turn, build a short “current state” view from the log and put that in the prompt instead of relying on the model to dig through the transcript.

This is basically Laban’s snowball strategy made structural.¹ Instead of restating requirements by hand, the harness always restates them, and a correction overwrites the old value in the view instead of sitting as one more line in a long chat.

2. A verify step after every side effect

Every tool call that changes something in the world gets followed by a read that checks it. You updated the address? Read the order back. You wrote a file? Check the file exists and has the content. You ran the tests? Look at the exit code, not the last line of output.

Here is the shape of it in Python. It’s small on purpose.

import time

log = []  # append-only

def act(name, do, check):
    log.append({"t": time.time(), "action": name, "status": "proposed"})
    try:
        result = do()
    except Exception as e:
        log.append({"t": time.time(), "action": name, "status": "failed", "err": str(e)})
        return None
    ok = check(result)  # re-read from the source of truth
    log.append({"t": time.time(), "action": name,
                "status": "verified" if ok else "unverified"})
    return result if ok else None

The model only ever gets told something is done if the log says verified. If it says unverified or failed, the agent has to say so to the user. That one rule would have caught my Dublin example, and it is the rule the upward-deception paper suggests models do not follow on their own.³

3. Keep “proposed” and “verified” as different words

This sounds tiny, and I think it matters a lot. When the model writes a plan, those steps go into the log as proposed. They only become done when the tool runs, and verified when the check passes. Summaries and compactions have to keep the status. A compaction that turns “ran migration (timed out)” into “ran migration” has destroyed the most important bit of information in that line.⁴

4. Freshness on every fact

Each fact in the state view carries how old it is and where it came from: user, tool, or model. Facts past some age, or facts the user has touched since, get re-read before the agent acts on them. That’s the freshness idea applied to an agent’s working memory. The rule I’d start with is boring: before any write, re-read whatever the write depends on.

5. Measure consistency, not just success

Run each task several times and report pass^k, like tau-bench does.² A state bug often shows up as flakiness before it shows up as a failure rate, because whether the agent loses a correction depends on small random differences in what it attends to. If your agent passes once and fails twice, that’s a clue it’s tracking state badly.

Where I’ve seen this in my own work

The closest I’ve come to building this was Dear Future Me’s voice app. An agent listens, decides what to keep, saves it to your notebook, and answers out loud.

The part I trusted was never the model’s own judgment that it had saved something. A FastAPI backend checks every request before anything gets written, and letters go out by SMS only to numbers that have confirmed a code first. That is a small version of “the harness decides what is true,” and it is what makes it reasonable to let a model write to something permanent.

The dangerous failures are the quiet ones, where a model confidently writes the wrong thing somewhere permanent. Looking back, almost every quiet failure I can think of fits one of the four kinds above.

A research project I worked on this summer pushed me the same way. It made me want the checks outside the model, in code, where I can see them.

What I don’t know yet

I’d like to know how often an agent could recover just by getting a clearer picture of the current situation. A stale tool result or a forgotten correction seems like a frustrating reason to fail when the information needed to continue is already available.

We could test that by keeping the model fixed and improving how updates reach it, including on tasks like those in τ-bench.² If performance improves, we’d learn something about the setup. If it doesn’t, we’d need to look more closely at what the model is struggling with. Either way, I want future agents to make it easier to see what they believe is happening before they act on it.

Sources