Metacognition in RL agents vs. LLMs vs. orchestrators

A reinforcement learning agent playing Super Mario, a language model answering a trivia question, and a coding agent running a test suite all have to answer some version of the same question: how am I doing right now? Am I about to fail, am I somewhere new, should I stop, should I try again?

Four days ago I wrote about what metacognition means for a model, and broke the word into six separate mechanisms, from confidence estimation to verbalized self-report. A few days before that I argued that the model is half the product and the harness around it is the other half. This post sits between those two. I want to compare three kinds of systems on one narrow question: where does the self-monitoring actually live?

I’ll judge each one on three things:

• Where it lives. Is the “how am I doing” signal a named number, something spread across hidden states, or code someone wrote?
• Is it inspectable? Can you look at it directly, or do you have to train a probe and hope?
• Is it causal? Does the signal actually change what the system does next, or is it just something we can read off after the fact?

That last one is the question I care about most, and it’s the one that sorts these three very differently.

RL agents: the monitoring is in the objective

Reinforcement learning is the oldest of the three, and in a weird way it is the one where self-monitoring is the most honest. That’s because the signals are part of the math from the start.

Value functions are a running guess about how things are going

Most RL agents learn a value function: an estimate of how much total future reward they expect from the state they’re in. Then every step they get to check that guess against reality. The gap between what they expected and what happened is the temporal-difference error, which in Sutton and Barto’s textbook looks like this:¹

delta = r + gamma * V(next_state) - V(state)

If delta is positive, things went better than expected. If it’s negative, worse. The agent uses that number to update its guess.

Here’s why I think this counts as a primitive kind of self-monitoring. V(state) is literally the agent’s belief about its own prospects, and delta is literally “I was wrong about how this would go, by this much.” It’s a named quantity. You can log it every step. And it’s causal by construction, because it’s what drives learning. Nobody had to discover it inside the network. It was put there on purpose.

The catch is that a value function only knows about expected reward. It doesn’t tell you how sure the agent is about that expectation. A state the agent has seen a million times and a state it has never seen can both come out with V = 5.

Ensembles: knowing what you don’t know

That’s the gap that ensemble methods fill. My favorite example is bootstrapped DQN from Osband et al. (2016).² Instead of one Q-network, you share a convolutional torso and split it into K separate heads, each one estimating action values. In their Atari experiments they used K = 10, and each head trains on a slightly different subset of the experience, picked by a random mask.

At the start of each episode the agent picks one head at random and follows it the whole episode. That’s the exploration trick: since the heads disagree about unfamiliar states, picking different heads sends the agent off in different, committed directions. The paper also points out that you can read the spread across heads as a measure of uncertainty. Their line for this is one I like a lot: bootstrapped DQN “can know what it doesn’t know.”² They noticed that the heads agree at the moments that clearly matter and stay diverse at less important steps.

So now there’s a second named signal. Disagreement across heads is inspectable (just look at the K numbers) and causal (it’s literally how the agent chooses what to explore).

Curiosity: surprise as a reward

The third RL signal is surprise. Pathak et al. (2017) gave an agent a forward model that tries to predict the next state from the current state and action, and rewarded it for how wrong that prediction was.³ The more the world surprised it, the more it wanted to go there.

They had a clever detail. If you measure surprise in raw pixels, the agent gets stuck staring at things that are inherently unpredictable but don’t matter. Their example was leaves blowing in a breeze. So they predicted in a learned feature space, trained so it only captures the parts of the screen the agent’s own actions can affect. With no game reward at all, their Mario agent learned to get through more than 30 percent of Level 1, and picked up avoiding enemies without being rewarded for it. The authors’ guess is that dying cuts off the new stuff the agent wants to see.³

And then the failure mode. Burda et al. (2018) took the thought experiment called the noisy TV problem literally: they put a TV in a 3D maze along with an action that changes the channel.⁴ A prediction-error agent finds that TV basically irresistible, because random channels are always surprising. Adding the TV drastically slowed learning.

That’s a really clean lesson about self-monitoring. The agent’s “I’m surprised” signal was real and causal, and it was measuring the wrong thing. Surprise isn’t the same as “there’s something here I could learn.” This is the part that stuck with me, because it’s the same kind of mistake my own project made, just in a different system.

Scorecard for RL

• Where: named quantities in the objective (value, TD error, ensemble spread, prediction error).
• Inspectable: yes. You can print them.
• Causal: yes, by design. They drive updates, exploration, or reward.

The cost is that these signals are narrow. They are about reward and state novelty, not about “is my reasoning sound.” But inside that narrow scope, RL is where monitoring is most clearly real.

LLMs: the monitoring is somewhere in there, maybe

A standard pretrained language model learns next-token prediction without an explicit self-error head. Post-training can add other objectives, so the training setup matters when we ask where its monitoring comes from.

There are three places people look.

The output distribution

The cheapest signal is the probability the model puts on its own answer. Kadavath et al. (2022) found that larger models are pretty well calibrated on multiple choice when you read the confidence straight off those probabilities, and that a model can be trained to estimate whether its own proposed answer is true.⁵ I covered this in the metacognition post, so I won’t redo it.

A more recent and clever version is semantic entropy from Farquhar et al. in Nature (2024).⁶ Sample the model several times on the same question, group the answers by meaning (so “Paris” and “It’s Paris” count as one answer), and compute the entropy over those groups. If the model keeps giving the same answer, low entropy. If it keeps changing its mind, it’s probably making something up. They aim it at a specific kind of hallucination they call confabulation: answers that are wrong and also arbitrary, changing with the random seed.

Honestly I think semantic entropy is the closest thing LLMs have to bootstrapped DQN’s heads. Both measure disagreement across several guesses. The difference is where it runs. In bootstrapped DQN the heads are part of the agent and the disagreement drives behavior. Semantic entropy is computed outside the model, by code that samples it and compares the samples. Unless the surrounding system feeds it back, the model does not see that number. Keep that in mind, because it comes back in the orchestrator section.

Hidden states

The second place to look is inside, with probes. There’s decent evidence that hidden states carry information about whether an answer is true, which I went through last time. The issue is always the same. A probe shows that we can read a signal. It doesn’t show that the model reads it and acts on it.

This is the exact wall I hit this summer. In a project I ran in June, which I wrote up here, I watched a model’s residual stream for drift during math solutions and stepped in when it crossed a threshold. The drift signal was real, but it tracked what kind of task the model was doing, not whether it was doing it right. On Mistral, intervening where my monitor fired fixed exactly as many problems on net as intervening at a random position. So “there’s a signal in the hidden states” and “the model is monitoring itself” turned out to be very different claims, at least for the signal I picked.

Chain of thought

The third place is the one you can read with your own eyes: the reasoning text. If a model writes “wait, that’s wrong, let me redo step 3,” that looks like monitoring happening in public.

The problem is that the text doesn’t have to be the real reason. Anthropic’s 2025 study slipped hints about the answer into prompts and checked whether reasoning models admitted using them. Averaged across hint types, Claude 3.7 Sonnet mentioned the hint 25 percent of the time and DeepSeek R1 39 percent of the time.⁷ In settings built to have reward hacks, the models exploited the hack in over 99 percent of cases and said so in the reasoning less than 2 percent of the time in most of the environments.⁷

That doesn’t make chain of thought useless as a monitoring surface. OpenAI’s Baker et al. (2025) had GPT-4o read the reasoning of o3-mini in coding environments and found that watching the chain of thought caught reward hacking much better than watching only the actions and outputs.⁸ Then they tried putting that monitor into the training reward. At low pressure it helped. At high pressure the model learned to hide its intent in the chain of thought and kept reward hacking anyway.⁸ A big multi-lab position paper later that year called chain-of-thought monitorability a “new and fragile opportunity” for exactly this reason.⁹

Notice where the monitor in that study lives, though. It’s a second model, wired up by code, reading o3-mini’s text from the outside. That’s an orchestrator.

Scorecard for LLMs

• Where: spread across hidden states, maybe in the output probabilities, maybe in the reasoning text.
• Inspectable: partly. Logits are easy, hidden states need probes, and the text can mislead.
• Causal: mostly unknown. Showing that an internal signal drives behavior needs interventions like patching or steering, and that work is still early.

Orchestrators: the monitoring is just code

The third kind of system is the one most people actually build with right now: a model inside a loop, with code deciding what happens between calls. LangGraph is a good concrete example, because it makes the control flow an explicit graph. You define nodes (call the model, run a tool, check a result) and edges between them, including conditional edges that pick the next node based on the current state.¹⁰

In that world, “metacognition” turns into very ordinary features.

Retries. LangGraph lets you attach a retry policy to a node, with a max_attempts setting that defaults to 3, counting the first try.¹¹ That’s “if this fails, try again, but not forever,” written down.

Asking a human. An interrupt pauses the graph at a chosen point, saves the state with a checkpointer, and waits for outside input before continuing.¹² That’s “I shouldn’t do this without checking,” except the decision about when to check is a line of code someone wrote, not the model sensing its own uncertainty.

Reflection loops. Reflexion (Shinn et al., 2023) has the agent write a short reflection on why an attempt failed, store it in memory, and try again with that note in context.¹³ It reached 91 percent pass@1 on HumanEval, against 80 percent for GPT-4 alone. But look at what triggers the reflection: feedback from outside, like failing unit tests. Compare that with Huang et al., who found that when a model has to decide on its own whether its reasoning answer is wrong, with no outside signal, self-correction tends to make things worse.¹⁴ The loop works when something external tells it when to fire.

Auditor agents. At the far end, the monitor is another agent. Anthropic’s 2025 work on alignment auditing agents built LLM agents that investigate a target model for hidden problems.¹⁵ A single investigator agent solved their auditing game 13 percent of the time, and a “super-agent” that combined many investigators’ reports got to 42 percent. Their caveats are worth reading too: the agents sometimes lost track over long investigations, fixated on early hypotheses, and were overconfident. Which is a little funny. The auditor has its own metacognition problem.

This lines up with how I build. On the Dear Future Me voice app, the model generates the reply, but a FastAPI backend checks every request before anything gets written, and letters only go out by SMS to numbers that have confirmed a code first. None of that depends on the model knowing when it might be wrong. The orchestration limits some consequences of a model mistake; it does not prevent every wrong save or message.

Scorecard for orchestrators

• Where: in plain code (tests, validators, retry limits, interrupts, a judge model).
• Inspectable: completely. It’s a graph you can read and a log you can replay.
• Causal: yes. An if statement decides what happens next.

Side by side

Put the three together and a pattern shows up.

RL agents have real, causal self-monitoring, but only for the narrow things their objective names. They can tell you they’re uncertain about a state. They can’t tell you their plan is dumb.

LLMs probably carry richer information about their own reliability, since they model so much of the world. But it’s spread across hidden states, the words they say about it aren’t reliable, and we mostly can’t show it drives behavior.

Orchestrators have monitoring that’s fully inspectable and fully causal, but it’s not really the system knowing anything about itself. It’s a developer’s guesses about failure modes, written as code. A retry limit doesn’t know why the call failed. An interrupt doesn’t know how uncertain the model is. They fire on rules.

And look at the most interesting LLM-side signals, semantic entropy and chain-of-thought monitoring. Both are computed by code running outside the model. The signal is about the model, but the monitoring loop lives in the orchestrator.

My view

What interests me most is whether a system can use a warning about its own answer to make a better decision. It might check a step, try a different approach, or ask for help. Where that warning comes from matters because it changes what we can inspect and test.

A monitor written in code is easier to trace. A signal learned inside a model could be useful too, if it notices problems the surrounding code misses. I’d be curious to train a model to predict its own mistakes and see whether changing that prediction changes its behavior. The hard part would be showing that it learned to recognize trouble, rather than another way of sounding careful.

Sources