What would convince me a model monitors itself?

Two days ago I wrote about what metacognition could mean for a language model. I broke the word into six mechanisms, ranked them by how much I’d trust a positive result, and ended with a short list of what would actually convince me. That list was loose. This post is me trying to make it strict.

So here is the question. Suppose someone, maybe me, says “we found the part of this model that knows when it’s about to be wrong.” What would they have to show before I believe them?

These are the six checks I’d start with, using methods from the papers below. I’d hold my own work to them, and it doesn’t clear them yet.

Start with a picture that is not proof

Here is a heatmap from Marks and Tegmark’s paper “The Geometry of Truth.”¹ They fed LLaMA-2-13B a statement like “Chicago is in Canada. This statement is:” and a true version with Toronto. Then they took internal activations from one run and patched them into the other, one layer and one token position at a time, and measured how much the model’s lean toward “TRUE” changed.

It’s a beautiful figure. You can see the city name matter in early layers, then something happen around “Canada” and the period in the middle layers, then the final token carry it from there on. It looks like you’re watching the model figure out that a statement is false.

And it is real evidence. But a heatmap like this is the start of an argument, not the end. It tells you where information that changes the output is sitting. It doesn’t tell you what that information is, whether it’s the same thing on a different kind of sentence, or whether the model would ever use it to change its own behavior. For metacognition, that last part is the whole claim.

A convincing picture can still be a poor explanation. Adebayo et al. showed in 2018 that some popular saliency maps for image models look almost the same when you randomize the model’s weights, meaning the picture wasn’t really reflecting what the model had learned.² Jain and Wallace argued the following year that attention weights in NLP models often don’t line up with other measures of which inputs matter, and that you can find very different attention patterns that give the same prediction.³ A heatmap is a summary someone chose to draw. It can’t tell you whether it’s lying.

So the bar has to be built from experiments where something could fail.

The bar

Here are the six parts, in the order I’d test them. Each one builds on the one before.

1. It predicts future errors, beyond the logits

The first requirement is that there is some internal representation that predicts the model is about to get something wrong, and that it predicts this better than what you could already read off the output.

That second clause matters more than it sounds. Kadavath et al. found that big models are already fairly well calibrated when you just look at the probability they put on their answer.⁴ So if my fancy internal signal predicts errors with the same accuracy as “the top token had low probability,” I haven’t found anything new. I’ve found the logits again, one layer earlier.

The tool here is a probe: a small classifier trained on hidden activations to predict some label. Alain and Bengio introduced linear probes in 2016 as a way to ask how much information about the label is linearly available at each layer.⁵ For this test the label is “this answer ends up wrong,” checked against the answer key, not against the model’s opinion of itself.

There’s a classic trap with probes. A flexible enough probe can learn the task by itself, so high accuracy might be the probe being clever instead of the model. Hewitt and Liang’s fix was control tasks: train the same probe on a nonsense version of the labels and report selectivity, the gap between real accuracy and control accuracy.⁶ I’d want something like that here, plus the logit baseline, plus a length baseline, because longer solutions are wrong more often and a probe will happily learn “this is long.”

What it looks like when this fails: my own project from this summer, which I wrote up here. I watched how far a model’s hidden state drifted during a math solution. On Mistral, where the model got enough problems wrong to actually test it, the drift score separated right from wrong answers at about 0.53, where a coin flip is 0.5. The signal was great at telling a math problem apart from a summarization task. It just had nothing to say about whether the math was right. That project fails part one, so nothing above it even gets asked.

2. It lives at a reproducible depth, before the words

Next, I’d want the signal localized. If it really is a monitoring signal, it should show up at a consistent range of layers, across random seeds and across different problems, and it should show up before the model writes anything that gives the error away.

The “before” part is the key. If the probe only lights up after the model has written “wait, that’s not right,” then all it’s reading is the model’s own text. That’s just self-report again, which I argued last time is the weakest evidence there is. I want the signal at the token where the mistake happens, or earlier, while the output still looks fine.

Alain and Bengio’s original finding was that, in the networks they looked at, linear separability of the labels tended to grow as you went deeper.⁵ That’s a useful reminder that “the probe works at layer 20” can just mean “everything is more readable at layer 20.” Localization means the error signal has its own profile across depth that’s different from the profile of everything else, and that the profile holds up when you rerun it.

This is also where the Attention Residuals paper from the Kimi team gets interesting to me. In a normal transformer, each layer’s output gets added into the residual stream with a fixed weight of one. AttnRes replaces that with a learned softmax over earlier layers’ outputs, so each layer chooses, depending on the input, how much to draw from each earlier depth.⁷ The authors built it to fix how hidden states grow with depth, not for interpretability. But it means the model is writing down, at every layer, which earlier depths it’s leaning on. If a monitoring signal lives at some depth, an architecture like that might literally show you when later layers start pulling from it. That’s a guess. I’d want to test it before I believed it.

3. It changes what the model looks up

This is the part people skip. A monitoring signal that nothing reads is a thermometer in an empty room. If the model monitors itself, then detecting a conflict should change what it does next, and the most concrete version of “what it does next” is what it goes and retrieves.

Retrieval here can mean two things. Outside the model, it means calling a search tool or a document store. Self-RAG (Asai et al.) trained models to emit special tokens that decide when to retrieve and to critique what they got back.⁸ That’s retrieval controlled by the model, which is great, but it’s trained behavior on the output. It doesn’t tell you what internal state triggers it.

Inside the model, retrieval means attention: which earlier tokens, and with something like AttnRes, which earlier layers, a position pulls from. My test would be this. Find cases where the model’s stored knowledge conflicts with something in the context, or where an earlier step conflicts with the problem statement. Check whether the signal from parts one and two fires. Then check whether, when it fires, the model’s retrieval pattern changes in a specific way, like attending back to the problem statement more, or calling a tool when it otherwise wouldn’t.

We know models handle conflicts in messy ways. Xie et al. found that LLMs can be very receptive to outside evidence that conflicts with what they “know,” as long as that evidence is coherent, but also show a kind of confirmation bias when some of the evidence agrees with their memory.⁹ What I want is the internal version: did a detectable conflict signal come first, and did retrieval change because of it? If retrieval does not change, that would weaken this particular hypothesis; the signal might still affect stopping or another behavior.

4. Necessity: take it away and the behavior breaks

Parts one through three are still correlations. To make a causal claim you have to intervene.

The first intervention is ablation: remove or scramble the signal and see if the behavior depends on it. If there’s an “I’m probably wrong” direction in the activations, project it out (or swap in its value from a different input) and check whether the model stops hedging, stops retrieving, and stops correcting itself on exactly the cases where it used to.

Heimersheim and Nanda’s practical guide to activation patching makes a distinction I found really clarifying.¹⁰ When you take a clean run and patch in activations from a corrupted run, called noising, you learn which parts were necessary to keep the behavior. When you go the other way, called denoising, you learn which parts were sufficient to bring it back. They also point out that these are not complements. One does not imply the other.

There’s a control I’d insist on here, and it’s the one that hurt me most this summer. Whatever intervention you run at the signal’s location, also run it at a random location. In my drift project the monitor’s net recovery on Mistral was +3 problems, and intervening at a random spot also got +3. If an ablation of the “monitor direction” breaks things exactly as much as ablating a random direction of the same size, you’ve learned that the model doesn’t like being poked, and nothing else.

5. Sufficiency: put it in and the behavior follows

The flip side. Take a run where the model is confidently right and doesn’t hedge. Patch in the signal from a run where it was about to be wrong. Does it now hedge, recheck, or retrieve, even though nothing about the problem changed?

This is the idea behind causal tracing in Meng et al.’s ROME paper.¹¹ They corrupted the subject of a factual prompt with noise, then restored clean hidden states one location at a time to see which ones brought back the right answer. They found that middle-layer MLPs at the last subject token mattered a lot for recalling facts. Vig et al. had earlier framed the same kind of experiment as causal mediation analysis, measuring how much of a model’s behavior flows through specific neurons and attention heads, and found the effects for gender bias concentrated in a fairly small part of the network.¹²

For metacognition, sufficiency is the most convincing single result I can imagine. If adding a direction to a confident, correct run makes the model say “let me double check that,” and adding a random direction of the same size doesn’t, then the model really is reading that direction. One honest caveat: a big enough push in almost any direction will knock a model off course. The test has to show the specific behavior you predicted, at the size of shift you actually see in natural runs, not just “the output got weird.”

6. It survives a change of format

Last, the signal has to generalize. If I find a monitoring signal on GSM8K word problems, I want to see the same direction, at roughly the same depth, doing the same job on a different format: multiple choice, code, a factual question, a problem written in a different style.

The Geometry of Truth paper is a good model for how to test this. They trained truth probes on one dataset and evaluated on others, and found that generalization got better with model size and when training included statements alongside their negations.¹ They also found that a probe trained on a dataset where “likely text” and “true text” came apart did badly, sometimes below chance where probability and truth were anti-correlated. That’s exactly the failure I’d worry about for an error signal: a probe that learned “this looks like the kind of problem I get wrong” instead of “I’m getting this one wrong.”

Kadavath et al. saw the same issue with their P(IK) predictor. It partly generalized across tasks, and calibration on new tasks was the weak spot.⁴ I would want to know how much the mechanism depends on the format of the input. A signal tied to one dataset’s formatting would support a much narrower claim.

Putting the pieces together: causal scrubbing

Each part above tests one claim. The strictest version tests the whole story at once, and that’s what causal scrubbing from Redwood Research is for.¹³ You write your hypothesis down precisely: these components compute this, and everything else doesn’t matter for this behavior. Then you resample every activation your hypothesis says is irrelevant, swapping it with the value from a different input that your hypothesis treats as equivalent, and you check whether the model’s performance survives. If your story is complete, the model shouldn’t notice. Whatever performance gets lost is the part of the behavior your story doesn’t explain.

I like this because it turns “I found the monitor” into a claim with a number attached. For a metacognition claim, the hypothesis would be something like: the error signal at layers X to Y is computed from these inputs, read by these heads, and drives hedging and retrieval. Scrub everything else. How much of the model’s self-correcting behavior is still there?

I don’t think anyone has done this for anything as complicated as self-monitoring in a big model, and I’m not sure it’s tractable yet. But it’s the right shape for the final test.

Why this order

The order isn’t random. Prediction comes first because without it there’s nothing to localize. Localization comes before use because you need to know where to look. Use comes before the causal tests because if the model never acts differently when the signal fires, there’s no behavior to break or restore. And generalization comes last because it’s the most expensive, and I don’t want to spend it on a signal that already failed something cheaper.

It also maps onto the ranking from my last post. Self-report sits at the bottom there because it’s the same machinery as any other text. Nothing in this bar asks the model what it thinks about itself. The model’s words only show up as one of the behaviors you measure in parts three through five, never as the evidence.

What I’d actually do next

I’d begin with a small question: can a model predict which of its answers are wrong before it sees the answer key? Then I’d change the signal that seems to carry that prediction and check whether its decisions change too. The interesting result would be a model asking for help on the problems where help actually improves its answer.

That would make uncertainty useful in a way that a cautious sentence on its own isn’t. I could imagine an assistant spending more time on a shaky step, checking a calculation, or handing a problem back to a person. Before trusting any of that, I’d want to see it work on unfamiliar problems where sounding confident is easy and being right is hard.

Sources