What metacognition means for an AI model, concretely

Ask a chatbot a hard question and it will sometimes say “I’m not completely sure, but I think the answer is 42.” That sentence sounds like self-awareness. It sounds like the model checked something inside itself, found it shaky, and told you.

Maybe it did. Or maybe it learned that people who write “I’m not completely sure” tend to get thanked for being honest, and it puts that phrase in front of answers that look like the hard kind. From the outside those two stories produce the exact same text. That gap is what this post is about.

I keep seeing “metacognition” used for language models as if it were one thing a model either has or doesn’t. I don’t think it is one thing. When I try to pin it down, it falls apart into at least six separate mechanisms, and each one can be tested in a different way. Some of them have real evidence behind them. One of them, the one people point to most, is honestly the least trustworthy.

Where the word comes from

The term comes from developmental psychology. John Flavell used it in a 1979 paper in American Psychologist about how children monitor their own memory and understanding.¹ The studies he described found that young kids know surprisingly little about how their own thinking works, and do little checking of it. A kid would say they had memorized a list when they clearly had not.

Flavell split the idea into knowledge about cognition (what I know about how I think, what tasks are hard for me, which strategies work) and the monitoring and control of cognition while it happens. That second half is the part I care about for models. Knowing facts about yourself is one thing. Using a signal about your own state to change what you do next is the interesting part.

Notice the kid example, too. The kid’s report (“I memorized it”) was wrong. Even in the original human research, self-report and actual monitoring were treated as different things that could disagree. That turns out to be the whole story for models as well.

Six things “metacognition” could mean

Here is the breakdown I use. For each one, I’ll say what it would mean concretely for a transformer, and how you’d check.

1. Confidence estimation

The model has some quantity that tracks how likely it is to be right. This is the most measurable one. You take a bunch of questions, record the model’s confidence on each, and check calibration: of all the answers where it said 70 percent, are about 70 percent actually correct?

The best known result here is Kadavath et al. (2022) from Anthropic, titled, nicely, “Language Models (Mostly) Know What They Know.”² They found that larger models are well calibrated on multiple choice and true/false questions when the question is formatted the right way. The confidence here is the probability the model assigns to the answer letter, read straight off the output distribution.

They went further. They had the model propose an answer and then estimate the probability that its own answer was true, which they called P(True). It worked reasonably well, and it worked better when the model got to look at several of its own sampled answers before judging one. They also trained the model to predict P(IK), the probability that it knows the answer at all, before seeing any proposed answer. That partially generalized across tasks, with calibration on new tasks being the weak spot.

One detail from the same paper I think gets skipped a lot. The RLHF-finetuned versions of their models looked badly miscalibrated, and a simple temperature adjustment (T = 2.5) mostly fixed it.² So the information was still in there. The way the chat model expressed it was just bent out of shape by training.

2. Error monitoring

Confidence is a number about a whole answer. Error monitoring is noticing, mid-process, that a specific step went wrong. In a model that would mean some internal signal that changes when a reasoning step contains a mistake, ideally near the step itself.

There is decent evidence that hidden states carry truth-related information. Azaria and Mitchell trained a small classifier on an LLM’s hidden-layer activations to predict whether a statement was true, and got 71 to 83 percent accuracy on a balanced test set depending on the model, including on statements the model generated itself.³ That is a probe, not the model monitoring itself. It shows the information is readable by us. It does not show the model reads it.

That distinction matters a lot and I’ll come back to it.

3. Strategy selection

Picking how to approach a problem based on what kind of problem it is and how capable you are. For a person: “this is long division, I should write it out instead of doing it in my head.” For a model, it might look like choosing to write out a chain of thought for a hard problem and answering directly for an easy one, or deciding to use a calculator tool.

This one is hard to separate from plain pattern matching. If a model writes out steps for anything that looks like a word problem, that is a habit triggered by surface features. Real strategy selection would mean the choice depends on the model’s own likely success, which means it would have to use something like signal 1.

4. Self-correction

Going back and fixing a mistake. This is the one people most want, because it would make models more reliable without outside help.

The cleanest test I know is Huang et al. (2023), “Large Language Models Cannot Self-Correct Reasoning Yet,” published at ICLR 2024.⁴ Their key move was separating two setups. In earlier work, the self-correction loop often used the correct answer to decide when to stop: if the answer was already right, don’t revise. That makes self-correction look great, because you never let the model break a right answer. With those oracle labels, GPT-3.5 went from 75.9 to 84.3 percent on GSM8K.

Then they removed the oracle. The model had to decide for itself whether to keep or change its answer. Accuracy went down across the board. GPT-3.5 went from 75.9 to 74.7 on GSM8K after two rounds, and on CommonSenseQA it collapsed from 75.8 to 41.8 after two rounds.⁴

Their explanation centers on the tested models’ difficulty judging whether their own reasoning was correct. Self-correction is downstream of error monitoring. If the monitoring signal is weak, the correction step mostly adds noise, and noise breaks right answers about as often as it fixes wrong ones.

5. Stopping and continuation control

Deciding when you’re done. When to stop thinking, when to keep going, when to give up and say you don’t know. For a reasoning model this is very concrete: how many tokens of thinking it spends on a problem.

Chen et al. (2024) looked at “overthinking” in o1-style reasoning models, where the model spends a lot of computation on simple problems for little benefit. The title of the paper is literally “Do NOT Think That Much for 2+3=?”⁵ If a model had good control over its own process, spending should scale with difficulty. When it doesn’t, that tells you the stopping decision isn’t tightly connected to the model’s actual uncertainty.

What I like about this mechanism is that it is a behavior with a cost attached. Stopping early on a hard problem is wrong in a measurable way. Thinking for a thousand tokens about 2 + 3 is wasteful in a measurable way. You can grade it.

6. Verbalized self-report

The model says something about its own state in words. “I’m not sure.” “I made an error in step 3.” “I’m about 80 percent confident.” Or, in a chain of thought, an explanation of why it chose an answer.

This is the one everyone notices, because it is the only one you can see without instruments. It is also, in my view, the least trustworthy of the six.

Why self-report is the weakest evidence

To be fair, verbalized confidence is not useless. Lin, Hilton and Evans (2022) finetuned GPT-3 to state its confidence in words (“90% confident”) on a set of math tasks they built called CalibratedMath, and those verbal numbers came out reasonably well calibrated, staying moderately calibrated under distribution shift.⁶ They argued the model was responding to its own uncertainty rather than just copying human examples. So words can carry a real signal, at least after targeted training.

But without that training, the picture is worse. Xiong et al. (ICLR 2024) tested how well you can get confidence out of LLMs just by asking, and found verbalized confidence tends to be overconfident, possibly because models imitate how humans talk about confidence.⁷ Failure prediction stayed close to chance, with AUROC in the 0.52 to 0.61 range.

The deeper problem shows up with chain-of-thought explanations. Turpin et al. (NeurIPS 2023) added biasing features to prompts, like reordering the few-shot examples so the correct answer was always “(A).”⁸ The models picked up on that and shifted their answers. Then they wrote explanations that rationalized the biased answer and almost never mentioned the real reason. Accuracy dropped by as much as 36 percent across 13 BIG-Bench Hard tasks, and the stated reasoning didn’t reflect why.

Lanham et al. (2023) tested faithfulness from another angle, by editing a model’s chain of thought (adding mistakes, paraphrasing it) and checking whether the final answer changed.⁹ How much the answer depended on the written reasoning varied a lot by task. And on most tasks they studied, larger and more capable models produced less faithful reasoning.

Put together, here is the point that matters most. A sentence like “I’m not sure” is an output. It is produced by the same next-token machinery as every other output, trained on human text full of people hedging. For it to count as metacognition, it has to be caused by an internal signal that actually tracks the model’s reliability, and that signal has to be the one driving the words. None of that is guaranteed. A model can have a perfectly good internal uncertainty signal that it never verbalizes accurately (the RLHF temperature result suggests something like that). And a model can say all the right humble things with nothing behind them.

So when I rank the six by how much I trust a positive result, it looks roughly like this:

most trustworthy
  confidence estimation   (graded against ground truth, at scale)
  stopping / continuation (behavior with a measurable cost)
  self-correction         (only if tested without oracle labels)
  error monitoring        (needs internals + causal tests)
  strategy selection      (hard to separate from habits)
  verbalized self-report  (same machinery as any other text)
least trustworthy

I put them in that order because I trust signals tested against ground truth and linked to behavior more than another fluent sentence about confidence.

What would actually convince me

So what evidence would I want before I believe a model monitors itself? I would start with the following checks.

There is an internal signal that predicts errors, graded against ground truth. Not against the model’s own judgment, not against another model’s judgment. The final answer is right or it isn’t. And it should beat simple baselines like output length or token probability.

The model uses that signal. This is the step most work skips. A probe reading a signal from the activations proves we can read it. To show the model reads it, you need a causal test: intervene on the signal (patch it, ablate it, steer it) and show that downstream behavior like stopping, revising, or hedging changes in the way you’d predict. If you flip the “I’m probably wrong” direction and the model doesn’t change anything, it isn’t monitoring with it.

The behavior helps without an oracle. Huang et al. made this point well. Any self-correction or self-check has to improve outcomes when the model alone decides when to act. Otherwise you’ve measured the oracle.

It beats a random version of itself. If intervening at random positions works as well as intervening where the “monitor” fires, the monitor isn’t telling you anything.

Self-report agrees with the internal signal. Only after all of the above would I care about the words. And then the interesting question is whether the verbal confidence tracks the internal signal, or the internal signal and the verbal output come apart. When they come apart, test both against ground truth; an internal signal can be miscalibrated too.

What I’m looking at next

I’d like to see a model notice a shaky step while there is still time to do something about it. Asking for help on the right problem would be much more interesting than adding a sentence about uncertainty to every answer.

The next question is how to tell whether that warning is useful. Does it show up more often before an actual mistake? If we change it, does the model make a different choice? I find the possibility exciting because it could make models easier to work with when the answer isn’t obvious. But it needs experiments that connect the warning to what the system actually does.

Sources