What I'm looking into this year

This is a living post. Senior year is underway, and this is a list of the questions I want to spend it on, what I’ve read so far, and what I’d build to get a better answer. I’ll update it as the answers change, or more likely as the questions do.

There are four of them: world models, optimizers, evaluating AI harnesses, and metacognition, which for me is really a question about interpretability. They look like separate topics. I think they share one thread, which I’ll get to at the end.

World models

A world model is a model that learns how an environment behaves, so it can predict what happens next when you take an action. The question I care about is simple to say and hard to answer: does training something to predict the world give it a usable picture of the world, one you can plan inside?

The paper that made this click for me was Ha and Schmidhuber’s “World Models” from 2018, where an agent learns to play inside the model’s own dream of the game and then transfers to the real one.¹ I wrote up the history from there, through DreamerV3² and Genie 3,³ and what I think it means for software agents, so I won’t repeat it here. Yann LeCun has argued for years that predicting in an abstract representation space, instead of predicting every pixel, is the way to get machines that actually understand their surroundings.⁴

What I still want to know is how you’d tell a world model is any good. A video that looks right and a model that has the physics right are different things, and the first is much easier to show off. A small project I’d like to try: train a tiny world model on a simple 2D physics sim, then test it on situations it never saw, like a ball hitting a wall at a new angle. Does it generalize the rule, or does it just interpolate between clips it remembers? That’s a question you can actually answer on a laptop.

I haven’t started on that yet. It’s on the list.

Optimizers

Every model I’ve written about on this blog is trained by some version of the same loop. You compute how the loss changes if you nudge each weight, and you step downhill. I walked through that from one neuron up in my neural network post. The optimizer is the rule for how big a step to take and in which direction. The question is whether we’re still using a good rule, or just a familiar one.

Over the summer I did a tour of optimizers and a deeper look at Adam, so this is the short version. The familiar rule is Adam, from 2014, which keeps a running average of each weight’s gradient and its size, and scales each step accordingly.⁵ Its cousin AdamW, which handles weight decay separately, is a common default for transformers.⁶ When I reread the original Transformer paper, an Adam-style optimizer was one of the things that survived basically unchanged since 2017 (that post is here).

The challenger I’ve been reading about is Muon, from Keller Jordan and collaborators in late 2024. It’s built for the 2D weight matrices in the hidden layers. It takes the usual momentum update and runs a few Newton-Schulz iterations on it to roughly orthogonalize it before applying it.⁷ In plain terms, it tries to make the update push in many directions evenly instead of letting a few directions dominate. It set speed records on the NanoGPT and CIFAR-10 speedrun benchmarks.⁷ Then Moonshot AI scaled it up to large language models and reported about 2x compute efficiency over AdamW in their scaling experiments.⁸

Here’s the paper that made me slow down, though. Wen, Hall, Ma, and Liang tuned AdamW baselines carefully and compared a bunch of new optimizers against them. A lot of the claimed speedups shrank. The best ones, matrix-based methods like Muon and Soap, gave about 1.4x at 0.1B parameters, but that fell to about 1.1x at 1.2B.⁹

That gap between “2x” and “1.1x” interests me because both results can be honest. It depends on how well you tuned the baseline and at what scale you measured. My view is that the optimizer question is partly a physics question (what is the geometry of the loss surface) and partly an evaluation question (did you give the old method a fair shot).

What I’d build: a small, careful comparison of AdamW and Muon on a tiny language model, where I spend as much effort tuning AdamW as tuning Muon, and report the speedup with error bars across seeds. It won’t settle anything at scale. It’d teach me what “well-tuned baseline” actually costs in practice.

Evaluating AI harnesses

A harness is everything around a model that turns it into something that does work: the prompt, the tools it can call, the loop that feeds results back, the memory, the retries. When people report that “model X scores Y on a coding benchmark,” they’re really reporting a model inside some harness. So how much of the score is the model, and how much is the harness?

The clearest evidence I’ve found is the SWE-agent paper. The authors argue that language-model agents are a new kind of user that needs its own interface, and they design one for editing files, searching a repo, and running tests.¹⁰ On SWE-bench Lite, GPT-4 Turbo inside their interface resolved 18.0% of issues. The same model with only a plain Linux shell resolved 11.0%.¹⁰ Same weights, different harness, a big change in the number.

I’ve felt a smaller version of this myself. At the VoiceOS hackathon, the hard part of a voice product, knowing when someone’s done talking, when to follow up, when to just save the thought, was handled by the platform around the model, not the model itself (the Dear Future Me post). The experience depended on more than the model call. The experience depended on everything else.

The other reason I care is the project I worked on this past summer, which became my MATH-AI workshop paper. I was testing a method that watches a model’s internal state for signs a math solution is going wrong. When I scored the runs by asking the model afterward whether its reasoning stayed coherent, my method looked great. When I checked the final answers against the answer key, it barely did anything, and a random trigger did about as well. The scoring setup was part of the harness, and it almost fooled me.

So my view is that “how good is this harness” is mostly a controls problem. You want to compare against doing nothing, against a dumb baseline harness, and against the same harness with each piece removed, and you want to score by outcomes you can check. What I’d build is a small evaluation kit that does exactly that: take one fixed model, swap harness pieces in and out (tools, retry loop, self-check step), and report how much each piece is worth on tasks with checkable answers.

Metacognition and interpretability

Metacognition is thinking about your own thinking. For a model, the practical version is: can it tell when it’s about to be wrong? And if it says it’s confident, is that report connected to anything real inside it?

There’s some good news. Kadavath and colleagues at Anthropic found in 2022 that larger models are well calibrated on many multiple-choice and true/false questions when they’re asked in the right format, and that models can be trained to predict whether they know the answer to a question. That last part only partly generalizes to new tasks.¹¹

A more recent paper goes inside the model. Jack Lindsey at Anthropic injected known concepts directly into a model’s activations and then asked the model whether it noticed anything. In some cases, models did notice and correctly named the injected concept. The paper’s own conclusion is careful: there’s some functional introspective awareness, and it’s highly unreliable and context-dependent.¹²

The interpretability side is the part I’m most drawn to. A model’s internal state is a big vector that every layer reads and writes, and the meaningful directions in it are packed together in superposition, which I wrote about in how a word gets meaning.¹³ If “I’m about to be wrong” is represented somewhere, it’s probably a direction, and finding it is an interpretability problem.

My MATH-AI paper ended in a negative result on this. The unsupervised drift score I used couldn’t tell a correct solution from an incorrect one. The model’s self-judgment was even worse, because it rated runs where I did nothing as coherent too. That tells me two things. A model’s own report about its reasoning can’t be taken at face value. And my particular signal did not reliably expose correctness, which leaves open whether another representation would.

What I’d build next is a supervised probe: train a simple linear classifier on the model’s activations to predict whether the final answer will be right, on a benchmark hard enough that the model fails often. If that works, the information is there and my method just didn’t find it. If it does not, I would still need to distinguish a weak probe or dataset from an absence of the information. Either way I learn something.

The thread

These projects keep bringing me back to the same curiosity: how much can I understand by building a small version and watching where it fails? A tiny model or a simple agent won’t answer every question about a bigger system, but it gives me something concrete to change and inspect.

I expect some of these ideas to lead nowhere. What would make the year worthwhile is coming away with better questions, and a clearer sense of which explanations survive an actual experiment. That’s what I want the writing here to follow as well.

Sources