The model is half the product. The harness is the other half.

In May 2024 a group at Princeton took one model, GPT-4 Turbo, and pointed it at real GitHub issues. With a retrieval setup and no agent loop, it fixed 2.67% of them. Given a plain shell to work in, it fixed 11%. Given a set of commands designed for how a language model actually reads and edits code, it fixed 18%.¹

Same weights, almost a 7x difference. Nothing about the model changed. What changed was the code wrapped around it.

That code has a name now: the harness. I spend most of my building time inside harnesses. I write code in Claude Code most days, and when I build apps on top of models, most of the work ends up in the gateway that sits between the model and everything it is allowed to touch. The longer I do this, the more I think the harness is half the product, and the half that gets the least careful measurement.

What a harness is

A language model, on its own, is a function from text to text. You give it tokens, it gives you tokens back. It cannot open a file, run a test, remember yesterday, or notice that it is about to do something dangerous. Everything that makes it feel like an agent is code somebody wrote around it.

Here is roughly what lives in that code:

• Tool calling. Which tools exist, how they are described to the model, what arguments they take, and what comes back when the model calls one.
• Memory and context. What goes into the prompt on each turn. The full history, a summary, the last few observations, a project notes file.
• Retrieval. How the harness finds the right files, docs, or past conversations and puts them in front of the model.
• Retries and error handling. What happens when a tool call fails, the output is malformed, or the model gets stuck repeating itself.
• Verification loops. Running the tests, linting the edit, checking the answer before calling the task done.
• Permissions. What the agent can do without asking, what needs a human yes, and what is never allowed.
• Evaluation and logging. Recording what happened so someone can tell later whether it worked.

Anthropic’s agent guide calls the core unit an “augmented LLM,” a model plus retrieval, tools, and memory.² I like that framing because it makes the point obvious. The augmentation is a separate thing you build, and you can build it well or badly.

Same model, different agent

The SWE-agent paper is the cleanest example I know of, so I want to stay on it for a minute.

The setup is SWE-bench, a benchmark of 2,294 real issues pulled from 12 popular Python repositories. The model gets the repo and the issue text, and has to produce a patch that makes the hidden tests pass. When SWE-bench came out in October 2023, the best model they tested, Claude 2, solved 1.96% of the issues.³

Yang et al. then asked a different question. Instead of making the model smarter, what if you made the computer easier for the model to use? They called this the agent-computer interface, or ACI, by analogy with human-computer interfaces. Humans got IDEs. Models deserve their own version.¹

The interface is not fancy. A file viewer that shows 100 lines at a time with line numbers. An edit command that replaces a range of lines. A search command that summarizes matches instead of dumping them. A linter that rejects an edit if it introduces a syntax error, so the model gets told right away instead of discovering it three steps later.

The ablations are the best part, because they show each small choice moving the number:

A couple of these surprised me. Showing the model the whole file did worse than showing it 100 lines (12.7% vs 18.0%). Showing it the full history did worse than showing it only the last five observations (15.0% vs 18.0%). More context made the agent worse. That runs against the intuition that you should just give the model everything, and it is a harness decision through and through.

Anthropic found the same thing from the other side. When they built their own SWE-bench agent, the model kept making mistakes with relative file paths after it changed directories. The fix was to change the tool so it always required absolute paths, and after that, they write, the model used it flawlessly.² No new training. One argument changed in one tool.

In January 2025 they put it plainly in their SWE-bench Verified write-up: agent performance “can vary significantly” based on the scaffolding, even with the same model, and the benchmark evaluates “an entire ‘agent’ system,” not the model alone.⁴ Their scaffold for that result was deliberately small, a prompt plus a bash tool and a file edit tool, and it got the upgraded Claude 3.5 Sonnet to 49%.

Two years of harness work

When people talk about agent progress since 2023, they mostly talk about models. I think if you look at the actual sequence of ideas, a huge share of it is harness work.

ReAct: think, act, look

ReAct, from Yao et al. in October 2022, is where I’d start.⁵ The idea is to have the model interleave reasoning traces with actions. It thinks out loud about what to do, calls a tool (a Wikipedia API in their experiments), reads the result, and thinks again. On ALFWorld and WebShop, two interactive benchmarks, it beat imitation and reinforcement learning baselines by 34 and 10 points of absolute success rate, with just one or two examples in the prompt.

The model was not retrained for this. ReAct is a prompting pattern plus a loop that executes actions and feeds observations back in. In other words, a harness. Almost every agent loop I have used since is a descendant of that thought, action, observation cycle.

Toolformer: teaching the model the protocol

Toolformer, from Schick et al. at Meta in February 2023, came at it from the model side.⁶ They trained a model to decide for itself when to call an API, which one, and with what arguments, using a calculator, a Q&A system, two search engines, a translator, and a calendar. The training was self-supervised and needed only a handful of demonstrations per tool. The result was often competitive with much larger models on zero-shot tasks.

I include it because it shows the boundary moving. Toolformer pushed some of the harness’s job into the weights: knowing when to reach for a tool. But the tools themselves, and the code that actually runs them and returns results, still live outside. Today’s models are trained to emit tool calls, and the harness still defines what those tools are.

MCP: standardizing the plug

By late 2024 every harness had its own way of connecting to tools and data, and every integration was custom. Anthropic released the Model Context Protocol on November 25, 2024, to fix that. In their words, every new data source “requires its own custom implementation,” which makes connected systems hard to scale.⁷ They shipped it with prebuilt servers for things like Google Drive, Slack, GitHub, Git, Postgres, and Puppeteer.

What MCP actually standardizes is worth being precise about, because people sometimes describe it as more than it is. The spec defines JSON-RPC 2.0 messages between three roles: hosts (the LLM application), clients (connectors inside the host), and servers (the things that provide context and capabilities). Servers can offer resources (data), prompts (templated messages), and tools (functions the model can call). Clients can offer servers sampling, roots, and elicitation, which is a way for a server to ask the user for more information. The spec says it takes inspiration from the Language Server Protocol.⁸

So MCP standardizes the plug between a harness and the outside world. It does not standardize the harness. It says nothing about when to retry, how much context to keep, or whether a tool’s output should be trusted. The spec actually says tool descriptions should be treated as untrusted unless they come from a trusted server, and that hosts must get user consent before invoking tools.⁸ That part, deciding what to trust and when to ask, is left to whoever builds the host. That gap is what this post is about.

Coding agents are harnesses you can buy

Then came the products. Claude Code launched as a research preview on February 24, 2025, as a terminal tool that can search and read code, edit files, run tests, and commit to GitHub.⁹ OpenAI released the open-source Codex CLI in April 2025 and a cloud Codex agent that runs tasks in sandboxed environments in May 2025.¹⁰ Cursor took the editor route and built the agent into a fork of VS Code.

You can run some of the same models inside all of these, and they still feel different to use. How they search a repo, how they decide a task is finished, whether they run the tests before telling you it works, when they stop and ask. Those differences are harness differences. When I switch between tools with the same model underneath, I am mostly comparing harnesses, whether or not I think about it that way.

Where the “metacognition” actually lives

I’m interested in whether a model can recognize when it is wrong or should stop. With an agent, we also have to ask how much of that behavior comes from the code around it.

My view is that much of the reliable checking I use in agents lives in the harness.

• “Did that work?” The model’s report alone is not enough. The harness can run the tests or the linter and put the result in front of it.
• “I’m stuck, try something else.” Usually a retry policy, a loop detector, or a step limit written in plain code.
• “Should I ask the user?” In Claude Code this is mostly a permission system. Some actions need approval and some do not, and that line is set by configuration, not by the model judging its own uncertainty.
• “Am I done?” Often a check the harness runs, or a stop condition, more than a calibrated internal sense of completion.

Building apps on top of models taught me this in a very concrete way. The dangerous failures are quiet ones: the model confidently writes the wrong thing somewhere permanent, or keeps going when it should stop and check. The fixes I end up trusting are in the harness, not the prompt: checks that run before anything gets saved, validation on what the model is allowed to call, limits on how many times it can retry. Stuff like that. None of it needs the model to know it is uncertain. The harness can limit the consequences of some mistakes, though validation alone cannot catch every wrong action.

I don’t think this is bad. Explicit control in code is inspectable and testable in a way an internal model state is not. But it means that if you want to know how “self-aware” an agent is, you are mostly asking about its harness. And that raises an awkward question about how we measure these things.

Model benchmarks vs agent benchmarks

A model benchmark tries to hold everything fixed except the weights. Same prompt format, same tools (often none), same scoring. MMLU-style multiple choice is the classic shape. You learn something about the model.

An agent benchmark like SWE-bench scores the whole system. The model, the prompt, the tools, the context strategy, the retry logic, the step budget. When a SWE-bench number goes up, you usually cannot tell from the number alone how much came from the model and how much from the harness. Anthropic says this directly about their own result.⁴

Even the benchmark has needed harness work. In August 2024 OpenAI and the SWE-bench authors released SWE-bench Verified, 500 tasks screened by human annotators after they found that many original tasks were underspecified or had unfair tests, plus a containerized evaluation setup to make runs more reliable.¹¹ So the harness that grades the agent matters too.

Kapoor et al.’s “AI Agents That Matter” makes the broader case that agent evaluation is in rough shape.¹² Their critiques: leaderboards focus on accuracy and ignore cost, so agents become needlessly expensive. Many benchmarks lack proper holdout sets, so agents overfit. Evaluation practices are not standardized, so results are hard to reproduce. And the needs of model developers and application developers get mixed together.

The gap: separating the harness from the model

It’s strange how different two products can feel when they use the same model. The surrounding software decides what the model sees, which tools it can use, and what happens after an error. I want to know how much better we could make that experience without changing the model at all.

A useful comparison would hold the model fixed and change one part of the setup at a time. Give it fresher context, a clearer tool response, or a way to check its work, then see whether it completes the task more reliably. I’d also want to see what happens when a tool fails halfway through.

That could make progress easier to understand. Instead of treating every improvement as evidence of a smarter model, we could see which changes actually helped the whole system. It would also give people building smaller products more concrete ideas to try.

Sources