How do you know your agent works?

On one of the most cited agent benchmarks, an agent that does absolutely nothing scores 38% on the airline tasks. The GPT-4o baseline in the original paper scored 35.2%.¹²

That number is the reason I wanted to write this post. In my last post I argued that the harness, all the code around a model, is half of what makes an agent good, and that its contribution is often hard to separate from the model’s. I ended by promising to work out what that would look like. Before you can benchmark a harness, though, you need a more basic skill: knowing whether an agent works at all. It turns out that is harder than it sounds, and the ways it goes wrong are pretty consistent.

This is the checklist I have ended up with. Most of it I learned from papers. Some of it I learned by getting fooled.

The way I got fooled

A project I ran this summer was about catching math mistakes inside a language model while it was still writing the solution. I wrote about it briefly before, and in full here. The short version: my first evaluation asked the model whether its own reasoning had stayed coherent after my method stepped in. That score said 82 to 95 percent “recovery” on Mistral.

Then I checked the final answers against the answer key. The real rate of wrong answers turning right was 5 to 20 percent. And the model gave high scores to runs where I had done nothing at all.

So a judge rated doing nothing as success. That one sentence is basically this whole post. Every item below is a way of making sure the thing you are measuring can actually tell your system apart from no system.

Checks 1 to 3: is it better than nothing?

1. Score the outcome, not the story

An agent produces two things: a result, and a description of what it did. The description is cheap. The agent can say it ran the tests, fixed the bug, and booked the flight. What you care about is whether the tests pass, whether the bug is gone, and whether the booking exists.

τ-bench (tau-bench) is a nice example of doing this on purpose. It puts an agent in a simulated customer service job, airline or retail, with API tools, a policy document, and a user played by another language model. At the end it does not ask anyone whether the conversation went well. It compares the final state of the database to the annotated goal state.² Either the right order got cancelled and the right refund issued, or it did not.

That is the gold standard when you can get it: a check that does not care how confident the agent sounded. For code, it is a test suite. For math, it is the answer key. For a booking agent, it is the database.

The catch is that outcome checks can also be wrong, just in a quieter way. I will come back to that.

2. Compare against doing nothing

It is easy to overlook a baseline this simple.

Back to that 38%. Zhu et al. went through a set of agent benchmarks looking for flaws in how tasks and rewards were set up. In τ-bench’s airline domain, 38% of the tasks are intentionally impossible: the correct behavior is to refuse, so success means leaving the database unchanged. An agent that returns empty responses leaves the database unchanged. So it passes all of them.¹

To be fair to τ-bench, testing whether an agent knows when to refuse is a reasonable thing to want, and newer models score well above 35.2%. The problem is only that a single success rate hides it. If you report 40% on airline without the do-nothing line next to it, nobody can tell that most of that 40% might be refusals that cost the agent nothing.

The same paper found similar holes elsewhere. On SWE-Lancer, they report an agent could score 100% without solving anything by overwriting the test files with a trivial assertion. On KernelBench, they estimate agent ability was overestimated by 31 percentage points because the fuzz tests only varied tensor values and never shapes. Overall they say these setup issues can cause performance to be misjudged by up to 100% in relative terms.¹

So the first row in any results table I make now is “do nothing.” If the agent cannot clearly beat it, nothing else in the table matters.

3. Compare against a random or trivial policy

Doing nothing is the floor. The next step up is a policy with no intelligence in it: act at random, or apply the simplest possible rule.

In my project, this was the result that actually stung. My method watched the model’s hidden state and stepped in when it looked like the reasoning was going wrong. On Mistral, it fixed a net 3 problems out of 56. Stepping in at a random point, without looking at anything, also fixed a net 3. Whatever was helping, it was not the part I was proud of. It was the act of interrupting at all.

The agent version of this is the simple baseline: the dumbest thing that could plausibly work. Kapoor et al. did this for coding agents on HumanEval and the comparison is revealing.³ They compared fancy published agent designs against three baselines:

• Retry: run GPT-4, and if the code fails the example tests, run it again, up to five times.
• Warming: the same, but raise the temperature a bit on each retry.
• Escalation: start with a cheap model and switch to a more expensive one only when tests fail.

Warming got 93.2% for $2.45. LATS, a tree-search agent, got 88.0% for $134.50. Reflexion got 87.8% for $3.90. The authors’ point is that claims like “planning helps” or “reflection helps” were mostly never tested against something like Retry, so we do not actually know which part of those designs was doing the work.³

The same pattern shows up on harder tasks. Agentless, from Xia et al., skipped the agent loop entirely and used a fixed pipeline: find the right files, generate candidate patches, test them, pick one. On SWE-bench Lite it resolved 32% of issues, which the authors report as the best result and lowest cost among open-source agents at the time.⁴ A pipeline with no autonomy at all was the bar the agents had to clear.

I think this is the single most useful habit in agent evaluation. Before claiming your clever component works, show that it beats the dumb version of itself.

Checks 4 to 6: is the measurement honest?

4. Ablate each component

An agent is a stack of parts: a planner, a memory strategy, a verifier, a retry policy, a set of tools. If the whole thing scores well, you still do not know which parts earn their keep.

An ablation removes one part at a time and reruns. The SWE-agent paper, which I spent a lot of the last post on, did this well. Same model, and each change to the interface moved the score: removing the linter, showing the whole file instead of 100 lines, keeping the full history instead of the last few observations.⁵ Some of those “improvements” made it worse.

For my project, the ablation was splitting the intervention into its two halves: rolling back some tokens, and inserting a “let me recheck” phrase. Testing each half alone is what told me the trigger itself was not adding much.

Three rules I try to follow with ablations:

• Change one thing at a time. If you remove the verifier and also change the prompt, you cannot isolate which change mattered.
• Run more than once. Agents are noisy. A 2-point gap from one run might be nothing.
• Report the ones that hurt. The ablations where removing a part makes things better are the most informative rows in the table.

5. Don’t let the system grade itself

Using a language model as a judge is tempting, because a lot of agent outputs have no answer key. Whether a summary was good or a frustrated customer was handled well is not something a unit test can check.

LLM judges are useful, and also biased in specific, documented ways. Zheng et al., who introduced MT-Bench and studied this carefully, found that strong judges like GPT-4 agree with human preferences more than 80% of the time, about as often as humans agree with each other.⁶ That is the good news. They also documented position bias (preferring whichever answer is shown first), verbosity bias (preferring longer answers), and self-enhancement bias (preferring answers that look like the judge’s own).

Panickssery et al. took that last one further. They showed models like GPT-4 and Llama 2 can recognize their own outputs at better than chance, and that LLM evaluators score their own outputs higher than others’ even when humans rate them as similar quality. When they fine-tuned models to be better at recognizing themselves, the self-preference got stronger, roughly in a straight line.⁷

So when an agent checks its own work with the same model, or you grade an agent with the model it runs on, you are setting up exactly the situation where these biases bite. My coherence score was the extreme version: the model grading itself, on its own reasoning, with no reference answer.

My rule now: an LLM judge is fine as a second opinion, but never the only signal. If I use one, I check it against ground truth on a sample first, swap the answer order to catch position bias, and make sure the baseline gets judged too. If the judge rates doing nothing highly, the judge is broken, not the baseline.

6. Make sure the test wasn’t in the training data

This one is specific to models trained on the whole internet. If the benchmark questions or answers were in training, a good score may partly reflect memorization rather than generalization.

The GSM1k paper is a clean example. Zhang et al. wrote 1,000 new grade-school math problems in the same style and difficulty as GSM8K. Several model families did up to 8% worse on the new set than on the old one, and how likely a model was to generate GSM8K examples verbatim correlated with how big its drop was.⁸ To be fair, they also found that many frontier models showed little sign of overfitting.

Coding benchmarks have the same issue in a sharper form. Liang et al. found that state-of-the-art models could name the buggy file in SWE-bench tasks 76% of the time from the issue text alone, with no access to the repository. On repositories outside SWE-bench, that dropped to 53%.⁹ They read the gap as a sign of memorization. Separately, Aleithan et al. went through successful SWE-bench patches and found that 32.67% had the solution sitting right in the issue report or comments, and 31.08% passed only because the tests were too weak. Filtering those out took SWE-Agent with GPT-4 from 12.47% to 3.97%.¹⁰

The practical takeaway: if you build your own eval, write fresh tasks, and keep some of them hidden from whoever is tuning the agent. Kapoor et al. found that most of the 17 agent benchmarks they surveyed lacked a proper held-out set.³ Without one, you end up tuning to the test.

Checks 7 to 9: cost, reliability, and the grader

7. Count the cost

Two agents with the same success rate are not equally good if one costs fifty times more. And an agent that makes ten attempts and keeps the best one is not comparable to an agent that makes one.

This is the other main point of Kapoor et al. They argue agent leaderboards should plot accuracy against cost, and only care about the systems on the Pareto frontier, the ones where nothing else is both cheaper and more accurate.³ In the chart above, LATS is nowhere near that frontier.

They also make a point I had not thought about: model developers and app developers need different cost measures. A researcher comparing models might care about compute. Someone building a product cares about actual dollars per task, which change over time, so they suggest also reporting token counts so people can recompute costs later.³

For harness work this matters a lot. A verification loop that adds 5 points of accuracy and triples the cost might be worth it, or might not. You cannot decide if you only report the 5 points.

8. Measure reliability, not just success

A customer service agent that works on a task 60% of the time is not 60% useful. Every customer gets a fresh roll of the dice.

τ-bench introduced a metric for this called pass^k (pass-hat-k). It is the probability that the agent succeeds on a task in all k independent tries, averaged over tasks.² Compare that to the more familiar pass@k, which counts a task as solved if any of k tries works:

n = trials per task, c = successful trials on that task

pass^k = average over tasks of  C(c, k) / C(n, k)       # all k succeed
pass@k = 1 - average over tasks of  C(n-c, k) / C(n, k)  # at least one succeeds

As k grows, pass@k goes up and pass^k goes down. One is “can it ever do this,” the other is “can I count on it.”

The paper reports that GPT-4o’s function-calling agent succeeded on less than half the tasks overall, and that its pass^8 in retail was under 25%.² So even on tasks it can do, running it eight times in a row usually hits a failure somewhere.

For anything a person relies on, I think pass^k is closer to the question that matters. pass@k is fine for something like code generation where a test suite picks the winner for you. It is a bad fit when there is no second try.

9. Check the grader

This is where outcome-based scoring comes back to bite. A test suite is a judge too, just a dumb one.

SWE-bench is graded by running the repository’s tests. When OpenAI and the SWE-bench authors released SWE-bench Verified in August 2024, they had 93 Python developers screen 1,699 tasks from the original set, and kept 500 that held up: clear issue descriptions, tests that actually check the fix.¹¹ That fixed a lot. It did not fix everything. Zhu et al. still describe Verified as using insufficient test cases, which means a patch can pass without really resolving the issue.¹

So the last check is to read some of the successes by hand. Not the failures, which everyone reads. The successes. Some apparent successes may come from deleting a test or special-casing an input. Correct refusals are useful too, but should be reported separately so they do not hide poor performance on actionable tasks.

The checklist

Put together, this is what I look for before I believe an agent result, including my own:

1. Scored on the outcome, checked against something the agent cannot talk its way past.
2. A “do nothing” row in the table.
3. A random or trivial baseline, like retry or a fixed pipeline.
4. Ablations, one component at a time, over several runs.
5. Any LLM judge validated against ground truth, with the baselines judged too.
6. Fresh or held-out tasks the agent was not tuned on.
7. Cost reported next to accuracy.
8. Reliability across repeated runs, not just best-of.
9. A human read of some of the successes.

None of this is new. Most of it comes straight from the papers below. But almost no agent demo I see does more than two or three of these, and I was not doing them either until a judge told me doing nothing was a success.

Back to the harness

I’d like to see an agent recover from a problem I deliberately put in its way. A tool returns something incomplete, a test fails for a reason unrelated to the change, or a file moves. Does it notice, and can it find a sensible next step?

Keeping the model fixed while changing those conditions would help show what the surrounding software contributes. It would also make a success easier to interpret: we could see which checks caught real mistakes and which only added work. That’s what I’d want from a better agent, especially on a task where I can’t watch every step myself.

Sources