FrontierMath, Erdős problems, and what "AI solved it" means

Over the last eighteen months, “AI solved a math problem” has been said about very different events. Sometimes it meant a model got a hidden numerical answer right on a benchmark. Sometimes it meant a model found a paper from 2003 that nobody on a problem website had read. And a few times it meant something much closer to what a mathematician would mean by “solved.”

I follow AI for math partly because I use these tools myself. The FrontierMath and Erdős episodes made me want a clearer account of what a model contributed: a reference, a formalization, a useful step, or a new proof.

Why a new math benchmark was needed

Math benchmarks have a short shelf life. GSM8K, a set of grade school word problems, and MATH, a set of competition problems, were both built in 2021 to be hard for language models. By the time Epoch AI released FrontierMath in November 2024, its paper described both as places where state-of-the-art models had reached near-perfect performance.¹

Saturation is only half the problem. The other half is contamination. Public benchmarks end up on the internet, and the internet ends up in training data. In 2024 a group of researchers wrote GSM1k, a fresh set of problems built to match GSM8K’s style and difficulty, and then compared scores. Some models showed “accuracy drops of up to 8%” on the new set, and the size of a model’s drop correlated with how likely it was to reproduce GSM8K examples word for word.² The same paper also found that many frontier models showed little overfitting, so this isn’t a story where every score is fake. It’s a story where a public score, by itself, can’t tell you which case you’re in.

FrontierMath was designed around both problems. The problems were new and unpublished, written by mathematicians (the paper says over 60 of them), and each had a single answer that a script could check, chosen so that guessing would almost never work.¹ The idea was simple: if nobody has seen the problems, a high score has to mean something.

At launch, the headline was the number. The abstract says “current state-of-the-art AI models solve under 2% of problems.”¹ The paper’s model comparison figure is almost empty.

The natural reading was a reality check: models were strong at contest math and nowhere near research math. That reading lasted about six weeks.

The disclosure

On December 20, 2024, OpenAI presented o3. Mark Chen said that “today, all offerings out there have less than 2%” on FrontierMath, and that o3 under aggressive test-time compute settings got over 25%.³⁴ That jump from 2% to 25% was the most cited number of the announcement.

Four weeks later, on January 18, 2025, a LessWrong user named Meemi posted a short note titled “FrontierMath was funded by OpenAI.” Meemi pointed out that OpenAI’s funding had only been made public on December 20, the day of the o3 announcement, that the mathematicians writing the problems had not been told, and that “the communication about this has been non-transparent.”⁴

Tamay Besiroglu, Epoch’s co-founder, replied in the same thread. He wrote, “We made a mistake in not being more transparent about OpenAI’s involvement.” He confirmed that OpenAI had access to “a large fraction of FrontierMath problems and solutions,” except for a hold-out set it had not seen, and said Epoch had “a verbal agreement that these materials will not be used in model training.”⁴ Meemi noted that the hold-out set was only about 50 problems and still being built.

The reaction from contributors was the part that stuck with me. Carina Hong, a Stanford math PhD student, said she had talked to six mathematicians who contributed problems and that they were unaware OpenAI would have exclusive access. “Most express they are not sure they would have contributed had they known,” she said.⁵ That is the cost of quiet funding. The people doing the hard intellectual work didn’t get to decide whether they wanted to be part of it.

Epoch published a clarification on January 23, 2025, by Besiroglu and Jaime Sevilla. It laid out the terms: OpenAI commissioned 300 problems, owns them, and can see their statements and solutions, except for a 50-problem holdout where OpenAI gets statements only. Epoch can run and publish evaluations but can’t share the problems without OpenAI’s permission. The post admits “we have not communicated clearly enough about the relationship between FrontierMath and OpenAI” and that communication with contributors “should have been more systematic and transparent.”⁶ Epoch also committed to disclosing sponsorship and data access terms from the start on future work.

I want to be fair here. Nothing that came out showed OpenAI training on the problems, and Epoch’s people were open about the mistake once it was raised. But a benchmark exists to be a neutral referee, and an arrangement where one participant funds the test and holds much of the answer key needs especially clear disclosure and independent checks. The verbal agreement is the weakest part. It might be honored perfectly. Nobody outside can check.

How the scores moved

Epoch ran its own evaluation once o3 was public. In April 2025, using an updated 290-problem version of the benchmark, it found o3 at around 10%, well under the December figure.³ Epoch suggested the difference “might be due to OpenAI evaluating with a more powerful internal scaffold,” along with more test-time compute or a different problem subset. TechCrunch noted that OpenAI’s December materials had also shown a lower-bound score close to what Epoch measured.³ So “25%” was a real result, from a configuration the public couldn’t run, on a version of the benchmark nobody outside could check.

After that, the numbers climbed fast. Epoch later added Tier 4, a smaller set of research-level problems. In March 2026, Epoch reported that GPT-5.4 Pro scored 50% on Tiers 1-3 and 38% on Tier 4.⁷ The access terms matter: OpenAI could see all 290 Tier 1-3 problems but only 237 solutions, plus 28 of the 48 Tier 4 problems and their solutions. Epoch held out the remaining material.

That holdout is what makes a contamination check possible, and the results are interesting. On Tiers 1-3, GPT-5.4 Pro solved 52% of the problems OpenAI had seen and 42% of the held-out ones. On Tier 4 it went the other way: 25% of the non-held-out set and 55% of the held-out set.⁷ Epoch says neither difference is statistically significant, and with only 20 held-out Tier 4 problems those percentages are very noisy.⁷ The honest summary is that the holdout doesn’t show clear evidence of contamination. It also doesn’t prove there’s none, because a 10-point gap on Tiers 1-3 is exactly the kind of thing you’d want a bigger sample to settle.

The same post has a detail that connects to the rest of this essay. On one Tier 4 problem, GPT-5.4 Pro “appeared to have found a preprint from 2011 which let it shortcut much of the intended work,” a preprint the problem’s author didn’t know about.⁷ The benchmark scored that as correct, because the answer was correct. Whether that is “solving” a research problem is the question the Erdős episode was about to make loud.

October 2025: ten problems that were already solved

Erdős left behind hundreds of problems, and Thomas Bloom runs erdosproblems.com, a database of them with each one marked open or solved. In early October 2025, people started running AI deep research tools across the list to look for literature. On October 12, Sebastien Bubeck at OpenAI posted that “gpt5-pro is superhuman at literature search,” because it had found that problem #339, listed as open, had been solved 20 years earlier.⁸ That claim was accurate and clearly worded.

On October 16, Terence Tao wrote a long Mathstodon thread on exactly this use case. He said he was “increasingly of the opinion that the most productive near-term adoptions of AI in mathematics will primarily come not from applying the most powerful models to the most challenging problems” but from medium-powered tools speeding up mundane work like literature review, where an expert can check the output.⁹ He reported six problems upgraded from open to solved this way.

The next day, October 17, OpenAI’s VP of science Kevin Weil posted that “GPT-5 found solutions to 10 (!) previously unsolved Erdős problems and made progress on 11 others.”¹⁰ Read cold, that sentence says GPT-5 proved ten new theorems. It didn’t. Bloom replied the same day, as the site’s maintainer, that it was “a dramatic misrepresentation.” In his words, “GPT-5 found references, which solved these problems, that I personally was unaware of.” And: “The ‘open’ status only means I personally am unaware of a paper which solves it.”¹¹ Weil deleted the post. Demis Hassabis replied to the thread with “this is embarrassing,” and Yann LeCun piled on.¹⁰ Bubeck deleted his own post repeating the claim and wrote that “only solutions in the literature were found that’s it,” adding that he still found it “very accelerating.”⁸

I think two things are true about this episode. The underlying result was useful. Finding a 20-year-old solution to a problem a whole website thought was open saves real people real time, which is Tao’s point. And the tweet was wrong in a way that matters, because it took a literature search and described it with the verb that means “proved.” The correction came from the person who maintained the list, not from the lab. That is the order you’d expect, and also the order I’d like to see reversed.

Then some of them were real

The story didn’t end as a joke, which is why I think it’s worth writing about now and not in October.

By late December 2025, models were producing proofs of problems marked open, and then people would find the problem had already been solved somewhere obscure. Tao described this on December 26 and offered a theory. Contamination might explain part of it, he said, but other tools failed to find those same connections, so he suspected models were now able to pick off problems “amenable to simple proofs using fairly standard techniques,” which are exactly the problems most likely to have been quietly solved already.¹²

Then problem #728. In early January 2026, a proof produced by GPT-5.2 Pro, repaired and formalized in Lean by Harmonic’s Aristotle, and steered by a few human participants, resolved the problem in the form the site’s community had reconstructed. Tao wrote on January 7 that it was “solved more or less autonomously by AI (after some feedback from an initial attempt),” with no matching result found in the literature, though similar results proven by similar methods were.¹³ He was careful about the caveats. The problem as Erdős stated it was misformulated, which partly explains why nobody had done it. In a later edit he added that the method turned out to be very close to a 2014 paper of Carl Pomerance, who released a note showing his methods also answer #728.¹³ Even so, the AI-generated solution was the first to address the problem explicitly.

A week later, on January 14, Tao posted the framing I find most useful. He saw “an empirical tradeoff between the level of AI involvement in the solution, and the difficulty or novelty of that solution.”¹⁴ Fully autonomous solutions tended to be short and standard. Harder results involved humans and AI working together, with the AI doing key calculations or steps. He guessed that the share of open Erdős problems simple enough for current tools with minimal human help was “on the order of 1-2%.”¹⁴

Tao also started a public wiki tracking AI contributions to Erdős problems, and I’d recommend it to anyone who wants to make claims in this area. It splits contributions into primary ones (AI standalone, AI alongside literature found afterward, AI building on known literature, AI collaborating with humans) and secondary ones (literature search, formalization, rewriting, computation). Its first disclaimer is “This page is not a benchmark.” Another reads “Absence of past progress may reflect obscurity rather than difficulty.”¹⁵ As of mid-June 2026, by my count the formalization table alone has over 170 entries and the literature search table over 80, which says something about where the volume of help actually is.

The most significant case so far is the one I’d least have predicted a year ago. In May 2026, an OpenAI internal model produced a counterexample to Erdős’s unit distance conjecture, which the wiki describes as a major open problem in an area with widespread expert attention.¹⁵ On May 20, nine mathematicians including Noga Alon, Thomas Bloom, Tim Gowers, and Melanie Matchett Wood posted “a short, digested, human-verified version of the recent OpenAI-generated counterexample.”¹⁶ Their abstract also says the argument relies crucially on ideas that “may, at least in retrospect, be attributed to” Ellenberg-Venkatesh, Golod-Shafarevich, and Hajir-Maire-Ramakrishna. That’s what a real result looks like in practice: a model finds a construction, and then serious mathematicians spend time checking it, compressing it, and placing it in the literature. The checking is part of the result, not an afterthought.

Four things people mean by “solved”

Tao’s spectrum and the wiki’s categories suggest four useful distinctions.

Lookup. The model finds an existing solution. This is the #339 case and the October episode. It’s valuable, and Tao’s argument for why is good: the expert can verify a reference quickly, so even a mediocre hit rate saves time.⁹ The right verb is “found.”

Formalization. The model turns a known proof, human or AI, into Lean code a computer can check. This is the biggest category on the wiki by far.¹⁵ It makes results more trustworthy, but the theorem was already proven. The right verb is “verified,” with the usual warning that a formal proof of a misstated theorem verifies the wrong thing.

Hint or step. The model supplies a calculation, a lemma, or the key idea, and humans assemble the rest. This is Tao’s collaborative middle, and I suspect it’s where most of the real research value is going to sit for a while. The right phrase is “with AI assistance,” plus a sentence on what the AI did.

New proof. The model produces a correct argument that isn’t in the literature, for the problem as intended, and experts confirm it. #728 roughly qualifies, with the Pomerance asterisk. The unit distance counterexample clearly does. This is the only one where I think “AI solved it” is the honest headline.

FrontierMath sits awkwardly across these. A benchmark answer is a number checked by a script. It can’t tell whether the model reasoned its way there, found a 2011 preprint, or got lucky on a guessproof problem. The GPT-5.4 preprint shortcut was scored as a solve, and under the benchmark’s rules it should have been.⁷ That’s fine for measuring capability. It’s not fine for describing what happened in words.

Who judges, and why incentives matter

In both episodes, the useful judgment came from the people closest to the problems. Bloom knew what “open” on his site meant.¹¹ Tao checked the #728 proof and then went back and noted the Pomerance overlap.¹³ For the unit distance result, the judgment took nine experts and a written paper.¹⁶ The misleading announcements are a reason to wait for people with domain knowledge to examine the result.

So my rule for myself, as someone reading these announcements: a claim that AI solved a problem counts when the maintainer of the problem, or someone with standing in that field, has checked it in public. A lab’s own post is evidence that something happened, not a verdict on what. The Erdős wiki’s advice page for people who think they’ve gotten an AI to solve a problem points the same way. One of its tips is simply “Avoid rushing to announce your results,” because AI proofs announced on social media have repeatedly needed corrections a few days later.¹⁵

A benchmark is only as good as the arrangement around it. FrontierMath’s problems were excellent. The setup around them, at least until January 2025, asked the public to trust that the company funding it, holding most of its answers, and announcing the biggest jump on it, would keep those answers out of training on a verbal promise.⁴ Epoch fixed what it could. It disclosed the terms, kept a holdout, and now reports held-out and non-held-out scores separately.⁶⁷ I think those fixes are real and that the held-out comparisons are the most informative numbers on the leaderboard.

The same incentives show up in the Erdős story without any money changing hands. A lab gets credit for “solved,” not for “found,” so the language drifts toward “solved.” The problem site has the opposite incentive: its value depends on its labels being right. I find the wiki useful because it records distinct contributions and caveats in one place, making the claims easier to check.

I’ve used these models a fair amount for math-adjacent research work, and my experience matches the literature-search story more than the theorem-proving one. Their best output for me has been a pointer to a paper I should have known about. Their less useful output has been a confident argument with a gap that needed checking.

Where I land

What excites me is the possibility of more people being able to explore a difficult mathematical idea before they know every tool it might need. A model that finds a useful connection could help someone get past a point where they would otherwise stop.

But once an answer appears, there is still a lot to learn from it. I’d want to know which step was new, why it worked, and whether the argument suggests anything beyond that one problem. If generating proofs becomes much easier, helping people understand them could become an even bigger part of mathematical work. That’s the future I find myself wondering about when I read these results.

Sources