ARTIFICIAL INTELLIGENCE
FrontierMath and the Difference Between Solving and Knowing
FrontierMath’s 2026 revisions show why mathematical AI evaluation needs checked answers, stable versions, explicit tools, and careful attribution.
Research checked
Newest cited publication
A proof-tree diagram illustrates the structure of formal derivation as background for assessing mathematical solutions and their verification.
Siddharthist / Wikimedia Commons. Adapted by GreatItself: corrected the transitive-closure premise to edge(y,z). · Image source · CC BY-SA 4.0
A mathematical answer can be straightforward to check and extremely difficult to discover. This asymmetry makes mathematics attractive for evaluating artificial intelligence: the evaluator need not be as inventive as the solver. Yet the apparently simple arrangement has demanding details. Someone must write a sound problem, implement a trustworthy checker, choose the permitted tools, and distinguish an autonomous solution from a collaboration. FrontierMath’s 2026 history brings those details into focus. Its revised closed-answer benchmark and its separate Open Problems project illuminate different kinds of achievement. Understanding the distinction is essential before treating a score as evidence that a system can conduct mathematical research.
Closed answers and open questions
The original FrontierMath paper appeared in November 2024. Its Tiers 1–4 collection uses unusually challenging problems written by mathematicians, with answers suitable for automated checking. The official project separates this from FrontierMath: Open Problems, which concerns mathematical challenges without known solutions at selection time. The two projects share an interest in difficult reasoning, but their success claims concern different relationships to existing knowledge. A hard known-answer question remains a known-answer question.
An independent analogy is a locked room whose owner knows the combination. Finding the combination can require impressive ingenuity without discovering a new fact for the owner. By contrast, finding a previously unknown object that satisfies a published specification can enlarge the shared record. Both are valuable achievements. They should not be collapsed into one scale merely because both involve mathematics. The evaluator’s prior knowledge, the solver’s access, and the novelty of the output determine what kind of achievement has occurred.
The June 2026 correction matters
On June 12, 2026, Epoch AI released FrontierMath v2, reporting that the update addressed errors in 42 percent of problems. Its Tier 4 v2 documentation lists a full revised collection of 338 problems: 295 in Tiers 1–3 and 43 in Tier 4. The changelog distinguishes corrections from removals. These are reported dataset changes, not a measured failure rate for any model and not a claim that all affected items had the same defect.
The right response is to preserve version identity. Suppose a toy benchmark replaces an impossible question and corrects an answer key. A model’s score can rise even if its responses never change. Conversely, removing unusually easy items can lower the score of an unchanged model. Neither movement alone reveals a capability trend. Running the same models on both versions, reporting common-item results, and describing the edits lets readers disentangle better measurement from better performance. A benchmark correction is scientific maintenance when the audit trail remains accessible.
Verification needs a specification
Epoch’s documented evaluation lets a model reason and execute Python, then submit an answer through a specified interface. The checker evaluates the returned mathematical object. This is a clear operational target, but it is narrower than judging every sentence of the solver’s explanation. A correct integer can accompany a flawed argument, just as a correct proof can be obscured by a formatting failure at submission. The evaluation boundary should say which of these properties it measures.
Consider a deliberately small example: find an integer whose square is 49. Returning 7 satisfies that wording, while returning only 7 would not completely answer a request for all integer solutions, because negative 7 also works. The computation is trivial; the specification changes what counts as success. At research scale the same distinction appears through hidden side conditions, admissibility requirements, or exact symbolic equivalence. Testing a verifier on deliberately invalid and alternative valid outputs can therefore be as important as testing it on the official answer.
A solver is more than a model name
The FrontierMath documentation spells out tool access, answer submission, runtime constraints, and token budgeting. These are part of the experimental apparatus. An evaluation with repeated code execution tests a model operating inside a computational loop. A comparison against an unaided conversational answer would change several variables at once. Useful reporting names the model snapshot and the whole surrounding procedure, including sampling, retries, and what happens when the system fails to submit.
Here is a general probability illustration, not a FrontierMath result. If independent attempts each succeed with probability 0.2, five attempts have a probability of at least one success equal to 1 minus 0.8 to the fifth power, about 67.2 percent. Choosing the best verified attempt can greatly improve a system result without changing single-attempt competence. Independence is an assumption, and model attempts often repeat mistakes. Reporting both the attempt budget and the observed correlation gives a far more useful picture than describing every selected success as an ordinary answer.
Unknown answers can still be checked
The Open Problems project selects challenges with bespoke computer verifiers for candidate solutions. Its official changelog records a July 31, 2026 expansion to 50 problems, while also describing removals when verifier fidelity or mathematical significance was inadequate. That history is especially instructive: a program that checks candidates is useful only if its acceptance criterion matches the intended mathematical achievement. An untrustworthy checker can turn optimization into exploitation of a loophole.
Imagine searching for a graph with a specified property. A candidate graph can be much cheaper to test than to find. But if the checker accidentally inspects only one connected component, a search procedure might discover a disconnected counterexample to the software rather than a solution to the mathematics. The repair is not simply a longer search budget. It requires a reviewed specification, independent checker tests, and explicit statements about what the accepted object proves. Automated verification moves intellectual work; it does not eliminate the need to examine assumptions.
Attribution changed in September
On September 16, 2026, Epoch introduced a human-plus-AI solution status for Open Problems, recognizing cases where AI was instrumental but did not solve the problem autonomously. The project explains that earlier binary labels did not adequately capture iterative collaboration. This is a methodological improvement with practical consequences. A human who selects promising approaches, supplies missing lemmas, or rejects misleading outputs can make a decisive contribution even when the machine generates most of the text.
For a prospective evaluation, record interventions as they happen. A timestamped log can distinguish an initial task description from a later hint that essentially supplies the central idea. It can also show genuine machine contributions that would otherwise disappear beneath a human-authored final exposition. Counting keystrokes is inadequate: a one-line observation may matter more than hundreds of lines of computation. The defensible unit of attribution is a contribution to the mathematical reasoning, supported by the available record and described with appropriate uncertainty.
From impressive answers to dependable practice
These developments suggest a practical reading strategy. First identify whether a result concerns the corrected closed-answer benchmark or Open Problems. Next inspect the actual problem version, checker, tool permissions, and computational budget. Finally examine the provenance of a claimed discovery and whether someone has independently checked the mathematical object. A score and a research contribution can support each other, but neither should borrow the other’s evidential strength without an explicit connection.
A dependable mathematical assistant would combine invention with disciplined verification: recognize a useful construction, test boundary cases, explain why the construction meets the specification, and welcome a check that could disprove it. FrontierMath provides several ways to measure pieces of that behavior. Its 2026 revisions also show why evaluators need the same discipline they ask of models. Better questions, better verifiers, and clearer credit are substantive progress even before another percentage increases. They make future success mean something precise enough to build upon.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIarXivPublished Checked
- FrontierMath Tiers 1–4Epoch AIChecked
- FrontierMath Tier 4 (v2)Epoch AIPublished Checked
- FrontierMath: Open ProblemsEpoch AIChecked
FOLLOW THE QUESTION
Study engineering foundations