ARTIFICIAL INTELLIGENCE
Humanity’s Last Exam and the Benchmark Ceiling
HLE-Diamond’s September 2026 release sharpens a larger question: how do we separate expert knowledge, reasoning, tools, and errors in an AI test?
Research checked
Newest cited publication
Illustrative ROC curves provide context for interpreting evaluation metrics. Humanity’s Last Exam uses its own question types, grading rules, and reported measures.
CMG Lee and Martin Thoma · Image source · CC BY-SA 4.0
An exceptionally hard examination can reveal a model’s limits while concealing the limits of the examination itself. A wrong answer may expose missing knowledge, a failed inference, an unreadable diagram, or an imperfect answer key. Humanity’s Last Exam makes these distinctions unusually visible because its questions sit near specialist boundaries. Its newest verified development at this article’s October 18, 2026 cutoff is HLE-Diamond, announced on September 22. The interesting change is not simply another score to compare. A refined benchmark asks us to reconsider what an examination actually measures, what its denominator contains, and how much confidence a single percentage deserves.
An exam with a revision history
The original HLE project began publicly in January 2025; its finalized public collection had 2,500 questions by April 3. A peer-reviewed account appeared in Nature on January 28, 2026. The organizers describe specialist questions across many disciplines, with multiple-choice and short-answer formats. These dates matter because an early paper, a finalized dataset, and a later model evaluation are different historical objects. An answer key may change between them even when the benchmark’s familiar name remains unchanged.
Consider an ordinary laboratory notebook. Replacing a miscalibrated sensor does not invalidate every earlier observation, but it changes how the observations should be interpreted. Benchmark revisions deserve the same treatment. Keep the original record, identify corrected items, and state which version produced each result. A rising score can reflect a stronger model, a cleaner examination, or both. Without that bookkeeping, apparent progress becomes an arithmetic mixture whose ingredients readers cannot recover.
What Diamond changes
HLE-Diamond is a refined 1,000-question subset, released by the Center for AI Safety and Scale AI after an extended cleaning process. Its announcement separates 500 reasoning questions from 500 knowledge questions. This is a new evaluation population, not a license to replace an older HLE percentage with a Diamond percentage. The release also distinguishes results without tools from results using web and code tools, and identifies the reasoning settings applied to the displayed comparisons.
A useful imaginary example shows why the partition matters. Suppose one system answers 400 of 500 reasoning questions and 200 of 500 knowledge questions correctly. Its overall score is 60 percent. A second system answers 300 in each partition and receives the same total. Those systems are interchangeable only if the application values both abilities identically. A research assistant that must recognize obscure terminology may favor the second profile; a solver supplied with trusted reference material might favor the first. The equal totals hide that choice.
Difficulty and noise are different ceilings
There are at least two ways an examination stops distinguishing capable systems. Questions can become too easy, producing a genuine performance ceiling. Alternatively, ambiguous wording or incorrect expected answers can prevent a defensible response from receiving credit. Making questions harder addresses the first issue; careful auditing addresses the second. Neither operation substitutes for the other. This distinction explains why cleaning a difficult dataset can be scientifically valuable even if the cleaned set contains fewer questions.
Imagine a ten-item miniature test containing nine sound problems and one defective item. A careful solver can be marked at most 90 percent despite perfect performance on every meaningful question. Another solver might receive the defective item’s expected answer by accident and appear better. The example does not estimate HLE’s defect rate. It isolates a general measurement problem: disagreement with a key is evidence about a model only after the key’s own validity has been considered. Difficulty should never make an evaluation immune to challenge.
Knowing when to stop answering
The HLE project also studies calibration: whether expressed confidence tracks actual correctness. Accuracy asks how often an answer is right. Calibration asks whether a collection of answers assigned, for example, 70 percent confidence is right roughly 70 percent of the time. These properties differ. A model can gain correct answers while remaining dangerously confident on its remaining mistakes. An especially challenging examination exposes that gap because persuasive explanations can accompany both insight and failure.
For an independent practical example, imagine two assistants reviewing unfamiliar archival material. Each gets six of ten interpretations right. One marks every interpretation as certain. The other correctly identifies four dubious cases and asks for expert review. Their examination accuracy matches, but their value inside a supervised workflow differs. The second output makes scarce human attention easier to allocate. Evaluation reports should therefore preserve the answer, confidence, abstention policy, and review decision rather than treating every unsuccessful response as an equally informative failure.
The examination boundary moves with tools
A closed-book model and a model supplied with search and executable code are different evaluated systems. Search can supply missing factual material; computation can check algebra or enumerate cases. A tool-assisted result can be highly useful without demonstrating that the underlying model stores the same knowledge internally. The HLE-Diamond announcement explicitly presents both settings. Readers should preserve that separation rather than promoting whichever setting gives the most impressive number into a universal capability claim.
Think of a question asking for a numerical property of a complicated construction. A mental solution, a short verified program, and a search result quoting an existing answer can all produce the same final text. They provide different evidence about reasoning and provenance. A useful run log shows which route occurred, the sources accessed, the computational budget, and whether external material contained the target answer. Tool access is a design choice to document, not a reason to discount every result or to ignore how it was obtained.
A score needs its surrounding record
The project publishes evaluation resources alongside the dataset. That availability makes a stronger reporting standard possible: identify the precise question collection, model snapshot, prompts, tool permissions, grading procedure, and treatment of missing outputs. If a reported run excludes image questions or uses a different subset, the subset belongs in the headline interpretation. A percentage calculated over a selected population cannot automatically speak for the questions left outside it.
Even a perfectly documented average needs uncertainty. A simple illustrative calculation treats 1,000 independent questions with 50 percent success as 1,000 coin-like observations. The standard error is about 1.6 percentage points. Real questions can share subject matter and failure patterns, so that calculation is not a complete confidence interval for HLE-Diamond. Its purpose is to resist reading tiny gaps as established superiority. Repeated runs, paired question-level comparisons, and subject breakdowns answer more than a ranking sorted to one decimal place.
What would count as a stronger conclusion?
A high score on expert questions supports a claim about performance on expert questions under specified conditions. It does not by itself establish autonomous discovery, dependable long projects, or sound judgment about which problems matter. The original HLE discussion acknowledges this boundary. An examination gives a question and a success condition to the system. Research often begins before either is available, with a decision about which uncertainty is worth investigating and what evidence would resolve it.
The next useful evidence is therefore plural. Clean, versioned questions can test academic competence. Prospective tasks can test unfamiliarity. Tool logs can explain where answers came from. Independent replication can test whether a result survives another evaluator. Longer projects can test whether a system notices and repairs its own mistaken assumptions. HLE-Diamond’s September release is valuable within this larger measurement practice. The most informative ceiling is one whose construction, cracks, and replacement date are visible to everyone looking up at it.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- Humanity's Last ExamCenter for AI Safety and Scale AIChecked
- A benchmark of expert-level academic questions to assess AI capabilitiesNaturePublished Checked
- Introducing HLE-DiamondCenter for AI Safety and Scale AIPublished Checked
- Humanity's Last ExamCenter for AI SafetyChecked
FOLLOW THE QUESTION
Study engineering foundations