SWE-bench Verified and the coding-agent measurement gap

The patch is only one part of the experiment

SWE-bench Verified was introduced in August 2024 as a human-reviewed collection of 500 repository issues. Its attraction was executable evaluation: a proposed change must address the issue while preserving behavior already covered by regression tests. OpenAI's original release describes expert screening intended to reduce ambiguous descriptions and unsuitable tests. That screening made the dataset more useful, but the word Verified names a particular review process at a particular time. It does not establish that every future solution rejected by its tests is necessarily wrong.

Consider an original example. A library should return unique search results without changing their first-occurrence order. One implementation uses a set plus an output list; another uses an ordered dictionary. If the public requirement specifies only the output, both implementations may satisfy it. A hidden test that imports a helper function used by the reference author would silently add an implementation requirement. A model could solve the user's problem and fail the experiment. Conversely, weak tests might approve code that works for the example and fails on empty input.

A selected audit is not a census

OpenAI's February 23, 2026 analysis examined 138 Verified problems that its o3 model did not reliably solve across repeated attempts. It reported material issues in 59.4% of that audited subset. The sampling condition matters: these were difficult residual failures, not a random sample of all 500 tasks. Applying that percentage to the entire benchmark would overstate what the audit measured. The report also presented examples of models reproducing reference-patch details, raising a distinct concern about exposure to evaluation material during training.

Test flaws and prior exposure can pull scores in opposite directions. A rigid evaluator can reject a fresh, correct implementation, while memorized details can help another system satisfy an unstated convention. The resulting percentage mixes problem-solving ability with the alignment between training history and test design. This is why subtracting a guessed contamination percentage cannot cleanly repair a leaderboard. One needs task-level evidence about what information the agent had, which behavior was required, and whether the acceptance rule actually checked it.

A replacement benchmark needs its own audit

SWE-Bench Pro, released in September 2025, broadened the challenge toward longer repository-level tasks and multiple repository splits. It did not thereby escape the need for specification review. On July 8, 2026, OpenAI reported a separate audit of its 731-task public split. The investigator pipeline and human review identified different totals of problematic tasks, supporting an approximate thirty-percent estimate rather than a single exact census. The company withdrew its February recommendation to adopt Pro. That is an important correction to any account stopping at the earlier announcement.

The general lesson is methodological. Increasing task length creates more room for meaningful engineering, but also more opportunities for mismatches between requirements and grading. A feature request can omit migration behavior, error handling, or a backward-compatibility condition that a later test expects. More complexity is useful only when the evaluator specifies which complexity belongs to the task. Otherwise, a harder benchmark may partly measure an agent's willingness to guess the maintainer's unwritten preferences.

Training exposure and execution-time leakage differ

The SWE-Bench Pro Verified preprint, first submitted September 8 and revised October 9, 2026, separates task repair from protection against answer retrieval during execution. Its authors reconstruct repositories, hide evaluation artifacts, filter identifying metadata, and restrict access to answer-bearing online locations. They describe a 731-instance release with corrections to 102 instances. Those are the authors' reported safeguards and repairs, not proof that every possible leakage channel is closed. The paper remains a preprint as of this article's cutoff.

This distinction changes the engineering response. If a model already memorized a future patch, blocking its network cannot erase that memory. If an otherwise fresh model can read a future commit from leftover Git objects, changing its training corpus will not fix the environment. A serious evaluation should investigate both. It must also preserve legitimate work: dependency installation and ordinary inspection should remain possible under clearly stated permissions. Isolation that breaks necessary tooling can introduce a new source of false failure while removing an old one.

Compare complete systems under declared budgets

A coding agent is a model combined with a driver, prompts, tools, context management, and a stopping policy. Give one system repeated attempts and another a single run, and their final success rates answer different questions. Give one the failing test names while hiding them from another, and even identical model weights face different tasks. A useful comparison therefore records the scaffold, allowed observations, dependency state, wall time, token usage, and number of independently sampled attempts alongside the resolved-task count.

An original budgeting example makes the point. Suppose one agent resolves sixty of one hundred tasks in one attempt, while another resolves seventy after five attempts. The second result may be more useful when compute is plentiful, but it does not demonstrate higher first-attempt reliability. Nor can the first figure be converted into the second by assuming independent failures: repeated attempts often get stuck on the same misunderstanding. Publishing per-task outcomes reveals whether extra compute expands coverage or merely retries a small cluster of favorable tasks.

Uncertainty should also be attached to the task sample. Two systems separated by a few resolved issues may exchange rank on another sample of repositories. Reporting a paired comparison is informative because it shows exactly which tasks changed outcome. Confidence intervals can summarize sampling uncertainty under stated assumptions, but they cannot account for an unknown amount of answer exposure or a grader that systematically rejects a class of valid programs. Statistical precision and evaluation validity solve different problems.

What convincing evidence would look like

For an unfamiliar codebase, a stronger evaluation would begin with a task statement reviewed independently of the reference patch. Reviewers would construct alternative valid implementations and deliberately incomplete ones. The grader should accept the former and reject the latter. Metamorphic checks can help: if sorting order is irrelevant, permuting an equivalent input should not change correctness; if it is relevant, the contract must say so. This is our proposed validation workflow, not a claim that the cited benchmarks already implement every step.

Public release should then preserve a versioned task manifest and disclose repairs. A score before a grading change should remain identifiable rather than being silently compared with the repaired score. Fresh tasks, held-out repositories, trajectory audits, and independently written acceptance tests address different weaknesses and work best together. The practical reading of the 2026 evidence is therefore specific: executable coding benchmarks remain valuable instruments, but their calibration is part of the research result. A patch's passing mark is credible only to the extent that its task, environment, and evaluator deserve that credibility.