ARTIFICIAL INTELLIGENCE
OSWorld 2.0 and the desktop-agent bottleneck
Long desktop workflows expose agent failures that short demos hide. OSWorld’s 2026 releases connect task completion, process evidence, and scientific artifacts.
Research checked
Newest cited publication
This general agent–environment loop is background for desktop interaction, where actions must be checked against changes in the interface.
Megajuice · Image source · CC0 1.0
A desktop demonstration can be mesmerizing: an agent opens a spreadsheet, finds a menu, changes a chart, and saves a file. The difficult question is what happens after the tenth dependent decision, when a notification changes the plan or an earlier assumption turns out to be wrong. OSWorld's 2026 developments push evaluation in that direction. The June OSWorld 2.0 release lengthened realistic workflows; September brought a versioned bug-fix release and separate research on process-level assessment and scientific applications. Together they provide a more informative picture than a single success video. They also require careful reading: OSWorld 2.0, OSWorld-Pro, and OSWorld-Science are distinct evaluations, with different tasks and grading rules. Their percentages should not be placed on one interchangeable scale.
Why longer workflows change the problem
The OSWorld 2.0 preprint was submitted on June 28, 2026, following the project's June 26 announcement, and revised on July 13. It describes 108 extended computer-use workflows, with a reported human median completion time around 1.6 hours. Under the paper's 500-step binary-completion setting, its best reported system completed 20.6% of tasks while obtaining a larger partial score. These are measurements for the named systems, task version, and budget in the paper. They are not an estimate of the fraction of all office work that artificial intelligence can perform.
Length matters because a workflow is a chain of state-dependent obligations. Imagine assembling a conference budget from a price list, a spreadsheet, and an email correction. Reading all three successfully is insufficient if the final spreadsheet still uses the obsolete venue price. Each action may look locally reasonable while the overall result violates a constraint. Repeatedly observing the current screen cannot solve that by itself. The agent must maintain which facts are authoritative, what has already changed, and which consequences must be revisited after new information arrives.
The screen is an observation, not the whole state
Graphical interfaces reveal only part of an application's state at any moment. A button can appear enabled while an upload is still processing. A file picker may show a recently used directory rather than the required destination. A spreadsheet cell may display a rounded value while retaining more precision internally. These are ordinary interface properties, but they create a difficult inference problem for automation. The agent needs observations that distinguish meaningful state changes from visual coincidence, and must sometimes inspect an artifact rather than trusting a screen.
An original example is a slide deck exported to a portable document format. Seeing the export dialog close does not establish that the correct pages were included or that fonts remained readable. A stronger procedure reopens the output, checks the expected page count, and inspects representative content. That verification costs extra actions. Evaluations with very tight action budgets may reward fast-looking interaction while penalizing prudent checking. The budget should therefore be reported alongside the success metric, and evaluators should distinguish work that stopped early from work whose final verification actually failed.
The September release makes reproducibility concrete
The official OSWorld-V2 repository announced its recommended osworld-v2.1 bug-fix release on September 16, 2026. The instructions pin the code, task files, task assets, public assets, websites, and provider images to matching references. They also explain that the earlier June websites ceased being hosted at their old address in September, so reproducing that release requires self-hosting the corresponding version. This is substantive experimental information: a model can receive a different effective task if the application state changes beneath an unchanged instruction.
A useful analogy is a physics experiment in which the instrument calibration changes between runs. Keeping the same sample label would not make the measurements comparable. For desktop agents, application versions, screen dimensions, account state, and initial files form part of the instrument. A reproducible report should retain enough information to reconstruct them. This does not require pretending environments never improve. It requires identifying which results belong to each release, so a repaired evaluator is not mistaken for an improvement in the agent that happens to be tested afterward.
Process scores reveal different kinds of failure
OSWorld-Pro, a separate preprint submitted September 21, 2026, introduces 305 tasks and 2,814 progressive subgoals, supported by 67,264 human annotations. It evaluates progress through dependent subgoals and studies model-based judges against human judgments. The motivation is diagnostic: two agents can both miss the final deliverable even though one failed immediately and the other completed most of the workflow. The process evidence helps locate the difference, while introducing its own question about the reliability of those intermediate labels.
Consider an agent that correctly imports data, computes the right totals, and then saves the wrong file type. An end-state grader may reasonably mark the requested deliverable incomplete. A process account can still distinguish that failure from importing the wrong dataset and producing plausible-looking totals. Those failures call for different repairs. More careful save-state checking addresses the former; provenance tracking addresses the latter. Partial credit becomes most useful when it guides this diagnosis. It becomes misleading when accumulated easy milestones are treated as equivalent to delivering a coherent final artifact.
Scientific applications raise the standard for artifacts
OSWorld-Science, submitted September 30, 2026, examines scientific software through 146 tasks across several domains and configurations. The authors describe workflows including molecular manipulation, image analysis, statistical computing, and physical simulation. Their task-specific evaluators inspect application state and generated artifacts such as molecular structures, masks, plots, and numerical outputs. The paper is a benchmark proposal and evaluation, not a demonstration that autonomous software use has produced a validated scientific discovery. Its contribution is making scientific goals more directly testable at the artifact level.
For a scientific plot, matching the appearance of an example image is a weak endpoint. A graph can have the right colors while using the wrong units, filtering the wrong rows, or plotting the wrong independent variable. A useful evaluator checks properties tied to the scientific task. Does the exported data contain the expected sample identifiers? Were the axes labeled consistently with the transformation? Are numerical tolerances justified by the computation? These checks test meaning beyond surface resemblance, although deciding the right tolerances still requires domain expertise.
Recovery should be measured, not edited out
The next convincing demonstration would preserve mistakes and recovery attempts in its trajectory. If a menu fails to open, does the agent verify focus before repeatedly clicking? If a required account state is missing, does it request clarification or invent a substitute? If an intermediate file is corrupt, can it restart from a known checkpoint without overwriting good work? These are proposed evaluation questions. A polished video that removes delays and retries cannot answer them, even when every frame shown is authentic.
The 2026 OSWorld line of work suggests a productive separation of evidence. End-state evaluation asks whether the user received the requested result. Process evaluation helps explain how the result emerged or why it failed. Artifact checks connect software interaction to the content that matters in a scientific or professional task. Version manifests make the experiment repeatable. None replaces the others. Desktop agents become more credible when they preserve constraints across long workflows, show their checks, and recover from surprises under conditions that another team can reproduce.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World TasksarXivPublished Checked
- OSWorld-V2: Benchmark releases and evaluation setupXLANG LabChecked
- OSWorld-Pro: Process-based Evaluation for Computer Use AgentsarXivPublished Checked
- OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific SoftwarearXivPublished Checked
FOLLOW THE QUESTION
Study engineering foundations