AI Agent OSWorld-Verified 85% — But We Didn’t Ask What It’s Compared Against

Does a score that says “better than humans” apply to our work too?

When you’re evaluating computer-use AI agents, one slide can rush the decision: OSWorld-Verified 85%, versus a human baseline of roughly 72%. Those numbers make it seem reasonable to hand over portal entry or ticket handling.

But if the score doesn’t state the benchmark version, task length, agent setup, or allowed number of steps, you can’t yet read it as a success rate for your own work. The publicly reported 85% and OSWorld 2.0’s 20.6%, in fact, differ in everything from the model to the tasks and scoring method. The gap is not evidence that AI capability suddenly collapsed. It’s a signal that you first need to confirm what was counted as complete, and under what conditions.

85% is not a “verified official score”

In an August 10, 2026 article, a16z highlighted Claude Fable 5’s 85% score on a third-party OSWorld-Verified leaderboard as of June 2026, explaining that it exceeded the roughly 72% human baseline. However, the leaderboard reviewed lists Fable 5 at 85.0% and Claude Opus 4.8 at 83.4%, while also stating that all 24 submitted results are self-reported and that zero results have been independently verified. a16z +1

Here, “Verified” is part of the benchmark name; it does not guarantee that each run on the table was independently verified. The public page also does not provide Fable 5’s original run report, per-task results, harness, step budget, or retry conditions. So it is more accurate to treat 85% as a third-party leaderboard snapshot cited by a16z at the time.

The 72% human figure also needs its source distinguished. The original 2024 OSWorld paper reported a 72.36% human success rate and a 12.24% best-model success rate across 369 tasks. Public materials do not establish whether this human evaluation was recalculated using exactly the same task set and protocol as the later OSWorld-Verified benchmark. arxiv

20.6% asks whether longer work was ‘fully completed’

OSWorld 2.0 is not the same test with a new name. It consists of 108 long-horizon tasks reflecting real work in research, content creation, software development, business, and finance. The median time for skilled humans to complete a task is about 1.6 hours, and 69.6% take more than an hour. Each task has an average of 27.25 scoring checkpoints. osworld-v2.xlang +1

On this test, Claude Opus 4.8 recorded a binary completion rate of 20.6% and a partial score of 54.8% with 500 steps, maximum thinking, and batched tool calls. This shows the difference: even if it makes more than half the progress, it does not count as fully complete if the final deliverable is not finished as required. osworld-v2.xlang +1

The execution burden also changes in longer work. Official materials say a Claude Opus 4.7 maximum-thinking run averaged about 318 tool calls, compared with roughly 30 in OSWorld 1.0. This is the result for a particular setup, not an average across all models, but it clearly shows why short app interactions and work that chains many steps together are hard to compare with one number. osworld-v2.xlang +1

A score is not a model name; it is a ‘bundle of evaluation conditions’

To compare benchmark results, you need at least the benchmark family, release, task set, model, agent harness, step budget, retries, human intervention, and scoring metric. If even one differs, the numbers may be answering different questions.

Don’t leave out the release either. OSWorld 2.0’s official repository currently recommends osworld-v2-2026.08.08 and instructs users to match the code, tasks, assets, and website version. This release revised 14 tasks and improved evaluation robustness and fairness for 31 tasks. Public materials do not confirm whether the paper’s 20.6% is reproduced unchanged on this revised release, so you should not put scores from different releases into one table and rank them. github +1

Today, line up the vendor score and your actual work one line at a time

You don’t need to build a new evaluation system first. Open the original link provided by the vendor and the real record for one candidate automation task, then compare them in the following order.

1. Create an ID card for the score.
   Write the benchmark name and release, task set, model, harness, binary-versus-partial score distinction, step budget, retries, and whether human intervention was involved on one line. Do not guess at unpublished items; leave them as “unconfirmed.” osworld-v2.xlang +2
2. Mark verification status separately.
   Distinguish an official evaluation, an independent reproduction, and a vendor self-report. Do not mark an individual result as verified just because “Verified” appears in the benchmark name. llm-stats
3. Document the length and decision branches of the actual work.
   For an insurance-portal claim, for example, record the number of apps used, checks against external materials, interim saves, questions when exceptions arise, and confirmation after submission. Also decide whether completion means seeing the submission screen or having the case processed with no follow-up correction.
4. Separate full completion from partial progress.
   In the pilot table, use separate columns for “final completion,” “partial completion,” “number of retries,” “human intervention,” and “failures discovered after execution.” Even with a high partial score, do not count it as automation completion if a person still has to redo the follow-up work. a16z +1
5. Downgrade scores with major condition gaps to reference values.
   If the release or harness is missing and execution traces are not public, use the score only to shortlist candidates, not as a performance guarantee. Make the actual purchase decision from a pilot run with the same work records and success criteria.

Vendor claim: OSWorld-Verified 85%
Source and verification status: Third-party leaderboard / self-reported
Benchmark release, harness, steps, retries: Unconfirmed
Our work: Insurance portal claim submission
Completion criterion: Completed processing with no follow-up correction, not merely seeing the submission screen
Escalation: Hand off to the responsible person if the policy number does not match or additional confirmation is requested

The success criterion after completing this record is not to trust or disprove a particular score. It is enough if you can describe the vendor score in reproducible conditions, show the difference between those conditions and your work, and separately count the pilot’s full-completion rate and human intervention.

The use cases a16z introduced through vendor interviews also focused on work with relatively clear scope and completion conditions, such as updating system records, moving data across portals, and handling tickets. The original article likewise identifies verification, permissions, error handling, and escalation as important operational layers. But because the featured vendor cases do not disclose the total number of attempts, failure rate, or independent verification materials, they should not be generalized into broad ROI evidence. a16z

If you want to dig deeper

OSWorld 2.0: Benchmarking computer-use agents on long-horizon real-world tasks This is the official project for checking how long-horizon tasks are structured and how binary completion rates differ from partial scores. osworld-v2.xlang.ai

OSWorld-V2 README A useful resource for understanding the currently recommended release and why code, task, and asset versions need to be pinned together. github.com

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments You can trace where the original 2024 OSWorld task count and human/model baselines came from. arxiv.org