Vol. XXV · Tools and Agents Plate 6 of 9computer use, GUI agents, screenshots, clicks, grounding, OSWorld, terminal, browserCE

Using a Computer.

No API: just the screen, the mouse and the keyboard, the way people do it.

Most software has no API an agent can call, but all of it has a screen. A computer-use agent receives a screenshot (or an accessibility tree, or terminal text), and acts by emitting coordinates to click, text to type, keys to press or commands to run; the harness performs the action on a real or virtual machine and returns the next screenshot. This makes every application available — and makes every step a perception problem: the model must find the right element in an image and output its position precisely, which benchmark analyses identify as the main source of failure (“grounding”).

OSWorld (Xie et al., 2024) gave 369 tasks in real desktop and web applications, each checked by a script against the machine’s final state: humans completed over 72.4% at the time and the best model 12.2%. The workshop models one part of the difficulty: the click lands at the intended point plus a Gaussian error, the target is a button of the chosen width, and a task needs the chosen number of clicks, each of which must land.

Below: the chance a click lands on its target against the target’s width, for the chosen pointing error; and the chance a whole task’s clicks all land.

What you will learn
Workshop
A square target of the chosen width in screen pixels; the click falls at the target’s centre plus independent Gaussian errors of the chosen standard deviation in x and y. Every click of the task must land. Real failures also include choosing the wrong element and not knowing the application — this models only precision. Screenshot cost: an assumed 1 500 tokens per screenshot, one per step.

Precision is a skill, and it compounds

With a pointing error of 10 pixels, a 24-pixel button is hit 59% of the time and a 15-click task succeeds 0.04% of the time; halve the error to 5 pixels and the click is nearly certain (96.7%) and the task succeeds 61% of the time. Shrink the button to 12 pixels at the original error and a single click lands only 20% of the time. The same compounding as plate XXV.5 applies, with grounding as a per-step failure mode that text agents do not have.

Computer-use agents have improved rapidly on such benchmarks since OSWorld reported 12.2% against humans’ 72%, through training on screenshots with target coordinates, larger and sharper image inputs, and harnesses that let the agent zoom or use keyboard shortcuts and accessibility information instead of pixels where possible. They remain slow and costly — every step is a screenshot of over a thousand tokens and a full model call — and they inherit every risk of an agent with a real machine’s permissions (plate XXV.8).

Where a structured interface exists — an API, a command line, an accessibility tree — agents use it: text actions are cheaper and more reliable than pixels. Screens are the fallback that makes everything else reachable.

Terms

Computer use
Acting through screenshots, clicks, typing and commands.
Grounding
Locating the right element in a screenshot precisely.
Accessibility tree
The structured description of on-screen elements an operating system provides.
State-based checking
Grading a task by inspecting the machine afterwards.
Computer use makes every application reachable and every step a perception problem; small pointing errors compound like any other.

Sources

  1. Xie et al., “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments”, NeurIPS 2024.
  2. Anthropic, “Introducing computer use”, October 2024.
  3. Zheng et al., “GPT-4V(ision) is a Generalist Web Agent, if Grounded”, ICML 2024.