Computer use

Anthropic released computer use as a public beta on 22 October 2024 with an upgraded Claude 3.5 Sonnet, and OpenAI followed in January 2025 with Operator and its Computer-Using Agent model; other labs and browser vendors have since shipped comparable features. Mechanically it is ordinary tool calling with a special tool definition. The client declares a computer tool with the display dimensions; the model returns tool-use blocks naming actions such as screenshot, left_click at (x, y), left_click_drag, scroll, type, key (a key combination) or wait; the client executes them against a real or virtual display and returns the result, a fresh screenshot for observation actions. The loop repeats until the model stops requesting tools.

Coordinates are expressed in the pixel space of the screenshot the model saw, which creates a classic bug. Vision APIs downscale large images (Anthropic's limit is roughly 1568 px on the long edge and about 1.15 megapixels), so a client that sends a 4K screenshot and clicks the returned coordinates unscaled will miss. Implementations either run the virtual display at a modest resolution such as 1280×800, which vendors recommend, or rescale screenshots down and coordinates back up. Recent tool versions let the model batch several actions per turn; the client should execute them in order and halt at the first failure, and ending a batch with a screenshot lets the model verify the outcome instead of assuming it.

Pixel-based control is the most general approach, reaching legacy thick clients, Citrix sessions and anything without an API, but it is slow, token-hungry and brittle compared with alternatives. Browser agents that read the DOM or accessibility tree, and classical robotic process automation with deterministic selectors, are faster and more repeatable when they apply. Benchmarks such as OSWorld (369 tasks across real desktop applications, with human success around 72%) and WebArena track progress; typical failures are misreading small text, acting before a page has loaded, losing track of state across many steps and clicking the visually similar wrong element.

The security posture follows from the fact that the screen is untrusted input and every click is a real action. Any web page, email or document rendered on screen can carry indirect prompt injection, and the agent inherits whatever sessions and credentials the machine holds. Vendor guidance is consistent: run the agent in a dedicated VM or container with minimal privileges, avoid exposing logged-in accounts or secrets, restrict network egress to an allowlist, and require human confirmation for consequential actions such as purchases, sending messages or accepting terms. Anthropic additionally runs classifiers over screenshots to flag likely injection attempts, which lowers but does not remove the risk.