Computer Use
also called Screen control, Browser agent
An agent that drives a real UI with clicks, typing and screenshots.
A capability where the model is given screenshots and control of mouse and keyboard, so it can operate software that has no API. Multimodal input plus low-level tools.
It is the general fallback when no integration exists, and it is fragile: slow, expensive in tokens per action, and sensitive to layout changes. Prefer an API whenever one exists.
Reach for it when there is no API: legacy internal tools, unfamiliar vendor interfaces, one-off data entry. Its costs are structural — it is slower than an API, it fails on visual nuance, and it inherits the full untrusted-input risk of whatever it is looking at. Treat it as a capability of last resort with the strongest guardrails.
Where it earns its keep
A quarterly task exists only as a click-path in an old admin console.
No API, no export, forty minutes of clicking. Computer use turns that into a script that runs while someone watches the first few times. Verify on a throwaway record before pointing it at production data.
Where it should not be used
Automating a mainstream SaaS tool that has a documented API.
The API is faster, more reliable and easier to test, and it fails with structured errors. Vision-based clicking is the fallback for the gaps the API leaves, not the primary integration mechanism.
- docsComputer use tool ↗
Screenshot, mouse and keyboard as a tool loop, with the safeguards that belong around it: isolated environments, minimal privileges, network limits, human confirmation.
- docsComputer use ↗
Two GUI-agent patterns — structured mouse and keyboard actions, or code driving a browser — and where the runtime owns screenshots, sessions, limits and permissions.
- blogIntroducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku ↗
The first public computer-use announcement, which characterised the capability as experimental and error-prone from day one.
Multimodal
A model that handles more than text — images, audio, video, screen pixels.
Tool
A function the model is allowed to call to affect the world.
Environment
The world outside the model that tools act upon.
Permission Request
The agent pausing to ask before doing something risky.