Anthropic introduces computer use with Claude

Anthropic announced computer use in beta alongside updated Claude 3.5 Sonnet and Claude 3.5 Haiku. The feature lets models interpret screenshots and interact with software through tool actions.

By Hypler editorial team · Source: Anthropic.

Original source date: .

Hypler publication: . Hypler updated: .

Estimated reading time: 5 minutes.

Topics: computer use, agents, authorization

Anthropic logo

Generated illustration of modular compute gates and an isolated coding workflow.
Generated illustration of modular compute gates and an isolated coding workflow. Credit: Hypler / AI-generated editorial illustration. Hypler editorial asset; not vendor or event photography

A beta that reaches the desktop

Anthropic announced computer use in beta on October 22, 2024 alongside updated Claude 3.5 Sonnet and Claude 3.5 Haiku. The feature lets models interpret screenshots and interact with software through tool actions. Anthropic described safeguards and limitations for the early release, which is different from granting unrestricted automation authority. Computer interaction matters because familiar user-interface actions can create durable changes: a click can submit a form, reveal account information, alter a record, or trigger a payment path. The announcement is a historical release account. An application considering such a capability still has to decide which environment exists, what actions are permitted, where secrets live, and who can approve a consequential step.

Screenshots are an untrusted input channel

A screenshot may contain rendered instructions, sensitive data, stale status, or visual ambiguity that a model interprets incorrectly. It can also omit the state needed to understand whether a button is safe to press. Treating pixels as direct authority creates a path for prompt injection through ordinary interface text. The tension is that computer use is valuable precisely when it can navigate existing software, while existing software may contain untrusted content and hidden side effects. A reliable design separates observation from action: the system can capture a view, classify it, and propose the next step, but a deterministic policy must still decide whether the target, control, and resulting operation are allowed.

Isolation gives action a boundary

A computer-use workflow should begin in an isolated environment with a declared account, network policy, data set, browser or desktop profile, session expiry, and allowed destinations. An action allowlist can constrain navigation, form fields, downloads, uploads, and system operations, while a confirmation rule protects external communication, deletion, purchases, and changes to shared state. Credentials should be scoped to the sandbox and unavailable to screenshots or general logs. The controller should validate target and parameters before each action rather than trusting a sequence of clicks because earlier steps appeared correct. This makes computer use a bounded execution capability rather than a chat feature with accidental access to a whole workstation.

Silent side effects are the failure mode

Automation can fail without a crash. A model might select the wrong tab, act on a stale page, follow an instruction rendered by untrusted content, or retry after a network delay and duplicate a submission. A screenshot-only log may show that something was visible without proving what action actually occurred. The service needs replayable receipts: environment identity, action intent, target, validated parameters, policy result, timestamp, output artifact, and final human disposition. It should distinguish observation failure, denied action, UI mismatch, timeout, partial completion, and irreversible result. That clarity lets an operator stop a run before it compensates for uncertainty with more authority.

A controlled browser drill

In a hypothetical Hypler browser-testing route, a model may inspect a staged public form inside an isolated profile and propose a navigation sequence. It can fill nonpersistent sample values, but the submit button requires a human confirmation and the environment has no payment, production, or personal-account access. Fixtures include an injected instruction in page text, a page layout that moves the button, a stale confirmation screen, and a request to upload a file outside the allowlist. The controller treats page text as data, rejects the upload, and ends the run when the UI no longer matches the expected action map. The drill illustrates execution controls; it does not describe production automation.

Control must outlast the demo

Readiness evidence should cover environment isolation, session expiry, destination allowlists, screenshot handling, action validation, confirmation gates, UI mismatch, retries, credential separation, and replayable logs. It should prove that a model or prompt update cannot grant access to a new application or weaken an existing stop condition. Anthropic’s computer-use announcement makes graphical interaction a design option. The grounded judgment is that automation earns wider use only after its environment, permissions, and recovery behavior are more explicit than the interface it is operating, with humans retaining authority over consequential outcomes.

Hypler analysis

Hypler’s view

A computer-use model can affect systems through ordinary interfaces, so a weak boundary becomes more expensive than in chat. Hypler should isolate the environment, validate every requested action, and make review possible before shared-state or external effects occur.

Hypler technical analysis

Engineering notes

computer interaction turns rendered pixels and page state into untrusted inputs and makes control errors different from chat errors. A supervised browser workflow needs allowlisted origins, disposable identity, read-only or narrowly scoped credentials, redacted visual evidence, normalized action logs, freshness checks between observation and click, and a stop policy for unexpected prompts. The model can identify a candidate record or draft a patch, but a separate service should validate field permissions and require approval before mutation. Evaluation must cover screenshot drift, overlays, stale DOM, retries, duplicate submission, and manual takeover. Success is a safe, explainable stopped run or review artifact as well as a completed task.

What to inspect

Browser session boundary
Inspect origin allowlist, disposable profile, credential scope, action permissions, screenshot redaction, retention rule, and stop conditions before navigation.
Perception-drift fixture
Move an overlay between screenshot and click or return stale page state; inspect freshness detection, blocked action, and recorded stop reason.
Mutation approval trace
For a proposed update, inspect selected record evidence, field-level validation, operator approval, idempotency handling, and final service response.

Scope and limits. A beta feature and its safeguards do not make screen interpretation reliable enough for unrestricted automation or grant authority to mutate external systems.

Terms and definitions

Prompt Injection
An attack where untrusted content tries to alter an AI system’s instructions or cause unsafe tool use.
Evaluation
A repeatable method for measuring a system against defined quality, safety, or task criteria.
Tool Calling
A model interface pattern where the model requests a structured operation from an external tool.
API Layer
The server-side interface that exposes defined data and actions to clients or integrations.

More on this subject

  • Anthropic introduces web search on the API

    Anthropic introduced web search as an API tool for supported Claude models, returning web results and citations for applications using current public sources.

  • Google announces the Agent Development Kit

    Google announced the Agent Development Kit, an open-source framework for building and evaluating multi-agent applications.

  • OpenAI introduces deep research

    OpenAI introduced deep research in ChatGPT as an agentic capability that searches, analyzes, and synthesizes multi-step research from online sources. The release described cited outputs and a browser-and-data-analysis workflow.

Generated illustration of isolated tool-execution fixtures and a controlled test gate; not deployed software.
Generated illustration of isolated tool-execution fixtures and a controlled test gate; not deployed software. Credit: Hypler / AI-generated editorial illustration. Hypler editorial asset; not vendor or event photography

Original source

Historical source briefing with Hypler engineering analysis. Source details reflect the original publication date; implementation examples are hypothetical. Technical analysis expanded October 24, 2026; this does not change the original source event or establish a new source verification. Editorial prose revised October 26, 2026; original event, publication and source-check dates are retained.

Related capabilities