Computer Use and Browser Agents

Computer-use agents operate software through the same interfaces available to a person. They are useful when a workflow has no practical API, but each action is slower and less reliable than a direct integration. The first design choice is whether the model should see pixels, structured UI data, or both.

Pixel, DOM, and hybrid control

The phrase covers two related but mechanically distinct architectures, and conflating them leads to the wrong tooling choice.

Pixel-input computer use. The model is given screenshots of a desktop or browser, and it returns structured actions in screen coordinates: {action: "left_click", coordinate: [612, 314]}, {action: "type", text: "Q3 budget"}. The harness executes the action against a real display (Xvfb, a Docker desktop, a VM, a remote browser via CDP) and returns the new screenshot. This is the model that Anthropic’s computer use tool implements. It’s general-purpose; works on any GUI, web or native; and pays for that generality with a vision-grounding tax: every click is the output of a multimodal model deciding what’s at that pixel.

Accessibility-tree / DOM-input computer use. The model is given a structured representation of the page or window; the DOM for a browser, the accessibility tree for a native app; annotated with stable identifiers, and it returns actions in terms of those identifiers: {action: "click", element: "submit-button"}, {action: "fill", element: "search-box", value: "Q3 budget"}. The harness executes against the DOM, not against pixels. This is the model that browser-use, Skyvern, and most “agentic browser” products use. It’s faster, cheaper, and more reliable on web; but it doesn’t work on native apps without an accessibility layer, and it falls over on visually-rich content (canvas elements, embedded PDFs, custom-rendered tables) that the DOM doesn’t expose semantically.

Hybrid input. Production systems blend both. The OSWorld benchmark, which evaluates desktop agents across 369 real tasks on real OSes, exposes four input modes: accessibility tree only, screenshot only, screenshot + accessibility tree, and set-of-marks (screenshot with element bounding boxes overlaid). Pixel + tree consistently outperforms either alone on the OSWorld leaderboard. The intuition is straightforward: the tree gives reliable element identity, the pixels give visual context for ambiguous cases (“the blue Submit button, not the gray one”).

Confusing these is the most common architectural mistake. Reaching for a screenshot loop when you’re automating a single SaaS app inside a browser is paying the vision-grounding tax for no reason; a DOM-driven library will be 5–10× cheaper and more reliable. Reaching for a DOM library to drive Excel, Photoshop, or a SAP fat client is a category error; there is no DOM.

Run the screenshot loop safely

The mechanical shape is the same agent loop from the agent loop article, with three substitutions: the action vocabulary is {screenshot, click, type, scroll, key, ...} instead of typed RPC, the observation is a base64-encoded PNG instead of a JSON result, and the loop driver owns coordinate scaling, a sandbox, and a screenshot retention policy.

Per-turn flow:

1. Loop driver captures a screenshot from the sandbox (Xvfb display, headless Chromium, remote VM).
2. Driver downsamples to fit the provider’s image limits; Anthropic constrains images to a maximum of 1568 pixels on the longest edge and ~1.15 megapixels total for pre-Opus-4.7 models; Opus 4.7 allows up to 2576 px on the long edge with 1:1 coordinate mapping. The downsample ratio must be applied in both directions: downscale the screenshot before sending, and upscale the coordinates Claude returns before executing the click. Mismatched scaling is the single most common cause of “the agent keeps clicking just below the button.”
3. Driver appends the screenshot to the conversation as a tool_result block, calls the model with the computer use beta header (computer-use-2025-11-24 for Opus 4.7 / Opus 4.6 / Sonnet 4.6 / Opus 4.5; computer-use-2025-01-24 for older models).
4. Model returns a tool_use block with an action and parameters: {action: "left_click", coordinate: [612, 314]}.
5. Driver executes the action against the sandbox via xdotool, pyautogui, Chrome DevTools Protocol, or platform-specific automation APIs.
6. Loop continues until the model returns a turn without a tool_use block, or until the iteration / token / wall-clock budget is hit.

Three points where most loops break.

Coordinate scaling. Outlined above. On macOS Retina displays the device pixel ratio doubles the screenshot resolution relative to logical coordinates, which compounds with the API’s input-size downsampling. The fix is to maintain a single explicit scale factor in the loop driver and apply it consistently on both legs.

Screenshot retention and prompt cache. Each screenshot is roughly 1,200–1,800 input tokens, so a 50-turn loop accumulates 60k–90k tokens of pixel data on top of the conversation. The naive fix; drop the oldest screenshot every turn; destroys prompt cache hit rates by mutating the prefix on every call. The right shape is batched eviction: keep the most recent 3 screenshots verbatim, evict in chunks of 25 turns or so, and place cache_control breakpoints after the system prompt and on the most recent tool_result blocks so the prefix stays byte-identical between evictions. The same JIT vs AOT context-engineering discipline applies; you’re managing a working set against a fixed token budget.

Action verification. Models sometimes assume their click landed and proceed. The fix is to explicitly ask the model, in the system prompt, to take a screenshot after each meaningful action and verify the outcome before continuing. Anthropic publishes this exact phrasing in their best-practices guide: “After each step, take a screenshot and carefully evaluate if you have achieved the right outcome.” Without this, a typo in the search box silently propagates through ten downstream steps before failing in a way that’s hard to attribute.

Choosing a model and harness

A quick orientation, May 2026:

• Anthropic computer use; beta on Opus 4.7, Opus 4.6, Sonnet 4.6, and Opus 4.5. The reference implementation ships as a Docker container with an Xvfb display, Firefox, LibreOffice, and the agent loop wired up. Opus 4.7 added a zoom action that lets the model request a high-resolution view of a sub-region; the visual equivalent of a database SELECT of a specific column. Internal benchmarks favor Opus 4.7 with effort: high extended thinking for the highest accuracy; Sonnet 4.6 with effort: medium for the best cost/accuracy ratio.
• OpenAI Operator / ChatGPT Atlas; Operator launched in January 2025 as a standalone CUA endpoint; Atlas, OpenAI’s ChatGPT-integrated Chromium-based browser, launched in late 2025 and as of May 2026 has Operator-style “Agent Mode” generally available on macOS for Plus, Pro, and Business tiers. Atlas runs the agent inside its own browser tab and grounds actions against the page DOM plus screenshots; a hybrid input mode by default.
• Browser-only DOM agents. browser-use is the dominant open-source Python library here, currently at 0.12.6 (April 2026), Python 3.11+. It exposes a clickable-element accessibility view via Playwright under the hood and lets you swap in any frontier model as the brain. Skyvern, Stagehand, and browserbase are the other production names. All four are betting on the same architectural premise: for web tasks, the DOM is a strictly better wire format than pixels.

The OSWorld-Verified leaderboard captures the trajectory. The Claude Mythos Preview currently leads at 79.6%, with GPT-5.5 at 78.7% and Claude Opus 4.7 at 78.0%. For reference, the human baseline is 72.36%; desktop agents are now above human level on this benchmark, having moved from ~20% a year ago. The benchmark has its critics (Epoch AI’s analysis is the most thoughtful), but the curve is real and steep.

Code: Python with the Anthropic computer use beta

A minimal sandboxed loop. This assumes you’ve stood up the reference Docker container or your own X11 sandbox with xdotool and screenshot helpers exposed.

1
2
3
4
5

pip install anthropic pillow pyautogui
# or use the reference Docker image:
# docker run -p 5900:5900 -p 6080:6080 -p 8501:8501 \
#   -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
#   ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158

import base64
import io
import math
import time
from anthropic import Anthropic
import pyautogui

# In production: run this loop inside a Docker container or VM, NOT on your host.
# Computer use treats any prompt injection on the rendered screen as
# an OS-level command. See /ai-engineering/guardrails/ for the input/output
# safety stack that complements sandboxing.
SCREEN_W, SCREEN_H = pyautogui.size()  # logical screen size
MAX_LONG_EDGE = 2576       # Opus 4.7 limit; use 1568 for older models
MAX_PIXELS = 1568 * 1568   # rough cap; per-model values in the docs

client = Anthropic()


def scale_factor(w: int, h: int) -> float:
    """Calculate downsample ratio to fit within the API's image limits."""
    long_edge_scale = MAX_LONG_EDGE / max(w, h)
    pixel_scale = math.sqrt(MAX_PIXELS / (w * h))
    return min(1.0, long_edge_scale, pixel_scale)


SCALE = scale_factor(SCREEN_W, SCREEN_H)
SENT_W, SENT_H = int(SCREEN_W * SCALE), int(SCREEN_H * SCALE)


def take_screenshot() -> str:
    """Capture, downscale, return as base64 PNG."""
    img = pyautogui.screenshot()
    img = img.resize((SENT_W, SENT_H))
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    return base64.standard_b64encode(buf.getvalue()).decode()


def execute_action(action: str, params: dict) -> dict:
    """Translate Claude's action into pyautogui calls. Coordinates come back
    in the *sent* image's pixel space; scale them up before clicking."""
    if action == "screenshot":
        return {"screenshot": take_screenshot()}

    if action == "left_click":
        x, y = params["coordinate"]
        pyautogui.click(x / SCALE, y / SCALE)
        time.sleep(0.3)  # let the UI settle before the next screenshot
        return {"screenshot": take_screenshot()}

    if action == "type":
        pyautogui.typewrite(params["text"], interval=0.02)
        return {"screenshot": take_screenshot()}

    if action == "key":
        pyautogui.hotkey(*params["text"].split("+"))
        return {"screenshot": take_screenshot()}

    if action == "scroll":
        x, y = params["coordinate"]
        amount = params["scroll_amount"] * (
            -1 if params["scroll_direction"] == "down" else 1
        )
        pyautogui.scroll(amount, x / SCALE, y / SCALE)
        return {"screenshot": take_screenshot()}

    return {"error": f"unhandled action: {action}"}


def to_tool_result(tool_use_id: str, result: dict) -> dict:
    if "error" in result:
        return {
            "type": "tool_result",
            "tool_use_id": tool_use_id,
            "content": result["error"],
            "is_error": True,
        }
    return {
        "type": "tool_result",
        "tool_use_id": tool_use_id,
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/png",
                    "data": result["screenshot"],
                },
            }
        ],
    }


def run(task: str, max_iters: int = 25):
    # Seed the conversation with an initial screenshot so the model
    # knows what's on screen.
    initial_shot = take_screenshot()
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": task},
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": initial_shot,
                    },
                },
            ],
        }
    ]

    system = (
        "You are a careful computer-use agent. After every meaningful action, "
        "take a screenshot and verify the outcome before continuing. If the "
        "result doesn't match your expectation, retry or replan rather than "
        "proceeding optimistically. Halt and ask for confirmation before any "
        "irreversible action (purchase, send, delete)."
    )

    for _ in range(max_iters):
        resp = client.beta.messages.create(
            model="claude-opus-4-7",
            max_tokens=4096,
            system=system,
            messages=messages,
            tools=[
                {
                    "type": "computer_20251124",
                    "name": "computer",
                    "display_width_px": SENT_W,
                    "display_height_px": SENT_H,
                    "enable_zoom": True,
                }
            ],
            betas=["computer-use-2025-11-24"],
        )
        messages.append({"role": "assistant", "content": resp.content})

        tool_uses = [b for b in resp.content if b.type == "tool_use"]
        if not tool_uses:
            # Model finished — surface the final text.
            text = "".join(b.text for b in resp.content if b.type == "text")
            return text

        tool_results = [
            to_tool_result(b.id, execute_action(b.input["action"], b.input))
            for b in tool_uses
        ]
        messages.append({"role": "user", "content": tool_results})

    return "halted: max iterations reached"


if __name__ == "__main__":
    print(run("Open Firefox and search for 'Anthropic computer use'."))

Several implementation details are easy to miss in the documentation.

The enable_zoom: true parameter on the tool definition turns on the zoom action for Opus 4.7. The model uses it when it needs to read small text or distinguish similar elements; far cheaper than re-screenshotting the entire display at a higher resolution.

The 0.3-second sleep after each click matters more than it looks. Without it, the next screenshot captures the screen mid-transition: half-rendered menus, half-loaded pages, animations partway through. The model then attempts to click the not-yet-rendered target.

The system prompt nudges the model toward verification and against optimistic continuation. This single prompt change is, empirically, one of the largest accuracy interventions you can make for under 50 tokens. Anthropic’s docs spell out this guidance for a reason.

Screenshot pruning isn’t implemented here for brevity. In a real loop you’d track screenshot positions in the messages array and replace older image blocks with text placeholders ("[screenshot from 8 turns ago; pruned]") in batched eviction passes, not one per turn; see the prompt caching discussion above.

There’s no sandbox boundary in this example because the goal is to show the loop shape. You should not run this against your host machine. Run it inside a VM, a Docker container with VNC, or against browserbase-style remote browser infrastructure. Anthropic’s reference implementation gives you a ready-built Docker container with the right defaults.

Further reading

• Anthropic; Computer use tool docs; the canonical reference for the beta header, the computer_20251124 and computer_20250124 tool schemas, the full action vocabulary, coordinate scaling, screenshot retention, and the model-specific best-practices tables. Read end-to-end before standing up a loop; the docs are dense but every paragraph is load-bearing.
• Anthropic; Best practices for computer and browser use; the benchmarked recommendations for resolution, extended-thinking effort, and screenshot management. The empirical guidance Anthropic doesn’t put in the main docs.
• OpenAI; Hardening Atlas against prompt injection; OpenAI’s published threat model and mitigation stack for the Atlas browser agent. The clearest single source on the prompt-injection attack surface in the wild.
• Epoch AI; What does OSWorld tell us about AI’s ability to use computers?; the most honest analysis of what the OSWorld numbers do and don’t say. Useful for sizing the gap between benchmark and production.