Runtime (withruntime.com) runs a full Linux desktop inside each sandbox's Firecracker microVM. Some software has no API. Examples include an old internal admin tool, a desktop application, and a vendor portal built for people who use a mouse. A computer-use agent works this software the way a person would, from screenshots. That makes it the most powerful kind of agent, and the one that most needs a machine of its own. This post connects Claude's computer toolset to a sandbox desktop. It covers the parts that trip up first versions, and it sets the limits that make it safe to leave running.
When does an agent need a desktop, not a browser API?
Less often than it seems. A screenshot loop is slower and costs more tokens than any structured interface. Use it only where nothing else reaches:
| Interface | Speed per step | What it reaches | Use it for |
|---|---|---|---|
| An API or a CLI | Milliseconds | What the vendor exposes | Everything it covers |
| A browser over CDP | Under a second | Web pages, through the page's structure | Web apps, forms, scraping |
| A desktop through screenshots | Seconds | Anything drawn on a screen | Desktop apps, canvas apps, odd portals |
Many teams use both of the lower rows. They run a CDP browser for the web steps
and a desktop for the one application that has only a window. The two can share
a sandbox, because a browser started with headless: false appears on the
desktop.
How does the computer toolset map onto a sandbox desktop?
Claude's computer toolset, computer_toolset_20260801, gives the model member
tools such as screenshot, left_click and type. The model calls them as
ordinary tool calls, often several in one reply. Your code performs each call.
Nearly every member maps to one call on Runtime's desktop:
| Member tool | Runtime call | Notes |
|---|---|---|
screenshot |
desktop.screenshot() |
PNG bytes, returned as an image |
left_click, right_click, middle_click |
desktop.click(x, y, { button }) |
At coordinate, or where the pointer is |
double_click |
desktop.doubleClick(x, y) |
|
mouse_move |
desktop.move(x, y) |
|
left_click_drag |
desktop.drag(start, end) |
|
left_mouse_down, left_mouse_up |
desktop.mouseDown(), desktop.mouseUp() |
For drags the model builds itself |
type |
desktop.type(text) |
Literal text |
key |
desktop.press(keys) |
Same key names: Return, ctrl+s |
scroll |
desktop.scroll(dy, { dx, x, y }) |
Direction and amount become wheel clicks |
cursor_position |
desktop.cursor() |
Returned as text |
wait |
A sleep in your code | |
zoom, triple_click, hold_key |
Turned off in the request | The toolset's configs turns them off |
Turning off a member is better than supporting it partly. The model then never asks for it. Otherwise, it asks and gets an error.
What does the whole agent look like?
The loop is an ordinary tool loop with three details of its own. Results carry
toolset_name: "computer". A screenshot goes back as an image. When one action
in a batch fails, the rest are reported as not run:
TypeScriptimport { writeFile } from "node:fs/promises";import Anthropic from "@anthropic-ai/sdk";import { Sandbox } from "withruntime";const client = new Anthropic(); // reads ANTHROPIC_API_KEYconst SYSTEM = "You operate a Linux desktop. Before any purchase, deletion, message to a person or " + "acceptance of terms, stop and describe what you are about to do instead of doing it.";type Point = [number, number];type Input = { coordinate?: Point; start_coordinate?: Point; text?: string; repeat?: number; scroll_direction?: "up" | "down" | "left" | "right"; scroll_amount?: number; duration?: number;};// One member tool call on the sandbox's desktop: a screenshot, a line of text, or "OK".async function act(desktop: Sandbox["desktop"], name: string, input: Input) { const [x, y] = input.coordinate ?? []; if (input.text && name.endsWith("click")) throw new Error("clicks with held keys are not supported"); const n = input.scroll_amount ?? 3; const dir = input.scroll_direction; switch (name) { case "screenshot": return desktop.screenshot(); case "left_click": await desktop.click(x, y); break; case "right_click": await desktop.rightClick(x, y); break; case "middle_click": await desktop.click(x, y, { button: "middle" }); break; case "double_click": await desktop.doubleClick(x, y); break; case "mouse_move": await desktop.move(x!, y!); break; case "left_click_drag": await desktop.drag(input.start_coordinate!, input.coordinate!); break; case "left_mouse_down": await desktop.mouseDown(); break; case "left_mouse_up": await desktop.mouseUp(); break; case "type": await desktop.type(input.text!); break; case "key": for (let i = 0; i < (input.repeat ?? 1); i++) await desktop.press(input.text!); break; case "scroll": await desktop.scroll(dir === "up" ? -n : dir === "down" ? n : 0, { dx: dir === "left" ? -n : dir === "right" ? n : 0, ...(x === undefined ? {} : { x, y }), }); break; case "cursor_position": { const c = await desktop.cursor(); return `X=${c.x}, Y=${c.y}`; } case "wait": await new Promise((r) => setTimeout(r, Math.min(input.duration ?? 1, 30) * 1000)); break; default: throw new Error(`${name} is not available`); } return "OK";}export async function computerUse(goal: string, maxTurns = 60): Promise<string> { await using sbx = await Sandbox.create({ diskMiB: 8192, timeoutSeconds: 3600, network: { internet: true, allow: ["portal.example.com", "*.example.com"] }, }); const { streamUrl } = await sbx.desktop.start({ width: 1280, height: 800 }); console.log("watch it live:", streamUrl); // private: the link carries its token const recording = await sbx.desktop.recordings.start({ fps: 5, maxMiB: 256 }); await sbx.desktop.open("https://portal.example.com"); const messages: Anthropic.MessageParam[] = [{ role: "user", content: goal }]; try { for (let turn = 0; turn < maxTurns; turn++) { const reply = await client.messages.create({ model: "claude-opus-5-5", max_tokens: 16_000, system: SYSTEM, tools: [ { type: "computer_toolset_20260801", configs: { zoom: { enabled: false }, triple_click: { enabled: false }, hold_key: { enabled: false }, }, }, ], messages, }); messages.push({ role: "assistant", content: reply.content }); if (reply.stop_reason !== "tool_use") return reply.content.flatMap((b) => (b.type === "text" ? [b.text] : [])).join("\n"); const results: Anthropic.ToolResultBlockParam[] = []; let failed = false; for (const call of reply.content) { if (call.type !== "tool_use") continue; const result = { type: "tool_result" as const, tool_use_id: call.id, toolset_name: "computer", }; if (failed) { results.push({ ...result, is_error: true, content: "Not executed: an earlier computer action in this turn failed.", }); continue; } try { const out = await act(sbx.desktop, call.name, call.input as Input); const data = typeof out === "string" ? "" : Buffer.from(out).toString("base64"); results.push({ ...result, content: typeof out === "string" ? [{ type: "text", text: out }] : [{ type: "image", source: { type: "base64", media_type: "image/png", data } }], }); } catch (error) { failed = true; results.push({ ...result, is_error: true, content: `Error: ${(error as Error).message}`, }); } } messages.push({ role: "user", content: results }); } return "Stopped at the turn limit."; } finally { await sbx.desktop.recordings.stop(recording.id); await writeFile( `computer-use-${sbx.id}.mp4`, await sbx.desktop.recordings.download(recording.id), ); }}Pythonimport base64import timeimport anthropicfrom withruntime import Sandboxclient = anthropic.Anthropic() # reads ANTHROPIC_API_KEYSYSTEM = ("You operate a Linux desktop. Before any purchase, deletion, message to a person or " "acceptance of terms, stop and describe what you are about to do instead of doing it.")TOOLS = [{"type": "computer_toolset_20260801", "configs": {"zoom": {"enabled": False}, "triple_click": {"enabled": False}, "hold_key": {"enabled": False}}}]def act(desktop, name: str, i: dict): """One member tool call on the sandbox's desktop: a screenshot, a line of text, or "OK".""" x, y = i.get("coordinate") or (None, None) if i.get("text") and name.endswith("click"): raise ValueError("clicks with held keys are not supported") n, direction = i.get("scroll_amount", 3), i.get("scroll_direction") if name == "screenshot": return desktop.screenshot() if name == "left_click": desktop.click(x, y) elif name == "right_click": desktop.right_click(x, y) elif name == "middle_click": desktop.click(x, y, button="middle") elif name == "double_click": desktop.double_click(x, y) elif name == "mouse_move": desktop.move(x, y) elif name == "left_click_drag": desktop.drag(tuple(i["start_coordinate"]), tuple(i["coordinate"])) elif name == "left_mouse_down": desktop.mouse_down() elif name == "left_mouse_up": desktop.mouse_up() elif name == "type": desktop.type(i["text"]) elif name == "key": for _ in range(i.get("repeat", 1)): desktop.press(i["text"]) elif name == "scroll": desktop.scroll({"up": -n, "down": n}.get(direction, 0), dx={"left": -n, "right": n}.get(direction, 0), x=x, y=y) elif name == "cursor_position": c = desktop.cursor() return f"X={c['x']}, Y={c['y']}" elif name == "wait": time.sleep(min(i.get("duration", 1), 30)) else: raise ValueError(f"{name} is not available") return "OK"def computer_use(goal: str, max_turns: int = 60) -> str: with Sandbox.create(disk_mib=8192, timeout_seconds=3600, network={"internet": True, "allow": ["portal.example.com", "*.example.com"]}) as sbx: print("watch it live:", sbx.desktop.start(width=1280, height=800)["streamUrl"]) recording = sbx.desktop.recordings.start(fps=5, max_mib=256) sbx.desktop.open("https://portal.example.com") messages: list[dict] = [{"role": "user", "content": goal}] try: for _ in range(max_turns): reply = client.messages.create(model="claude-opus-5-5", max_tokens=16_000, system=SYSTEM, tools=TOOLS, messages=messages) messages.append({"role": "assistant", "content": reply.content}) if reply.stop_reason != "tool_use": return "\n".join(b.text for b in reply.content if b.type == "text") results, failed = [], False for call in (b for b in reply.content if b.type == "tool_use"): result = {"type": "tool_result", "tool_use_id": call.id, "toolset_name": "computer"} if failed: results.append({**result, "is_error": True, "content": "Not executed: an earlier computer action in this turn failed."}) continue try: out = act(sbx.desktop, call.name, call.input) except Exception as error: failed = True results.append({**result, "is_error": True, "content": f"Error: {error}"}) continue if isinstance(out, bytes): image = {"type": "base64", "media_type": "image/png", "data": base64.b64encode(out).decode()} results.append({**result, "content": [{"type": "image", "source": image}]}) else: results.append({**result, "content": [{"type": "text", "text": out}]}) messages.append({"role": "user", "content": results}) return "Stopped at the turn limit." finally: sbx.desktop.recordings.stop(recording["id"]) with open(f"computer-use-{sbx.id}.mp4", "wb") as f: f.write(sbx.desktop.recordings.download(recording["id"]))Why is the screen exactly 1280 by 800?
Your screen must match the screenshots you send. The model's clicks use the coordinates of those screenshots, and the API does not shrink images for you. A screenshot larger than the model's image limit is refused. If you shrink a screenshot yourself, every coordinate the model returns must be scaled up before the click. Skip that step once, and the agent clicks a little left of every button. That looks like a model failure, not a bug in your code.
The simplest fix is to make the screen the size you send. 1280 by 800 is well inside the limit for current Claude models. It matches how most web applications are designed, and it keeps each screenshot to a modest number of tokens. Coordinates then pass straight through with no arithmetic. Larger screens make each step slower and more expensive, and they do not make the model more accurate.
What should a batch of actions do when one fails?
Stop. The model often sends a short plan in one reply: click the search box, type a query, press Return, and take a screenshot. If the click fails, the typed query lands somewhere else. Pressing Return may then submit a form that nobody meant to submit. The loop above runs the calls in order. After the first error, it marks each remaining call "Not executed". Every call still gets a result, and the model checks the screen again before it tries anything else.
How long can one run go on?
Every screenshot stays in the transcript, so a long task adds an image to its context at each step. Start with the turn limit: a task that needs more than sixty steps is usually two tasks. Do not delete old screenshots from the messages yourself. On current Claude models, editing earlier turns invalidates the model's later reasoning. When a run must go longer, use the API's tool result clearing instead. It removes old results on the server side.
How do you keep a computer-use agent from doing harm?
The model acts on whatever it sees. That includes text written by strangers: a web page, an email in the portal, or a document. Instructions in that text can steer it (prompt injection to code execution). Limit what a steered agent can reach:
- Its own machine. The desktop is inside the sandbox's microVM, so a download it opens or a program it starts never touches yours.
- A short network allow list. Name only the sites the task needs, so a page cannot send the agent somewhere new (allow only some hosts).
- No logged-in browser of yours. Sign in with an account made for the agent. Give that account the least access the task needs.
- A person for irreversible actions. The system prompt tells the model to stop and describe the action before a purchase, a deletion, a message to a person, or new terms. Your code sends that description to a human, not back to the model.
- A record. The recording is an MP4 of every step. It stays on the sandbox's disk until your code downloads it. When the agent does something unexpected, you can watch exactly what it saw. The live link lets you watch while the agent runs.
How fast is it, and what does it cost?
The desktop is ready on every server. The first desktop start in a sandbox installs nothing, and it takes about a second. Start it once. Then take a memory snapshot of the sandbox, with the desktop running, the application open, and the agent's account signed in. Each task then starts from the snapshot, already at the first screen (snapshots and forks).
After that, the model sets the pace. Each step is one model call with a screenshot in it. A step usually takes a few seconds, while the desktop's own actions return in a fraction of that time. A 40-step task takes a few minutes. During that time the sandbox mostly waits. CPU is billed only when used, at $0.025 per vCPU-hour. Memory is billed at $0.0075 per GiB-hour while the sandbox runs. The model's image tokens are the larger cost. That is one more reason to keep the screen at 1280 by 800 rather than larger.
How do you test it without spending tokens?
Replay. Save the model's replies from one good run. Then feed them back through
act against a fresh sandbox from the same snapshot, and compare the final
screenshot with the original. A replay that diverges means the application
changed. That is what you want to learn before the next real run.
The same idea, applied to whole agent runs, is in
reproducible agent evals.
In short
- Use a desktop only where no API or browser interface reaches; it is the slowest and most capable interface.
- Map each computer toolset member to one desktop call, and turn off the members you do not support.
- Make the screen the size you send, so coordinates need no scaling.
- Stop a batch at its first failed action, and mark the rest as not run.
- Contain it: its own microVM, a short allow list, an account made for it, a person for irreversible steps, and a recording of every run.
Run it on Runtime
Runtime costs 42% to 88% less than fourteen other sandbox providers for an agent that mostly waits on a model (compare costs). Every Runtime sandbox can run a desktop, record it to MP4 and share a private live view. Read how to control a desktop, then create an account at withruntime.com and point the agent above at the application that has no API.