On Runtime (withruntime.com) one sandbox per task keeps its state between tool calls, so only the first call pays a start. One cold start rarely matters. Forty in a row, each with a setup step and a network round trip, can matter more than the model. This post works through the arithmetic for one task. It shows where the time goes, and it gives a small harness to measure your own loop.
Where does an agent's time go?
Three places: the model thinking, the tools running, and the overhead of sending each tool call to a machine and back. The first two are the work. The third is the subject of this post. It grows with the number of calls, not with how hard the task is.
Take one ordinary coding task: 40 tool calls, one after another, each a short
command such as grep, cat or a single test file. Say the model takes 4
seconds per turn, which is typical for a mid-sized model answering with a tool
call. The model's share is fixed:
- Model time
- 40 turns × 4 s
- 160 s
Everything below adds to those 160 seconds. The user waits for all of it.
What does each architecture add?
The cost depends on how often the loop starts a machine and how much setup each start repeats. Call a new sandbox's time to its first result S, a command in a running sandbox C, and a wake through the next command W (the speed guide lists an operation's time once it is within 2× of its floor; time them yourself from where your loop runs). Here are four common architectures:
| Architecture | What the loop does | Overhead added |
|---|---|---|
| New sandbox per call, with setup | Create, clone, install (say 20 s), run, delete | 40 × (S + 20 s) |
| New sandbox per call, no setup | Pure computation with nothing to install | 40 × S |
| One sandbox per task | Create once, send every call to the running machine | S + 40 × C |
| One per conversation, paused between | 40 calls in 8 user messages; each message wakes it once | 8 × W + 32 × C |
The setup is the main cost: 40 × 20 s is 800 seconds, five times the model's 160, whatever the start time. Each fresh machine repeats it, because a fresh machine has no memory of the last call. Keep state, and only S, C and W are left.
Why does the network count 40 times?
Every tool call is a round trip from your agent loop to the sandbox and back. The loop waits for each one. Your loop adds the distance between it and the sandbox, once per call.
Runtime runs in one region, in Virginia. From a laptop one round trip of, say, 70 ms away, 40 calls add 40 × 70 ms = 2.8 s per task; from another continent, add your own round-trip time, once per call. The fix is cheap. Run the agent loop on a server in a US East region, close to the sandboxes. Let only the final answer travel to the user. A model API call is one round trip per turn either way, so moving the loop costs nothing there.
Why does the slow tail matter more than the median?
Because a task with many calls almost always hits a slow one. If each call has a 5% chance of landing at or beyond the 95th percentile, a task of 40 calls has at least one such call with this probability:
- 1 − 0.95^40
- 87%
So most of your tasks feel the p95, and long sessions feel the p99. That is why a provider's median alone tells you little about an agent loop (what p95 latency means). Ask for the tail numbers, and measure them yourself.
How do you measure your own loop?
Time every tool call where your loop makes it. Then look at the distribution, not the average. This harness runs a sequence of commands the way an agent would, one after another. It reports the median, the p95 and the total:
TypeScriptimport { Sandbox } from "withruntime";const calls = [ "ls", "cat /etc/os-release", "python3 -c 'print(sum(range(1000)))'", "grep -c processor /proc/cpuinfo", "df -h /workspace",];await using sbx = await Sandbox.create();const times: number[] = [];for (let round = 0; round < 8; round++) { for (const command of calls) { const started = performance.now(); await sbx.exec(command, { timeoutMs: 30_000 }); times.push(performance.now() - started); }}times.sort((a, b) => a - b);const at = (q: number) => Math.round(times[Math.min(times.length - 1, Math.floor(q * times.length))]!);const total = Math.round(times.reduce((sum, t) => sum + t, 0));console.log(`${times.length} calls: median ${at(0.5)} ms, p95 ${at(0.95)} ms, total ${total} ms`);Pythonimport timefrom withruntime import Sandboxcalls = [ "ls", "cat /etc/os-release", "python3 -c 'print(sum(range(1000)))'", "grep -c processor /proc/cpuinfo", "df -h /workspace",]with Sandbox.create() as sbx: times: list[float] = [] for _ in range(8): for command in calls: started = time.perf_counter() sbx.exec(command, timeout_ms=30_000) times.append((time.perf_counter() - started) * 1000) times.sort() def at(q: float) -> int: return round(times[min(len(times) - 1, int(q * len(times)))]) print(f"{len(times)} calls: median {at(0.5)} ms, p95 {at(0.95)} ms, total {round(sum(times))} ms")Run it from where your agent loop runs in production, not from your laptop, and compare the total with your model's time for the same number of turns. If the sandbox share is more than a few percent, one of the fixes below will find it.
What actually shrinks the overhead?
Most of the gain comes from the first two items; the rest are refinements.
- Keep one sandbox per task or conversation. Never use one per call. State survives, setup runs once, and each call costs a command, not a start.
- Bake setup into an image. Dependencies installed at build time cost nothing at create time, so even the first call of a task is fast (custom images).
- Run the loop near the sandboxes. Each call costs one round trip, and the calls multiply that cost. This is the overhead you can control most cheaply.
- Pause between messages, not between calls. A wake costs more than a command. Pause when the user is reading, and let the 300 seconds idle pause handle the rest (pause, don't delete).
- Run parallel tool calls in parallel. When a model asks for three reads at the same time, send them concurrently. The overhead becomes the slowest call, not the sum.
- Let one call do more. A tool that accepts a short script instead of a
single command turns five round trips into one. Models handle
cd app && npm test -- --run authas easily as three separate calls.
Item 6 has a limit. A call that does too much returns too much output, and the model reads all of it. Combine steps that always run together. Keep separate the steps whose output the model needs to decide the next one.
In short
- Per-call overhead multiplies by the number of tool calls, so a task of 40 calls feels it 40 times.
- Repeated setup on fresh machines is what costs minutes, more than the start itself.
- Every call is a round trip, so run the agent loop near the sandboxes.
- Many calls means most tasks hit the p95: measure the tail, not just the median.
- Keep one sandbox per task or conversation, bake setup into an image, and pause only between messages.
Run it on Runtime
Runtime costs 42% to 88% less than fourteen other sandbox providers for an agent that mostly waits on a model (compare costs). CPU is billed at $0.025 per vCPU-hour of actual use, so a sandbox kept for a whole task costs little while the model thinks. Each month includes 100 free hours of a 1 GB machine, and no card is required. Sign in, or read get started and the pricing.