Runtime (withruntime.com) answers each question below with something you can check: a Firecracker microVM per sandbox, speed measured each minute from outside, and $0.025 per vCPU-hour of CPU actually used. Each provider's landing page says it is fast, secure and cheap. These twelve questions go past those words. Each one includes a follow-up question that separates a direct answer from a scripted one, and a way to check the answer yourself in a few minutes.
1. Does every sandbox get its own kernel?
It should. Code that shares a kernel with other tenants is one kernel bug away from them.
Follow up: "What exactly is shared between two customers' sandboxes on the
same server?" A container shares the host kernel. A user-space kernel shares
less. A microVM shares only the hypervisor.
Check it: get the isolation technology in writing, not only the word
"isolated". Inside a sandbox, uname -r shows the kernel your code runs on.
Runtime: every sandbox is a Firecracker microVM with its own Linux kernel
(microVM vs container).
2. How fast is a new sandbox, and how was that measured?
Ask for a median and a 95th percentile. Time each one from the create request to the first command finishing. The word "ready" can mean many things.
Follow up: "Is that timed on your servers or from a client? Is it the API answering, or a command having run?" A create call that returns before the machine can run anything moves the wait into your first command. Check it: time a create and a first command a hundred times from where your agent runs. Look at the slowest five. Runtime: every operation is timed each minute from outside, through the public API: the API's own time and the client's, at p50, p95 and p99, with the samples and the code. An operation's figures are published once it is within 2× of its physical floor (speed).
3. What does an hour of waiting cost?
It should cost less than an hour of work, if the provider bills CPU by use. Agents spend most of their time waiting for a model, so this question sets the bill.
Follow up: "Do you bill CPU on what the sandbox uses, or on what it was allocated, for every second it runs?" Then ask about minimums per sandbox and plan fees. Check it: price one of your real runs at the provider's published rates. Count the running time, the CPU-seconds used and the memory held. Runtime: a sandbox with 2 vCPU and 4 GiB costs $0.03125 an hour while it waits, and $0.08 an hour with both CPUs busy. There is no plan fee. On the pricing guide's example job, it costs 42% to 88% less than fourteen other providers (what 1,000 agent runs cost).
4. Does pause keep memory, and how fast is the wake?
A pause that keeps only files forces the next turn to restart every running process. A pause that keeps memory hides the gap.
Follow up: "Does a paused sandbox wake by itself on the next command, and how long is paused state kept?" Check it: start a process that holds a value in memory. Pause the sandbox, wake it, and ask the process for the value. A clean boot that looks fine does not prove the memory was kept. Runtime: a pause keeps files, memory and processes. Any command or file call also wakes it. Paid accounts keep paused state while they have credit (pause, don't delete).
5. What can a sandbox reach on the network?
By default, a sandbox reaches nothing private. You should be able to limit the rest to a list.
Follow up: "Where is the rule enforced? Can root inside the sandbox change it?" A rule enforced inside the machine is a rule the agent can remove. Check it: from inside the sandbox, request the cloud metadata address, a private address, and a host that your list should block. Runtime: a proxy on the host carries every connection. Private and internal addresses are refused for every sandbox. An allow list limits the rest, and root inside the sandbox cannot change it (why the sandbox shouldn't reach the whole internet).
6. Can code use an API key without being able to read it?
It should. An agent that can read a key can be talked into printing it.
Follow up: "Is the secret injected as an environment variable, or added to
requests outside the sandbox?" An environment variable is convenient storage.
Only the second option keeps the value out of the agent's reach.
Check it: store a secret. Then run env in the sandbox and search for it.
Runtime: a secret appears inside the sandbox as a placeholder. The host's
proxy adds the real value only to HTTPS requests to the hosts you name
(secrets sandboxes never see).
7. How long can one sandbox run?
A sandbox should run as long as the work takes. A hard cap on lifetime forces you to checkpoint and restart long jobs on the provider's schedule, not yours.
Follow up: "What happens at the limit: a stop with warning, or a kill? Is
the limit on running time or on wall-clock time, pauses included?"
Check it: read the create call's options for a maximum. Then run a job for
longer than an hour.
Runtime: there is no time limit unless you set one (timeoutSeconds). A
sandbox runs while it works, and it pauses itself after 300 seconds with
nothing happening.
8. How large can one sandbox be, and how many run at once?
Large enough for your heaviest task. Enough of them for your busiest hour.
Follow up: "Is concurrency a hard cap, or do extra creates wait? Can a sandbox have a GPU?" Check it: start as many sandboxes as your peak needs at the same time. Count how many are running one minute later. Runtime: up to 16 vCPUs and 64 GiB of memory per sandbox, and 100 sandboxes at the same time on a paid account. Creates beyond that limit wait for capacity instead of failing. Runtime runs on CPUs only. It has no GPU sandboxes, so run GPU work on another service.
9. Can you copy a running sandbox?
You should be able to, with memory included. Best-of-N attempts, evals and tree search all start from one prepared machine.
Follow up: "Does a fork copy running processes, or only the disk? How many
copies per call?" Check it: start a server in a sandbox and fork it. Send a
request to the copy without starting anything. Runtime: one fork call
returns up to 100 running copies, each with files, memory and
processes.
10. Can you bring your own image?
You should be able to start from any Dockerfile or registry image. The agent then does not spend its first minute installing your stack.
Follow up: "Are images versioned, and can a sandbox switch images without losing its files?" Check it: build your real project's image and time a create from it. Runtime: custom images come from a package list, from any public or private registry image, or from a Dockerfile. Images are versioned and tagged, and they can have start and ready commands (images).
11. What is promised when the service is down?
Get a number in writing, measured from outside the provider. Get a credit back when the number is missed.
Follow up: "Is the credit automatic, or do I have to file a claim? Where is uptime published?" Check it: read the terms. Then find the status page and its history. Runtime: paid accounts have a promised API uptime of 99% each month. Uptime is checked from outside every two minutes. If a month falls below the promise, Runtime returns 10% of that month's charges as credit. No claim is needed (uptime promise, status).
12. How hard is it to leave, or to arrive?
Moving in or out should take a day of work. Plain Linux, plain HTTP and standard images keep it that way.
Follow up: "Is there anything in my code that only your platform can run?" Check it: list every provider call in your code. Count the ones with no equivalent elsewhere. Runtime: sandboxes are ordinary Linux machines, and images are Dockerfiles. Code written for E2B, Daytona, Vercel Sandbox or Blaxel runs after one import change (switch to Runtime).
How do you check five of these in a minute?
Run one script against each candidate. It times the first command from your side of the network. Then it asks the sandbox for its kernel, its size and what it can reach:
TypeScriptimport { Sandbox } from "withruntime";const started = performance.now();await using sbx = await Sandbox.create();await sbx.exec("true");console.log(`first command: ${Math.round(performance.now() - started)} ms, network included`);const checks: Record<string, string> = { kernel: "uname -r", "vCPUs and memory": "nproc && free -m | awk '/Mem:/ {print $2 \" MiB\"}'", "who am I": "id -un && sudo -n true && echo 'sudo works'", "cloud metadata": "curl -sf -m 5 http://169.254.169.254/ || echo refused", "private network": "curl -sf -m 5 http://10.0.0.1/ || echo refused",};for (const [name, command] of Object.entries(checks)) { const result = await sbx.exec(command, { timeoutMs: 15_000 }); console.log(`${name}: ${result.stdout.trim().replaceAll("\n", ", ")}`);}Pythonimport timefrom withruntime import Sandboxstarted = time.perf_counter()with Sandbox.create() as sbx: sbx.exec("true") print(f"first command: {(time.perf_counter() - started) * 1000:.0f} ms, network included") checks = { "kernel": "uname -r", "vCPUs and memory": "nproc && free -m | awk '/Mem:/ {print $2 \" MiB\"}'", "who am I": "id -un && sudo -n true && echo 'sudo works'", "cloud metadata": "curl -sf -m 5 http://169.254.169.254/ || echo refused", "private network": "curl -sf -m 5 http://10.0.0.1/ || echo refused", } for name, command in checks.items(): result = sbx.exec(command, timeout_ms=15_000) print(f"{name}: {', '.join(result.stdout.strip().splitlines())}")Run it ten times and keep the slowest first command, not the fastest. Port the five checks to each provider's SDK. The shell commands stay the same.
What does the scorecard look like?
One line per question. The middle column shows the answer to look for:
| # | Question | The answer to look for | Runtime |
|---|---|---|---|
| 1 | Own kernel per sandbox? | Yes, a microVM | Firecracker microVM |
| 2 | Speed, and how measured? | Median and p95 to a first command | Timed each minute from outside, samples public |
| 3 | Cost of an idle hour? | Far below a busy hour | $0.03125 idle, $0.08 busy |
| 4 | Pause keeps memory? | Yes, with a wake on demand | Yes, and any call wakes it |
| 5 | Network rules where? | Outside the sandbox | On the host; root cannot change them |
| 6 | Keys the code cannot read? | Injected outside the sandbox | Placeholder, value added by the proxy |
| 7 | Longest run? | No cap you did not set | None unless you set one |
| 8 | Size and concurrency? | Your peak, with room | 16 vCPUs, 100 at the same time |
| 9 | Copy a running sandbox? | Memory and processes included | Up to 100 copies per call |
| 10 | Your own image? | Any Dockerfile or registry image | Yes, versioned |
| 11 | Uptime in writing? | A number and automatic credit | 99%, 10% back |
| 12 | Easy to leave? | Plain Linux, standard images | One import to switch from E2B and others |
The guide to choosing an agent sandbox turns these into a full evaluation plan, with the failure cases worth testing.
In short
- Ask for medians and p95s, timed to a first command, and check them from where your agent runs.
- Ask whether CPU is billed on use; an agent mostly waits, and waiting should be cheap.
- Ask where network rules and secrets are enforced; inside the sandbox is not enough.
- Ask what pause keeps and what a fork copies; memory is the difference.
- Get uptime and switching terms in writing before you depend on either.
Run it on Runtime
A Runtime sandbox is a Firecracker microVM. It costs $0.03125 an hour while it waits, at 2 vCPU and 4 GiB. It pauses itself with its memory kept. Start with 100 free hours of a 1 GB machine every month, with no card needed: sign in and run the script above, or read get started and the pricing.