On Runtime every Harbor trial runs in its own microVM, built from the task's own Dockerfile, and you pay for the CPU it uses. Harbor is the framework behind Terminal-Bench 2.0: it runs an agent on a task inside an environment, then runs the task's verifier and records a score. Docker on one machine is its default environment, so a run's parallelism is limited by that machine's cores and disk. With Runtime (withruntime.com) each trial is a machine with its own kernel, and a paid account runs 100 at the same time. Harbor 0.25.0 was the current PyPI release on 25 October 2026.
Run a task, then a benchmark
Terminalpip install "withruntime[harbor]"harbor run -t hello-world/hello-world -a oracle -e withruntime.harbor:RuntimeEnvironmentharbor run -d terminal-bench@2.0 -a oracle -e withruntime.harbor:RuntimeEnvironment --n-concurrent 8Harbor loads any environment by its import path (module.path:ClassName). This
works with the Harbor you already have. The oracle agent runs each task's
reference solution. It checks the environment without paying for a model. Once
it passes, swap in your agent and model with -a and -m. The key comes from
RUNTIME_API_KEY or the one npx withruntime login saved.
What happens to a task's Dockerfile?
Runtime builds it once as a image and reuses it. The image is
named harbor-<task> and tagged with a hash of the environment. A second run of
the same task starts from it immediately. A changed Dockerfile builds a new
image. --force-build builds it again. To skip the build, pass a ready image
with --ek image=<name:tag>.
| Task setting | In the trial's sandbox |
|---|---|
Dockerfile or docker_image |
A Runtime image, built once and kept |
cpus, memory_mb, storage_mb |
The sandbox's vCPU, memory and disk |
workdir or the Dockerfile's WORKDIR |
Where every command starts |
| Commands as root | Run through the sandbox user's passwordless sudo |
| Network policies | no-network and host allow-lists, changeable mid-trial |
| A long trial | The sandbox runs as long as the trial does |
How many trials at once?
Trials run as many at a time as --n-concurrent asks for, up to your account's
limit. An account without credit runs four sandboxes at the same time.
With credit, it runs 100. Builds run one at a time until the
account holds credit, then four. Build a benchmark's images before a wide run.
Harbor stops each sandbox when its trial ends. runtime images ls lists the
images the run kept.
What does a Terminal-Bench pass cost?
Runtime bills the CPU a trial uses, not the CPUs it holds, plus memory while it runs. A trial that keeps 2 vCPUs half busy for 10 minutes with 4 GiB costs about $0.01, so 100 such trials cost about $0.92. Agents spend most of a trial waiting on the model. While they wait, the trial bills memory only (pricing).
When a task's image lacks sudo
Harbor runs commands as root. A Runtime sandbox runs them as uid 1000 through
sudo. Runtime adds sudo to every image it builds, so this matters only for
a ready image passed with --ek image=. The environment checks at the start of
the trial. If sudo is missing, it stops the trial and prints the line to add
to the Dockerfile:
dockerfileRUN apt-get update && apt-get install -y sudo && echo '#1000 ALL=(ALL:ALL) NOPASSWD: ALL' > /etc/sudoers.d/runtimeDocker Compose tasks with sidecar services are not supported yet. A task with one image runs as is.
Related
- Runtime for Harbor and Terminal-Bench, every option
- SWE-bench-style evals in parallel sandboxes
- What is SWE-bench?
- Inspect AI evals on Runtime
- RL environments in sandboxes
Sources
- harbor on PyPI, version 0.25.0,
harbor run --helpread 25 October 2026 - Terminal-Bench, read 25 October 2026
On 2 October 2026, with a key on production, harbor run of the hello-world
task with the oracle agent on this environment scored 1.0.
Facts on this page were checked on 25 October 2026.