Sandbox creates no longer wait on an unresponsive host
When the host chosen for a new sandbox stops responding before it starts creating the sandbox, POST /v1/sandboxes places the sandbox on another host, usually within 2 seconds, instead of waiting for that host for up to several minutes. If that host had already started creating the sandbox, the request waits for it, so that each Idempotency-Key still creates at most one sandbox. If the cell cannot confirm where the sandbox may be created, the request returns 503; retry with the same Idempotency-Key.
Stopping a sandbox always completes
A stop that could not be recorded used to leave the sandbox in stopping permanently, and every request on it returned 500 operation_failed. The stop is now retried automatically every 5 minutes until the sandbox is stopped; meanwhile requests on it still return 500 operation_failed immediately, its files are kept, and nothing is charged. Sandboxes already stuck in stopping are stopped by the same retry.
Previews wait for a server that is still starting
Within 60 seconds after a sandbox is created or wakes, a preview or TCP port request to a port that is not listening yet now waits up to 10 seconds for the port to open, and is forwarded as soon as it does. Previously it returned 502 with x-runtime-preview-error: port_closed immediately, so a health check sent right after create failed until the server started. After 10 seconds the request still returns 502 port_closed. Requests to a port that is already listening are not delayed.
Forks of a sandbox in use no longer fail with sandbox_not_paused
If a command or other request woke the source between the fork's pause and the start of its copy, POST /v1/sandboxes/{id}:fork returned 409 sandbox_not_paused. Requests that wake a sandbox automatically now wait until the copy has started, up to one minute, and then proceed as before.
Postgres read replicas follow the standby
On a Postgres database with spec.standby set to true, each read replica now streams its changes from the standby instead of the primary, so no replica is ever ahead of the standby. When the standby is promoted after a host loss, the replicas keep running at the same host names and continue to receive changes from the new primary without being recreated. A database whose host is lost while its state is paused or stopped now also fails over to its standby, then pauses or stops, and its read replicas are removed; previously its standby and read replicas kept running, and were billed, until the database was deleted.
Exec returns all of its output
POST /v1/sandboxes/{id}:exec now returns every byte a command writes to stdout and stderr, up to 256 MiB of each stream in a JSON response and with no limit in a streamed one. Previously the JSON response kept at most 64 KiB of each stream for a command of 60 seconds or less, and 1 MiB for a longer one. The response adds stdoutBytes and stderrBytes, the bytes each stream wrote, and truncated. The new maxOutputBytes parameter limits the response to that many bytes of each stream; a stream it cuts has its truncation flag set to true. The TypeScript, Python, Go, Ruby and Java SDKs return the same output and take the same limit (maxOutputBytes, max_output_bytes, MaxOutputBytes). withruntime sandboxes exec takes --max-output-bytes. runtime_sandboxes_exec over MCP returns stdoutBytes, stderrBytes and truncated. Sandboxes started before this change keep the 64 KiB limit until they are stopped and started.
Forks succeed when the source is woken during the fork
POST /v1/sandboxes/{id}:fork pauses the source while it copies it. If another request woke the source during that time (for example, a command run on it), the fork returned 409 not_paused, said the source stayed paused, and left its copies running. The fork now returns its copies, and the source keeps running.
Apps and Functions accept a start command
spec.command on /v2/apps and /v2/functions sets the process each copy runs, as an argument list, with PORT set to spec.port. When the process exits, it is restarted automatically, and its last 8 KiB of output is kept in the copy at ~/.runtime/process-<version>.status. A version that fails its health check includes that output in status.failure. When command is omitted, the image's start command runs, as before.
A failed fork deletes the copies it started
When POST /v1/sandboxes/{id}:fork fails after some copies have started, the copies that started are now deleted and no longer appear in GET /v1/sandboxes. The error's details.startedSandboxIds still names them, and details.copiesDeleted is true. When the error is a 5xx, a failure on Runtime's side (for example, no capacity for a later copy, a copy that does not reach running in time, or the source cannot be woken), none of their running time is billed and details.copiesBilled is false. When it is a 4xx, caused by the request or the account (a limit or quota, or a copy or the source deleted or changed during the fork), the time they ran is billed. A copy whose create completed after the fork had already failed is deleted too. A copy that could not be deleted at once is named outside details.deletedSandboxIds, details.copiesDeleted is false, and it is deleted within 15 minutes. If the server handling the fork stops before it returns, the fork is marked failed with a 500 once it has run longer than any fork can (15 minutes, plus about 20 seconds for each GiB of the source's memory), and its copies are deleted unbilled within 15 minutes after that. A fork that does not answer within the request deadline is a failure on Runtime's side, and its copies are not billed; if your client closes the connection before the fork answers, the copies are deleted and billed for the time they ran. Previously these copies kept running, and were billed, until you stopped them.
Reads are served while a cell's storage is full
When a cell's drives are short of space, GET requests now succeed instead of returning 503 api_unavailable, including a project's first request after a front restarts. Requests that change state return 503 capacity_unavailable with a Retry-After header; retry them with the same Idempotency-Key.
Fixed: file shares and streams for sandboxes created with POST /v2/sandboxes
In regions served by the new API, a sandbox created with spec.fileShares could fail with "mounts file shares, which this cell's machines do not", and GET /v2/sandboxes/{id}/streams returned 404 for any sandbox created through /v2. Volumes and file shares are now mounted as they are for POST /v1/sandboxes, and exec and pty streams reach the sandbox. One file share can be mounted read-write in several sandboxes on different hosts at once.
Postgres synchronous standby with automatic failover
Set spec.standby to true on a Postgres database to keep a synchronous standby on another rack. A commit is acknowledged only once it is on the standby too. If the database's host is lost, the standby is promoted in its place without a request: the database keeps its ID, host name, port and password, holds every acknowledged commit, and accepts writes again within seconds; a new standby is then made on another rack, and until it runs, commits are acknowledged without a copy on another rack. status.standby.phase is running once every commit waits for the standby. The standby is billed as a database of the same size that never pauses.
Password sign-in is faster
POST /v2/auth/sessions with an email and password now checks the password while the sign-in attempt is recorded, instead of after it, and reads the clock alongside the account lookup. On cell 1 the median sign-in dropped from about 47 ms to about 35 ms. Responses, throttling, and the attempt count are unchanged: an attempt is always recorded before the response is sent, and a throttled email is refused before its password is checked.
getOrCreate always returns the one sandbox that holds a name
Concurrent POST /v1/sandboxes requests with getOrCreate: true and the same name now return the same sandbox, including when that sandbox is paused, stopped with its disk kept, or being moved between regions. Previously, under load, a second sandbox could be created under the same name, and later calls by that name could return either one. A create with a name that cannot be placed at that moment, for example while capacity is short, returns 503 busy with Retry-After and creates nothing; retry it. In the Vercel-compatible TypeScript SDK, Sandbox.getOrCreate now creates a missing sandbox with getOrCreate, and so does get_or_create_sandbox in the Vercel-compatible Python SDK. Concurrent calls for the same name return the same sandbox instead of failing with 409 name_taken. resume: false still leaves a found sandbox paused.
Sandbox list pagination returns every page
GET /v1/sandboxes no longer returns 400 invalid_request (cursor) on the second or later page for organizations whose sandboxes run in more than one cell. Each sandbox is returned exactly once, oldest first, and nextCursor stays an opaque string to pass back unchanged. withruntime sandboxes ls and every SDK's list iterator page through the full list.
A terminal that cannot open names the missing file
When a sandbox's image cannot open a terminal for a pty process or stream (for example, an image without a devpts at /dev/pts), the error now names the file that failed, such as /dev/ptmx: No such file or directory, instead of the system error alone.
idempotency_key_reused errors include details on every route
A 422 idempotency_key_reused now always returns details.firstUsedAt and details.operation, naming when the key was first used and for which operation. Some routes previously omitted details.
A fork retried with its Idempotency-Key after a 503 now creates its copies
When POST /v1/sandboxes/{id}:fork returns 503 because its copies could not be opened, nothing is created, and sending the same request again with the same Idempotency-Key now forks the sandbox, instead of returning 500 operation_failed ("its copies were withdrawn") on every retry. A fork that succeeded returns its original response to a retry with its key, including after its copies are deleted, and never forks again.
Pull request checks into non-default branches use at most half of an account's CI/CD machines
Checks of pull requests whose base is a branch other than the repository's production branch now hold at most half of the machines the account can run at once, combined. The other half stays available for checks of the production branch and of pull requests into it, so those start immediately even when many pull requests into another branch are being checked. A check waiting on this limit shows the reason in its jobs: how many machines checks into other branches hold and the most they may hold.
Requests that a secret's rules do not allow now return 403 from Runtime
A request that carries an egress secret's placeholder in a credential header, to a method and path that none of the secret's rules allows, is no longer sent to the destination with the placeholder. It returns 403 Forbidden with X-Runtime-Egress: secret-rule-not-matched, and the body names the secret, the method and the path. Previously the destination received the placeholder and typically returned 401 with no explanation. The sandbox.secret_not_applied event with reason rule_not_matched is still sent.
A process start whose program is missing returns 422 program_not_found
When the program to run, or the interpreter it names (a script's #! line or a binary's loader), does not exist in the sandbox (for example, an image without /usr/bin/env, through which a command named without a path runs), POST /v1/sandboxes/{id}/processes now returns 422 with the code program_not_found and the missing path in details.path, instead of 502 guest_failed. A cwd that does not exist returns 422 cwd_not_found. Nothing runs, and a retry returns the same response.
Interpreter 503 busy errors include their cause
When a code interpreter request returns 503 with code busy, error.details.cause now describes why the request could not be confirmed, for example that the interpreter process exited before it answered. Use it to tell a transient condition that a retry resolves from one that persists.
Commands start faster in new sandboxes
POST /v1/sandboxes/{id}:exec and process starts no longer wait for the API to look up the organization's egress secret placeholders. The sandbox supplies them from its host before every command, so each command still receives them. A command in a sandbox seconds old previously waited about 1.4 s for that lookup.
Egress secrets reach every new sandbox
A sandbox whose start carried no organization in its network rules trusted the egress CA but received none of its organization's secrets: their variables were missing from commands, and requests to the secrets' hosts were sent without credentials. Such a sandbox now gets its secrets like any other.
SDK methods for workflow runs, steps and events
The TypeScript, Python, Go, Java and Ruby SDKs add workflows.start and workflows.runs for a workflow's runs, jobs.start for a job's run, jobRuns.getStep and jobRuns.recordStep for a run's durable steps, and jobEvents.send for events. In TypeScript, withruntime/workflows exports them, and runSteps() authenticates with the run's RUNTIME_RUN_TOKEN. PUT /v2/job-runs/{id}/steps/{name} with none of output, error, sleepUntil and event now records the step with a null result instead of returning 400, so a step that returns nothing is kept and does not run again.
Run tokens for job and workflow runs
Each run gets a token in its RUNTIME_RUN_TOKEN environment variable. Send it as Authorization: Bearer to GET and PUT /v2/job-runs/{id}/steps/{name} for the run's own steps, so a workflow no longer needs an API key in its env. A run token works only for its own run's steps.
Idempotency keys for job and workflow runs expire after 24 hours
An Idempotency-Key sent to POST /v2/workflows/{id}/runs or POST /v2/jobs/{id}/runs returns the run it started for 24 hours; after that, the same key starts a new run.
Durable steps and events for workflows in the v2 API
POST /v2/workflows/{id}/runs starts a workflow run with an input (and POST /v2/jobs/{id}/runs a job's run), once per Idempotency-Key. A run's code records each step's result with PUT /v2/job-runs/{id}/steps/{name} and reads it back with GET, so a continued run replays it; a step can sleep until a time or wait for an event, sent with POST /v2/job-events, and its run continues when the time or event arrives.
Sandboxes: mount file shares
POST /v2/sandboxes accepts spec.fileShares, a list of up to 16 {name, path, readOnly} entries; each file share of the project is mounted at its path before the sandbox's phase is running, and again each time it resumes or starts. The API key needs file-shares:read on each share, and file-shares:write on each one not mounted readOnly: a share the key cannot read, or that does not exist, returns 404; one mounted read-write by a key that may only read it returns 403. fileShares is set when the sandbox is created and cannot be changed. A sandbox with file shares cannot be forked, and a sandbox started from a snapshot mounts none. Writes to a share mounted readOnly return EROFS. Each path must be absolute, outside /proc, /sys and /dev, and neither at nor inside another share's path; otherwise the request returns 400.
Alerts: count rules
PUT /v2/alerts/rules/{id} now accepts rules with "query": {"kind": "count", "selector": "...", "window_ms": ...}. A count rule counts the log lines that match a LogQL stream selector and line filters (for example {service_name="api"} |= "error") within the window, and fires when the count crosses its condition. Each line is counted once, as soon as it is accepted, so a count rule's webhook is sent about 10 ms after the line that fires it.
Sandboxes: call the API without a key
A sandbox created with spec.permissions (for example ["sandboxes:list", "sandboxes:create"]) can call https://api.withruntime.com without an API key. Its host adds a short-lived token for that sandbox to each request and replaces any Authorization header the sandbox sends, so no credential is ever stored in the sandbox. Each action must be one the creating key allows; otherwise the create returns 403. A fork gets its own identity with its source's permissions. permissions cannot be changed after a sandbox is created.
Alerts: delete a rule
DELETE /v2/alerts/rules/{id} deletes a rule and returns {"id": ..., "deleted": true}, or 404 when no such rule exists. A rule can be deleted while its alerts are firing: evaluation stops within a second, and each firing alert is sent to the rule's webhooks once more with status resolved. The limit of 100 rules per organization now also holds when rules are created concurrently; a create past the limit returns 400.
Logs: faster live tail
GET /v2/logs/v1/logs/tail now delivers a line about 15 ms after it is accepted, down from about 100 ms. A tail now waits for new lines instead of checking for them on an interval.
CI/CD setup can be a list of layers
setup: in runtime.yml now also accepts a list of up to 16 commands, run in order. A push to the production branch keeps each finished layer as a snapshot, keyed to that layer, the layers before it, the machine size and the cache files. A later setup starts from the deepest layer that is ready and runs only the layers after it, so a change to the last layer does not run the earlier ones again, and a setup that is stopped or fails partway keeps the layers it finished. Pull request checks start from these snapshots but never write them. A setup: of one command works as before.
Logs: live tail
GET /v2/logs/v1/logs/tail?query= now streams matching log lines as Server-Sent Events, resumable with Last-Event-ID. A tail stays open for as long as the API key that opened it is valid. It ends with an end event with code tail_unauthorized once the key is revoked or no longer grants logs:read. If the key cannot be verified for two minutes, the tail ends with code tail_unverified; open it again with the ID of its last event. The Loki WebSocket tail (/v2/logs/loki/api/v1/tail) still returns 501.
CI/CD: checks a release waits on get machines first
When an account's checks are short of machines, a pull request from the repository's own branches into the production branch gets them first, then a push to the production branch, each oldest first, then every other check. While one of these waits, the account's lower checks hand it machines as their running tasks end. A pull request from a fork is ordered with every other check. To skip pull requests into a branch whose commits are tested another way, such as a staging branch, list the branches to check in the production branch's config: pull_requests: { branches: [main] }.