# AI agents that run for hours: time limits, checkpoints and recovery

An agent that runs for hours needs a time limit on every machine, a checkpoint after every step, and a supervisor that restarts it.

**On Runtime (withruntime.com), an agent started with `spawn` belongs to its
sandbox, not to your connection. Your server can deploy or crash during the
run.** Short agent tasks fail rarely enough to retry by hand. A six-hour run
meets every kind of failure: a deploy restarts your worker, the model API
returns 529 for ten minutes, the agent loops on one test, and the context
window fills. This post shows how to keep a run of hours going through these
failures. It covers a crash you can reproduce, the supervisor that recovers
from it, and the cost of the safety net.

## What goes wrong in a run that lasts hours?

Every failure in a short run can also happen in a long run, and a long run
has more chances for it. Long runs also have two failures that short runs do
not have: a full context window and a time limit.

| Failure                    | How you notice                      | What you do                                          | What is lost                    |
| -------------------------- | ----------------------------------- | ---------------------------------------------------- | ------------------------------- |
| Your worker restarts       | Nothing; the agent keeps running    | Reconnect by the ids you stored                      | Nothing                         |
| Network drop to the API    | A call errors                       | Retry; the agent never noticed                       | Nothing                         |
| Model API overloaded       | The agent's own retries             | Back off inside the agent, minutes not seconds       | Time                            |
| Agent stuck in a loop      | Checkpoint unchanged for 15 minutes | Kill it, restart from the checkpoint with a note     | Work since the checkpoint       |
| Agent crashes              | Process exited with an error        | Restart from the checkpoint                          | Work since the checkpoint       |
| Context window full        | The agent's own token count         | Summarize into the checkpoint, start a fresh context | Nothing, if the summary is good |
| Sandbox time limit reached | The sandbox is no longer running    | New sandbox from the checkpoint, if budget is left   | Work since the checkpoint       |
| The sandbox fails          | The sandbox does not answer         | New sandbox from the checkpoint                      | Work since the checkpoint       |

Read the last column. Every failure costs at most the work since the last
checkpoint, as long as two things hold: the checkpoint is written after every
step, and a copy of it is stored outside the sandbox. Everything else in this post
follows from that.

## Why does a time limit help a long run?

A run with no ceiling cannot fail at a low cost. An agent stuck in a loop at
2 a.m. keeps running, and keeps costing money, until someone wakes up. Set
`timeoutSeconds` on every sandbox, and the worst case has a known cost. When
the limit is reached, the sandbox stops, whatever the agent was doing. Treat
it as a crash: the supervisor starts a new sandbox from the last checkpoint
if the run's overall budget allows.

These are two different limits, and they are easy to confuse:

- **The sandbox time limit** is `timeoutSeconds` on each sandbox: how long one
  sandbox may run before it is replaced.
- **The run budget** lives in your supervisor: total hours, total restarts and
  total spend for the task, across every sandbox it uses.

A [daily spending limit](/how-to/set-a-daily-spending-limit) on the account
sits above both. It caps the total cost of a bad day across every run.

## What goes in a checkpoint?

A checkpoint holds everything the agent needs to continue in a new process
with no memory of the earlier run: the plan, the index of the next step, a
short summary of what it has learned, and the key of every outside action it
has taken. Code changes go to a git branch, committed and pushed at each step.
The checkpoint file stays small, a few kilobytes, so copying it outside the
sandbox every minute costs little.

Two rules make checkpoints safe:

1. **Write it atomically.** Write to a temporary file, then rename it over the
   old one. If a crash happens during the write, the previous checkpoint
   survives. A half-written new checkpoint is never used.
2. **Make outside actions idempotent.** A crash can happen after an action and
   before its record is saved. Give each step's outside action a key that stays
   fixed for that step. The retry then finds the action already done.

People skip the second rule most often. The program below runs as written and
shows why it matters. The agent opens a pull request in step 4 and dies before
saving. Without the key, the retry would open a second pull request:

```ts
import { existsSync, mkdtempSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";

const STATE = join(mkdtempSync(join(tmpdir(), "agent-")), "checkpoint.json");
const STEPS = [
  "read the code",
  "write the migration",
  "run the tests",
  "open the pull request",
  "post a summary",
];
type Checkpoint = { next: number; notes: string[] };

// The outside world: an API that takes idempotency keys, or a check such as
// "does this branch already have a pull request?"
const opened = new Set<string>();
function act(step: number, key: string): string {
  if (step < 3) return "done";
  if (opened.has(key)) return `skipped, already done under ${key}`;
  opened.add(key);
  return `done under ${key}`;
}

function load(): Checkpoint {
  return existsSync(STATE) ? JSON.parse(readFileSync(STATE, "utf8")) : { next: 0, notes: [] };
}
function save(cp: Checkpoint) {
  writeFileSync(`${STATE}.tmp`, JSON.stringify(cp));
  renameSync(`${STATE}.tmp`, STATE); // atomic: a crash mid-write keeps the old checkpoint
}

function run(attempt: number, dieAfter?: number) {
  const cp = load();
  console.log(`attempt ${attempt}: resuming at step ${cp.next + 1}`);
  for (let step = cp.next; step < STEPS.length; step++) {
    const result = act(step, `run-981-step-${step + 1}`); // the key is fixed per step, not per attempt
    console.log(`  step ${step + 1}, ${STEPS[step]}: ${result}`);
    if (step === dieAfter) return console.log("  the process dies before saving the checkpoint");
    cp.next = step + 1;
    cp.notes.push(STEPS[step]!);
    save(cp);
  }
  console.log(`  finished; ${opened.size} outside actions taken in total`);
}

run(1, 3);
run(2);
```

```python
import json
import os
import tempfile

STATE = os.path.join(tempfile.mkdtemp(prefix="agent-"), "checkpoint.json")
STEPS = ["read the code", "write the migration", "run the tests", "open the pull request", "post a summary"]

# The outside world: an API that takes idempotency keys, or a check such as
# "does this branch already have a pull request?"
opened = set()


def act(step, key):
    if step < 3:
        return "done"
    if key in opened:
        return f"skipped, already done under {key}"
    opened.add(key)
    return f"done under {key}"


def load():
    if os.path.exists(STATE):
        with open(STATE) as f:
            return json.load(f)
    return {"next": 0, "notes": []}


def save(cp):
    with open(STATE + ".tmp", "w") as f:
        json.dump(cp, f)
    os.replace(STATE + ".tmp", STATE)  # atomic: a crash mid-write keeps the old checkpoint


def run(attempt, die_after=None):
    cp = load()
    print(f"attempt {attempt}: resuming at step {cp['next'] + 1}")
    for step in range(cp["next"], len(STEPS)):
        result = act(step, f"run-981-step-{step + 1}")  # the key is fixed per step, not per attempt
        print(f"  step {step + 1}, {STEPS[step]}: {result}")
        if step == die_after:
            print("  the process dies before saving the checkpoint")
            return
        cp["next"] = step + 1
        cp["notes"].append(STEPS[step])
        save(cp)
    print(f"  finished; {len(opened)} outside actions taken in total")


run(1, 3)
run(2)
```

It prints:

```
attempt 1: resuming at step 1
  step 1, read the code: done
  step 2, write the migration: done
  step 3, run the tests: done
  step 4, open the pull request: done under run-981-step-4
  the process dies before saving the checkpoint
attempt 2: resuming at step 4
  step 4, open the pull request: skipped, already done under run-981-step-4
  step 5, post a summary: done under run-981-step-5
  finished; 2 outside actions taken in total
```

The checkpoint still listed step 4 as to do, which was accurate. The key
stopped the duplicate. GitHub has no idempotency keys for pull requests, so
for pull requests the check is a lookup instead: does this branch already have
an open pull request? Stripe and many other APIs take an `Idempotency-Key`
header directly.

A checkpoint also handles a full context window. When the agent's history
nears its limit, the agent writes a summary into the checkpoint. Then it exits
with a code that tells the supervisor to restart it. The new process reads the
plan and the summary, not four hours of tool output, and it usually works
better.

## What does the supervisor look like?

Your backend runs a loop that copies the checkpoint out of the sandbox every
minute. It watches for progress. It replaces the sandbox when the agent exits,
stalls, or runs out of time:

```ts check
import { Sandbox } from "withruntime";

const RUN = "run-981";
const REPO = "https://github.com/acme/app.git";
const AGENT = "python3 /workspace/agent/main.py --resume /workspace/checkpoint.json";
const MACHINE_SECONDS = 3600; // each sandbox's time limit: an hour
const MAX_RESTARTS = 5; // the run's budget
const STALL_MS = 15 * 60_000;

let saved = ""; // load the last copy from your database here

async function start(checkpoint: string) {
  const sbx = await Sandbox.create({
    vcpu: 2,
    memoryMiB: 4096,
    timeoutSeconds: MACHINE_SECONDS,
    labels: { run: RUN },
  });
  const branch = `agent/${RUN}`;
  await sbx.exec(
    `git clone ${REPO} app && cd app && (git switch ${branch} || git switch -c ${branch})`,
    {
      cwd: "/workspace",
      check: true,
    },
  );
  await sbx.files.upload("./agent", "/workspace/agent");
  if (checkpoint) await sbx.files.write("/workspace/checkpoint.json", checkpoint);
  const agent = await sbx.spawn(AGENT, { cwd: "/workspace/app" });
  return { sbx, agentId: agent.id }; // store both, so a restarted supervisor reconnects
}

let { sbx, agentId } = await start(saved);
let lastProgress = Date.now();
for (let restarts = 0; ;) {
  await new Promise((resolve) => setTimeout(resolve, 60_000));
  try {
    const checkpoint = await sbx.files.readText("/workspace/checkpoint.json");
    if (checkpoint !== saved) {
      saved = checkpoint; // and write it to your database: the copy off the machine
      lastProgress = Date.now();
    }
    const agent = await sbx.processes.get(agentId);
    if (agent.state === "exited" && agent.exitCode === 0) break; // finished
    if (agent.state !== "running") throw new Error(`agent ${agent.state}, exit ${agent.exitCode}`);
    if (Date.now() - lastProgress > STALL_MS) throw new Error("no progress for 15 minutes");
  } catch (error) {
    if (++restarts > MAX_RESTARTS) throw error; // out of budget: a person looks
    console.log(`restart ${restarts} from the last checkpoint: ${error}`);
    await sbx.pause().catch(() => {}); // paused, its files stay for a look later
    ({ sbx, agentId } = await start(saved));
    lastProgress = Date.now();
  }
}
await sbx.stop();
```

```python check
import time

from withruntime import Sandbox

RUN = "run-981"
REPO = "https://github.com/acme/app.git"
AGENT = "python3 /workspace/agent/main.py --resume /workspace/checkpoint.json"
MACHINE_SECONDS = 3600  # each sandbox's time limit: an hour
MAX_RESTARTS = 5  # the run's budget
STALL_SECONDS = 15 * 60

saved = ""  # load the last copy from your database here


def start(checkpoint):
    sbx = Sandbox.create(vcpu=2, memory_mib=4096, timeout_seconds=MACHINE_SECONDS,
                         labels={"run": RUN})
    branch = f"agent/{RUN}"
    sbx.exec(f"git clone {REPO} app && cd app && (git switch {branch} || git switch -c {branch})",
             cwd="/workspace", check=True)
    sbx.files.upload("./agent", "/workspace/agent")
    if checkpoint:
        sbx.files.write("/workspace/checkpoint.json", checkpoint)
    agent = sbx.spawn(AGENT, cwd="/workspace/app")
    return sbx, agent.id  # store both, so a restarted supervisor reconnects


sbx, agent_id = start(saved)
last_progress = time.monotonic()
restarts = 0
while True:
    time.sleep(60)
    try:
        checkpoint = sbx.files.read_text("/workspace/checkpoint.json")
        if checkpoint != saved:
            saved = checkpoint  # and write it to your database: the copy off the machine
            last_progress = time.monotonic()
        agent = sbx.process(agent_id)
        if agent.state == "exited" and agent.exit_code == 0:
            break  # finished
        if agent.state != "running":
            raise RuntimeError(f"agent {agent.state}, exit {agent.exit_code}")
        if time.monotonic() - last_progress > STALL_SECONDS:
            raise RuntimeError("no progress for 15 minutes")
    except Exception as error:
        restarts += 1
        if restarts > MAX_RESTARTS:
            raise  # out of budget: a person looks
        print(f"restart {restarts} from the last checkpoint: {error}")
        try:
            sbx.pause()  # paused, its files stay for a look later
        except Exception:
            pass
        sbx, agent_id = start(saved)
        last_progress = time.monotonic()
sbx.stop()
```

One `catch` covers every failure in the table, because from here they all
look the same: the sandbox stops answering, the agent stops running, or the
checkpoint stops changing. A paused sandbox keeps its files. After the run is
safe, you can open the sandbox that stalled and read its logs. The
[long-running agents guide](/use-cases/long-running-agents) covers
reattaching to the agent's output stream from another process.

## When should you snapshot instead?

Use a snapshot when the state that matters is not in files. Examples are a
database the agent filled, a dev server it configured, or a warm cache that
took an hour to build. A [snapshot](/how-to/snapshot-a-sandbox) keeps the
whole sandbox, including memory and running processes. Taking a snapshot
pauses the sandbox during the capture. Take the snapshot between steps, not
during a model call. For most coding agents, a git branch and a small
checkpoint file are enough, and they cost nothing to keep.

## What does the safety net cost?

The safety net costs little next to the run. Take a six-hour run on 2 vCPU and
4 GiB, where the agent's tools average 0.3 of a vCPU. Add two restarts, and
each restart spends three minutes cloning and reinstalling:

```
The run:      6 h × (0.3 vCPU × $0.025 + 4 GiB × $0.0075)       = $0.2250
Two restarts: 2 × 0.05 h × (2 vCPU × $0.025 + 4 GiB × $0.0075) = $0.0080
```

The restarts add about 4% to the
sandbox cost. The model's tokens for six hours cost far more than either
line. The checkpoint saves the hours of model work that a restart from zero
would repeat ([pricing](/pricing)).

## In short

- Every failure in a long run costs at most the work since the last checkpoint, if the checkpoint is written at each step and copied outside the sandbox.
- Put a time limit on every sandbox. Keep the run budget (hours, restarts and spend) in your supervisor.
- Write checkpoints atomically. Give every outside action a key that is fixed per step, so a retry never repeats it.
- A supervisor that copies the checkpoint every minute and restarts the sandbox on a failure, a stall, or a time limit handles all of these with one code path.
- Take a snapshot when the state lives in memory or in running processes. A git branch and a small file cover most coding agents.

## Run it on Runtime

Runtime costs 42% to 88% less than fourteen other sandbox providers for an
agent that mostly waits on a model ([compare costs](/how-to/compare-your-costs)). Runtime runs an agent started with `spawn` independently of your connection, lets you give every sandbox a time limit, and restores a whole sandbox from a snapshot. Start
with 100 free hours of a 1 GB machine every month, with no card needed: [sign in](/sign-in) or read
[get started](/docs/start).