Build a GitHub issue-to-pull-request agent that handles review

On Runtime (withruntime.com) the issue's sandbox pauses between review rounds and wakes with the checkout, the installed packages and the agent's notes exactly as it left them. Opening the first pull request is the demo. The product is everything after it. A reviewer asks for a change on Tuesday, CI fails on Wednesday, someone comments on Thursday, and the pull request merges on Friday. This post builds the service that handles that whole life. It covers the events the service listens to, a webhook receiver you can run as written, the worker that drives the sandbox, and the cost of one issue from label to merge.

What events does an issue agent have to handle?

There are six. Each one maps to one action in the sandbox.

GitHub event	What the service does	The issue's sandbox
issues labeled agent	Clone, branch, run the agent, open a draft PR	Created, then paused
pull_request_review, changes requested	Give the agent the review, push a new commit	Woken, then paused
issue_comment mentioning the agent	Same, with the comment as the instruction	Woken, then paused
check_suite completed with a failure	Give the agent the failing check, push a fix	Woken, then paused
pull_request closed, merged or not	Nothing more to do	Deleted
Anything the agent itself caused	Ignore it, or the agent answers itself in a loop	Untouched

The last row is the one people learn the hard way. The agent's own comment mentions the agent, and its own push triggers CI. A service that reacts to everything ends up talking to itself all night.

Why keep one sandbox per issue for the whole pull request?

Each review round needs the working tree that the last round left. Rebuilding that tree costs more than keeping it. A fresh machine for each round means a new clone, a new npm install or pip install, cold test caches, and an agent that has lost its scratch notes. That adds minutes to each round before the agent reads the review.

A paused sandbox keeps all of it, memory included. It costs only storage while the reviewer is busy elsewhere. The sandbox's name comes from the repository and issue number, so getOrCreate finds it on every event. Any request wakes it (pause and resume). A paused sandbox is kept as long as your account has credit. A pull request that sits for weeks therefore wakes the same sandbox. If the sandbox was deleted in the meantime, a fresh one replaces it, and that sandbox checks out the branch the earlier one pushed.

How do you receive GitHub's webhooks safely?

Check the signature against the raw body. Skip deliveries you have already seen. Ignore the agent's own actions. Respond quickly. GitHub signs each delivery with HMAC-SHA256 in X-Hub-Signature-256, names it with a unique X-GitHub-Delivery id, may deliver the same one again, and gives up on a receiver that takes longer than ten seconds. So the receiver only verifies and queues. A worker does the slow part. This receiver runs as written. It replays seven deliveries against itself:

TypeScriptimport { createHmac, timingSafeEqual } from "node:crypto";const SECRET = "the-webhook-secret-from-your-github-app";const BOT = "acme-agent[bot]";const seen = new Set<string>(); // keep delivery ids in your database in productionconst prToIssue = new Map([[51, 42]]); // saved when the agent opens pull request #51type Payload = {  action?: string;  sender?: { login: string };  label?: { name: string };  issue?: { number: number; pull_request?: object };  comment?: { body: string };  review?: { state: string; body: string | null };  pull_request?: { head: { ref: string } };  check_suite?: { conclusion: string | null; head_branch: string | null };};type Job = { kind: "start" | "revise" | "fix-checks" | "cleanup"; issue: number; note: string };function signed(body: string, signature: string | null): boolean {  const expected = Buffer.from(`sha256=${createHmac("sha256", SECRET).update(body).digest("hex")}`);  const given = Buffer.from(signature ?? "");  return given.length === expected.length && timingSafeEqual(given, expected);}const issueOf = (branch: string | null | undefined) =>  Number(/^agent\/issue-(\d+)$/.exec(branch ?? "")?.[1] ?? 0);function route(event: string, p: Payload): Job | string {  if (p.sender?.login === BOT) return "ignored: the agent's own action"; // no loops  if (event === "issues" && p.action === "labeled" && p.label?.name === "agent")    return { kind: "start", issue: p.issue!.number, note: "" };  if (event === "pull_request_review" && p.review?.state === "changes_requested")    return { kind: "revise", issue: issueOf(p.pull_request?.head.ref), note: p.review.body ?? "" };  if (    event === "issue_comment" &&    p.action === "created" &&    p.comment?.body.includes("@acme-agent")  )    return {      kind: "revise",      issue: prToIssue.get(p.issue!.number) ?? p.issue!.number,      note: p.comment.body,    };  if (    event === "check_suite" &&    p.check_suite?.conclusion === "failure" &&    issueOf(p.check_suite.head_branch)  )    return { kind: "fix-checks", issue: issueOf(p.check_suite.head_branch), note: "" };  if (event === "pull_request" && p.action === "closed" && issueOf(p.pull_request?.head.ref))    return { kind: "cleanup", issue: issueOf(p.pull_request?.head.ref), note: "" };  return "ignored: not for the agent";}function receive(delivery: string, event: string, body: string, signature: string | null): string {  if (!signed(body, signature)) return "401 bad signature";  if (seen.has(delivery)) return "200 duplicate delivery, skipped";  seen.add(delivery);  const job = route(event, JSON.parse(body));  return typeof job === "string" ? `200 ${job}` : `202 queued ${job.kind} for issue #${job.issue}`;}const sign = (body: string) => `sha256=${createHmac("sha256", SECRET).update(body).digest("hex")}`;const tests: [string, string, object, boolean?][] = [  [    "d1",    "issues",    {      action: "labeled",      label: { name: "agent" },      issue: { number: 42 },      sender: { login: "maya" },    },  ],  [    "d1",    "issues",    {      action: "labeled",      label: { name: "agent" },      issue: { number: 42 },      sender: { login: "maya" },    },  ],  [    "d2",    "issue_comment",    {      action: "created",      issue: { number: 42 },      comment: { body: "Done, @acme-agent opened a PR" },      sender: { login: BOT },    },  ],  [    "d3",    "issue_comment",    {      action: "created",      issue: { number: 51, pull_request: {} },      comment: { body: "@acme-agent add a test" },      sender: { login: "maya" },    },  ],  [    "d4",    "pull_request_review",    {      review: { state: "changes_requested", body: "Handle empty input" },      pull_request: { head: { ref: "agent/issue-42" } },      sender: { login: "maya" },    },  ],  [    "d5",    "check_suite",    {      action: "completed",      check_suite: { conclusion: "failure", head_branch: "agent/issue-42" },      sender: { login: "github-actions[bot]" },    },  ],  [    "d6",    "pull_request",    {      action: "closed",      pull_request: { head: { ref: "agent/issue-42" } },      sender: { login: "maya" },    },  ],  ["d7", "issues", { action: "labeled", label: { name: "agent" }, issue: { number: 7 } }, false],];for (const [delivery, event, payload, valid = true] of tests) {  const body = JSON.stringify(payload);  console.log(    `${delivery} ${event}: ${receive(delivery, event, body, valid ? sign(body) : "sha256=0")}`,  );}

Pythonimport hashlibimport hmacimport jsonimport reSECRET = b"the-webhook-secret-from-your-github-app"BOT = "acme-agent[bot]"seen = set()  # keep delivery ids in your database in productionpr_to_issue = {51: 42}  # saved when the agent opens pull request #51def signed(body: bytes, signature: str | None) -> bool:    expected = "sha256=" + hmac.new(SECRET, body, hashlib.sha256).hexdigest()    return hmac.compare_digest(expected, signature or "")def issue_of(branch):    m = re.fullmatch(r"agent/issue-(\d+)", branch or "")    return int(m.group(1)) if m else 0def route(event, p):    if p.get("sender", {}).get("login") == BOT:        return "ignored: the agent's own action"  # no loops    if event == "issues" and p.get("action") == "labeled" and p.get("label", {}).get("name") == "agent":        return {"kind": "start", "issue": p["issue"]["number"], "note": ""}    if event == "pull_request_review" and p.get("review", {}).get("state") == "changes_requested":        return {"kind": "revise", "issue": issue_of(p["pull_request"]["head"]["ref"]),                "note": p["review"].get("body") or ""}    if event == "issue_comment" and p.get("action") == "created" and "@acme-agent" in p["comment"]["body"]:        n = p["issue"]["number"]        return {"kind": "revise", "issue": pr_to_issue.get(n, n), "note": p["comment"]["body"]}    suite = p.get("check_suite") or {}    if event == "check_suite" and suite.get("conclusion") == "failure" and issue_of(suite.get("head_branch")):        return {"kind": "fix-checks", "issue": issue_of(suite["head_branch"]), "note": ""}    ref = (p.get("pull_request") or {}).get("head", {}).get("ref")    if event == "pull_request" and p.get("action") == "closed" and issue_of(ref):        return {"kind": "cleanup", "issue": issue_of(ref), "note": ""}    return "ignored: not for the agent"def receive(delivery, event, body: bytes, signature):    if not signed(body, signature):        return "401 bad signature"    if delivery in seen:        return "200 duplicate delivery, skipped"    seen.add(delivery)    job = route(event, json.loads(body))    return f"200 {job}" if isinstance(job, str) else f"202 queued {job['kind']} for issue #{job['issue']}"def sign(body: bytes) -> str:    return "sha256=" + hmac.new(SECRET, body, hashlib.sha256).hexdigest()labeled = {"action": "labeled", "label": {"name": "agent"}, "issue": {"number": 42}, "sender": {"login": "maya"}}tests = [    ("d1", "issues", labeled, True),    ("d1", "issues", labeled, True),    ("d2", "issue_comment", {"action": "created", "issue": {"number": 42},                             "comment": {"body": "Done, @acme-agent opened a PR"}, "sender": {"login": BOT}}, True),    ("d3", "issue_comment", {"action": "created", "issue": {"number": 51, "pull_request": {}},                             "comment": {"body": "@acme-agent add a test"}, "sender": {"login": "maya"}}, True),    ("d4", "pull_request_review", {"review": {"state": "changes_requested", "body": "Handle empty input"},                                   "pull_request": {"head": {"ref": "agent/issue-42"}}, "sender": {"login": "maya"}}, True),    ("d5", "check_suite", {"action": "completed", "check_suite": {"conclusion": "failure", "head_branch": "agent/issue-42"},                           "sender": {"login": "github-actions[bot]"}}, True),    ("d6", "pull_request", {"action": "closed", "pull_request": {"head": {"ref": "agent/issue-42"}},                            "sender": {"login": "maya"}}, True),    ("d7", "issues", {**labeled, "issue": {"number": 7}}, False),]for delivery, event, payload, valid in tests:    body = json.dumps(payload).encode()    print(f"{delivery} {event}: {receive(delivery, event, body, sign(body) if valid else 'sha256=0')}")

Both languages print the same seven lines:

Textd1 issues: 202 queued start for issue #42d1 issues: 200 duplicate delivery, skippedd2 issue_comment: 200 ignored: the agent's own actiond3 issue_comment: 202 queued revise for issue #42d4 pull_request_review: 202 queued revise for issue #42d5 check_suite: 202 queued fix-checks for issue #42d6 pull_request: 202 queued cleanup for issue #42d7 issues: 401 bad signature

Three details are easy to miss. The comparison runs in constant time, so an attacker cannot guess the signature byte by byte. A comment on a pull request arrives as an issue_comment with the pull request's number. So the service keeps a map from pull request to issue. The branch name, agent/issue-42, is how check and close events find their issue.

What does the worker do for each job?

The worker finds or creates the issue's sandbox, writes the task, runs the agent, runs the tests, pushes, and reports to GitHub. Then it pauses the sandbox until the next event. The model's key and the token for git push are Runtime secrets, set up once as in the overnight Claude Code post. The sandbox therefore holds only placeholders. Your service keeps its own GitHub token, for opening the pull request and commenting:

TypeScriptimport { Runtime, Sandbox } from "withruntime";const REPO = "acme/app";const runtime = new Runtime();const headers = {  authorization: `Bearer ${process.env.GITHUB_TOKEN}`,  accept: "application/vnd.github+json",};async function github(path: string, body?: object) {  const res = await fetch(`https://api.github.com/repos/${REPO}${path}`, {    method: body ? "POST" : "GET",    headers,    body: body ? JSON.stringify(body) : undefined,  });  if (!res.ok) throw new Error(`GitHub ${path}: ${res.status} ${await res.text()}`);  return res.json();}type Job = { kind: "start" | "revise" | "fix-checks" | "cleanup"; issue: number; note: string };export async function work(job: Job): Promise<number | undefined> {  const name = `${REPO.replace("/", "-")}-issue-${job.issue}`;  if (job.kind === "cleanup") {    for await (const sbx of await runtime.sandboxes.list({ name })) await sbx.delete();    return;  }  const branch = `agent/issue-${job.issue}`;  const sbx = await Sandbox.getOrCreate(name, {    vcpu: 2,    memoryMiB: 4096,    labels: { repo: REPO.replace("/", "-"), issue: String(job.issue) },  });  const app = { cwd: "/workspace/app" };  if (!sbx.info.reused) {    await sbx.exec(["git", "clone", `https://github.com/${REPO}.git`, "/workspace/app"], {      check: true,    });    // Picks up the branch if an earlier sandbox for this issue already pushed it.    await sbx.exec(`git switch ${branch} || git switch -c ${branch}`, { ...app, check: true });  }  const issue = await github(`/issues/${job.issue}`);  const why = {    start: "Resolve the issue.",    revise: "Address this review:",    "fix-checks": "CI failed. Read the failing check and fix it.",  };  await sbx.files.write(    "/workspace/TASK.md",    `# ${issue.title}\n\n${issue.body ?? ""}\n\n## Now\n\n${why[job.kind]}\n\n${job.note}\n`,  );  await sbx.exec(process.env.AGENT_COMMAND!, { ...app, timeoutMs: 1_800_000 }); // a round's ceiling  if (await sbx.files.exists("/workspace/QUESTION.md")) {    const question = await sbx.files.readText("/workspace/QUESTION.md");    await sbx.exec("rm /workspace/QUESTION.md");    await github(`/issues/${job.issue}/comments`, { body: `Before I go on: ${question}` });    await sbx.pause();    return;  }  const tests = await sbx.exec("npm test 2>&1 | tail -20", { ...app, timeoutMs: 600_000 });  const commit = `git add -A && git -c user.name="Acme Agent" -c user.email=agent@acme.dev commit -qm ${JSON.stringify(`#${job.issue}: ${job.kind}`)} && git push -q -u origin ${branch}`;  const pushed = await sbx.exec(commit, app);  if (pushed.exitCode === 0 && job.kind === "start") {    const pr = await github("/pulls", {      title: issue.title,      head: branch,      base: "main",      draft: true,      body: `Closes #${job.issue}\n\nTests:\n\n\`\`\`\n${tests.stdout}\n\`\`\``,    });    await sbx.pause();    return pr.number; // save pull request -> issue for comments on the pull request  }  const said = pushed.exitCode === 0 ? "Pushed a new commit." : "No change made this round.";  await github(`/issues/${job.issue}/comments`, {    body: `${said}\n\n\`\`\`\n${tests.stdout}\n\`\`\``,  });  await sbx.pause();}

Pythonimport jsonimport osimport urllib.requestfrom withruntime import Runtime, SandboxREPO = "acme/app"runtime = Runtime()HEADERS = {"authorization": f"Bearer {os.environ['GITHUB_TOKEN']}",           "accept": "application/vnd.github+json", "content-type": "application/json"}WHY = {"start": "Resolve the issue.", "revise": "Address this review:",       "fix-checks": "CI failed. Read the failing check and fix it."}def github(path, body=None):    req = urllib.request.Request(f"https://api.github.com/repos/{REPO}{path}", headers=HEADERS,                                 data=json.dumps(body).encode() if body else None,                                 method="POST" if body else "GET")    with urllib.request.urlopen(req, timeout=30) as res:        return json.load(res)def work(job):    name = f"{REPO.replace('/', '-')}-issue-{job['issue']}"    if job["kind"] == "cleanup":        for sbx in runtime.sandboxes.list(name=name):            sbx.delete()        return None    branch = f"agent/issue-{job['issue']}"    sbx = Sandbox.get_or_create(name, vcpu=2, memory_mib=4096,                                labels={"repo": REPO.replace("/", "-"), "issue": str(job["issue"])})    app = "/workspace/app"    if not sbx.info.get("reused"):        sbx.exec(["git", "clone", f"https://github.com/{REPO}.git", app], check=True)        # Picks up the branch if an earlier sandbox for this issue already pushed it.        sbx.exec(f"git switch {branch} || git switch -c {branch}", cwd=app, check=True)    issue = github(f"/issues/{job['issue']}")    sbx.files.write("/workspace/TASK.md", f"# {issue['title']}\n\n{issue.get('body') or ''}\n\n"                    f"## Now\n\n{WHY[job['kind']]}\n\n{job['note']}\n")    sbx.exec(os.environ["AGENT_COMMAND"], cwd=app, timeout_ms=1_800_000)  # a round's ceiling    if sbx.files.exists("/workspace/QUESTION.md"):        question = sbx.files.read_text("/workspace/QUESTION.md")        sbx.exec("rm /workspace/QUESTION.md")        github(f"/issues/{job['issue']}/comments", {"body": f"Before I go on: {question}"})        sbx.pause()        return None    tests = sbx.exec("npm test 2>&1 | tail -20", cwd=app, timeout_ms=600_000)    message = json.dumps(f"#{job['issue']}: {job['kind']}")    pushed = sbx.exec(f'git add -A && git -c user.name="Acme Agent" -c user.email=agent@acme.dev '                      f"commit -qm {message} && git push -q -u origin {branch}", cwd=app)    if pushed.exit_code == 0 and job["kind"] == "start":        pr = github("/pulls", {"title": issue["title"], "head": branch, "base": "main", "draft": True,                               "body": f"Closes #{job['issue']}\n\nTests:\n\n```\n{tests.stdout}\n```"})        sbx.pause()        return pr["number"]  # save pull request -> issue for comments on the pull request    said = "Pushed a new commit." if pushed.exit_code == 0 else "No change made this round."    github(f"/issues/{job['issue']}/comments", {"body": f"{said}\n\n```\n{tests.stdout}\n```"})    sbx.pause()    return None

AGENT_COMMAND is the command for any coding agent you run headless. The agent reads /workspace/TASK.md. That can be Claude Code, Codex or your own loop (coding agent sandbox). The worker's tests are a report for the reviewer, not a gate. Your normal CI still runs on the pull request (CI for agent pull requests).

What should the agent be told?

The issue, the round's instruction, and a way out. Put these three rules in the agent's standing instructions:

• Stay on the issue. Change only what it needs; do not edit CI configuration, lock files or tests you were not asked to touch.
• Prove it. Run the tests before finishing, and add one that fails without your change.
• Ask instead of guessing. If the issue is ambiguous, write the question to /workspace/QUESTION.md and stop.

The last rule is what makes the agent safe to point at real issues. An agent that guesses opens a confident pull request for the wrong reading. An agent that asks gets a one-line answer from the issue's author, and the comment wakes the sandbox to continue.

How do you keep it from doing damage?

Make a pull request the only thing it can produce. Protect the default branch so that every change needs an approved review. The agent's token can push its own branches, and nothing lands without a person. Open pull requests as drafts. Cap each round with the agent command's timeout. Cap the number of rounds per issue in your service (five is plenty). Set a daily spending limit on the account, so that a runaway day has a ceiling. The sandbox's network rules can be narrowed to GitHub, your package registry and the model API.

What does one issue cost?

Take one issue from label to merge. It has a first round of 20 minutes and three review rounds of 8 minutes each. All rounds run on 2 vCPU and 4 GiB, with the agent's tools averaging half a vCPU. Between rounds, the sandbox is paused for two days in total, at 1.5 GB:

CPU

44 min × 0.5 vCPU × $0.025 / 60

$0.0092

Memory

44 min × 4 GiB × $0.0075 / 60

$0.0220

Paused

1.5 GB × $0.08 × 2 days / 30 days

$0.0080

Total

$0.0392

The model's tokens for those 44 minutes will cost many times more. A hundred issues a month is about $3.92 in sandboxes (pricing).

In short

• An issue agent is an event handler: label, review, comment, failed check and close, each mapped to one sandbox action.
• Keep one sandbox per issue for the pull request's life, paused between rounds, so each review starts where the last one stopped.
• Verify GitHub's signature, skip repeated deliveries, ignore the agent's own events, and respond within ten seconds.
• Let the agent ask a question instead of guessing, and make a reviewed pull request the only thing it can produce.
• One issue with three review rounds costs about $0.0392 in sandbox time.

Run it on Runtime

Runtime costs 42% to 88% less than fourteen other sandbox providers for an agent that mostly waits on a model (compare costs). Each Runtime sandbox is named after its issue, and getOrCreate finds it again. Between rounds, it pauses and costs only storage. Secrets keep the GitHub and model keys out of the machine. Each month includes 100 free hours of a 1 GB machine, and no card is required. Sign in or read get started.