# What is SWE-bench?

SWE-bench is a benchmark of real GitHub issues. A model gets a codebase and an
issue, and writes a patch. The repository's tests judge the patch.

**Every SWE-bench attempt needs its own machine with the repository at the right
commit. On Runtime, that machine is a Firecracker microVM. It bills the CPU the
tests use.** A paid account runs 100 sandboxes at once. A task
image built once gives every attempt a ready start
([agent evals and SWE-bench](/use-cases/agent-evals-and-swe-bench)).

## Why it matters for AI agents

SWE-bench measures coding agents on real bugs from real projects. It does not
use puzzles written for a test. In the project's words: "Given a codebase and an
issue, a language model is tasked with generating a patch that resolves the
described problem." A result is a count of issues resolved. A higher score
means more real bugs fixed, with the project's own tests as the judge.

That makes it an infrastructure problem as much as a model problem. Each task
has its own repository, version and dependencies. A fair run gives every
attempt a clean copy of all three.

## SWE-bench in facts

| Dataset                | Size, as SWE-bench states it | What it is                             |
| ---------------------- | ---------------------------- | -------------------------------------- |
| SWE-bench              | 2,294 instances              | The full set of GitHub issues          |
| SWE-bench Lite         | 534 instances                | "Smaller subset for quick evaluations" |
| SWE-bench Verified     | 500 instances                | "Expert-verified solvable problems"    |
| SWE-bench Multimodal   | 100 dev instances (500 test) | Problem statements with image assets   |
| SWE-bench Multilingual | 300 instances                | A multilingual set                     |

Each instance has a repository, a `base_commit`, the `problem_statement`, the
gold `patch`, a `test_patch`, and two test lists: `FAIL_TO_PASS` and
`PASS_TO_PASS`. SWE-bench reports an instance as resolved when "the patch made
the required tests pass". It was accepted to ICLR 2024 as an oral
presentation. It gained a fully containerized Docker harness on 27 June 2024.
It released Verified on 13 August 2024.

## How a run works

1. Build an environment for each task: the repository at `base_commit`, with
   its dependencies installed.
2. Give the agent the problem statement, and let it produce a patch.
3. Apply the patch, run the repository's tests, and record each attempt as
   resolved or unresolved.

The official harness runs step 3 in Docker. `--max_workers` sets how many
tasks run side by side. Its guide's examples use 8 for Lite and 12 for the
full set. On one machine, that number stops where its cores and disk do.

## Running it on Runtime

- **One microVM per attempt.** Each has its own kernel and disk, so tests
  cannot disturb each other.
- **Task images.** Build them free from a Dockerfile, any registry image or a
  recipe. Pin them with `name@version` ([custom images](/docs/images)).
- **Internet off** with `network: { internet: false }`, so an attempt cannot
  fetch the fix.
- **Forks** of 1 to 100 running copies, for pass@k, from one
  prepared machine ([sandbox forks](/glossary/sandbox-fork)).
- **Measured CPU:** $0.025 per vCPU-hour used and $0.0075 per
  GiB-hour of memory, so time spent waiting on the model costs little
  ([pricing](/docs/pricing)).

The full loop, in TypeScript and Python, with a cost for 500
attempts, is in [how to run SWE-bench-style agent evals](/use-cases/agent-evals-and-swe-bench).

## Related

- [RL environments](/use-cases/rl-environments)
- [Coding agent sandbox](/use-cases/coding-agent-sandbox)
- [What is a sandbox snapshot?](/glossary/sandbox-snapshot)
- [What is an agent sandbox?](/glossary/agent-sandbox)

## Sources

Read 25 October 2026.

- [SWE-bench documentation](https://www.swebench.com/SWE-bench/): the definition, the harness and its milestones
- [SWE-bench datasets](https://www.swebench.com/SWE-bench/guides/datasets/): dataset sizes and instance fields
- [SWE-bench evaluation](https://www.swebench.com/SWE-bench/guides/evaluation/): resolved and unresolved, and `--max_workers`

Facts on this page were checked on 25 October 2026.