# How to query large CSV and Parquet files with DuckDB in a sandbox

Install the `duckdb` Python package in the sandbox, upload the files or mount the bucket, run the SQL there and download the result.

**On Runtime, an agent's SQL runs in a machine of its own, next to the data.
Memory is the cheap part.** Memory costs $0.0075 per GiB-hour. A 4 vCPU,
16 GiB sandbox therefore costs $0.22 an hour with all four CPUs busy, and less
while the model decides on its next query. A bucket mounts as a folder, and the
sandbox never holds the bucket's key. DuckDB 2.0.0 was the current release on
25 October 2026.

## Upload files and run a query

Put the query in a script, run the script in the sandbox, and download the output:

```ts check
import { Sandbox } from "withruntime";

const query = `
import duckdb
con = duckdb.connect()
con.execute("SET memory_limit = '3GB'")
con.execute("SET temp_directory = '/workspace/duckdb-tmp'")
con.execute("""
    COPY (
        SELECT customer_id, sum(amount) AS total
        FROM read_csv('/workspace/data/*.csv')
        GROUP BY customer_id
        ORDER BY total DESC
    ) TO '/workspace/out/totals.parquet' (FORMAT parquet)
""")
print(con.execute("SELECT count(*) FROM '/workspace/out/totals.parquet'").fetchone()[0], "customers")
`;

await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });
await sbx.exec("pip install -q duckdb==2.0.0", { check: true, timeoutMs: 300_000 });
await sbx.files.upload("./data", "/workspace/data");
await sbx.files.write("/workspace/query.py", query);
await sbx.files.mkdir("/workspace/out");

await sbx.network.set({ internet: false }); // the query needs nothing from outside
const run = await sbx.exec(["python3", "/workspace/query.py"], {
  check: true,
  timeoutMs: 1_200_000,
});
console.log(run.stdout);
await sbx.files.download("/workspace/out/totals.parquet", "./totals.parquet");
```

```python check
from withruntime import Sandbox

QUERY = '''
import duckdb
con = duckdb.connect()
con.execute("SET memory_limit = '3GB'")
con.execute("SET temp_directory = '/workspace/duckdb-tmp'")
con.execute("""
    COPY (
        SELECT customer_id, sum(amount) AS total
        FROM read_csv('/workspace/data/*.csv')
        GROUP BY customer_id
        ORDER BY total DESC
    ) TO '/workspace/out/totals.parquet' (FORMAT parquet)
""")
print(con.execute("SELECT count(*) FROM '/workspace/out/totals.parquet'").fetchone()[0], "customers")
'''

with Sandbox.create(timeout_seconds=1800) as sbx:
    sbx.exec("pip install -q duckdb==2.0.0", check=True, timeout_ms=300_000)
    sbx.files.upload("./data", "/workspace/data")
    sbx.files.write("/workspace/query.py", QUERY)
    sbx.files.mkdir("/workspace/out")

    sbx.network.set(internet=False)
    print(sbx.exec(["python3", "/workspace/query.py"], check=True, timeout_ms=1_200_000).stdout)
    sbx.files.download("/workspace/out/totals.parquet", "totals.parquet")
```

The CSV, Parquet and JSON readers are built into the Python package. In a test
on 25 October 2026, `duckdb_extensions()` listed `parquet` and `json` as
statically linked. So the query runs with the internet off. Reading `https://`
or `s3://` URLs directly needs the `httpfs` extension. DuckDB downloads that
extension the first time, so leave the network on for that.

## Memory, disk and spilling

DuckDB processes data larger than memory by writing intermediate results to
its temp directory. Two of its defaults decide how far that goes. Both follow
from the sandbox's size:

| DuckDB setting            | Default                 | In a default sandbox (4 GiB memory, 4 GiB disk) |
| ------------------------- | ----------------------- | ----------------------------------------------- |
| `memory_limit`            | 80% of RAM              | About 3.2 GiB                                   |
| `max_temp_directory_size` | 90% of available disk   | Most of the roughly 2.5 GiB free                |
| `threads`                 | The number of CPU cores | 2, one per vCPU                                 |

`SET memory_limit = '3GB'` counts decimal gigabytes. DuckDB reports it back as
2.7 GiB. For a larger job, size the sandbox, not the setting:

- **Memory:** `memoryMiB` sets the sandbox's memory at create. By default,
  DuckDB uses 80% of it.
- **Disk:** `diskMiB` holds the input files, the output and anything DuckDB
  spills. Without credit, a sandbox can have up to 10 GiB.
- **CPU:** `vcpu` raises the thread count. You pay for the CPU the query
  actually uses.

## Query a bucket without copying it

A bucket in S3, R2 or Google Cloud Storage can appear as a folder. Store the
bucket's key once as a secret. The sandbox's mount client signs requests with a
placeholder. The host then signs them again with the real key
([mount your own bucket](/docs/storage#mount-your-own-bucket)):

```ts check
import { Sandbox } from "withruntime";

await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });
await sbx.mounts.add({
  provider: "s3",
  bucket: "acme-events",
  region: "eu-west-1",
  path: "/data",
  secret: "EVENTS_BUCKET",
  readOnly: true,
});
await sbx.exec("pip install -q duckdb==2.0.0", { check: true, timeoutMs: 300_000 });
const run = await sbx.exec([
  "python3",
  "-c",
  "import duckdb; print(duckdb.sql(\"SELECT event, count(*) FROM '/data/2026/09/*.parquet' GROUP BY event\"))",
]);
console.log(run.stdout);
```

`readOnly` means a query cannot change the bucket, whatever SQL the agent
writes.

## Let the model write the SQL

Pass the model's query to a small runner on standard input. Do not put it
inside a shell command. The runner returns a bounded number of rows as JSON:

```python check
from withruntime import Sandbox

RUNNER = '''
import json, sys, duckdb
cursor = duckdb.connect().execute(sys.stdin.read())
columns = [column[0] for column in cursor.description or []]
print(json.dumps({"columns": columns, "rows": cursor.fetchmany(50)}, default=str))
'''

def run_sql(sbx, sql: str) -> str:
    result = sbx.exec(["python3", "/workspace/run_sql.py"], stdin=sql, timeout_ms=300_000)
    return result.stdout if result.exit_code == 0 else result.stderr

with Sandbox.create() as sbx:
    sbx.exec("pip install -q duckdb==2.0.0", check=True, timeout_ms=300_000)
    sbx.files.write("/workspace/run_sql.py", RUNNER)
    print(run_sql(sbx, "SELECT 42 AS answer"))
```

A DuckDB error, such as `Parser Error: syntax error at or near "SELEC"`, comes
back on `stderr` with the position marked. The model needs that position to
correct its query. Fifty rows keep the reply short. A query that needs the
whole result should `COPY` it to a file that you download. The runner reads
rows from the cursor `execute` returns, because DuckDB 2.0 relations no longer
have `fetchone()` or `fetchmany()`.

## Start with DuckDB installed

A recipe image installs the package once, for every sandbox you create later:

```ts check
import { Runtime } from "withruntime";

const runtime = new Runtime();
await runtime.images.build({ name: "duckdb", recipe: { pip: ["duckdb==2.0.0"] } });
await using sbx = await runtime.sandboxes.create({ image: "duckdb", memoryMiB: 8192 });
```

The default image already includes `pandas` 2.1.4, so `relation.df()` works
without installing it.

## Related

- [A data analysis agent that runs Python safely](/use-cases/data-analysis-agent)
- [SQLite in a sandbox](/integrations/sqlite)
- [Run untrusted Python code safely](/languages/python)
- [Turn off sandbox internet](/how-to/turn-off-sandbox-internet)

## Sources

Checked 25 October 2026.

- [DuckDB configuration](https://duckdb.org/docs/2.0/configuration/overview.html):
  `memory_limit` (80% of RAM), `temp_directory`, `max_temp_directory_size`
  (90% of available disk), `threads` (the number of CPU cores)
- [duckdb on PyPI](https://pypi.org/project/duckdb/): version 2.0.0, Python
  3.11 or later
- A local test of duckdb 2.0.0 on 25 October 2026: `duckdb_extensions()`
  lists `parquet`, `json` and `icu` as statically linked and `httpfs` as not
  installed; `SET memory_limit = '3GB'` reads back as 2.7 GiB

Facts on this page were checked on 25 October 2026.