Can coding agents write high-quality infrastructure code?
InfraBench asks agents to extend the proprietary infra code-generation framework nplus1 uses to build and run infrastructure for its clients, across Terraform, GCP, Vercel and CI. Each task is real work on that framework. We grade it by generating and running real projects from the agent's changes, not by comparing against a reference solution.
Infrastructure · Platform engineering
- Tasks
- 2
- Public tasks
- 2
- Models evaluated
- 3
- Published trials
- 10
- Mean reward
- Average of task means across the task versions that match the dataset’s rules.
- ± SE
- Standard error across task means; directional when a dataset has few tasks.
| Model · harness | Mean reward | ± SE | Tasks | Trials |
|---|---|---|---|---|
| GPT 5.6 SolOpenCode | 80.0%Task means: 80.0%. | — | 1 / 2 | 1 |
| Muse Spark 1.3 ContributorOpenCode | 76.3%Task means: 59.3%, 93.3%. | 17.0 pp | 2 / 2 | 8 |
| gemini-3.1-pro-previewOpenCode | 48.0%Task means: 48.0%. | — | 1 / 2 | 1 |
How InfraBench works
1 · The agent gets a framework
A runnable copy of n1x — the code-generation framework that stamps out client applications from composable plugins — plus one engineering instruction taken from real capability work.
# Add composable AG Grid and AG Charts # Enterprise support to n1x Introduce a `web-ag` capability that requires `web` … registers AllEnterpriseModule exactly once, including across development reloads. … Keep the capability isolated and lifecycle-safe: - `web-ag` must work without `infra`. - Repeated apply operations must be idempotent. - Removing `web-ag` must clean its managed output without removing PostHog …
2 · It ships a plugin
The deliverable is a declarative plugin: a manifest plus managed templates, composed through the framework's existing seams. The core engine and unrelated capabilities stay untouched.
{ "name": "web-ag", "requires": ["web"], "replace": [ { "from": "files/src/_lib/ag.ts", "to": "web/src/_lib/ag.ts" } ], "mergeJson": { "web/package.json": { "dependencies": { "ag-charts-enterprise": "14.0.0", "ag-grid-enterprise": "36.0.0", … } } }, … }3 · Grading is generated behavior
Generate → Execute → Recompose → Remove
The verifier stamps out real projects from the modified framework — alone, combined with other plugins in both orders, reapplied, and removed — and runs the generated code with stubs. Structure is never string-matched against a reference patch.
4% core engine and unrelated capabilities are unchanged 4% web-ag resolves web and renders its managed helper 1% web-ag pins ag-charts-community 1% web-ag pins ag-charts-enterprise + 28 more weighted checks
Pass rate (%) · 11.1% of 9 scored trials pass
Median steps per trial · 2 tasks
Elapsed runtime · 10 timed trials