Can coding agents rebuild an allocation engine from its run history?
Modeled on a real optimization engine for a top-5 global data center developer, AllocBench asks agents to rebuild the allocation engine from nothing but its run history. None of the rules are documented, so the agent has to infer each one from past runs. We grade whether its solver reaches the best possible plan. Each task represents a distinct set of inputs.
Optimization
- Tasks
- 2
- Public tasks
- 2
- Models evaluated
- 3
- Published trials
- 29
- Leading model
- GPT 5.6 SolOpenCode
- Leading mean reward
- 69.5%± 30.5 pp
- Mean reward
- Average of task means across the task versions that match the dataset’s rules.
- ± SE
- Standard error across task means; directional when a dataset has few tasks.
| Model · harness | Mean reward | ± SE | Tasks | Trials |
|---|---|---|---|---|
| GPT 5.6 SolOpenCode | 69.5%Task means: 39.0%, 100.0%. | 30.5 pp | 2 / 2 | 4 |
| Muse Spark 1.3 ContributorOpenCode | 56.1%Task means: 55.8%, 56.5%. | 0.3 pp | 2 / 2 | 22 |
| gemini-3.1-pro-previewOpenCode | 45.0%Task means: 39.0%, 51.0%. | 6.0 pp | 2 / 2 | 3 |
How AllocBench works
1 · The agent gets a planning cycle
A supply-allocation scenario from an anonymized data-center operator: equipment lots with ERP pool noise and missing costs, deal-phase demand with required-on-site dates, and a config whose switches — supplier pins, the single-supplier rule, locked phases — are named but never explained. The retired engine's decision policy was never written down; the only spec is its run history, a salted archive of captured request/plan pairs the agent must reverse-engineer.
{ "config": { "objective": "timelineRisk", "enforceSingleSupplier": true, "supplierPins": [ { "phaseId": "phase-tus02-2", "equipmentTypeId": "eq-gen", "supplierId": "sup-duneland" } ], "lockedPhaseIds": ["phase-msn04-1"] }, "demand": [ { "id": "req-msn04-1-pdu", "quantityRequired": 36, "needByDate": "2027-04-01", … }, … ], "supply": [ { "id": "lot-0016", "inventoryPool": " ops spare ", "unitCost": null, "availabilityDate": "2027-06-15", … }, … ] }2 · It ships an allocation engine
Not a plan — a solver. The deliverable is a pure function from request to assignments that reproduces the legacy engine's behavior: its lexicographic objective (maximize fill, then minimize lateness, then cost), its pricing of unpriced lots, its pool exclusions — every rule the archive evidences.
# allocator/solve.py def solve(request: dict) -> dict: """Return an allocation response for an allocation request.""" … return { "assignments": [ { "demandId": "req-msn04-1-pdu", "lotId": "lot-0142", "quantity": 24 }, … ] }3 · Grading is what the plan achieves
Run → Check rules → Compare → Perturb
The verifier re-derives feasibility from the contract, then compares the plan's fill, lateness, and cost against the golden optimum — any optimal plan passes, whichever lots it picks. Then the world changes: new, cheaper lots land and one more phase locks. The engine re-runs, and locked phases must come back byte-for-byte even though re-optimizing them would look better. That freeze rule exists because a real operator once watched a locked phase move.
Pass rate (%) · 17.2% of 29 scored trials pass
Median steps per trial · 2 tasks
Elapsed runtime · 29 timed trials