Reproduce a full planning season: four re-run cycles as lots land and locks accumulate.
- Version
- Category
- software-engineering
- Difficulty
- expert
- Model configurations
- 3
- Attempts
- 13
- Public trials
- 134 passed
- Median runtime
- 16m 7s
Instruction
# Replay a planning season The old planning module was retired last quarter and its allocation engine's source did not survive the migration. Its captured production runs are in `audit/` — request/plan pairs, exactly as the engine consumed and produced them. The decision policy was never written down; the archive is the only spec. The engine never ran once; it ran all season. After the opening cycle in `data/scenario.json`, lots keep landing and reviewed phases keep locking — each cycle's plan becomes the next cycle's commitments. The verifier replays four such cycles against your engine, so a small policy error early compounds into every later cycle. Implement the engine at `allocator/solve.py` (entry point in `allocator/README.md`) so it reproduces the legacy engine's planning behavior on any request, and use it to plan `data/scenario.json`. The engine re-runs as the world changes — new lots land and phases lock between runs — and grading is outcome equivalence with the legacy engine, not matching its exact lot choices. Check plan structure with `python3 contracts/validate.py <request.json> <response.json>`.
Public trials
Results by model
| Model · harness | Mean reward | Range | ± SE | Attempts |
|---|---|---|---|---|
| GPT 5.6 SolOpenCode | 100.0% | 100% – 100% | — | 1 |
| Muse Spark 1.3 ContributorOpenCode | 55.8% | 0% – 100% | 12.5 pp | 11 |
| gemini-3.1-pro-previewOpenCode | 51.0% | 51% – 51% | — | 1 |