Rework a GEO analytics app's hourly collection job and edition generation so scheduled runs complete on the first tick, retrying transient provider failures within the tick, not retrying unfixable ones, and giving up on partial runs within hours.
- Version
- Category
- Engineering
- Model configurations
- 1
- Attempts
- 8
- Public trials
- 164 passed
- Median runtime
- 20m 5s
Instruction
# Make scheduled runs complete on the first tick One workspace's 9am scheduled run sat `partial` all day and no edition appeared. One provider had no API key in production, so every call to it failed. The hourly job kept re-claiming the run to retry the missing responses, and it would only have given up, and generated an edition from what it had, 24 hours after the slot. The same day's logs show the other ways runs end partial: Perplexity and ChatGPT rate limits that outlasted the collectors' own retries and then went through on a later hourly tick, and a few DeepSeek answers that came back with no text and were fine when asked again. From the engineer who owns the pipeline: > taking a look at our overall retry harness for runing these queries (not > just on deepseek) - would you reapproach/rebuild that to improve chance of > us getting completes rather than partials (without hanging retrying for > hours etc) (and without overcomplicating it like crazy) What we want: - Ordinary provider trouble (rate limits, server errors, an empty answer) is absorbed within the tick, so a scheduled run completes the first time. That includes trouble that outlasts the collectors' current retries, as that day's rate limits did: the response is still collected on the same tick, not left for a later one. - Failures that no retry can fix (a missing or rejected API key, a model the provider doesn't know) aren't retried pointlessly. - A run that is still partial after its tick keeps being picked up by later hourly ticks for a while. When we provisioned the missing key, the next tick collected just the missing responses and completed the run, and that must keep working. But a slot shouldn't hang for hours on end before we give up and generate the edition from what was collected. - Editions are still generated only from completed runs, or from a partial run once we've given up on it. - Ticks will take longer now. A run that is still being worked on must never look abandoned to the next hourly tick. The same goes for what builds the edition after collection. Once that run finally completed, the month's edition failed twice on AI Insights. Those calls (xhigh reasoning over a 100–150k-token topic corpus) run at roughly two minutes: the ones that succeeded took 95–120 seconds, and the 120-second limit cut off the rest on every attempt. A slow insight call shouldn't be abandoned. One analysis that fails once shouldn't sink the whole edition and leave it waiting for the hourly retry either. ## Environment PostgreSQL on localhost:5432 and the local database endpoint the app's driver uses are running when you start, and the dev env files point at the dev database.
Public trials
Results by model
| Model · harness | Mean reward | Range | ± SE | Attempts |
|---|---|---|---|---|
| Frontier model ACursor CLI | 50.0% | 0% – 100% | 18.9 pp | 8 |