scheduled-run-retry-harness

Public task · Field Engineering

← Field Engineering

Rework a GEO analytics app's hourly collection job and edition generation so scheduled runs complete on the first tick, retrying transient provider failures within the tick, not retrying unfixable ones, and giving up on partial runs within hours.

Version
Category
Engineering
Model configurations
1
Attempts
8
Public trials
164 passed
Median runtime
20m 5s

Instruction

# Make scheduled runs complete on the first tick

One workspace's 9am scheduled run sat `partial` all day and no edition
appeared. One provider had no API key in production, so every call to it
failed. The hourly job kept re-claiming the run to retry the missing
responses, and it would only have given up, and generated an edition from
what it had, 24 hours after the slot. The same day's logs show the other ways
runs end partial: Perplexity and ChatGPT rate limits that outlasted the
collectors' own retries and then went through on a later hourly tick, and a
few DeepSeek answers that came back with no text and were fine when asked
again.

From the engineer who owns the pipeline:

> taking a look at our overall retry harness for runing these queries (not
> just on deepseek) - would you reapproach/rebuild that to improve chance of
> us getting completes rather than partials (without hanging retrying for
> hours etc) (and without overcomplicating it like crazy)

What we want:

- Ordinary provider trouble (rate limits, server errors, an empty answer) is
  absorbed within the tick, so a scheduled run completes the first time.
  That includes trouble that outlasts the collectors' current retries, as
  that day's rate limits did: the response is still collected on the same
  tick, not left for a later one.
- Failures that no retry can fix (a missing or rejected API key, a model the
  provider doesn't know) aren't retried pointlessly.
- A run that is still partial after its tick keeps being picked up by later
  hourly ticks for a while. When we provisioned the missing key, the next
  tick collected just the missing responses and completed the run, and that
  must keep working. But a slot shouldn't hang for hours on end before we
  give up and generate the edition from what was collected.
- Editions are still generated only from completed runs, or from a partial
  run once we've given up on it.
- Ticks will take longer now. A run that is still being worked on must never
  look abandoned to the next hourly tick.

The same goes for what builds the edition after collection. Once that run
finally completed, the month's edition failed twice on AI Insights. Those
calls (xhigh reasoning over a 100–150k-token topic corpus) run at roughly two
minutes: the ones that succeeded took 95–120 seconds, and the 120-second
limit cut off the rest on every attempt. A slow insight call shouldn't be
abandoned. One analysis that fails once shouldn't sink the whole edition and
leave it waiting for the hourly retry either.

## Environment

PostgreSQL on localhost:5432 and the local database endpoint the app's
driver uses are running when you start, and the dev env files point at the
dev database.

Public trials

Results by model

Model · harnessMean rewardRange± SEAttempts
Frontier model ACursor CLI50.0%0% – 100%18.9 pp8