Conversation
The nightly export failed again. Can you take a look?
Sure. Which job is it, and when did it last succeed?
billing-export. It last succeeded on Tuesday.
Since Wednesday, every run has stopped after 30 seconds waiting on the reports service. The service was slow, not down: its median response time went from 4 to 41 seconds.
So we just raise the timeout?
Raising it would get tonight's run through, but the job would still fail on the next slow night. I'd retry each page with backoff, cap the attempts at five, and alert when a run takes longer than an hour.
Do that, and open a pull request.
Done. The pull request adds the retry and the alert, with a test that replays Wednesday's responses.