Retry transient GitHub read failures before failing a Pass #786

This was generated by AI during triage.

Problem

A single transient GitHub failure on a read fails a whole Pass and sends a failure notification. On 2026-10-19 at 00:02:34 UTC, the scheduled Architect run for thirdshift on stephens failed 12 seconds in, before its Architecture review started:

Result:       review failed
Review:       failed
Cause:        gh issue list failed: HTTP 504: 504 Gateway Timeout (https://api.github.com/graphql)
Host:         stephens
Took:         12s

The next minute's passes ran normally. The same 504 also appears in ~/.thirdshift/logs/cron.log at 18:45:14 UTC the same day. With 15 scheduled passes a minute on one host, occasional GitHub 5xx responses will keep producing failure notifications that need no action.

Current behavior

GitHub::gh_json (src/github.rs) bails on the first nonzero gh exit. CI check reads already classify failures with check_read::is_transient (HTTP 429/5xx, EOF, connection resets) and retry in src/ci.rs, but no other read uses it.

Proposed behavior

• Retry read-only gh calls (issue/PR/label listing and viewing) that fail with a transient classification, using a small bounded budget (e.g. 3 attempts with increasing delay, well under a minute total), then fail with the last cause as today.
• Reuse is_transient. Only HTTP 429 that is not GraphQL quota exhaustion should be retried, since Defer scheduled passes during GitHub quota exhaustion and reduce idle polling #739 covers quota.
• Never retry mutations (label edits, comments, PR creation, merges).
• Record each retry as a progress/log line so recovered failures stay visible.
• Command interruption cancels the wait.

Validation targets

• A fake gh that returns HTTP 504 once, then succeeds, lets the Pass continue with no failure notification.
• A persistent 504 fails within the budget, with the original cause.
• Authentication, 404, validation and quota errors fail on the first attempt.
• No mutation is ever issued twice.

Related

• Report Security run preflight failures without claiming the audit ran #740 / PR Report Security pre-audit failures without claiming an audit ran #767: reports a Security run's pre-audit failure honestly. The Architect run equivalent (review failed before any review ran) is not covered there; see Defer scheduled passes during GitHub quota exhaustion and reduce idle polling #739.
• Defer scheduled passes during GitHub quota exhaustion and reduce idle polling #739: quota-aware deferral (a different failure class).
• Recover from temporary Harness service failures within a bounded retry budget #584: transient Harness service failures (same idea, for model sessions).