Repository navigation
Two machines' Pickup runs can take the same Ready issue at once #712
Description
This was generated by AI during triage.
Split from #695.
Problem
Across machines, the Claim on GitHub is the only coordination, by design: ADR 0009 says an issue "is taken at most once across passes by state on GitHub", and "the lock only covers one machine". That holds once the in-progress label is on the issue. It does not hold in the window between a Pickup run reading an issue as Ready and its Claim being written. A Pickup run on a second machine can read the same issue as Ready in that window. Then two Runs work on one issue, competing for its Issue branch and pull request.
This hasn't been seen yet. It is a gap in reasoning, not a reported failure. Verify it before building a fix: check whether making the Claim, or creating the Issue branch, already fails safely for the second machine.
What done looks like
When Pickup runs on two machines race for one Ready issue, exactly one Run proceeds. The other is skipped, or moves to the next Ready issue, without touching the first Run's branch or pull request.
Design questions
- Is the race real? GitHub's label edits are not compare-and-swap. A push of the Issue branch is atomic and might already act as the tiebreaker.
- If it is real, the options: claim, then re-read and back off if someone else claimed too; treat pushing the Issue branch as the lock; or something else.
Background
The Temporal research in #694 lists coordination across machines among the things Temporal would add (Workflow Id uniqueness). It recommends not adopting Temporal now.
Activity
JacobStephens2 commentedon Oct 17, 2026
The Oct 11–16 2026 Run retrospective provides observed evidence for this issue, whose original body says the race has not yet been seen.
The report attributes about five hours of aggregate overhead across Runs #658, #666, #671, #672, and #707 to a second writer on the same Issue branches. Its evidence includes first foreign commits within minutes of this host's Claim (#658: 21:24:44 after a 21:20:05 Claim; #707: commit 506f9b7 at 15:47:41 after a 15:42:07 Claim), the same GitHub identity and Built-with marker, and an accounting of local Codex sessions/schedulers. #672's code was discarded; #707 lost 10 of 12 changed files. These are the report's conclusions, not a fresh independent reproduction in this filing session. The other host's identity remains unknown.
Two follow-ups belong here:
Acceptance remains exactly one Run proceeding for one Ready issue, with the loser leaving its branch/PR untouched. Related retro issues cover empty-commit ping-pong and lifecycle stop rules independently.
Full report and finding index: #721 (finding 1, Host; race-safe Claims also affect Factory).
JacobStephens2 commentedon Oct 19, 2026
Observed in production: both hosts ran #740 at the same time on 2026-10-17.
stephensstarted a Pickup run on Report Security run preflight failures without claiming the audit ran #740 at 18:21 UTC (~/.thirdshift/logs/JacobStephens2/thirdshift/commands/pickup/740-20261017T182101+0000.log).thirdshiftstarted one on the same issue in the same minute (~/.thirdshift/logs/JacobStephens2/thirdshift/commands/pickup/740-20261017T142101-0400.log, local time −04:00).issue-740. One session had to merge the other's work (commitf9c8639"Merge concurrent issue-740 work while preserving both implementations intent").stephensended its Run failed at 19:42:56 UTC and marked PR Report Security pre-audit failures without claiming an audit ran #767 as a draft (4fa7cef).thirdshift's Failed run preservation push at 19:22:28 UTC was then rejected as non-fast-forward. Its local branch was left behind in a worktree with no code changes, only an older version bump.Net effect: double the Harness spend, a draft PR, a review in the PR body that was done on an older commit than the PR's current head, and a Claim holding a slot in the Claim limit until a person sorts it out.