Skip to content

Re-run an Abandoned job instead of starting a CI-fix Repair - #930

Merged
JacobStephens2 merged 2 commits into
mainfrom
issue-908
Oct 21, 2026
Merged

JacobStephens2 merged 2 commits into
mainfrom
issue-908

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

A GitHub Actions job that GitHub ended as failure while none of its steps failed, and at least one step had not completed (a lost runner), is now an Abandoned job. If every one of the branch's own failures on a head is an Abandoned job, the Repair loop re-runs them with no Repair, once per head for the Delivery. It then watches CI again behind the Check re-run's old-attempt guard.

 CheckReruns::assess(head)
   unchanged after a CI-fix Repair, allowance unspent → Check re-run (ADR 0010)
   otherwise → watch(head)
+    watch_to_end: own failures all Actions jobs concluded `failure`?
+      read each job's steps (GET actions/jobs/<id>), stopping at the first not abandoned
+    rerun_abandoned_jobs_once
+      own failures all Abandoned jobs, head has had neither re-run
+        "<check> lost its runner on <sha>; re-running it"
+        gh run rerun <run> --failed, per workflow run
+        wait for the old check run ids to leave, then watch
+        refused / no new attempt → the red CI as first watched
 red on own → CI-fix Repair (as today)
  • Mixed abandoned and ordinary failures, a refused or unseen re-run, an unreadable job, and a second abandonment on the same head all start a CI-fix Repair, as today. After that Repair, an unchanged head still gets its Check re-run, so a head may get both re-runs, in that order.
  • No Repair starts and no budget is spent. No annotation or log text is read.
  • ADR 0020 records the decision, including why it isn't the "re-run before any Repair" option that ADR 0010 rejected. The glossary gains Abandoned job, and Check re-run points to it. The guide (step 6) and the generated site/docs/index.html are updated.

Seams under test:

  • ci::is_abandoned: the classifier.
  • ci watch and re-run through the in-memory Outside: step reads, marking, request and new-attempt watch, refusal.
  • repair_loop::take_to_goal through the scripted Outside.
  • One end-to-end Run through the fake gh (tests/ci.rs), whose fake now answers repos/<repo>/actions/jobs/<id>.

Evidence

  • Before: an abandoned job's red CI went straight to CI red; starting Repair 1 of 5.
    After: an_abandoned_job_is_re_run_with_no_repair_and_green_is_success (tests/ci.rs): only the implement session runs, gh run rerun 900 --failed is requested, stderr reads smoke / wsl lost its runner on <sha>; re-running it then CI passed, and the Run exits 0.

  • Acceptance criteria, in-process:

    • a_failed_job_with_no_failed_step_and_a_step_not_completed_is_abandoned
    • an_abandoned_job_alone_is_re_run_with_no_repair_and_green_reaches_the_goal
    • an_abandoned_job_abandoned_again_or_not_re_run_gets_the_ci_fix_repair_then_its_check_re_run
    • abandoned_jobs_beside_ordinary_own_failures_get_the_ci_fix_repair_as_any_red_ci
    • a_head_whose_abandoned_jobs_were_re_run_gets_no_second_re_run_after_a_refused_merge
  • Review finding P1 had no test, so I wrote two, and both failed on 672f9b7:

    • a_watch_reads_steps_only_while_every_own_failure_may_yet_be_an_abandoned_job: left: [1, 2, 3].
    • a_job_whose_steps_cannot_be_read_is_not_abandoned_and_ci_stays_red: Err … HTTP 403.

    Both pass now.

  • cargo test: 2187 passed, 0 failed. cargo clippy --all-targets -- -D warnings and cargo fmt --check are clean. cd site && npm run docs:check is clean.

Merge Danger

Door: two-way

Blast Radius: CI watch

  • Every red watch whose own failures are all Actions jobs concluded failure now makes up to one jobs-API read per job.
  • A read that fails only adds a progress line.
  • Revert to restore the old behaviour.

Review coverage

Both axes reviewed main...672f9b7 and read every changed file. site/docs/index.html is excluded as generated output: its source is docs/guide.md, and npm run docs:check is clean. The follow-up commit only touched files both reviewers had read.

Unaddressed findings

Standards

  • S2: move abandonment off ActionsJob onto FailedChecks. Declined: that means editing every FailedChecks literal, about 30 across src/ci.rs, src/prompt.rs, src/prompts_page.rs, src/base_fix.rs and src/delivery/repair_loop.rs. The field has one production constructor (ActionsJob::of, src/github.rs), and its doc says it is false until the CI watch tells it.
  • S3 (part): rename CheckReruns and consumed. Declined: these are existing names. The struct doc now names both allowances, and each field is documented. The rest of S3 was fixed.
  • S6: one re-run entry point taking a kind, instead of a parallel rerun_abandoned at each layer. Declined: ADR 0020 keeps the two re-runs separate, with different triggers and refusal handling. On refusal, a Check re-run watches the whole head (ADR 0010), while an Abandoned-job re-run gives None; see if !rerun_checks.all_requested { return Ok(None); } in rerun_abandoned_through, src/ci.rs. The shared part, request_reruns and watch_new_attempts, is already extracted.

Spec

  • P2: after a partial refusal, a CI-fix Repair runs while the accepted workflow run's new attempt is still in progress. Declined: the reviewer marks it consistent with the spec ("If the re-run is refused … carry on exactly as today for red CI"), and the all_requested check above does exactly that. ADR 0010's refusal path re-reads every check on the head, so a later Check re-run still settles correctly.

Closes #908

🤖 Generated with Claude Code

Built with claude · claude-opus-5-5 · medium

Security review outcome

No unaddressed introduced findings.

JacobStephens2 and others added 2 commits October 21, 2026 05:54
A GitHub Actions job GitHub ended as failed with no failed step and a step
not completed is an Abandoned job. When every one of the branch's own
failures on a head is one, the Repair loop re-runs them once per head for
the Delivery, with no Repair, and watches CI again behind the Check
re-run's guard. Mixed failures, a refused or unseen re-run, or a second
abandonment take the CI-fix Repair as before. ADR 0020 records why this is
not the re-run before every Repair that ADR 0010 rejected.

Closes #908

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…adable job is not abandoned

Steps are read only while every own failure may yet be an Abandoned job,
stopping at the first that is not, and a failed read is told and taken as
not abandoned, so red CI still gets its CI-fix Repair rather than failing
the Run. Names now use the glossary's Abandoned job, the re-run request
says whether every workflow run was requested, and the two per-head
allowances are documented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JacobStephens2
JacobStephens2 merged commit 2d8007b into main Oct 21, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-908 branch October 21, 2026 10:37
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-run a job that lost its runner instead of starting a CI-fix Repair

1 participant