Repository navigation
Description
Overview
Since the 2026-10-23 token consolidation (6f5f95e), the nightly release driver runs a 1.5–4.5 h job on a single GitHub App installation token that lives 60 minutes. Every scheduled night from 10-24 to 11-04 died at the 60-minute mark with HTTP 401: Bad credentials while watching the Stage 3 Heart integrate run, and paged "Stage 3 failed" although the integrate run passed on most of those nights. No nightly has reached the readiness gate since the 2026-10-22 anchor; the 11-01 release was manual. NIGHTLY_RELEASES was set to false on 2026-11-04 until this lands, so the false pages stop.
The token policy (PyAutoMind/policy/github_tokens.md) already states the rule: "The token lives 1 h, so mint it right before a long loop, not at the top of the job." This task makes the driver obey it, in policy (no PAT), with the smallest change that also makes a lost credential page honestly.
Plan
- Give the driver its own in-policy token mint: a stdlib helper that signs the App JWT with the runner's
openssland exchanges it for an installation token with exactly the four permissions the workflow documents. - Refresh the token before every GitHub call that follows a long wait: inside the run watch, before artifact downloads and the capture-heads reads, before the Stage 3 and live-release dispatches, and before the anchor write.
- Replace the single blocking
gh run watchwith a one-minute polling loop that refreshes on schedule and retries a bounded number of API failures with a forced re-mint. - Separate "could not observe the run" from "the run failed": the former pages as a driver credential error and exits 1 (workflow outcome
driver-error); the latter keeps today's Stage 2 / Stage 3 pages. - Keep the workflow's initial mint step (a failed mint still stops the night before any gate); pass the App client id and private key to the driver step for re-minting only.
- One fake-
ghtest of the watch loop; the Witness is a manualdry_run=truedispatch that passes the 60-minute mark with no 401 and reaches step 5.
Tier: judge — merge mode: human /prm
Detailed implementation plan
Affected Repositories
- PyAutoBrain (primary, only)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./PyAutoBrain | main (28fb917) | clean |
Conflict guard clear. Unregistered worktree feature/pyautodna-stack-management exists (warning only).
Suggested branch: feature/nightly-token-refresh
Work Classification: Library (organ)
Worktree root: ~/Code/PyAutoLabs-wt/nightly-token-refresh/
Implementation Steps
bin/gh_app_token.py(new, stdlib only). ReadsAPP_CLIENT_ID/APP_PRIVATE_KEYfrom the environment; RS256 JWT viaopenssl dgst -sha256 -signon a temp file (never logged);GET /app/installations→ the PyAutoLabs install;POST /app/installations/{id}/access_tokenswith{"permissions": {"actions": "write", "contents": "read", "issues": "read", "variables": "write"}}; prints only the token. HTTP goes through one injectable function so tests never touch the network.agents/conductors/release/_nightly_lib.sh(new, sourced by the driver):refresh_gh_token [--force]— no-op when the App credentials are absent (local runs on a personalghunchanged); otherwise re-mints when the token is older than 45 min (GH_TOKEN_MINTED_AT, epoch) and exportsGH_TOKEN.await_run <repo> <run_id>— every 60 s: refresh,gh run view --json status,conclusion; on aghfailure force a re-mint and count; 5 consecutive failures → return 4 (unobservable); completed → 0 on success, 2 otherwise. Extracted so it can be unit-tested (the driver runs top-level code when sourced).agents/conductors/release/nightly.sh:dispatch_and_awaitusesawait_run, refreshes beforegh run download, propagates 4. Stage 2, Stage 3 and step 6 callers add the sibling branch: on 4,page "driver lost its GitHub credentials while watching <run> — the run was not judged"andexit 1. Addrefresh_gh_tokenbefore capture-heads, the step 4cci_status.sh/open_prs.shchecks,released_todayandadvance_anchor. Header: document the 60-minute token lifetime and the refresh rule..github/workflows/nightly-release.yml: driver step env gainsAPP_CLIENT_ID: ${{ vars.APP_CLIENT_ID }},APP_PRIVATE_KEY: ${{ secrets.APP_PRIVATE_KEY }},GH_TOKEN_MINTED_AT(set fromdate +%sin the mint step's successor); correct the timing comment (the token lives 60 min; the job up to 4.5 h). Outcome contract unchanged.tests/test_nightly_await.py(new): fakeghon a lean PATH (pattern:tests/test_branch_sweep.py::_gh_free_path) scripted for (a) 401 twice then success → 0, (b) 401 forever → 4, (c) conclusion failure → 2; andrefresh_gh_tokenis a no-op without credentials. Light check intests/test_gh_app_token.pythat the JWT header/claims are well-formed and the permission dict equals the workflow's mint.- Witness (ship-time): one manual
nightly-release.ymldispatch withdry_run=truewhose log shows Stage 2 and Stage 3 complete, noHTTP 401, andstep 5 — readiness gatereached. (Heart will likely be STALE, so the run ends at exit 3 — that is the gate, not the bug.)
Key Files
agents/conductors/release/nightly.sh— the driver;dispatch_and_await~L313, Stage 2 ~L378, Stage 3 ~L410, step 6 ~L470.github/workflows/nightly-release.yml— the single mint at L101 and the driver step env at L114PyAutoMind/policy/github_tokens.md— the rule this obeys; the "App mints" table row for Brainnightly-releaseis unchanged (same permissions)
Rejected alternatives
- Restore
PAT_PYAUTOLABS: the secret no longer exists and the 10-28 policy forbids PATs in CI. - Split the driver into per-stage jobs each minting a token: large restructure, artifact plumbing, breaks the single-driver outcome contract.
- Watch with
GITHUB_TOKEN: step 6's live dispatch still needs actions write on PyAutoHands.
Trade-off flagged
The App private key enters the driver step's environment. The same job already holds it for the mint action; the driver is deterministic shell with no tracing and never echoes its environment.
Original Prompt
Click to expand starting prompt
Nightly release driver dies at the 60-minute mark: the single App token expires during the Stage 3 gh run watch, and the 401 is paged as "Stage 3 failed"
Type: bug
Target: PyAutoBrain
Repos:
- PyAutoBrain
Themes: - release-validation
- ci
Difficulty: medium
Autonomy: supervised
Priority: high
Status: formalised
Consequence: judge
Witness: a manually dispatchednightly-release.ymlrun (dry_run=true) reaches step 5 (the readiness gate) with Stage 2 and Stage 3 both completed and noHTTP 401anywhere in its log, more than 60 minutes after the token mint; and a driver test shows agh run watchthat fails while the watched run's conclusion issuccessis classifieddriver-error(exit 1, paged as a credential failure), neverblocked(exit 2, "Stage 3 failed").
Lane: any
Review-minutes: 10
Unattended: never
Filed: 2026-11-04
Observed
Every scheduled nightly from 2026-10-29 to 2026-11-04 ends the same way, about 60
minutes after the job starts, while the driver is watching the PyAutoHeart
release-integrate.yml run (Stage 3):
failed to get run: HTTP 401: Bad credentials (https://api.github.com/repos/PyAutoLabs/PyAutoHeart/actions/runs/41804300905?exclude_pull_requests=true)
nightly: failed run's artifact 'release-stage-report' not downloadable
🚨 *nightly release stopped* — Stage 3 (release-fidelity integration) failed — <...runs/41804300905|run>
##[warning]The driver stopped at a gate (exit 2); no release was made. This is the gate working
Token expired, skipping token revocation
| Nightly run (PyAutoBrain) | Start | 401 at | Heart integrate run it was watching | Its real conclusion |
|---|---|---|---|---|
| 40840009792 | 2026-10-29 08:08 | 09:08 | 40842185049 | success |
| 41024762702 | 2026-10-30 08:11 | 09:11 | 41026647430 | failure (autogalaxy guides, real) |
| 41183039794 | 2026-10-31 08:03 | 09:04 | 41184365143 | failure (autogalaxy guides, real) |
| 41290517021 | 2026-11-01 08:50 | 09:51 | 41291876649 | success |
| 41416875388 | 2026-11-02 08:58 | 09:58 | 41419508621 | success |
| 41608230314 | 2026-11-03 09:06 | 10:06 | 41611102721 | success |
| 41801624733 | 2026-11-04 08:43 | 09:43 | 41804300905 | success |
Five of the seven nights the integrate run actually passed; the page said it failed.
No nightly has reached step 5 (the readiness gate) since the activity anchor
NIGHTLY_LAST_WINDOW_END = 2026-10-22T07:13:10Z, so the 2026-11-01 release had to
be dispatched by hand. The standing grant is armed (NIGHTLY_RELEASES=true) but
inoperative.
Cause
@PyAutoBrain/.github/workflows/nightly-release.yml mints one GitHub App
installation token (actions/create-github-app-token@v3, step app_token) and
hands it to @PyAutoBrain/agents/conductors/release/nightly.sh as GH_TOKEN for
the whole run. App installation tokens live 60 minutes. The driver's Stage 2
rehearsal takes about 20 minutes and the Stage 3 integrate run 40 to 80 minutes,
so the token expires inside dispatch_and_await's gh run watch (nightly.sh,
around line 341). gh run watch returns non-zero, the function returns 2, and the
Stage 3 branch at line ~410 pages "Stage 3 (release-fidelity integration) failed"
and exits 2 — the outcome the workflow classifies as blocked ("the gate
working").
The step 0b token sanity check (gh api rate_limit, line ~167) only catches a
token that is dead at the start; it cannot see mid-run expiry. Nothing
distinguishes "the watched run failed" from "I lost the ability to watch".
Ask
Two defects, one PR:
- Keep a valid token for the whole run. Options to weigh in the plan, not
decided here: mint a fresh token before Stage 3 (and before step 6) — e.g. the
driver re-mints fromAPP_CLIENT_ID/APP_PRIVATE_KEYvia the App JWT flow
when the token is older than ~50 minutes, or the workflow splits the stages
into separate jobs that each mint their own token and pass artifacts between
them, ordispatch_and_awaitretriesgh run watchafter re-minting on a 401.
Whatever is chosen, the token's permission set must stay exactly the one the
workflow documents (actions write, contents read, issues read, variables
write) — no PAT fallback, no widening. - Classify a lost watch honestly. When
gh run watchfails, re-read the
run'sconclusionwith a working token before judging:success→ continue;
failure→ the existing Stage 3 page; could not read → page as a driver
credential error (exit 1,driver-error), never as a gate stop. The Slack
page and the Actions warning must not say a stage failed when it passed.
Add a test beside the existing driver tests (tests/ in PyAutoBrain) for the
classification, and verify the token fix with one manual dry_run=true dispatch
that runs past the 60-minute mark.
Not in scope
- The
NIGHTLY_LAST_WINDOW_ENDanchor is behaving as designed (it advances only on
a shipped or empty window); after the fix the first green night will release
the whole window since 2026-10-22, which is intended. - The 10-30 and 10-31 integrate failures were real (autogalaxy guides scripts) and
are already fixed on main. - Heart's own readiness verdict is unaffected; it reads the rehearsal and
integrate artifacts regardless of the driver.
Evidence pointers
- Nightly runs: https://github.com/PyAutoLabs/PyAutoBrain/actions/workflows/nightly-release.yml
- Latest: https://github.com/PyAutoLabs/PyAutoBrain/actions/runs/41801624733
- Heart integrate it was watching (passed): https://github.com/PyAutoLabs/PyAutoHeart/actions/runs/41804300905
- Found 2026-11-04 in the release check-in session (release board review).
Mind prompt: draft/bug/pyautobrain/nightly_app_token_expires_during_stage3_watch.md
Activity
Jammy2211 commentedon Nov 4, 2026
Shipped as #578 (branch
feature/nightly-token-refresh, commit ca69c14). Targeted tests 23 passed; full Brain suite 1558 passed, 1 skipped. Judge tier: merge by human/prm.After merge, before re-arming: run one manual
nightly-release.ymldispatch withdry_run=trueand confirm the log reachesstep 5 — readiness gatewith noHTTP 401(the Witness). Then setNIGHTLY_RELEASES=true(it isfalsesince 2026-11-04 18:00 UTC).fix(nightly): APP_OWNER comes from the workflow, not the helper
Jammy2211 commentedon Nov 4, 2026
Shipped — PR #578 merged 2026-11-04 18:34 UTC (merge commit 61a3f87). CI green on both legs after the tenant-firewall fix (61a9e08:
APP_OWNERnow comes from the workflow, not the helper).The nightly stays paused (
NIGHTLY_RELEASES=false). Before re-arming, run the Witness: one manualnightly-release.ymldispatch withdry_run=truethat reachesstep 5 — readiness gatewith noHTTP 401in its log. Then setNIGHTLY_RELEASES=true.Jammy2211 commentedon Nov 4, 2026
Witness PASSED — manual
nightly-release.ymldispatchdry_run=trueon main after #578: https://github.com/PyAutoLabs/PyAutoBrain/actions/runs/41889750490 (18:42 → 19:58 UTC, 76 min).== nightly: GitHub token refreshedat 19:27 (45 min after the mint); Stage 3 run 41892492030 watched to completion at 19:54; zeroHTTP 401in the log.step 5 — readiness gate: RELEASE READINESS ✓ GREEN score 100 (release-ci profile);🧪 nightly dry-run complete — would have dispatched 2026.11.4.1.actions_variablespermission key.NIGHTLY_RELEASESis stillfalse; re-arming is the human's call (the next scheduled night would ship the window since 2026-10-22).Jammy2211 commentedon Nov 4, 2026
Re-armed:
NIGHTLY_RELEASES=trueset 2026-11-04 20:07 UTC by the human after the Witness passed. The next scheduled night (02:00 UTC) runs live against the window since the 2026-10-22 anchor.