Skip to content

fix(nightly): refresh the App token across stages; a lost watch is a driver error #577

Closed

Description

@Jammy2211

Overview

Since the 2026-10-23 token consolidation (6f5f95e), the nightly release driver runs a 1.5–4.5 h job on a single GitHub App installation token that lives 60 minutes. Every scheduled night from 10-24 to 11-04 died at the 60-minute mark with HTTP 401: Bad credentials while watching the Stage 3 Heart integrate run, and paged "Stage 3 failed" although the integrate run passed on most of those nights. No nightly has reached the readiness gate since the 2026-10-22 anchor; the 11-01 release was manual. NIGHTLY_RELEASES was set to false on 2026-11-04 until this lands, so the false pages stop.

The token policy (PyAutoMind/policy/github_tokens.md) already states the rule: "The token lives 1 h, so mint it right before a long loop, not at the top of the job." This task makes the driver obey it, in policy (no PAT), with the smallest change that also makes a lost credential page honestly.

Plan

  • Give the driver its own in-policy token mint: a stdlib helper that signs the App JWT with the runner's openssl and exchanges it for an installation token with exactly the four permissions the workflow documents.
  • Refresh the token before every GitHub call that follows a long wait: inside the run watch, before artifact downloads and the capture-heads reads, before the Stage 3 and live-release dispatches, and before the anchor write.
  • Replace the single blocking gh run watch with a one-minute polling loop that refreshes on schedule and retries a bounded number of API failures with a forced re-mint.
  • Separate "could not observe the run" from "the run failed": the former pages as a driver credential error and exits 1 (workflow outcome driver-error); the latter keeps today's Stage 2 / Stage 3 pages.
  • Keep the workflow's initial mint step (a failed mint still stops the night before any gate); pass the App client id and private key to the driver step for re-minting only.
  • One fake-gh test of the watch loop; the Witness is a manual dry_run=true dispatch that passes the 60-minute mark with no 401 and reaches step 5.

Tier: judge — merge mode: human /prm

Detailed implementation plan

Affected Repositories

  • PyAutoBrain (primary, only)

Branch Survey

Repository Current Branch Dirty?
./PyAutoBrain main (28fb917) clean

Conflict guard clear. Unregistered worktree feature/pyautodna-stack-management exists (warning only).

Suggested branch: feature/nightly-token-refresh
Work Classification: Library (organ)
Worktree root: ~/Code/PyAutoLabs-wt/nightly-token-refresh/

Implementation Steps

  1. bin/gh_app_token.py (new, stdlib only). Reads APP_CLIENT_ID / APP_PRIVATE_KEY from the environment; RS256 JWT via openssl dgst -sha256 -sign on a temp file (never logged); GET /app/installations → the PyAutoLabs install; POST /app/installations/{id}/access_tokens with {"permissions": {"actions": "write", "contents": "read", "issues": "read", "variables": "write"}}; prints only the token. HTTP goes through one injectable function so tests never touch the network.
  2. agents/conductors/release/_nightly_lib.sh (new, sourced by the driver): refresh_gh_token [--force] — no-op when the App credentials are absent (local runs on a personal gh unchanged); otherwise re-mints when the token is older than 45 min (GH_TOKEN_MINTED_AT, epoch) and exports GH_TOKEN. await_run <repo> <run_id> — every 60 s: refresh, gh run view --json status,conclusion; on a gh failure force a re-mint and count; 5 consecutive failures → return 4 (unobservable); completed → 0 on success, 2 otherwise. Extracted so it can be unit-tested (the driver runs top-level code when sourced).
  3. agents/conductors/release/nightly.sh: dispatch_and_await uses await_run, refreshes before gh run download, propagates 4. Stage 2, Stage 3 and step 6 callers add the sibling branch: on 4, page "driver lost its GitHub credentials while watching <run> — the run was not judged" and exit 1. Add refresh_gh_token before capture-heads, the step 4c ci_status.sh/open_prs.sh checks, released_today and advance_anchor. Header: document the 60-minute token lifetime and the refresh rule.
  4. .github/workflows/nightly-release.yml: driver step env gains APP_CLIENT_ID: ${{ vars.APP_CLIENT_ID }}, APP_PRIVATE_KEY: ${{ secrets.APP_PRIVATE_KEY }}, GH_TOKEN_MINTED_AT (set from date +%s in the mint step's successor); correct the timing comment (the token lives 60 min; the job up to 4.5 h). Outcome contract unchanged.
  5. tests/test_nightly_await.py (new): fake gh on a lean PATH (pattern: tests/test_branch_sweep.py::_gh_free_path) scripted for (a) 401 twice then success → 0, (b) 401 forever → 4, (c) conclusion failure → 2; and refresh_gh_token is a no-op without credentials. Light check in tests/test_gh_app_token.py that the JWT header/claims are well-formed and the permission dict equals the workflow's mint.
  6. Witness (ship-time): one manual nightly-release.yml dispatch with dry_run=true whose log shows Stage 2 and Stage 3 complete, no HTTP 401, and step 5 — readiness gate reached. (Heart will likely be STALE, so the run ends at exit 3 — that is the gate, not the bug.)

Key Files

  • agents/conductors/release/nightly.sh — the driver; dispatch_and_await ~L313, Stage 2 ~L378, Stage 3 ~L410, step 6 ~L470
  • .github/workflows/nightly-release.yml — the single mint at L101 and the driver step env at L114
  • PyAutoMind/policy/github_tokens.md — the rule this obeys; the "App mints" table row for Brain nightly-release is unchanged (same permissions)

Rejected alternatives

  • Restore PAT_PYAUTOLABS: the secret no longer exists and the 10-28 policy forbids PATs in CI.
  • Split the driver into per-stage jobs each minting a token: large restructure, artifact plumbing, breaks the single-driver outcome contract.
  • Watch with GITHUB_TOKEN: step 6's live dispatch still needs actions write on PyAutoHands.

Trade-off flagged

The App private key enters the driver step's environment. The same job already holds it for the mint action; the driver is deterministic shell with no tracing and never echoes its environment.

Original Prompt

Click to expand starting prompt

Nightly release driver dies at the 60-minute mark: the single App token expires during the Stage 3 gh run watch, and the 401 is paged as "Stage 3 failed"

Type: bug
Target: PyAutoBrain
Repos:

  • PyAutoBrain
    Themes:
  • release-validation
  • ci
    Difficulty: medium
    Autonomy: supervised
    Priority: high
    Status: formalised
    Consequence: judge
    Witness: a manually dispatched nightly-release.yml run (dry_run=true) reaches step 5 (the readiness gate) with Stage 2 and Stage 3 both completed and no HTTP 401 anywhere in its log, more than 60 minutes after the token mint; and a driver test shows a gh run watch that fails while the watched run's conclusion is success is classified driver-error (exit 1, paged as a credential failure), never blocked (exit 2, "Stage 3 failed").
    Lane: any
    Review-minutes: 10
    Unattended: never
    Filed: 2026-11-04

Observed

Every scheduled nightly from 2026-10-29 to 2026-11-04 ends the same way, about 60
minutes after the job starts, while the driver is watching the PyAutoHeart
release-integrate.yml run (Stage 3):

failed to get run: HTTP 401: Bad credentials (https://api.github.com/repos/PyAutoLabs/PyAutoHeart/actions/runs/41804300905?exclude_pull_requests=true)
nightly: failed run's artifact 'release-stage-report' not downloadable
🚨 *nightly release stopped* — Stage 3 (release-fidelity integration) failed — <...runs/41804300905|run>
##[warning]The driver stopped at a gate (exit 2); no release was made. This is the gate working
Token expired, skipping token revocation
Nightly run (PyAutoBrain) Start 401 at Heart integrate run it was watching Its real conclusion
40840009792 2026-10-29 08:08 09:08 40842185049 success
41024762702 2026-10-30 08:11 09:11 41026647430 failure (autogalaxy guides, real)
41183039794 2026-10-31 08:03 09:04 41184365143 failure (autogalaxy guides, real)
41290517021 2026-11-01 08:50 09:51 41291876649 success
41416875388 2026-11-02 08:58 09:58 41419508621 success
41608230314 2026-11-03 09:06 10:06 41611102721 success
41801624733 2026-11-04 08:43 09:43 41804300905 success

Five of the seven nights the integrate run actually passed; the page said it failed.
No nightly has reached step 5 (the readiness gate) since the activity anchor
NIGHTLY_LAST_WINDOW_END = 2026-10-22T07:13:10Z, so the 2026-11-01 release had to
be dispatched by hand. The standing grant is armed (NIGHTLY_RELEASES=true) but
inoperative.

Cause

@PyAutoBrain/.github/workflows/nightly-release.yml mints one GitHub App
installation token (actions/create-github-app-token@v3, step app_token) and
hands it to @PyAutoBrain/agents/conductors/release/nightly.sh as GH_TOKEN for
the whole run. App installation tokens live 60 minutes. The driver's Stage 2
rehearsal takes about 20 minutes and the Stage 3 integrate run 40 to 80 minutes,
so the token expires inside dispatch_and_await's gh run watch (nightly.sh,
around line 341). gh run watch returns non-zero, the function returns 2, and the
Stage 3 branch at line ~410 pages "Stage 3 (release-fidelity integration) failed"
and exits 2 — the outcome the workflow classifies as blocked ("the gate
working").

The step 0b token sanity check (gh api rate_limit, line ~167) only catches a
token that is dead at the start; it cannot see mid-run expiry. Nothing
distinguishes "the watched run failed" from "I lost the ability to watch".

Ask

Two defects, one PR:

  1. Keep a valid token for the whole run. Options to weigh in the plan, not
    decided here: mint a fresh token before Stage 3 (and before step 6) — e.g. the
    driver re-mints from APP_CLIENT_ID / APP_PRIVATE_KEY via the App JWT flow
    when the token is older than ~50 minutes, or the workflow splits the stages
    into separate jobs that each mint their own token and pass artifacts between
    them, or dispatch_and_await retries gh run watch after re-minting on a 401.
    Whatever is chosen, the token's permission set must stay exactly the one the
    workflow documents (actions write, contents read, issues read, variables
    write) — no PAT fallback, no widening.
  2. Classify a lost watch honestly. When gh run watch fails, re-read the
    run's conclusion with a working token before judging: success → continue;
    failure → the existing Stage 3 page; could not read → page as a driver
    credential error (exit 1, driver-error), never as a gate stop. The Slack
    page and the Actions warning must not say a stage failed when it passed.

Add a test beside the existing driver tests (tests/ in PyAutoBrain) for the
classification, and verify the token fix with one manual dry_run=true dispatch
that runs past the 60-minute mark.

Not in scope

  • The NIGHTLY_LAST_WINDOW_END anchor is behaving as designed (it advances only on
    a shipped or empty window); after the fix the first green night will release
    the whole window since 2026-10-22, which is intended.
  • The 10-30 and 10-31 integrate failures were real (autogalaxy guides scripts) and
    are already fixed on main.
  • Heart's own readiness verdict is unaffected; it reads the rehearsal and
    integrate artifacts regardless of the driver.

Evidence pointers

Mind prompt: draft/bug/pyautobrain/nightly_app_token_expires_during_stage3_watch.md

Activity

  1. Jammy2211 commented on Nov 4, 2026

    @Jammy2211
    ContributorAuthor

    Shipped as #578 (branch feature/nightly-token-refresh, commit ca69c14). Targeted tests 23 passed; full Brain suite 1558 passed, 1 skipped. Judge tier: merge by human /prm.

    After merge, before re-arming: run one manual nightly-release.yml dispatch with dry_run=true and confirm the log reaches step 5 — readiness gate with no HTTP 401 (the Witness). Then set NIGHTLY_RELEASES=true (it is false since 2026-11-04 18:00 UTC).

  2. Jammy2211 commented on Nov 4, 2026

    @Jammy2211
    ContributorAuthor

    Shipped — PR #578 merged 2026-11-04 18:34 UTC (merge commit 61a3f87). CI green on both legs after the tenant-firewall fix (61a9e08: APP_OWNER now comes from the workflow, not the helper).

    The nightly stays paused (NIGHTLY_RELEASES=false). Before re-arming, run the Witness: one manual nightly-release.yml dispatch with dry_run=true that reaches step 5 — readiness gate with no HTTP 401 in its log. Then set NIGHTLY_RELEASES=true.

  3. Jammy2211 commented on Nov 4, 2026

    @Jammy2211
    ContributorAuthor

    Witness PASSED — manual nightly-release.yml dispatch dry_run=true on main after #578: https://github.com/PyAutoLabs/PyAutoBrain/actions/runs/41889750490 (18:42 → 19:58 UTC, 76 min).

    • == nightly: GitHub token refreshed at 19:27 (45 min after the mint); Stage 3 run 41892492030 watched to completion at 19:54; zero HTTP 401 in the log.
    • Reached step 5 — readiness gate: RELEASE READINESS ✓ GREEN score 100 (release-ci profile); 🧪 nightly dry-run complete — would have dispatched 2026.11.4.1.
    • The mint accepted the actions_variables permission key.

    NIGHTLY_RELEASES is still false; re-arming is the human's call (the next scheduled night would ship the window since 2026-10-22).

  4. Jammy2211 commented on Nov 4, 2026

    @Jammy2211
    ContributorAuthor

    Re-armed: NIGHTLY_RELEASES=true set 2026-11-04 20:07 UTC by the human after the Witness passed. The next scheduled night (02:00 UTC) runs live against the window since the 2026-10-22 anchor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions