Repository navigation
fix(nightly): refresh the App token across stages; a lost watch is a driver error - #578
Merged
Merged
Conversation
…driver error The nightly driver ran a 1.5-4.5 h job on one App installation token that lives 60 minutes (since 6f5f95e, 2026-10-23). Every scheduled night from 10-24 to 11-04 died at the 60-minute mark with 'HTTP 401: Bad credentials' inside the Stage 3 gh run watch and paged 'Stage 3 failed' over a run that had passed. - bin/gh_app_token.py: stdlib + openssl mint of a fresh installation token with the workflow's exact permission set (actions write, contents read, issues read, actions_variables write). - agents/conductors/release/_nightly_lib.sh: refresh_gh_token re-mints once the token is 45 min old; await_run polls a run (refreshing as it goes) and returns 4 when the run cannot be observed after five failed reads. - nightly.sh: refresh before every dispatch, download, read and the anchor write; a lost watch pages as a driver error (exit 1), never a stage failure. - nightly-release.yml: the driver step receives APP_CLIENT_ID/APP_PRIVATE_KEY for re-minting only; the initial mint step and the outcome contract stand. Closes #577 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The tenant firewall forbids a new organ file from naming the GitHub owner. The mint helper now requires APP_OWNER (set in nightly-release.yml beside the mint step's own owner:), and the test uses a neutral owner. Refs #577 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The nightly release driver runs for hours on one GitHub App installation token that lives 60 minutes (since 6f5f95e, 2026-10-23). Every scheduled night from 10-24 to 11-04 died at the 60-minute mark with
HTTP 401: Bad credentialswhile watching the Stage 3 Heart integrate run, and paged "Stage 3 failed" although that run passed on most nights. No nightly reached the readiness gate after the 2026-10-22 anchor; the 11-01 release was manual.NIGHTLY_RELEASESisfalsesince 2026-11-04 until this lands.This obeys the token policy as written ("mint it right before a long loop, not at the top of the job") with no PAT and the same four permissions:
bin/gh_app_token.py— stdlib +opensslmint of a fresh installation token; prints only the token, never logs the key or JWT.agents/conductors/release/_nightly_lib.sh—refresh_gh_tokenre-mints once the token is 45 min old (no-op without App credentials, so local runs on a personalghare unchanged);await_runpolls a run every minute, refreshing as it goes, and returns 4 after five consecutive failed reads.agents/conductors/release/nightly.sh— refresh before every dispatch, download, read and the anchor write; a watch that still loses its credentials pages as a driver error (exit 1, workflow outcomedriver-error), never as a stage failure. Step 6 usesnotifyrather thanpageon that path because the live release may have published..github/workflows/nightly-release.yml— the driver step receivesAPP_CLIENT_ID/APP_PRIVATE_KEYfor re-minting only; the initial mint step and the OUTCOME CONTRACT are unchanged.Fixes #577.
API Changes
None. No library or public Python API is touched; the driver's exit-code contract gains one documented meaning for exit 1 (a lost watch).
dispatch_and_awaitgains return code 4 (internal to the driver).Test Plan
python3 -m pytest -q tests/test_gh_app_token.py tests/test_nightly_await.py tests/test_activity_gate.py— 23 passedpython3 -m pytest -q -x(whole Brain suite) — 1558 passed, 1 skippedbash -non the driver and the new libnightly-release.ymldispatch withdry_run=truewhose log shows Stage 2 and Stage 3 complete, noHTTP 401, andstep 5 — readiness gatereached (Heart will likely be STALE, so it ends at exit 3 — that is the gate, not the bug). Then setNIGHTLY_RELEASES=true.Decisions flagged for review
actions_variables, matching the installation's own name (e0f88ca, 2026-10-23) and whatcreate-github-app-tokensends. If GitHub rejects it (HTTP 422 in the mint's stderr), the dry-run Witness will show it; the constant and the parity test change together.Full API Changes (for automation & release notes)
Added
bin/gh_app_token.py—make_jwt(client_id, private_key, now=None),mint(client_id, private_key, owner="PyAutoLabs", request=_request),INSTALLATION_PERMISSIONSagents/conductors/release/_nightly_lib.sh—refresh_gh_token [--force],await_run <repo> <run_id>; env overridesNIGHTLY_POLL_SECONDS,NIGHTLY_MINT_CMD(tests)Removed
gh run watchindispatch_and_awaitGenerated by the PyAutoLabs agent workflow.
🤖 Generated with Claude Code