release: require green checks and promote only tested immutable artifacts

[defangdevs]

Audit baseline: 0cc18b8, verified against current master on 2026-09-10.

Trigger and impact. The branch rules API returned only a pull-request rule with zero required approvals and resolved conversations; it returned no required status checks. The public-template workflow runs on pushes to master independently of CI and deployment tests. Thus broken or insufficiently reviewed code can become the public install default. Self-update also follows the repository branch rather than a separately promoted release.

Evidence. The rule snapshot is included in the evidence bundle. See publication trigger, independent deploy-test trigger, and update target resolution. Publication also resolves an external unstable channel during the publishing job; an immutable source commit alone is not a complete release manifest.

Smallest useful fix. Require a stable aggregate CI status and one independent approving review for release-impacting changes. Ensure path-filtered jobs cannot leave the required aggregate pending forever. Build a candidate once, record its exact source/dependency/template identities, run deployment tests against that candidate, then explicitly promote those same artifacts. Point supported installation and update paths to that promoted release. Start with explicit promotion this week; a new release orchestration framework is unnecessary.

Acceptance. A failing or missing check blocks merge/promotion; a failed fresh-boot test leaves public defaults unchanged; an installed box reports the same revision and dependency pins as the tested candidate. Demonstrate rollback.

Priority: P1, a release blocker for the affected supported capability.

Scope: agent-box owns this integration and its fix. Do not change vendored/upstream behavior that already matches its documented contract.

[defangdevs]

PR #648 addresses this. Four of the five pieces are in it and tested; one needs a repo admin, and one is a question only a maintainer can answer. Both are below, precisely.

What #648 does

• Step 2 first, because step 1 is impossible without it. The path filters moved off the workflow triggers into .github/path-filters/*.paths, and each gated workflow now ends in a terminal job that runs with if: always(). So CI gate, AWS template gate and Azure template gate are reported on every push and every pull request - including one that touches nothing they cover, where the gate says "nothing to run" instead of saying nothing at all. A path-filtered job could never have been required: it reports no check run when it does not start, so the PR would pend forever.
• Step 3. scripts/release_manifest.py records the candidate's rev, module SRI hash, flake ref, flake.lock hash, the resolved nixpkgs channel snapshot url + hash, and a hash per template - once. promote.yml refuses a candidate whose gates are not green for that exact sha (an absent gate is a failure), builds the manifest, boots that candidate in deploy-test with those pins, publishes the same manifest's pins, and only then tags it and cuts a Release carrying the manifest.
• Step 4, install half. publish-template.yml no longer triggers on push and computes nothing; the S3 templates carry a promoted candidate's pins, so a 1-click launch starts at the tested release.
• Rollback, demonstrated in the PR body against a scratch repo with a real remote.

One thing worth recording from the investigation, because the issue's phrasing understates it: deploy-test pinned AgentBoxRev/AgentBoxSha256 and left AgentNixpkgsUrl/AgentNixpkgsSha256 empty, while publish-template resolved that channel itself. So it was not merely that publishing ran independently of the deployment test - the two never described the same box, even when both were green.

Needs a repo admin: add the three gates as required status checks

I have the token rights to do this and did not, on purpose: the gates do not exist on any currently-open PR's branch, so requiring them today would make the five in-flight PRs unmergeable. #648 has to land first, and each open PR then needs its branch updated once (a push, or the "Update branch" button) so its merge commit picks up the new workflows.

After #648 merges, this is the exact call. The ruleset is 21491459; a PUT replaces the whole rules array, so the existing pull_request rule is repeated verbatim below and must not be dropped. 15368 is the GitHub Actions integration id, confirmed against real check runs on this repo.

gh api -X PUT repos/defangdevs/agent-box/rulesets/21491459 --input - <<'JSON'
{
  "name": "Require green checks and conversation resolution",
  "target": "branch",
  "enforcement": "active",
  "conditions": { "ref_name": { "include": ["~DEFAULT_BRANCH"], "exclude": [] } },
  "rules": [
    {
      "type": "pull_request",
      "parameters": {
        "required_approving_review_count": 0,
        "dismiss_stale_reviews_on_push": false,
        "required_reviewers": [],
        "require_code_owner_review": false,
        "require_last_push_approval": false,
        "required_review_thread_resolution": true,
        "require_extra_approval_for_unattributed_changes": true,
        "allowed_merge_methods": ["merge", "squash", "rebase"]
      }
    },
    {
      "type": "required_status_checks",
      "parameters": {
        "strict_required_status_checks_policy": false,
        "do_not_enforce_on_create": false,
        "required_status_checks": [
          { "context": "CI gate", "integration_id": 15368 },
          { "context": "AWS template gate", "integration_id": 15368 },
          { "context": "Azure template gate", "integration_id": 15368 }
        ]
      }
    }
  ]
}
JSON

Verify with gh api repos/defangdevs/agent-box/rules/branches/master.

Two settings in there are deliberate choices, not defaults:

• strict_required_status_checks_policy: false ("branch need not be up to date before merging"). true forces a rebase of every PR whenever master moves, which on this repo means constant churn across parallel agent PRs. It also buys less here than it looks: pull_request runs build the merge commit, so CI already tests the PR's result against the current base. Set it to true only if you want the stricter serialization.
• "context": "CI gate" matches the job's name:, not its key. If those names are ever changed in the workflows, the ruleset silently starts requiring a check nothing reports - which is the failure this whole issue is about, so the three names are load-bearing and the workflows say so in comments.

Needs a maintainer decision: the approving-review count

The issue asks for "one independent approving review", and I have not set it, because on this repo it is a policy change with a cost rather than a mechanical fix: defangdevs (this box) authors nearly every PR, so required_approving_review_count: 1 means a human must approve every change before anything can land, including the routine ones. That may well be right for a P1 security posture - it is just not a call I should make silently.

Three shapes, in the same rule as above:

1. 0 (unchanged). Green checks gate merges; nothing gates review. Weakest against the "under-reviewed code" half of this issue, and note that require_extra_approval_for_unattributed_changes: true is already on, which covers some of it.
2. 1. What the issue asks for. Blocks autonomous merging entirely.
3. 1 on a separate ruleset scoped to release-impacting paths (deploy/**, modules/**, .github/workflows/**), with 0 elsewhere. Closest to the issue's own wording ("for release-impacting changes") and keeps routine work moving. Costs a second ruleset.

My recommendation is 3, with 2 as the answer if you would rather not maintain a path-scoped ruleset. Either way promote.yml is a second, independent gate that a ruleset bypass does not reach: it re-checks all three gates for the exact candidate sha before anything becomes a public default.

One more, if a ruleset is ever added over release

promote.yml's record job pushes the release-* tag and the release branch pointer with contents: write. The current ruleset only targets ~DEFAULT_BRANCH, so nothing conflicts today - but if release is later protected, the Actions token needs a bypass entry or promotion will fail at its last step, after publishing has already happened.

The one part left open by design

Boxes' self-update still follows the tracked branch (master), not the promoted release. The mechanism needs no code - agentbox update --branch release, or selfUpdate.branch = "release" - and #648 documents it, but changing the shipped default changes what every existing box updates to, and it needs a promotion to exist first. Happy to do it as a follow-up once there is a release pointer to follow; say the word and I will open it.

[defangdevs]

Code review completed on #648 at 1f930e9. Five findings are posted inline: #648 (review)

The issue must remain open: manual publication / skip-deploy bypasses, rebuilding the rollback manifest against a moving channel, hashing the dispatch checkout instead of the selected candidate, no Lightsail/native fresh-boot gate for the published Lightsail artifact, and the rollback ancestry check occurring after publication. The local manifest and path-filter suites pass, but do not exercise these workflow paths. Required repository checks/release update policy also remain explicit acceptance work as already noted by the author.