An action a release depends on is rehearsed on pull requests, or named - #186
Merged
Merged
Conversation
Dependabot opened three major bumps and the honest answer to "can we take them?" was "we do not know". That is a poor answer from a supply-chain tool, so it was measured instead. All three PRs showed 26 checks green. For `actions/checkout` that means something: roughly twenty jobs run it. For `oras-project/setup-oras` the green was VACUOUS — the action appears only in `deposit-layer.yml`, which is `workflow_dispatch`, and the OCI systest downloads its own pinned `oras` binary rather than using the action. Twenty-six checks passed and not one had executed the thing being upgraded. That is this repository's recurring shape: a check that first speaks after the tag is documentation. An action first exercised by the release is first exercised at the point of no return. `action_coverage.rs` makes the gap mechanical. Every action a tag-time workflow depends on must either be exercised by a workflow that runs on pull requests, or be named UNREHEARSABLE with the reason it cannot be. It found two: `setup-oras` and `download-artifact`. Both are now rehearsed with the SAME action at the SAME pin the release uses: * `oci-roundtrip.sh` already prefers an `oras` on PATH, so installing it with the action turns the existing push/pull round trip into a real rehearsal — the action installs the client, and the client pushes and pulls a genuine layer through a genuine registry. A step asserts the binary is actually on PATH first, because a silent fall-back to the script's own download would rehearse nothing while still passing. * the coverage job downloads the artifact it just uploaded and compares bytes. `release.yml` collects every platform's archives this way; an action that returns a different SHAPE would publish the wrong file set. Two remain unrehearsable, each named with why: `attest-build-provenance` mints a real attestation on a public transparency log, and `crates-io-auth-action` exchanges OIDC for a real publish token — for both, performing the operation IS the privileged act, so a rehearsal would not be a test. A second gate refuses a stale entry, so the excuse list cannot outlive its reasons. Both gates negative-controlled. The first draft of the parser missed the `- uses:` spelling and reported `cosign-installer` as unrehearsed when ci.yml had been running it all along — a wrong answer, not a smaller one, which is why the fix is called out in the code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019TNtfRjLNhEz82G2ggeeNu
avrabe
force-pushed
the
ci/release-rehearsal
branch
from
September 24, 2026 10:05
554efe8 to
1a2beb1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prompted by the three open dependabot majors and the unsatisfying answer "we
cannot upgrade because we do not know".
What was measured
All three PRs show 26 checks green. That means different things:
actions/checkout4→7cosign-installer3→4rivetjob verifies a blob with itsetup-oras1→2setup-orasappears only indeposit-layer.yml(workflow_dispatch), and theOCI systest downloads its own pinned
orasrather than using the action. Sotwenty-six checks passed without executing the thing being upgraded, and the
break would have surfaced at deposit time.
The gate
action_coverage.rs: every action a tag-time workflow depends on must beexercised by a workflow that runs on pull requests, or be named
UNREHEARSABLEwith a reason. It found two gaps —setup-orasanddownload-artifact— both now rehearsed with the same action at the same pinthe release uses:
oci-roundtrip.shalready prefers anorason PATH, soinstalling it via the action turns the existing round trip into the
strongest rehearsal available: the action installs the client, the client
pushes and pulls a real layer through a real registry. A step first asserts
the binary is on PATH, because a silent fall-back to the script's own
download would rehearse nothing while still passing.
uploaded and compares bytes.
release.ymlcollects every platform'sarchives this way; an action returning a different shape would publish the
wrong file set.
Two stay unrehearsable and say why:
attest-build-provenancemints a realattestation on a public transparency log, and
crates-io-auth-actionexchanges OIDC for a real publish token. For both, performing the operation
is the privileged act. A second test refuses a stale entry so the excuse
list cannot outlive its reasons.
Controls
Both negative-controlled: removing the oras rehearsal reopens the gap; a
fictional excused action is caught as stale.
The parser's first draft missed the
- uses:spelling and reportedcosign-installeras unrehearsed while ci.yml had been running it all along —a wrong answer, not a smaller one. The fix is called out in the code,
because a coverage tool that under-reports is worse than none.
What this unblocks
After this lands, #158 can be decided on evidence: rebase it and the systest
either round-trips a layer through oras v2 or it does not.
Gate: fmt · clippy -D warnings · 1030 tests, 0 failed · trace-gate OK.
🤖 Generated with Claude Code
https://claude.ai/code/session_019TNtfRjLNhEz82G2ggeeNu