Skip to content

feat: gate pull requests on changed-line coverage and report it - #66

Open
aryanjasala wants to merge 3 commits into
feat/ci-resultfrom
feat/coverage-report
Open

aryanjasala wants to merge 3 commits into
feat/ci-resultfrom
feat/coverage-report

Conversation

@aryanjasala

@aryanjasala aryanjasala commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

What this PR does

The workflow half of #36. With coverage: true, PHPUnit and Jest measure coverage, wp-tooling coverage-gate fails the job when too few of the lines the PR changed are covered, the HTML report is uploaded as an artifact, and an opt-in workflow posts the summary as a comment on the PR. With coverage off, every job runs the same steps as before.

Closes

Closes #36

Changes

  • ci-test-php.yml and ci-test-js.yml gain coverage (default false), coverage-threshold (80), coverage-artifact-name, base-ref and wp-tooling-ref. With coverage on:

    • the checkout keeps full history for the merge-base diff;
    • PHPUnit writes coverage/clover.xml and coverage/html (Xdebug inside wp-env, PCOV when standalone); Jest writes coverage/lcov.info and its HTML report to an absolute --coverageDirectory, because Jest resolves it against a rootDir the starters move;
    • coverage-gate runs with --output github: step outputs, annotations, and a per-file table of missed lines in the job summary, labelled with the job (PHPUnit, PHP 8.3, WP 6.8, plugins/acme);
    • the coverage/ directory, including that table as summary.md, is uploaded even when the gate or the tests fail, which is when it is wanted most (14-day retention).
  • wp-env and Xdebug. wp-env bakes Xdebug into the image it builds, and any later wp-env start without --xdebug rebuilds it without. The starters run PHPUnit in a separate .wp-env.tests.json environment that their pretest:php hook starts. So with coverage on, the workflow starts that environment itself with --xdebug=coverage and the default command becomes npm run --ignore-scripts test:php --, which stops the hook from restarting it. Stop wp-env stops the same config.

  • wp-ci.yml gains coverage and coverage-threshold. It measures test-js and only the first test-php matrix cell (several cells would multiply the cost and could disagree), names the artifacts per unit (coverage-php, coverage-js, suffixed with the directory in a monorepo), forwards base-ref and wp-tooling-ref, and fails fast in validate-inputs when test-php-exclude drops that first cell.

  • New ci-coverage-comment.yml posts every summary.md of the run as one PR comment and updates it on later runs (or says nothing was measured). Fork PRs get a notice instead, since their token is read-only, and a refused comment (a Dependabot PR, say) is a warning, not a failure.

  • Examples for all three workflows and a README Coverage section.

  • Review fixes (543bbc6):

    • the comment's marker carries a hash of the caller's workflow name, and only a github-actions[bot] comment is ever edited, so a contributor's comment that quotes the marker is left alone and two caller workflows in one repository keep separate comments (checked against a stubbed gh: it edits the bot's own comment, and posts a new one when only a human's or another workflow's exists);
    • standalone runs set pcov.directory to the unit, because PCOV otherwise records only src/, lib/ or app/ when one exists and the starters keep PHP in inc/ (not yet run on a real runner);
    • base-ref and wp-tooling-ref in wp-ci.yml say they also drive the coverage gate, and that a branch is written origin/<branch>.

How I verified

$ actionlint .github/workflows/*.yml examples/*.yml
$ bin/check-workflows.sh
Workflow/example contract check passed.
$ pipx run yamllint .   # only the existing warning on main
  • validate-inputs with coverage on: a plain matrix passes and emits coverage-php-artifact=coverage-php; test-php-exclude dropping the first cell ({"php":"8.2","wp":"6.7"} or just {"php":"8.2"}) fails with the error; working-dir: themes/acme gives coverage-php-themes-acme.
  • coverage-gate from @rtcamp/wp-tooling 2.0.0 on a scratch repo, to confirm the summary format the label rewrite expects (### Changed-line coverage as the first line).
  • The comment script with a stubbed gh on all four paths: new comment, update of an existing comment to "No coverage was measured", no summaries and no earlier comment (no call), fork PR (notice, no call).

Not done yet, and needed before merge: the end-to-end check from #36 on a real consumer (a covered change passes, a later commit that adds an untested branch fails and names the lines, a new untested file shows at 0%, then the fix passes). Docker was unavailable when I built this, so the wp-env and PCOV paths have only been linted, not run.

Acceptance criteria

Runtime behavior

Code quality

  • Lint: zero errors
  • Static analysis: zero errors (zizmor: two new adhoc-packages notes for the wp-tooling install, the same pattern ci-detect-changes.yml already uses)
  • Tests pass (step scripts and the comment script against the cases above)

Housekeeping

  • No CHANGELOG.md edit: release-please writes it from the PR title
  • examples/<workflow>.yml added or updated to match any input change
  • yamllint, actionlint and bin/check-workflows.sh pass locally

Reviewer notes

ci-test-php.yml and ci-test-js.yml take an opt-in coverage input (and
wp-ci.yml forwards coverage and coverage-threshold). With it on, the
tests run with coverage, @rtcamp/wp-tooling's coverage-gate checks the
changed lines against the threshold and writes a job summary, and the
report is uploaded as a coverage-php / coverage-js artifact.

ci-coverage-comment.yml is a separate workflow that turns those
artifacts into one pull request comment. It is its own opt-in because
it needs pull-requests: write, and adding that to a job callers already
use would fail validation for every caller that doesn't grant it.

Uses @rtcamp/wp-tooling 2.0.0's coverage-gate, already on npm.
The coverage comment was the first one starting with a fixed marker,
whoever wrote it, so it could overwrite a contributor's comment, and two
caller workflows in one repository overwrote each other's. The marker
now carries a hash of the caller's workflow name, and only a comment
from github-actions[bot] is ever edited.

Standalone runs set pcov.directory to the unit: PCOV otherwise records
only src/, lib/ or app/ when one exists, and the starters keep PHP in
inc/. base-ref and wp-tooling-ref now say they also drive the coverage
gate, and that a branch is written origin/<branch>.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant