-
Notifications
You must be signed in to change notification settings - Fork 0
ci(a11y): add PR checks with a shrink-only axe baseline #38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| # Pull-request checks for ChatCPT. | ||
| # | ||
| # Includes an axe-core accessibility gate against a checked-in shrink-only | ||
| # baseline (a11y-audits/tools/baseline.json). The baseline may only ever | ||
| # shrink — new rules or higher node counts fail the build. | ||
| # | ||
| # IMPORTANT: axe cannot detect live-region over-announcement (finding A1 in | ||
| # a11y-audits/8-5-26/audit.md). A green badge here is a floor, not WCAG | ||
| # conformance. Manual VoiceOver checks remain mandatory for live-region and | ||
| # focus work. | ||
|
|
||
| name: PR | ||
|
|
||
| on: | ||
| pull_request: | ||
|
|
||
| jobs: | ||
| check: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Populate design system submodule | ||
| run: git submodule update --init | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '20' | ||
| cache: 'npm' | ||
|
|
||
| - name: Install dependencies | ||
| run: npm ci | ||
|
|
||
| - name: Unit tests | ||
| run: npm test | ||
|
|
||
| - name: Build frontend bundle | ||
| run: npm run build | ||
|
|
||
| - name: Install accessibility audit tools | ||
| working-directory: a11y-audits/tools | ||
| run: npm ci | ||
|
|
||
| - name: Baseline helper unit tests | ||
| working-directory: a11y-audits/tools | ||
| run: npm test | ||
|
|
||
| - name: Install Playwright Chromium | ||
| working-directory: a11y-audits/tools | ||
| run: npx playwright install chromium --with-deps | ||
|
|
||
| - name: Start app server | ||
| # No Octavus credentials in CI — audit.mjs stubs /api/* when A11Y_CI=1. | ||
| # A placeholder AGENT_ID keeps the server from warning; the stub never | ||
| # reaches the create-session path. | ||
| run: | | ||
| PORT=3100 AGENT_TARGET=dev OCTAVUS_AGENT_ID=ci-placeholder \ | ||
| node server.js > /tmp/chatcpt-ci-server.log 2>&1 & | ||
| echo $! > /tmp/chatcpt-ci-server.pid | ||
| for i in $(seq 1 30); do | ||
| if curl -sf http://127.0.0.1:3100/ >/dev/null; then | ||
| echo "Server ready" | ||
| exit 0 | ||
| fi | ||
| sleep 1 | ||
| done | ||
| echo "Server failed to start" >&2 | ||
| cat /tmp/chatcpt-ci-server.log >&2 | ||
| exit 1 | ||
|
|
||
| - name: Axe scan (light + dark) against shrink-only baseline | ||
| working-directory: a11y-audits/tools | ||
| env: | ||
| A11Y_CI: '1' | ||
| A11Y_BROWSER_CHANNEL: bundled | ||
| A11Y_BASE_URL: http://127.0.0.1:3100 | ||
| A11Y_OUT: a11y-out-ci | ||
| A11Y_BASELINE: ${{ github.workspace }}/a11y-audits/tools/baseline.json | ||
| run: npm run audit | ||
|
|
||
| - name: Upload axe report | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: axe-report | ||
| path: a11y-audits/tools/a11y-out-ci/ | ||
| if-no-files-found: ignore | ||
|
|
||
| - name: Stop app server | ||
| if: always() | ||
| run: | | ||
| if [ -f /tmp/chatcpt-ci-server.pid ]; then | ||
| kill "$(cat /tmp/chatcpt-ci-server.pid)" || true | ||
| fi | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,3 @@ | ||
| node_modules/ | ||
| a11y-out/ | ||
| a11y-out-ci/ |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,7 +2,7 @@ | |
|
|
||
| The Playwright + axe-core scripts that produced the evidence in | ||
| [`../8-5-26/audit.md`](../8-5-26/audit.md). Kept in the repository so findings can be | ||
| re-measured after each fix, and because `audit.mjs` is the intended starting point for the | ||
| re-measured after each fix, and because `audit.mjs` drives the | ||
| CI accessibility gate ([#36](https://github.com/CodeSignal/learn_cosmo-chat/issues/36)). | ||
|
|
||
| These have their own dependency tree. They are **not** part of the application build and are | ||
|
|
@@ -37,10 +37,37 @@ npm run audit | |
| | `A11Y_BASE_URL` | `http://localhost:3100` | Where the app is running | | ||
| | `A11Y_OUT` | `a11y-out` | Directory for screenshots and `report.json` | | ||
| | `A11Y_BROWSER_CHANNEL` | `chrome` | Set to `bundled` to use Playwright's own Chromium | | ||
| | `A11Y_CI` | unset | `1` stubs `/api/*`, skips the live-agent send flow, and enables the baseline gate | | ||
| | `A11Y_BASELINE` | `./baseline.json` when `A11Y_CI=1` | Path to the shrink-only axe baseline | | ||
| | `A11Y_UPDATE_BASELINE` | unset | `1` rewrites the baseline from the current run | | ||
|
|
||
| `A11Y_BROWSER_CHANNEL` defaults to system Chrome because Playwright's bundled Chromium was | ||
| missing on the audit machine. In CI, run `playwright install chromium` and set it to `bundled`. | ||
|
|
||
| ### CI gate | ||
|
|
||
| `.github/workflows/pr.yml` runs: | ||
|
|
||
| ```bash | ||
| npm run audit:ci | ||
| ``` | ||
|
Comment on lines
+49
to
+53
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Correct the documented workflow command. This section states that 🤖 Prompt for AI Agents |
||
|
|
||
| That compares the axe results for empty / settings / settings-with-dropdown in both color | ||
| schemes against [`baseline.json`](./baseline.json). The baseline is **shrink-only**: a new | ||
| rule or a higher node count fails the build; a fix that removes violations should update | ||
| `baseline.json` in the same PR so the floor ratchets down. | ||
|
|
||
| ```bash | ||
| # after a fix that clears axe violations: | ||
| npm run audit:update-baseline | ||
| ``` | ||
|
|
||
| CI does not exercise streaming or populated-conversation states (no live agent). Those | ||
| axe results matched the empty-state shell in the original audit, and the settings / | ||
| dropdown states carry the distinctive rules (`aria-input-field-name`, the extra | ||
| `region` nodes, dark-mode contrast). Re-run the full `npm run audit` locally when a | ||
| fix needs the live-agent states. | ||
|
|
||
| ## What each script does | ||
|
|
||
| **`audit.mjs`** — the main sweep. For each of light and dark mode it walks the app through five | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: CodeSignal/learn_cosmo-chat
Length of output: 240
🏁 Script executed:
Repository: CodeSignal/learn_cosmo-chat
Length of output: 12994
Restrict the workflow token and disable checkout credential persistence.
The workflow runs pull-request-controlled scripts after checkout. Set
permissions: contents: readandpersist-credentials: falseforactions/checkout@v4.🧰 Tools
🪛 zizmor (1.29.0)
[warning] 22-23: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Source: Linters/SAST tools