Skill eval suite Phase 2: prose-register, choice-only gate, and CI

[nonrational]

Problem

Phase 1 (#58) left the eval suite unfit to gate anything. A keyword-overlap check failed a different correct discrimination answer every run. prose-register's rank and structural cases couldn't run at all. Nothing ran in CI, and the judge needed an ANTHROPIC_API_KEY that bills the API.

Motivation

A flaky or never-run check teaches everyone to ignore it. Phase 2 makes the suite something CI can run on every skill PR. Real keyed runs, not just unit tests, drove the design: they turned up four promptfoo behaviors the offline tests couldn't see.

Proposed Solution

Spec revisions (decisions 5–9) and the plan are in docs/superpowers/specs/2026-08-29-skill-eval-suite-design.md and docs/superpowers/plans/2026-09-19-skill-eval-suite-phase2.md. One commit per concern:

• Judge through claude -p. Subject and judge share one spawn helper and one subscription credential, CLAUDE_CODE_OAUTH_TOKEN. The judge runs with tools and skills off, outside the repo, and not in --bare mode, which ignores OAuth. Spawned sessions drop TMUX/TMUX_PANE so they don't ring the developer's tmux bell through a Stop hook.
• Only the discrimination choice gates hard. The choice assert is tagged metric: choice. The stated rule becomes a judged llm-rubric that sees the case's grading notes. check-gate reads per-assert results, so detection and judged checks count toward the 90% floor. make eval sets PROMPTFOO_FAILED_TEST_EXIT_CODE=0 so the gate actually runs after a failure.
• prose-register fully supported. Rank and structural types, per-skill prompt framing, and rubric criteria read from each case. Prompts and rubrics are wrapped in {% raw %}, because promptfoo renders both as nunjucks and Hugo shortcodes broke that. The rank answer key is passed as a string, because promptfoo splits list-valued vars into one test per element. The old run-evals.mjs is deleted.
• Offline checks in preflight. eval-validate and the unit tests (60) block in the existing CI job.
• Lockfile refresh. An npm update within promptfoo's ranges (promptfoo stays pinned at 0.122.2) clears the transitive advisories behind Dependabot's Bump sharp and @huggingface/transformers in /evals #64–Bump smol-toml and @openai/codex-security in /evals #66, plus fast-uri, qs, and fflate. What's left is extract-zip, which has no upstream fix and arrives through an optional promptfoo dependency the suite never calls. CI pins Node 24, because npm before 11.11 rejects this npm-11 lockfile as out of sync.
• Advisory evals workflow. It selects the evaluated skills a PR touches (every skill when evals/ changes), links each skill into the runner's ~/.claude/skills (the repo has no root .claude/), runs make eval, and uploads results. Fork PRs and a missing secret skip with a notice. It's hardened with read-only permissions, persist-credentials: false and concurrency cancellation.

Feedback

• The CI proof passed. This PR changes evals/, so its own evals run tested token auth and skill discovery on a bare runner. Neither skill had any subject errors. code-comment-register scored 15/16 and passed.
• prose-register is red, on purpose. CI scored it 12/22. Local runs: code-comment-register 16/16. prose-register 15/22, with hard failures on disc-09 and disc-11. Those two cases (plus disc-04, which flipped once in four runs) have grading notes that call the alternative answers defensible. They stay hard until the cases are revised. Even then, prose sits below the 90% floor, because its detection cases are expected to fail. That's the main input for threshold tuning, recorded in the spec's After Phase 2.
• Local runs aren't isolated from user config. They load user rules, plugins and Stop hooks. CI's bare $HOME doesn't. The spec records this as a follow-up. CI's prose score (12/22, with extra hard failures on disc-01 and disc-12) came in below the local 15/22, which fits that gap. One run isn't enough to tell.
• Worth a close look: the gate rule in evals/bin/check-gate.mjs, and the judge's rule rubric in evals/lib/prompts.mjs.