test(evals): add a plugin eval suite for the github-screenshots skill

[Zach Dunn (zachdunn)]

Adds an eval suite under evals/ that measures whether the github-screenshots skill actually gets an agent to capture and host visual evidence of a UI change — and whether it stays quiet when it shouldn't.

What's here

Seven cases, run with claude plugin eval:

• Five "should fire" — a docs width change, a thumbnail resize, a three-part sidebar tweak, a thumbnail link bug, and a rendered alert email. None of the prompts asks for a screenshot; the skill has to fire off its own "having just changed something visual" clause.
• Two "should NOT fire" — CLI shell-completion debugging, and a webhook outage investigation that happens to mention screenshots.

Every prompt is lifted from a real Claude, Codex, or Grok transcript on this machine, not invented. The one that matters most is the docs-width case: that's the PR where a screenshot only got attached after a manual nudge.

Supporting pieces: a shared git fixture (fixture/scaffold.sh) so each case's premise is true on disk, a mock of the hosted MCP server so no run touches production, and run.sh, which wraps two local workarounds (see its header comments).

How to run

evals/run.sh --ablation with-without

Cheap single-case iteration: evals/run.sh --case 'docs*' --ablation none --runs 1.

Read NOTES.md before trusting a number

The first full run (Sonnet, 3 runs/case, 42 runs) exposed four defects in the suite itself, all written up in evals/NOTES.md:

1. The with/without delta is not yet sound — the baseline arm has no capture tool at all, so a positive delta is guaranteed by construction.
2. Two graders referenced the wrong MCP namespace (mcp__uploads__put rather than mcp__plugin_uploads_uploads__put). Fixed here, not yet re-run.
3. The mock serves tools with no descriptions, so agents invented arguments.
4. Case 07 is a weak negative — its prompt names the screenshots bot, so engaging with the skill there is defensible.

The one clean signal it did produce is in #964: the skill fires inconsistently on identical prompts.

Notes

• evals/results/ is gitignored — it holds full agent transcripts.
• The suite measures Sonnet. Codex and Grok load this skill too and can't be exercised by this harness.
• Nothing under skills/ or .claude-plugin/ is touched; this is additive.

Refs #964. Related: #960 (the plugin currently fails to load from the marketplace, which this suite sidesteps by loading it by path).

[changeset-bot]

⚠️ No Changeset found

Latest commit: fc15c10

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (2)

• coderabbit:review
• review

🚫 Excluded labels (none allowed) (1)

• wip

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: daa03129-dcd0-4476-90f4-8939136a34b2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}