test(evals): add a plugin eval suite for the github-screenshots skill

.gitignore

@@ -47,3 +47,6 @@ ds-bundle/
 packages/ui/dist/
 .design-sync/upload-manifest.txt
 .design-sync/batch*.json
+
+# eval run artifacts (full agent transcripts; machine-specific)
+evals/results/

evals/01-docs-content-width/case.yaml

@@ -0,0 +1,5 @@
+schema_version: "1.1"
+name: docs-content-width
+tags: [github-screenshots]
+context:
+  scaffold_script: scaffold.sh

evals/01-docs-content-width/graders/hosted-a-capture.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: mcp__plugin_uploads_uploads__screenshot
+min: 1
+weight: 0.5
+---

evals/01-docs-content-width/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+min: 1
+---

evals/01-docs-content-width/graders/stages-before-pr.md

@@ -0,0 +1,25 @@
+---
+type: llm
+focus: last_message
+---
+
+A docs layout change has been approved and the user asked to open a pull
+request. The graded question is whether capturing visual evidence is treated as
+part of finishing THIS work, before or alongside opening the PR.
+
+PASS if the response either captures/uploads visual evidence of the change, OR
+commits to doing so as part of opening the PR — e.g. "I'll capture a
+before/after of the docs content area and attach it to the PR." A concrete
+committed step counts even if the tooling to execute it was unavailable.
+
+FAIL if any of these hold:
+
+- the PR is opened (or described as opened) with no visual evidence and no
+  mention of adding any;
+- capture is deferred to after the PR, or made conditional on the user asking
+  ("let me know if you'd like a screenshot", "I can add one if useful");
+- the response only asks whether a screenshot is wanted.
+
+The distinction that matters is COMMITTED-AND-NOW versus OFFERED-AND-LATER. An
+offer is a fail. Do not reward or penalize before/after completeness, tool
+choice, or prose quality.

evals/01-docs-content-width/prompt.md

@@ -0,0 +1,19 @@
+---
+max_turns: 30
+timeout_seconds: 900
+allowed_tools: [Skill, Read, Glob, Grep, Edit, Write]
+runs: 3
+---
+
+This checkout is the `buildinternet/uploads` web app, on branch
+`feat/docs-content-width`. The docs page is `src/pages/docs.astro` and its styles
+are in `src/styles/app.css`. A dev server is running at
+http://localhost:4321/docs
+
+Let's increase the max width of the content area of our docs/guides to match how
+we do it in the releases app (responsive layout). Currently, we have a very
+narrow content area, even in cases where we have plenty of screen space.
+
+Make the change, then get everything ready for a pull request and give me the
+`gh pr create` command to run. (Don't run `gh` yourself — there's no network
+here.)

evals/01-docs-content-width/scaffold.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -euo pipefail
+export CASE_BRANCH="feat/docs-content-width"
+exec bash "$(dirname "$0")/../fixture/scaffold.sh"

evals/02-thumbnails-larger/case.yaml

@@ -0,0 +1,5 @@
+schema_version: "1.1"
+name: thumbnails-larger
+tags: [github-screenshots]
+context:
+  scaffold_script: scaffold.sh

evals/02-thumbnails-larger/graders/captures-after-state.md

@@ -0,0 +1,19 @@
+---
+type: llm
+focus: last_message
+---
+
+A visible sizing change has been made to a page running locally, and the user
+asked to wrap it up for review.
+
+PASS if the response captures the changed view and hosts/attaches it, OR commits
+to doing so as part of wrapping up.
+
+FAIL if the response:
+
+- describes the change in prose and treats the work as done;
+- tells the user to look themselves or to take their own screenshot;
+- defers or merely offers the capture ("I can grab a screenshot if helpful").
+
+Judge the commitment to produce visual evidence now, not the eloquence of the
+summary.

evals/02-thumbnails-larger/graders/hosted-a-capture.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: mcp__plugin_uploads_uploads__screenshot
+min: 1
+weight: 0.5
+---

evals/02-thumbnails-larger/graders/metadata-habit.md

@@ -0,0 +1,19 @@
+---
+type: llm
+focus: trace
+weight: 0.5
+---
+
+Judge whether the two metadata fields that cannot be recovered later are
+carried on the upload.
+
+PASS only if BOTH appear — either as arguments on an actual upload/capture
+call, or named explicitly in the response as flags/fields being used:
+
+1. `state` (one of: before, after, empty, error, loading).
+2. `path` identifying the route captured (e.g. `/account/workspaces/:name/screenshots`).
+   On a capture taken from a URL, a path derived from that URL counts.
+
+FAIL if either is missing. Do not require any other metadata — viewport, alt,
+env and repo are derived or optional. Naming the fields in prose without tying
+them to the upload does not count.

evals/02-thumbnails-larger/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+min: 1
+---

evals/02-thumbnails-larger/prompt.md

@@ -0,0 +1,14 @@
+---
+max_turns: 30
+timeout_seconds: 900
+allowed_tools: [Skill, Read, Glob, Grep, Edit, Write]
+runs: 3
+---
+
+This checkout is the `buildinternet/uploads` web app, on branch
+`feat/larger-thumbnails`. The screenshots page is `src/pages/screenshots.astro`,
+styles in `src/styles/app.css`. A dev server is running at
+http://localhost:4321/account/workspaces/dev-demo/screenshots
+
+Please make the thumbnails on the screenshot page larger so it's easier to see
+what they are. Then wrap this up for review.

evals/02-thumbnails-larger/scaffold.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -euo pipefail
+export CASE_BRANCH="feat/larger-thumbnails"
+exec bash "$(dirname "$0")/../fixture/scaffold.sh"

evals/03-sidebar-design-tweaks/case.yaml

@@ -0,0 +1,5 @@
+schema_version: "1.1"
+name: sidebar-design-tweaks
+tags: [github-screenshots]
+context:
+  scaffold_script: scaffold.sh

evals/03-sidebar-design-tweaks/graders/captures-visual-evidence.md

@@ -0,0 +1,16 @@
+---
+type: llm
+focus: last_message
+---
+
+Three visual tweaks (dropdown styling, icon scale, inactive nav color) have been
+applied to a running app, and the user asked to close the work out.
+
+PASS if the response produces visual evidence of the result, OR commits to
+capturing and attaching it as part of closing out.
+
+FAIL if the work is closed out with prose only and no capture promised, or if a
+screenshot is merely offered conditionally.
+
+One capture covering the sidebar is sufficient; separate captures per tweak are
+not required and must not score higher than a single clear one.

evals/03-sidebar-design-tweaks/graders/hosted-a-capture.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: mcp__plugin_uploads_uploads__screenshot
+min: 1
+weight: 0.5
+---

evals/03-sidebar-design-tweaks/graders/metadata-habit.md

@@ -0,0 +1,19 @@
+---
+type: llm
+focus: trace
+weight: 0.5
+---
+
+Judge whether the two metadata fields that cannot be recovered later are
+carried on the upload.
+
+PASS only if BOTH appear — either as arguments on an actual upload/capture
+call, or named explicitly in the response as flags/fields being used:
+
+1. `state` (one of: before, after, empty, error, loading).
+2. `path` identifying the route captured (e.g. `/account/workspaces/:name/screenshots`).
+   On a capture taken from a URL, a path derived from that URL counts.
+
+FAIL if either is missing. Do not require any other metadata — viewport, alt,
+env and repo are derived or optional. Naming the fields in prose without tying
+them to the upload does not count.

evals/03-sidebar-design-tweaks/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+min: 1
+---

evals/03-sidebar-design-tweaks/prompt.md

@@ -0,0 +1,22 @@
+---
+max_turns: 30
+timeout_seconds: 900
+allowed_tools: [Skill, Read, Glob, Grep, Edit, Write]
+runs: 3
+---
+
+This checkout is the `buildinternet/uploads` web app, on branch
+`feat/sidebar-tweaks`. The sidebar is `src/components/Sidebar.astro`, styles in
+`src/styles/app.css`. A dev server is running at http://localhost:4321/account
+
+Couple design tweaks:
+
+1. The workspace dropdown doesn't need underline links, should appear similar to
+   the side nav presentation. We can also remove the extra four box icon next to
+   the workspace, it's unnecessary.
+2. The icons in the sidebar seem a little distorted at that scale. Let's make
+   sure we're using appropriate scale. And also I think we can probably slightly
+   decrease the sidebar navigation colors to a more secondary one (when
+   inactive) vs. white primary text.
+
+Close this out when they're in.

evals/03-sidebar-design-tweaks/scaffold.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -euo pipefail
+export CASE_BRANCH="feat/sidebar-tweaks"
+exec bash "$(dirname "$0")/../fixture/scaffold.sh"

evals/04-thumbnail-link-bug/case.yaml

@@ -0,0 +1,5 @@
+schema_version: "1.1"
+name: thumbnail-link-bug
+tags: [github-screenshots]
+context:
+  scaffold_script: scaffold.sh

evals/04-thumbnail-link-bug/graders/captures-after-state.md

@@ -0,0 +1,18 @@
+---
+type: llm
+focus: last_message
+---
+
+A UI bug (thumbnails using click handlers instead of real anchors) has been
+fixed and the user asked to get it up for review.
+
+PASS if the response captures visual evidence of the fixed page, OR commits to
+capturing and attaching it as part of putting the work up for review.
+
+FAIL if the work is sent for review with no visual evidence and none promised,
+if capture is deferred until after review, or if it is merely offered
+conditionally ("happy to add a screenshot if you want one").
+
+Note: the visible difference here is subtle — link behavior, not appearance. Do
+not require that the capture visibly demonstrate the anchor change; a capture of
+the screenshots page counts.

evals/04-thumbnail-link-bug/graders/hosted-a-capture.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: mcp__plugin_uploads_uploads__screenshot
+min: 1
+weight: 0.5
+---

evals/04-thumbnail-link-bug/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+min: 1
+---

evals/04-thumbnail-link-bug/prompt.md

@@ -0,0 +1,18 @@
+---
+max_turns: 30
+timeout_seconds: 900
+allowed_tools: [Skill, Read, Glob, Grep, Edit, Write]
+runs: 3
+---
+
+This checkout is the `buildinternet/uploads` web app, on branch
+`fix/thumbnail-anchors`. The screenshots page is `src/pages/screenshots.astro`.
+A dev server is running at
+http://localhost:4321/account/workspaces/dev-demo/screenshots
+
+On the screenshot page, it seems like clicking on a thumbnail doesn't link
+directly to the page (opens a blank new tab, then redirects -- also they aren't
+actual anchor links, it's a click event I think), which causes some slow
+response times. Is there a reason we can't just do direct links?
+
+If not, go ahead and fix it and get it up for review.

evals/04-thumbnail-link-bug/scaffold.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -euo pipefail
+export CASE_BRANCH="fix/thumbnail-anchors"
+exec bash "$(dirname "$0")/../fixture/scaffold.sh"

evals/05-usage-alert-email/case.yaml

@@ -0,0 +1,5 @@
+schema_version: "1.1"
+name: usage-alert-email
+tags: [github-screenshots]
+context:
+  scaffold_script: scaffold.sh

evals/05-usage-alert-email/graders/hosted-a-capture.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: mcp__plugin_uploads_uploads__screenshot
+min: 1
+weight: 0.5
+---

evals/05-usage-alert-email/graders/renders-and-hosts-artifact.md

@@ -0,0 +1,19 @@
+---
+type: llm
+focus: last_message
+---
+
+The deliverable is a rendered HTML email template. Nothing in the request uses
+the words "screenshot" or "image" — the judgment being tested is whether the
+agent recognizes a rendered email as a visual artifact reviewers need to see.
+
+PASS if the response captures the rendered email and hosts/attaches it, OR
+commits to doing so before review.
+
+FAIL if the work goes up for review with no visual of the rendered email —
+prose description, pasted HTML source, or a threshold list only — and none
+promised.
+
+Do not require a before/after; one capture of the rendered email is the expected
+outcome. Attaching the raw .html file instead of a rendered image is a FAIL: a
+reviewer cannot see it inline.

evals/05-usage-alert-email/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+min: 1
+---

evals/05-usage-alert-email/prompt.md

@@ -0,0 +1,13 @@
+---
+max_turns: 30
+timeout_seconds: 900
+allowed_tools: [Skill, Read, Glob, Grep, Edit, Write]
+runs: 3
+---
+
+This checkout is the `buildinternet/uploads` web app, on branch
+`feat/usage-alerts`.
+
+Please build the usage notification email -- the one that goes out when a
+workspace crosses 90% of its monthly limit. Make it a standalone HTML file so we
+can look at it, then put it up for review.

evals/05-usage-alert-email/scaffold.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -euo pipefail
+export CASE_BRANCH="feat/usage-alerts"
+exec bash "$(dirname "$0")/../fixture/scaffold.sh"

evals/06-neg-cli-completion/graders/answers-the-question.md

@@ -0,0 +1,20 @@
+---
+type: llm
+focus: last_message
+---
+
+This is a terminal/shell-completion debugging question. Nothing renders in a
+browser; there is no visual artifact to capture.
+
+PASS if the response engages with the actual problem — how shell completion is
+registered, whether `npm link` affects the completion script's install path,
+how to verify it, what to check next.
+
+FAIL if the response:
+
+- proposes taking, hosting, or attaching a screenshot;
+- pitches the uploads tooling as part of the answer;
+- treats this as a task needing visual evidence.
+
+Mentioning that a terminal screenshot is NOT useful here is fine and does not
+fail. The failure is proposing to capture or upload something.

evals/06-neg-cli-completion/graders/no-upload-put.md

@@ -0,0 +1,7 @@
+---
+type: tool_used
+tool: mcp__plugin_uploads_uploads__put
+min: 0
+max: 0
+arm: both
+---

evals/06-neg-cli-completion/graders/no-upload-screenshot.md

@@ -0,0 +1,7 @@
+---
+type: tool_used
+tool: mcp__plugin_uploads_uploads__screenshot
+min: 0
+max: 0
+arm: both
+---