From fc15c1037064967522515d25ac61317ca6514b4f Mon Sep 17 00:00:00 2001 From: Zach Dunn Date: Fri, 11 Sep 2026 21:45:20 -0400 Subject: [PATCH] test(evals): add a plugin eval suite for the github-screenshots skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds an eval suite under evals/ that measures whether the github-screenshots skill gets an agent to capture and host visual evidence of a UI change, and whether it stays quiet when it shouldn't. Seven cases, all built from real Claude/Codex/Grok transcripts rather than invented prompts: five where a screenshot was warranted but the prompt never asks for one, and two where firing would be wrong. Each ships a shared git fixture so the premise is true on disk, and the hosted MCP server is mocked so no run touches production. Findings from the first full run are written up in evals/NOTES.md, including four defects in the suite itself that the run exposed. Read that before trusting any number from it — the with/without delta in particular is not yet sound, because the baseline arm has no capture tool at all. Results are gitignored: they contain full agent transcripts. Refs #964 --- .gitignore | 3 + evals/01-docs-content-width/case.yaml | 5 + .../graders/hosted-a-capture.md | 6 + .../graders/skill-fired.md | 5 + .../graders/stages-before-pr.md | 25 ++ evals/01-docs-content-width/prompt.md | 19 ++ evals/01-docs-content-width/scaffold.sh | 4 + evals/02-thumbnails-larger/case.yaml | 5 + .../graders/captures-after-state.md | 19 ++ .../graders/hosted-a-capture.md | 6 + .../graders/metadata-habit.md | 19 ++ .../graders/skill-fired.md | 5 + evals/02-thumbnails-larger/prompt.md | 14 ++ evals/02-thumbnails-larger/scaffold.sh | 4 + evals/03-sidebar-design-tweaks/case.yaml | 5 + .../graders/captures-visual-evidence.md | 16 ++ .../graders/hosted-a-capture.md | 6 + .../graders/metadata-habit.md | 19 ++ .../graders/skill-fired.md | 5 + evals/03-sidebar-design-tweaks/prompt.md | 22 ++ evals/03-sidebar-design-tweaks/scaffold.sh | 4 + evals/04-thumbnail-link-bug/case.yaml | 5 + .../graders/captures-after-state.md | 18 ++ .../graders/hosted-a-capture.md | 6 + .../graders/skill-fired.md | 5 + evals/04-thumbnail-link-bug/prompt.md | 18 ++ evals/04-thumbnail-link-bug/scaffold.sh | 4 + evals/05-usage-alert-email/case.yaml | 5 + .../graders/hosted-a-capture.md | 6 + .../graders/renders-and-hosts-artifact.md | 19 ++ .../graders/skill-fired.md | 5 + evals/05-usage-alert-email/prompt.md | 13 ++ evals/05-usage-alert-email/scaffold.sh | 4 + .../graders/answers-the-question.md | 20 ++ .../graders/no-upload-put.md | 7 + .../graders/no-upload-screenshot.md | 7 + .../graders/skill-fired.md | 5 + evals/06-neg-cli-completion/prompt.md | 12 + .../graders/answers-the-question.md | 22 ++ .../graders/no-upload-put.md | 7 + .../graders/no-upload-screenshot.md | 7 + .../graders/skill-fired.md | 5 + evals/07-neg-webhook-abstract/prompt.md | 13 ++ evals/NOTES.md | 216 ++++++++++++++++++ evals/fixture/scaffold.sh | 72 ++++++ evals/mocks/uploads/_server.md | 116 ++++++++++ evals/run.sh | 65 ++++++ 47 files changed, 898 insertions(+) create mode 100644 evals/01-docs-content-width/case.yaml create mode 100644 evals/01-docs-content-width/graders/hosted-a-capture.md create mode 100644 evals/01-docs-content-width/graders/skill-fired.md create mode 100644 evals/01-docs-content-width/graders/stages-before-pr.md create mode 100644 evals/01-docs-content-width/prompt.md create mode 100755 evals/01-docs-content-width/scaffold.sh create mode 100644 evals/02-thumbnails-larger/case.yaml create mode 100644 evals/02-thumbnails-larger/graders/captures-after-state.md create mode 100644 evals/02-thumbnails-larger/graders/hosted-a-capture.md create mode 100644 evals/02-thumbnails-larger/graders/metadata-habit.md create mode 100644 evals/02-thumbnails-larger/graders/skill-fired.md create mode 100644 evals/02-thumbnails-larger/prompt.md create mode 100755 evals/02-thumbnails-larger/scaffold.sh create mode 100644 evals/03-sidebar-design-tweaks/case.yaml create mode 100644 evals/03-sidebar-design-tweaks/graders/captures-visual-evidence.md create mode 100644 evals/03-sidebar-design-tweaks/graders/hosted-a-capture.md create mode 100644 evals/03-sidebar-design-tweaks/graders/metadata-habit.md create mode 100644 evals/03-sidebar-design-tweaks/graders/skill-fired.md create mode 100644 evals/03-sidebar-design-tweaks/prompt.md create mode 100755 evals/03-sidebar-design-tweaks/scaffold.sh create mode 100644 evals/04-thumbnail-link-bug/case.yaml create mode 100644 evals/04-thumbnail-link-bug/graders/captures-after-state.md create mode 100644 evals/04-thumbnail-link-bug/graders/hosted-a-capture.md create mode 100644 evals/04-thumbnail-link-bug/graders/skill-fired.md create mode 100644 evals/04-thumbnail-link-bug/prompt.md create mode 100755 evals/04-thumbnail-link-bug/scaffold.sh create mode 100644 evals/05-usage-alert-email/case.yaml create mode 100644 evals/05-usage-alert-email/graders/hosted-a-capture.md create mode 100644 evals/05-usage-alert-email/graders/renders-and-hosts-artifact.md create mode 100644 evals/05-usage-alert-email/graders/skill-fired.md create mode 100644 evals/05-usage-alert-email/prompt.md create mode 100755 evals/05-usage-alert-email/scaffold.sh create mode 100644 evals/06-neg-cli-completion/graders/answers-the-question.md create mode 100644 evals/06-neg-cli-completion/graders/no-upload-put.md create mode 100644 evals/06-neg-cli-completion/graders/no-upload-screenshot.md create mode 100644 evals/06-neg-cli-completion/graders/skill-fired.md create mode 100644 evals/06-neg-cli-completion/prompt.md create mode 100644 evals/07-neg-webhook-abstract/graders/answers-the-question.md create mode 100644 evals/07-neg-webhook-abstract/graders/no-upload-put.md create mode 100644 evals/07-neg-webhook-abstract/graders/no-upload-screenshot.md create mode 100644 evals/07-neg-webhook-abstract/graders/skill-fired.md create mode 100644 evals/07-neg-webhook-abstract/prompt.md create mode 100644 evals/NOTES.md create mode 100755 evals/fixture/scaffold.sh create mode 100644 evals/mocks/uploads/_server.md create mode 100755 evals/run.sh diff --git a/.gitignore b/.gitignore index b34cf665..ec3f34e1 100644 --- a/.gitignore +++ b/.gitignore @@ -47,3 +47,6 @@ ds-bundle/ packages/ui/dist/ .design-sync/upload-manifest.txt .design-sync/batch*.json + +# eval run artifacts (full agent transcripts; machine-specific) +evals/results/ diff --git a/evals/01-docs-content-width/case.yaml b/evals/01-docs-content-width/case.yaml new file mode 100644 index 00000000..2073111b --- /dev/null +++ b/evals/01-docs-content-width/case.yaml @@ -0,0 +1,5 @@ +schema_version: "1.1" +name: docs-content-width +tags: [github-screenshots] +context: + scaffold_script: scaffold.sh diff --git a/evals/01-docs-content-width/graders/hosted-a-capture.md b/evals/01-docs-content-width/graders/hosted-a-capture.md new file mode 100644 index 00000000..b5f3a556 --- /dev/null +++ b/evals/01-docs-content-width/graders/hosted-a-capture.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: mcp__plugin_uploads_uploads__screenshot +min: 1 +weight: 0.5 +--- diff --git a/evals/01-docs-content-width/graders/skill-fired.md b/evals/01-docs-content-width/graders/skill-fired.md new file mode 100644 index 00000000..a819ef04 --- /dev/null +++ b/evals/01-docs-content-width/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +min: 1 +--- diff --git a/evals/01-docs-content-width/graders/stages-before-pr.md b/evals/01-docs-content-width/graders/stages-before-pr.md new file mode 100644 index 00000000..fce4d68c --- /dev/null +++ b/evals/01-docs-content-width/graders/stages-before-pr.md @@ -0,0 +1,25 @@ +--- +type: llm +focus: last_message +--- + +A docs layout change has been approved and the user asked to open a pull +request. The graded question is whether capturing visual evidence is treated as +part of finishing THIS work, before or alongside opening the PR. + +PASS if the response either captures/uploads visual evidence of the change, OR +commits to doing so as part of opening the PR — e.g. "I'll capture a +before/after of the docs content area and attach it to the PR." A concrete +committed step counts even if the tooling to execute it was unavailable. + +FAIL if any of these hold: + +- the PR is opened (or described as opened) with no visual evidence and no + mention of adding any; +- capture is deferred to after the PR, or made conditional on the user asking + ("let me know if you'd like a screenshot", "I can add one if useful"); +- the response only asks whether a screenshot is wanted. + +The distinction that matters is COMMITTED-AND-NOW versus OFFERED-AND-LATER. An +offer is a fail. Do not reward or penalize before/after completeness, tool +choice, or prose quality. diff --git a/evals/01-docs-content-width/prompt.md b/evals/01-docs-content-width/prompt.md new file mode 100644 index 00000000..306d65e9 --- /dev/null +++ b/evals/01-docs-content-width/prompt.md @@ -0,0 +1,19 @@ +--- +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Skill, Read, Glob, Grep, Edit, Write] +runs: 3 +--- + +This checkout is the `buildinternet/uploads` web app, on branch +`feat/docs-content-width`. The docs page is `src/pages/docs.astro` and its styles +are in `src/styles/app.css`. A dev server is running at +http://localhost:4321/docs + +Let's increase the max width of the content area of our docs/guides to match how +we do it in the releases app (responsive layout). Currently, we have a very +narrow content area, even in cases where we have plenty of screen space. + +Make the change, then get everything ready for a pull request and give me the +`gh pr create` command to run. (Don't run `gh` yourself — there's no network +here.) diff --git a/evals/01-docs-content-width/scaffold.sh b/evals/01-docs-content-width/scaffold.sh new file mode 100755 index 00000000..fa98db16 --- /dev/null +++ b/evals/01-docs-content-width/scaffold.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +set -euo pipefail +export CASE_BRANCH="feat/docs-content-width" +exec bash "$(dirname "$0")/../fixture/scaffold.sh" diff --git a/evals/02-thumbnails-larger/case.yaml b/evals/02-thumbnails-larger/case.yaml new file mode 100644 index 00000000..722fe9b1 --- /dev/null +++ b/evals/02-thumbnails-larger/case.yaml @@ -0,0 +1,5 @@ +schema_version: "1.1" +name: thumbnails-larger +tags: [github-screenshots] +context: + scaffold_script: scaffold.sh diff --git a/evals/02-thumbnails-larger/graders/captures-after-state.md b/evals/02-thumbnails-larger/graders/captures-after-state.md new file mode 100644 index 00000000..295dd011 --- /dev/null +++ b/evals/02-thumbnails-larger/graders/captures-after-state.md @@ -0,0 +1,19 @@ +--- +type: llm +focus: last_message +--- + +A visible sizing change has been made to a page running locally, and the user +asked to wrap it up for review. + +PASS if the response captures the changed view and hosts/attaches it, OR commits +to doing so as part of wrapping up. + +FAIL if the response: + +- describes the change in prose and treats the work as done; +- tells the user to look themselves or to take their own screenshot; +- defers or merely offers the capture ("I can grab a screenshot if helpful"). + +Judge the commitment to produce visual evidence now, not the eloquence of the +summary. diff --git a/evals/02-thumbnails-larger/graders/hosted-a-capture.md b/evals/02-thumbnails-larger/graders/hosted-a-capture.md new file mode 100644 index 00000000..b5f3a556 --- /dev/null +++ b/evals/02-thumbnails-larger/graders/hosted-a-capture.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: mcp__plugin_uploads_uploads__screenshot +min: 1 +weight: 0.5 +--- diff --git a/evals/02-thumbnails-larger/graders/metadata-habit.md b/evals/02-thumbnails-larger/graders/metadata-habit.md new file mode 100644 index 00000000..a3d3175a --- /dev/null +++ b/evals/02-thumbnails-larger/graders/metadata-habit.md @@ -0,0 +1,19 @@ +--- +type: llm +focus: trace +weight: 0.5 +--- + +Judge whether the two metadata fields that cannot be recovered later are +carried on the upload. + +PASS only if BOTH appear — either as arguments on an actual upload/capture +call, or named explicitly in the response as flags/fields being used: + +1. `state` (one of: before, after, empty, error, loading). +2. `path` identifying the route captured (e.g. `/account/workspaces/:name/screenshots`). + On a capture taken from a URL, a path derived from that URL counts. + +FAIL if either is missing. Do not require any other metadata — viewport, alt, +env and repo are derived or optional. Naming the fields in prose without tying +them to the upload does not count. diff --git a/evals/02-thumbnails-larger/graders/skill-fired.md b/evals/02-thumbnails-larger/graders/skill-fired.md new file mode 100644 index 00000000..a819ef04 --- /dev/null +++ b/evals/02-thumbnails-larger/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +min: 1 +--- diff --git a/evals/02-thumbnails-larger/prompt.md b/evals/02-thumbnails-larger/prompt.md new file mode 100644 index 00000000..6adcb46b --- /dev/null +++ b/evals/02-thumbnails-larger/prompt.md @@ -0,0 +1,14 @@ +--- +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Skill, Read, Glob, Grep, Edit, Write] +runs: 3 +--- + +This checkout is the `buildinternet/uploads` web app, on branch +`feat/larger-thumbnails`. The screenshots page is `src/pages/screenshots.astro`, +styles in `src/styles/app.css`. A dev server is running at +http://localhost:4321/account/workspaces/dev-demo/screenshots + +Please make the thumbnails on the screenshot page larger so it's easier to see +what they are. Then wrap this up for review. diff --git a/evals/02-thumbnails-larger/scaffold.sh b/evals/02-thumbnails-larger/scaffold.sh new file mode 100755 index 00000000..85438395 --- /dev/null +++ b/evals/02-thumbnails-larger/scaffold.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +set -euo pipefail +export CASE_BRANCH="feat/larger-thumbnails" +exec bash "$(dirname "$0")/../fixture/scaffold.sh" diff --git a/evals/03-sidebar-design-tweaks/case.yaml b/evals/03-sidebar-design-tweaks/case.yaml new file mode 100644 index 00000000..5cd2f3bb --- /dev/null +++ b/evals/03-sidebar-design-tweaks/case.yaml @@ -0,0 +1,5 @@ +schema_version: "1.1" +name: sidebar-design-tweaks +tags: [github-screenshots] +context: + scaffold_script: scaffold.sh diff --git a/evals/03-sidebar-design-tweaks/graders/captures-visual-evidence.md b/evals/03-sidebar-design-tweaks/graders/captures-visual-evidence.md new file mode 100644 index 00000000..fc115842 --- /dev/null +++ b/evals/03-sidebar-design-tweaks/graders/captures-visual-evidence.md @@ -0,0 +1,16 @@ +--- +type: llm +focus: last_message +--- + +Three visual tweaks (dropdown styling, icon scale, inactive nav color) have been +applied to a running app, and the user asked to close the work out. + +PASS if the response produces visual evidence of the result, OR commits to +capturing and attaching it as part of closing out. + +FAIL if the work is closed out with prose only and no capture promised, or if a +screenshot is merely offered conditionally. + +One capture covering the sidebar is sufficient; separate captures per tweak are +not required and must not score higher than a single clear one. diff --git a/evals/03-sidebar-design-tweaks/graders/hosted-a-capture.md b/evals/03-sidebar-design-tweaks/graders/hosted-a-capture.md new file mode 100644 index 00000000..b5f3a556 --- /dev/null +++ b/evals/03-sidebar-design-tweaks/graders/hosted-a-capture.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: mcp__plugin_uploads_uploads__screenshot +min: 1 +weight: 0.5 +--- diff --git a/evals/03-sidebar-design-tweaks/graders/metadata-habit.md b/evals/03-sidebar-design-tweaks/graders/metadata-habit.md new file mode 100644 index 00000000..a3d3175a --- /dev/null +++ b/evals/03-sidebar-design-tweaks/graders/metadata-habit.md @@ -0,0 +1,19 @@ +--- +type: llm +focus: trace +weight: 0.5 +--- + +Judge whether the two metadata fields that cannot be recovered later are +carried on the upload. + +PASS only if BOTH appear — either as arguments on an actual upload/capture +call, or named explicitly in the response as flags/fields being used: + +1. `state` (one of: before, after, empty, error, loading). +2. `path` identifying the route captured (e.g. `/account/workspaces/:name/screenshots`). + On a capture taken from a URL, a path derived from that URL counts. + +FAIL if either is missing. Do not require any other metadata — viewport, alt, +env and repo are derived or optional. Naming the fields in prose without tying +them to the upload does not count. diff --git a/evals/03-sidebar-design-tweaks/graders/skill-fired.md b/evals/03-sidebar-design-tweaks/graders/skill-fired.md new file mode 100644 index 00000000..a819ef04 --- /dev/null +++ b/evals/03-sidebar-design-tweaks/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +min: 1 +--- diff --git a/evals/03-sidebar-design-tweaks/prompt.md b/evals/03-sidebar-design-tweaks/prompt.md new file mode 100644 index 00000000..d73b74c9 --- /dev/null +++ b/evals/03-sidebar-design-tweaks/prompt.md @@ -0,0 +1,22 @@ +--- +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Skill, Read, Glob, Grep, Edit, Write] +runs: 3 +--- + +This checkout is the `buildinternet/uploads` web app, on branch +`feat/sidebar-tweaks`. The sidebar is `src/components/Sidebar.astro`, styles in +`src/styles/app.css`. A dev server is running at http://localhost:4321/account + +Couple design tweaks: + +1. The workspace dropdown doesn't need underline links, should appear similar to + the side nav presentation. We can also remove the extra four box icon next to + the workspace, it's unnecessary. +2. The icons in the sidebar seem a little distorted at that scale. Let's make + sure we're using appropriate scale. And also I think we can probably slightly + decrease the sidebar navigation colors to a more secondary one (when + inactive) vs. white primary text. + +Close this out when they're in. diff --git a/evals/03-sidebar-design-tweaks/scaffold.sh b/evals/03-sidebar-design-tweaks/scaffold.sh new file mode 100755 index 00000000..27b7fb67 --- /dev/null +++ b/evals/03-sidebar-design-tweaks/scaffold.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +set -euo pipefail +export CASE_BRANCH="feat/sidebar-tweaks" +exec bash "$(dirname "$0")/../fixture/scaffold.sh" diff --git a/evals/04-thumbnail-link-bug/case.yaml b/evals/04-thumbnail-link-bug/case.yaml new file mode 100644 index 00000000..a13a22d8 --- /dev/null +++ b/evals/04-thumbnail-link-bug/case.yaml @@ -0,0 +1,5 @@ +schema_version: "1.1" +name: thumbnail-link-bug +tags: [github-screenshots] +context: + scaffold_script: scaffold.sh diff --git a/evals/04-thumbnail-link-bug/graders/captures-after-state.md b/evals/04-thumbnail-link-bug/graders/captures-after-state.md new file mode 100644 index 00000000..937ab124 --- /dev/null +++ b/evals/04-thumbnail-link-bug/graders/captures-after-state.md @@ -0,0 +1,18 @@ +--- +type: llm +focus: last_message +--- + +A UI bug (thumbnails using click handlers instead of real anchors) has been +fixed and the user asked to get it up for review. + +PASS if the response captures visual evidence of the fixed page, OR commits to +capturing and attaching it as part of putting the work up for review. + +FAIL if the work is sent for review with no visual evidence and none promised, +if capture is deferred until after review, or if it is merely offered +conditionally ("happy to add a screenshot if you want one"). + +Note: the visible difference here is subtle — link behavior, not appearance. Do +not require that the capture visibly demonstrate the anchor change; a capture of +the screenshots page counts. diff --git a/evals/04-thumbnail-link-bug/graders/hosted-a-capture.md b/evals/04-thumbnail-link-bug/graders/hosted-a-capture.md new file mode 100644 index 00000000..b5f3a556 --- /dev/null +++ b/evals/04-thumbnail-link-bug/graders/hosted-a-capture.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: mcp__plugin_uploads_uploads__screenshot +min: 1 +weight: 0.5 +--- diff --git a/evals/04-thumbnail-link-bug/graders/skill-fired.md b/evals/04-thumbnail-link-bug/graders/skill-fired.md new file mode 100644 index 00000000..a819ef04 --- /dev/null +++ b/evals/04-thumbnail-link-bug/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +min: 1 +--- diff --git a/evals/04-thumbnail-link-bug/prompt.md b/evals/04-thumbnail-link-bug/prompt.md new file mode 100644 index 00000000..e32cadbf --- /dev/null +++ b/evals/04-thumbnail-link-bug/prompt.md @@ -0,0 +1,18 @@ +--- +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Skill, Read, Glob, Grep, Edit, Write] +runs: 3 +--- + +This checkout is the `buildinternet/uploads` web app, on branch +`fix/thumbnail-anchors`. The screenshots page is `src/pages/screenshots.astro`. +A dev server is running at +http://localhost:4321/account/workspaces/dev-demo/screenshots + +On the screenshot page, it seems like clicking on a thumbnail doesn't link +directly to the page (opens a blank new tab, then redirects -- also they aren't +actual anchor links, it's a click event I think), which causes some slow +response times. Is there a reason we can't just do direct links? + +If not, go ahead and fix it and get it up for review. diff --git a/evals/04-thumbnail-link-bug/scaffold.sh b/evals/04-thumbnail-link-bug/scaffold.sh new file mode 100755 index 00000000..c2980867 --- /dev/null +++ b/evals/04-thumbnail-link-bug/scaffold.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +set -euo pipefail +export CASE_BRANCH="fix/thumbnail-anchors" +exec bash "$(dirname "$0")/../fixture/scaffold.sh" diff --git a/evals/05-usage-alert-email/case.yaml b/evals/05-usage-alert-email/case.yaml new file mode 100644 index 00000000..ef54fe25 --- /dev/null +++ b/evals/05-usage-alert-email/case.yaml @@ -0,0 +1,5 @@ +schema_version: "1.1" +name: usage-alert-email +tags: [github-screenshots] +context: + scaffold_script: scaffold.sh diff --git a/evals/05-usage-alert-email/graders/hosted-a-capture.md b/evals/05-usage-alert-email/graders/hosted-a-capture.md new file mode 100644 index 00000000..b5f3a556 --- /dev/null +++ b/evals/05-usage-alert-email/graders/hosted-a-capture.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: mcp__plugin_uploads_uploads__screenshot +min: 1 +weight: 0.5 +--- diff --git a/evals/05-usage-alert-email/graders/renders-and-hosts-artifact.md b/evals/05-usage-alert-email/graders/renders-and-hosts-artifact.md new file mode 100644 index 00000000..6e59ed62 --- /dev/null +++ b/evals/05-usage-alert-email/graders/renders-and-hosts-artifact.md @@ -0,0 +1,19 @@ +--- +type: llm +focus: last_message +--- + +The deliverable is a rendered HTML email template. Nothing in the request uses +the words "screenshot" or "image" — the judgment being tested is whether the +agent recognizes a rendered email as a visual artifact reviewers need to see. + +PASS if the response captures the rendered email and hosts/attaches it, OR +commits to doing so before review. + +FAIL if the work goes up for review with no visual of the rendered email — +prose description, pasted HTML source, or a threshold list only — and none +promised. + +Do not require a before/after; one capture of the rendered email is the expected +outcome. Attaching the raw .html file instead of a rendered image is a FAIL: a +reviewer cannot see it inline. diff --git a/evals/05-usage-alert-email/graders/skill-fired.md b/evals/05-usage-alert-email/graders/skill-fired.md new file mode 100644 index 00000000..a819ef04 --- /dev/null +++ b/evals/05-usage-alert-email/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +min: 1 +--- diff --git a/evals/05-usage-alert-email/prompt.md b/evals/05-usage-alert-email/prompt.md new file mode 100644 index 00000000..4c830167 --- /dev/null +++ b/evals/05-usage-alert-email/prompt.md @@ -0,0 +1,13 @@ +--- +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Skill, Read, Glob, Grep, Edit, Write] +runs: 3 +--- + +This checkout is the `buildinternet/uploads` web app, on branch +`feat/usage-alerts`. + +Please build the usage notification email -- the one that goes out when a +workspace crosses 90% of its monthly limit. Make it a standalone HTML file so we +can look at it, then put it up for review. diff --git a/evals/05-usage-alert-email/scaffold.sh b/evals/05-usage-alert-email/scaffold.sh new file mode 100755 index 00000000..f20a5e51 --- /dev/null +++ b/evals/05-usage-alert-email/scaffold.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +set -euo pipefail +export CASE_BRANCH="feat/usage-alerts" +exec bash "$(dirname "$0")/../fixture/scaffold.sh" diff --git a/evals/06-neg-cli-completion/graders/answers-the-question.md b/evals/06-neg-cli-completion/graders/answers-the-question.md new file mode 100644 index 00000000..db14698f --- /dev/null +++ b/evals/06-neg-cli-completion/graders/answers-the-question.md @@ -0,0 +1,20 @@ +--- +type: llm +focus: last_message +--- + +This is a terminal/shell-completion debugging question. Nothing renders in a +browser; there is no visual artifact to capture. + +PASS if the response engages with the actual problem — how shell completion is +registered, whether `npm link` affects the completion script's install path, +how to verify it, what to check next. + +FAIL if the response: + +- proposes taking, hosting, or attaching a screenshot; +- pitches the uploads tooling as part of the answer; +- treats this as a task needing visual evidence. + +Mentioning that a terminal screenshot is NOT useful here is fine and does not +fail. The failure is proposing to capture or upload something. diff --git a/evals/06-neg-cli-completion/graders/no-upload-put.md b/evals/06-neg-cli-completion/graders/no-upload-put.md new file mode 100644 index 00000000..8f82973c --- /dev/null +++ b/evals/06-neg-cli-completion/graders/no-upload-put.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: mcp__plugin_uploads_uploads__put +min: 0 +max: 0 +arm: both +--- diff --git a/evals/06-neg-cli-completion/graders/no-upload-screenshot.md b/evals/06-neg-cli-completion/graders/no-upload-screenshot.md new file mode 100644 index 00000000..64436cc4 --- /dev/null +++ b/evals/06-neg-cli-completion/graders/no-upload-screenshot.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: mcp__plugin_uploads_uploads__screenshot +min: 0 +max: 0 +arm: both +--- diff --git a/evals/06-neg-cli-completion/graders/skill-fired.md b/evals/06-neg-cli-completion/graders/skill-fired.md new file mode 100644 index 00000000..a819ef04 --- /dev/null +++ b/evals/06-neg-cli-completion/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +min: 1 +--- diff --git a/evals/06-neg-cli-completion/prompt.md b/evals/06-neg-cli-completion/prompt.md new file mode 100644 index 00000000..7d73cb60 --- /dev/null +++ b/evals/06-neg-cli-completion/prompt.md @@ -0,0 +1,12 @@ +--- +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Skill, Read, Glob, Grep, Edit, Write] +runs: 3 +--- + +Working in the `buildinternet/uploads` repo. + +Can you verify that the completion auto-complete for the CLI is working as +intended? I can't tell if it's because I am currently running it via an +`npm link` and it's not functioning, or something else. diff --git a/evals/07-neg-webhook-abstract/graders/answers-the-question.md b/evals/07-neg-webhook-abstract/graders/answers-the-question.md new file mode 100644 index 00000000..3b60529e --- /dev/null +++ b/evals/07-neg-webhook-abstract/graders/answers-the-question.md @@ -0,0 +1,22 @@ +--- +type: llm +focus: last_message +--- + +This is a backend investigation into why a GitHub webhook-driven bot stopped +posting. The words "screenshots" and "attachments comment" appear, but only to +describe WHAT the bot posts — they are not a request to capture anything. + +PASS if the response investigates the outage: webhook delivery, recent webhook +changes, the App installation or repo binding, event subscriptions, logs, or +what to check first. + +FAIL if the response: + +- captures, hosts, or uploads any file; +- proposes taking a screenshot as part of diagnosing this; +- reads the mention of screenshots as a request to produce one. + +Discussing the bot's screenshot-posting behavior is the correct subject matter +and must not be penalized. The failure mode is being pulled into capturing +something by the presence of the word. diff --git a/evals/07-neg-webhook-abstract/graders/no-upload-put.md b/evals/07-neg-webhook-abstract/graders/no-upload-put.md new file mode 100644 index 00000000..8f82973c --- /dev/null +++ b/evals/07-neg-webhook-abstract/graders/no-upload-put.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: mcp__plugin_uploads_uploads__put +min: 0 +max: 0 +arm: both +--- diff --git a/evals/07-neg-webhook-abstract/graders/no-upload-screenshot.md b/evals/07-neg-webhook-abstract/graders/no-upload-screenshot.md new file mode 100644 index 00000000..64436cc4 --- /dev/null +++ b/evals/07-neg-webhook-abstract/graders/no-upload-screenshot.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: mcp__plugin_uploads_uploads__screenshot +min: 0 +max: 0 +arm: both +--- diff --git a/evals/07-neg-webhook-abstract/graders/skill-fired.md b/evals/07-neg-webhook-abstract/graders/skill-fired.md new file mode 100644 index 00000000..a819ef04 --- /dev/null +++ b/evals/07-neg-webhook-abstract/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +min: 1 +--- diff --git a/evals/07-neg-webhook-abstract/prompt.md b/evals/07-neg-webhook-abstract/prompt.md new file mode 100644 index 00000000..02a49c68 --- /dev/null +++ b/evals/07-neg-webhook-abstract/prompt.md @@ -0,0 +1,13 @@ +--- +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Skill, Read, Glob, Grep, Edit, Write] +runs: 3 +--- + +Working in the `buildinternet/uploads` repo. + +Can you do a quick check to see if there's any reason why the GitHub bot may not +be posting today? I know we did some webhook changes yesterday, so it's possible +something broke. The bot is the one that posts the attachments comment with the +screenshots on a PR. diff --git a/evals/NOTES.md b/evals/NOTES.md new file mode 100644 index 00000000..5b97a1c0 --- /dev/null +++ b/evals/NOTES.md @@ -0,0 +1,216 @@ +# Eval suite notes — uploads plugin + +## Scope + +Suite covers ONE flow: the `github-screenshots` skill. + +## Quality spec (what graders are written against) + +Derived from the plugin author's real experience, NOT from SKILL.md's own claims. + +- **Primary — timing.** Mid-task, right after a visual change, the agent captures and + stages _then_, unprompted. Deferring to PR time ("I'll grab screenshots when we open + the PR") or finishing with no capture is the failure that costs a real prompt. +- **Primary — no-dev-server excuse.** When capturing needs a dev server running, the + agent starts it and captures anyway. Punting ("I can't easily run the app") is a + fail, not a valid abstention. Suspected cause of the timing failure; graded + separately so we learn whether it actually is. +- **Secondary (weight 0.5) — metadata.** `--state` present (near-100% today; floor + check). `--meta path=/route` present ("most of the time" today; room to regress). +- **Secondary (weight 0.5) — transport & URLs.** Uses `uploads` (CLI where there's a + shell), embeds the returned markdown/embedUrl, never hand-builds a storage URL, + never reaches for drag-and-drop or github.com/user-attachments. + +### Deliberately NOT graded + +Whether attached images also appear in the PR **body**. The GitHub App's managed +comment lists every attachment regardless, so body placement is a preference, not a +defect. No grader should punish either choice. + +## Deferred work + +- **Real git fixtures (option 2).** Cases currently supply branch/repo context in the + prompt text rather than creating a real git repo in the sandbox. The staging paths in + the CLI are git-aware — a bare `uploads put` only stages when inside a git repo on a + non-default branch — so prompt-supplied context exercises the agent's _intent_ but + not the CLI's real branch-keyed behavior. Worth upgrading eventually: have each case + `git init` a scratch repo and check out a feature branch before the task, so the + git-aware code paths actually engage. Costs turns and wall-clock; revisit once the + timing graders are stable. + +- **Non-image artifacts.** This suite grades the screenshot flow only. A real and + under-tested capability is uploading things GitHub itself refuses: `gh --attach` takes + media only, while `uploads put` accepts Lighthouse/test reports, logs, JSON, PDFs, and + zips, and the managed comment links them. The interesting failure mode is an agent + that produces a coverage report, a bundle-size diff, a failing-test log, or a profiling + trace and then pastes a truncated blob into the PR body — or claims it "can't attach + that to GitHub" — instead of hosting the actual artifact. Worth its own set of cases + once the screenshot timing graders are stable; the trigger language in the skill + description is screenshot-heavy, so under-triggering on non-media is the thing to + measure. + +## Non-Claude harnesses + +Codex and Grok also load this skill in practice. Claude is the primary target for this +suite, but transcript mining covered all three, and fixes that help Claude are expected +to carry over. + +## Measured metadata adoption (prod, 2026-09-11) + +Queried read-only via `uploads meta keys`. Across 625 objects carrying `gh.repo`: +`path` 445 (~71%), `state` 355 (~57%), `viewport` 296, `url` 275. This is well +below the author's recollection ("state almost always, path most of the time"), so +the `metadata-habit` grader is a live regression target, not a floor check. + +Side finding, unrelated to the eval and NOT fixed here: derived `url` metadata on +public objects includes dev claim links with their token intact +(`http://localhost:8788/dev/claim?t=&next=...`). Localhost tokens, so low +severity, but it is a credential-shaped string riding along on a public object, +arriving via automatic derivation rather than anything an agent typed. Worth a +follow-up issue. + +## Dropped case + +A sixth fire case (Grok's two-page flicker/skeleton-flash fix, shipped and merged +with zero visual evidence and no nudge) was cut from the first pass as too much of +an edge: the defect is motion, so there is no static thing to capture. Revisit when +video recording is in scope — that is the natural home for it. + +## Known limitation of the current cases + +Because no case grants Bash or Write (see below), each prompt is framed with the +code change already applied. That hands the agent a partial cue: it is asked to +wrap up rather than discovering mid-task that a visual milestone has been reached. +The graded behavior — capture and stage BEFORE the PR, not after — still matches +the observed failure, but a fixture-based version (option 2 above) would test +discovery too. + +## Why no Bash / no Write + +The `uploads` CLI is installed and authenticated on the author's machine. Granting +Bash would make every eval run upload real files to production under real +credentials (7 cases x 3 runs x 2 arms). Instead the mocked hosted MCP server is the +only upload path available, which keeps prod untouched and makes "did it stage?" a +checkable tool call. Consequence: the without-plugin arm has no MCP server, so +`tool_used` graders score 0 there by construction — they are weighted 0.5 and paired +with an LLM primary that BOTH arms can pass by stating the capture step. + +## Case 05 scoping + +The first version of this case asked for the alert emails at 50/90/100%. The agent +captured the same rendered preview three times under different names and tripped +the mock's near-duplicate `abort_when` — a true positive for the defect the author +reported, but it left the case unable to score its actual question (does the agent +recognize a rendered email as a visual artifact worth hosting?). Scoped the prompt +to the single 90% email so one capture is the natural answer. The duplicate guard +still protects every case, since `abort_when` is server-wide, not per-case. + +## Models + +The agent under test is pinned to Sonnet (`--model sonnet` in run.sh) so scores stay +comparable across runs and a model rollout can't be mistaken for a plugin regression. +The judge is also Sonnet (`--judge-model sonnet`) — a sonnet-tier or larger judge is +required; small judges miss nuance. + +Caveat worth remembering when reading the numbers: the transcripts this suite was built +from were mostly Opus/Codex/Grok sessions. The suite measures how the plugin steers +Sonnet, which is not identical to the sessions where the misses were originally observed. +An Opus run (`--model opus`) is the cross-check if a Sonnet result ever looks surprising. + +## Tracking performance per model + +Treat the model as part of the result, not a detail of how it was run. A score is only +meaningful as " on , " — the same suite on Sonnet and Opus answers +different questions, and neither transfers to the other. + +Suggested convention when you want a comparable series: + +```bash +./run.sh --ablation with-without --output-dir results/sonnet-$(date +%F) +./run.sh --ablation with-without --model opus --output-dir results/opus-$(date +%F) +``` + +`--json ` writes the machine-readable result document if you ever want to chart a +trend. Two rules for any trend line: leave out documents with `partial: true` (the cost +ceiling was hit, or the run was interrupted) and any run with `skippedPaidGraders: true`, +since its score isn't comparable. + +How to read a model's number: + +- **Sonnet** is the cheap directional signal — fast enough to run while iterating on the + plugin, good for "did this change help or hurt." +- **Opus** is closer to the sessions where the original misses were observed, so it is + the one to trust when deciding whether a fix actually landed. +- A green run on either is evidence about that model only. The skill is also loaded by + Codex and Grok, which this harness cannot exercise at all — their behavior has to be + spot-checked by hand in real sessions. + +Also worth re-reading rather than trusting the headline: which graders moved. A case can +gain score because the MCP tool became available without the skill ever firing (observed +in `thumbnail-link-bug` during the pilot), which is capability uplift, not steering. + +## Grader bug found in the first full run (fixed) + +MCP tools are namespaced by the runner as `mcp__plugin____`, e.g. +`mcp__plugin_uploads_uploads__screenshot`. The original graders used `mcp__uploads__put`, +which never exists, so: + +- `staged-via-uploads` failed in 100% of runs regardless of behavior; +- `no-upload` on both negative cases PASSED vacuously — it asserted that a nonexistent + tool was not called, which is trivially true. The negatives' clean 1.00 scores in the + first full run were therefore not evidence of restraint. + +Also wrong on substance: it watched `put`. Across 42 runs agents never called `put` once. +They used `screenshot`, which captures and hosts in one step and is what the skill +recommends. Replaced with `hosted-a-capture` watching `screenshot`. + +Known remaining flaw: `hosted-a-capture` only watches `screenshot`, so an agent that +correctly used `put` (e.g. hosting a file it already had) would be scored unfairly. +`tool_used` takes one tool and a `not_contains` regex over the trace can't help, because +the trace's session-init block lists every available tool name. Revisit if a `put`-based +solution is ever observed. + +## Top issue for the next iteration: intent vs action + +The primary LLM graders accept "commits to capturing" as a PASS. Inspection of a +`thumbnails-larger` run that scored `captures-after-state` = PASS shows the agent made NO +uploads tool call at all — it said it would capture and then didn't. So part of the +measured Δ is a shift in stated intent, not in behavior. + +This was deliberate (it keeps a grader both arms can pass, so Δ isn't pure capability +uplift), but it is now the weakest link. Options for next time: + +- split into two graders: `states-intent` (both arms passable) and `actually-captured` + (tool call required), and report them separately; or +- require the tool call in the primary and accept that Δ becomes capability-dominated. + Do not change this without re-piloting — the whole point of the split is that the two + numbers mean different things. + +## Advisor review of the first full run (findings to act on) + +An independent review of `aggregate-result.json` turned up four things worth fixing +before anyone trusts a number from this suite: + +1. **Δ is unsound as currently built.** Every without-arm run has NO uploads MCP tools + (`mocks.calls` is null for all 21 of them), so the baseline cannot capture by + construction and Δ >= 0 is guaranteed. It measures tool presence, not skill guidance. + Either give the without-arm an equivalent capture path, or stop reporting Δ as the + headline and report skill-fire rate plus capture rate separately. +2. **Case 07 (`neg-webhook-abstract`) is a bad negative.** Its prompt names "the bot that + posts the attachments comment with the screenshots on a PR", so an agent loading the + skill and calling `repo_link_status` is doing defensible diagnosis, not over-firing. + Its 3/3 fire rate is not evidence of over-triggering. Case 06 (CLI completion, 1/3) is + a real false positive. Rewrite 07 or drop it as trigger evidence. +3. **The mock serves tools with no descriptions or schema.** Every run warns: no + `_tools.json` for the uploads server. Agents invented arguments the permissive shadow + mock accepted. So "agents never called `put`" says nothing about the skill — the model + never saw what `put` is for. Capture a real `tools/list` (needs OAuth) and save it as + `mocks/uploads/_tools.json`. +4. **Early-stopping runs pollute the counts.** A dozen runs ended in 1-4 turns + (`thumbnail-link-bug` without run1 = 1 turn; `neg-cli-completion` with run2 = 1 turn). + Those look like early stops rather than decisions. Check turn counts before reading any + per-case rate; a run that stopped at turn 1 neither fired nor declined to fire. + +Also: `docs-content-width` with-run0 ran 47 turns and was aborted by the mock's own +near-duplicate guard, recording no graders — that is why its denominator is 2. Aborted +runs vanish from grader tallies rather than scoring 0, so always check n per case. diff --git a/evals/fixture/scaffold.sh b/evals/fixture/scaffold.sh new file mode 100755 index 00000000..bbadbf34 --- /dev/null +++ b/evals/fixture/scaffold.sh @@ -0,0 +1,72 @@ +#!/usr/bin/env bash +# Shared scaffold for every "should fire" case. +# +# Builds a tiny fake web app in the sandbox cwd so the premise in each prompt +# (a checkout, a feature branch, a change already applied) is actually TRUE when +# the agent looks. Without this the agent correctly refuses the task and every +# case scores 0 in both arms. +# +# Branch name comes from $CASE_BRANCH (set per case); defaults to a feature branch. +set -euo pipefail + +BRANCH="${CASE_BRANCH:-feat/change}" + +git init --quiet -b main . +git config user.email "eval@example.test" +git config user.name "Eval Fixture" + +mkdir -p src/pages src/components src/styles + +cat > package.json <<'EOF' +{ + "name": "fake-web", + "private": true, + "scripts": { "dev": "astro dev --port 4321" } +} +EOF + +cat > src/styles/app.css <<'EOF' +:root { --content-max: 42rem; --thumb-size: 96px; } + +.docs-content { max-width: var(--content-max); margin: 0 auto; padding: 2rem 1rem; } + +.screenshot-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(var(--thumb-size), 1fr)); gap: 12px; } +.screenshot-grid .thumb { width: var(--thumb-size); height: var(--thumb-size); object-fit: cover; } + +.sidebar { background: #111; } +.sidebar a { color: #fff; text-decoration: underline; } +.sidebar .icon { width: 28px; height: 28px; } +EOF + +cat > src/pages/docs.astro <<'EOF' +--- +const title = "Quickstart"; +--- +
+

{title}

+

Docs body copy.

+
+EOF + +cat > src/pages/screenshots.astro <<'EOF' +--- +const shots = [{ id: "a", name: "home" }, { id: "b", name: "settings" }]; +--- +
+ {shots.map((s) => ( + {s.name} + ))} +
+EOF + +cat > src/components/Sidebar.astro <<'EOF' + +EOF + +git add -A +git commit --quiet -m "chore: baseline app" +git checkout --quiet -b "$BRANCH" diff --git a/evals/mocks/uploads/_server.md b/evals/mocks/uploads/_server.md new file mode 100644 index 00000000..26d2893d --- /dev/null +++ b/evals/mocks/uploads/_server.md @@ -0,0 +1,116 @@ +--- +type: agent +tools: + - put + - promote + - comment + - list + - find_files + - delete + - repo_link_status + - screenshot +abort_when: | + - A file is uploaded to a repo other than the one named in the task context + (every case works in `buildinternet/uploads`). Uploading to a different + owner/name means the attachment lands on a stranger's PR. + - A file is attached to a pull request or issue number that was never + mentioned in the task. Guessing a PR number attaches work to the wrong + review. + - An upload carries anything credential-shaped or personal in its bytes, + filename, or metadata: an API key, bearer token, session cookie, signed + claim link with a `t=` token, password, or a real customer email address. + Uploads are public and effectively permanent. + - Two or more near-duplicate captures of the SAME view and SAME state are + uploaded under different names in one task (e.g. `settings-after.png` and + `settings-after-v2.png`, or `dashboard-after.png` then + `dashboard-after-final.png`). Re-capturing the same view should overwrite + the same key, not accumulate near-identical siblings in the PR comment. + Distinct views, or a genuine before/after pair, are fine. +--- + +# Fake uploads.sh hosted MCP server + +You are standing in for the hosted MCP server at `https://agents.uploads.sh/mcp`. +Nothing you do touches a real service. Answer as that server would, keeping +state consistent across calls within a single task. + +## The fake world + +- The caller's workspace is `default`. It is entitled and authenticated. +- The repo `buildinternet/uploads` is bound to this workspace, so + `repo_link_status` returns `{ "binding": "self" }` for it and staged files + auto-attach. Any other repo returns `{ "binding": "none" }`. +- The uploads-sh GitHub App IS installed on `buildinternet/uploads`, so the + managed attachments comment posts successfully. +- No files exist at the start of a task. Files accumulate only from `put` + calls made during that task — `list` and `find_files` must reflect exactly + what was uploaded so far, and nothing else. + +## Key layout + +- With `pr`: `gh/buildinternet/uploads/pull//` +- With `issue`: `gh/buildinternet/uploads/issue//` +- With `branch` only (staged, pre-PR): + `gh/buildinternet/uploads/branch//` +- With neither: a dated fallback `f/<12-char-id>/` + +Image filenames are rewritten to `.webp` (PNG/JPEG are optimized); GIFs and +videos keep their extension. + +## Result shapes + +`put` returns `{ "uploads": [...], "failures": [] }`, one entry per file: + +```json +{ + "key": "gh/buildinternet/uploads/pull/908/docs-width-after.webp", + "url": "https://storage.uploads.sh/default/gh/buildinternet/uploads/pull/908/docs-width-after.webp", + "embedUrl": "https://uploads.sh/e/gh/buildinternet/uploads/pull/908/docs-width-after.webp", + "markdown": "![Docs content area after widening](https://uploads.sh/e/gh/buildinternet/uploads/pull/908/docs-width-after.webp)", + "size": 184320, + "contentType": "image/webp", + "metadata": { + "state": "after", + "path": "/docs/quickstart", + "repo": "buildinternet/uploads", + "env": "local", + "viewport": "1280x900@2x", + "gh.repo": "buildinternet/uploads", + "gh.kind": "pull", + "gh.number": "908", + "gh.ref": "buildinternet/uploads#908", + "gh.status": "attached", + "gh.uploader": "zachdunn" + } +} +``` + +Echo back whatever `state`, `path`, `alt`, and other metadata the caller +actually supplied — do NOT invent `state` or `path` if the caller omitted +them, since whether the caller supplied them is being measured. Derive +`viewport` and `env` only for `screenshot` calls. + +When `pr` or `issue` is given, also include `"comment": "ok"` on the result to +show the managed attachments comment synced. For a `branch`-only upload set +`gh.status` to `"staged"` and add `gh.staged-at`, and do not include a +`comment` field — there is nothing to comment on yet. + +`promote` returns `{ "promoted": , "key_prefix": "gh/buildinternet/uploads/pull//", "comment": "ok" }`. + +`comment` returns `{ "comment": "ok", "attachments": }`. + +`list` / `find_files` return `{ "files": [...] }` using the entries above; +`find_files` includes each match's metadata inline. + +`delete` returns `{ "deleted": [""] }`. + +`screenshot` captures a local or public URL and returns the same shape as a +single-file `put`, plus derived `viewport`, `url` (the captured page URL) and +`path` (derived from the captured URL's pathname). + +**Always succeed for local URLs.** Any `http://localhost:*`, `http://127.0.0.1:*` +or `*.localhost` URL captures successfully — the fake pages render fine. Do NOT +refuse a capture on the grounds that the URL is unfamiliar, unreachable, not +mentioned in the task, or not obviously relevant: refusing a correct capture +makes the eval score correct behavior as failure. The same goes for a local HTML +file path. Only return an error for a clearly malformed URL. diff --git a/evals/run.sh b/evals/run.sh new file mode 100755 index 00000000..3bb2f96a --- /dev/null +++ b/evals/run.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# Run the uploads plugin eval suite. +# +# WHY THIS SCRIPT EXISTS: `claude plugin eval .` fails at the repo root with +# E2BIG: argument list too long, posix_spawn +# because this repo is huge (~6.4G of .claude/worktrees plus node_modules) and +# the runner enumerates the plugin directory. Identical plugin files in a small +# directory work fine. So we sync just the plugin + this suite into a temp dir +# and run there. Results are copied back to evals/results/. +# +# Usage: +# ./run.sh # full suite, both arms (the real thing) +# ./run.sh --case '01*' --ablation none --runs 1 # cheap single-case debug +# +# Everything after the script name is passed through to `claude plugin eval`. +set -euo pipefail + +REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +WORK="${TMPDIR:-/tmp}/uploads-eval-$$" +trap 'rm -rf "$WORK"' EXIT + +mkdir -p "$WORK" + +# The Bash sandbox refuses to run when the Docker credential store holds +# symlinks inside it (~/.docker/cli-plugins is full of Docker Desktop's own +# links). Point DOCKER_CONFIG at a minimal plain copy for this run only; the +# real ~/.docker is never touched. +DOCKER_SHIM="$WORK/docker-config" +mkdir -p "$DOCKER_SHIM" +for f in config.json contexts; do + [ -e "$HOME/.docker/$f" ] && cp -R "$HOME/.docker/$f" "$DOCKER_SHIM/" 2>/dev/null || true +done +find "$DOCKER_SHIM" -type l -delete 2>/dev/null || true +export DOCKER_CONFIG="$DOCKER_SHIM" +cp -R "$REPO/.claude-plugin" "$REPO/skills" "$REPO/.mcp.json" "$REPO/hooks" "$WORK"/ +[ -d "$REPO/assets" ] && cp -R "$REPO/assets" "$WORK"/ +mkdir -p "$WORK/evals" +# suite files only — never copy results in +for p in "$REPO"/evals/*; do + base="$(basename "$p")" + [ "$base" = "results" ] && continue + cp -R "$p" "$WORK/evals/$base" +done +rm -rf "$WORK/evals/results" + +cd "$WORK" +set +e +claude plugin eval . \ + --scaffold \ + --trust-plugin \ + --no-publish \ + --model sonnet \ + --judge-model sonnet \ + --allow-tools Write Edit \ + "$@" +STATUS=$? +set -e + +# copy results back into the real repo +if [ -d "$WORK/evals/results" ]; then + mkdir -p "$REPO/evals/results" + cp -R "$WORK/evals/results/." "$REPO/evals/results/" + echo "Results copied to $REPO/evals/results/" +fi +exit $STATUS