Add a ClawBench benchmark for Kernel MCP

[rgarcia]

What this adds

This PR adds a reproducible ClawBench benchmark and repository-native acceptance workflow for the browser-control tools exposed by kernel-mcp-server.

ClawBench supplies 129 real-website tasks and evaluates the browser request representing completion. Harbor runs each task with a stock agent and writes numeric rewards plus an ATIF trajectory. Hypeman provides one isolated environment per trial.

Each Kernel MCP trial:

1. builds this checkout into a source-pinned benchmark image;
2. creates one stealth Kernel browser through ClawBench;
3. starts Redis and the local Next.js MCP server in the trial environment;
4. exposes only get_connection_context and execute_playwright_code through mcp-remote;
5. requires every browser call to reuse ClawBench's exact session;
6. lets ClawBench score the request, finalize the replay, and delete the browser.

ClawBench remains the scorer and lifecycle owner. The additional kernel_mcp_valid metric confirms that a trial used the local source build and exact ClawBench browser.

Pull-request and scheduled benchmarks

.github/workflows/benchmark-clawbench.yml supports:

• a weekly full-suite run against main;
• manual runs for a same-repository PR, ref, full suite, or individual task;
• the exact PR comment /benchmark clawbench from an organization member or repository collaborator.

PR runs build the candidate and merge-base SHAs independently, then run both arms concurrently with a workflow-pinned task set, agent, model, judge, and concurrency. One bot comment is updated with each arm's observed configuration, lenient, strict, intercepted, infrastructure, validity, call, duration, and cost results plus candidate-minus-baseline deltas; incomplete arms produce a prominent warning instead of a partial comparison.

The comment trigger executes only the exact allowlisted command, rejects fork PRs and untrusted commenters, and resolves immutable SHAs through GitHub's API. Actions are pinned by SHA, checkout credentials are not persisted, shell inputs are passed through environment variables, and the workflow uses least-privilege job permissions.

Braintrust publication

bun run benchmark:publish reads completed Harbor job directories and publishes all synchronized arms into one private Braintrust experiment using the public Braintrust API.

• experiment, row, and child-span IDs are deterministic, so publishing the same job directories again replaces the same data and refreshes experiment metadata instead of duplicating it;
• infrastructure failures emit only infrastructure/ungraded scores and never task-quality scores;
• rows contain task identity, numeric rewards, arm/source provenance, bounded errors, and timing/token/call/cost metrics;
• ATIF agent and tool activity becomes child spans;
• environment secrets, credential-shaped strings, token/JWT query parameters, cookie headers, live-view paths, password fields, and email addresses are redacted before publication;
• task instructions, ground truth, browser session URLs, and recordings are not placed on experiment rows or public PR comments.

bun run benchmark:report produces the same redacted Markdown and JSON summaries used by the workflow.

Run locally

./benchmarks/harbor/build-image.sh

export CLAWBENCH_REPO=../ClawBench
./benchmarks/harbor/clawbench/run.sh codex \
  v2-1134-chapter-finder-redcross

Run all tasks:

HARBOR_N_CONCURRENT=10 \
  ./benchmarks/harbor/clawbench/run.sh codex all

See benchmarks/harbor/README.md for credentials, model/version overrides, result paths, reporting, and publication commands.

Validation

• bun test — 261 passed
• bunx tsc --noEmit --incremental false — clean
• production Next.js build — passed with the benchmark image's build-time environment
• targeted Prettier check — clean
• Python compilation and shell syntax checks — clean
• Actionlint — clean
• Zizmor workflow audit — no findings
• generated Red Cross Harbor task transformation — passed against pinned ClawBench source
• Harbor result parser exercised against an existing 129-task job
• publisher tests cover reward selection, numeric scoring, infrastructure isolation, deterministic rows/spans, metadata-refreshing re-publication, incomplete-arm reporting, and redaction

The earlier three-arm full-suite validation retained and deletion-verified all 387 browser replays. The description-only benchmark candidate now lives in the dependent follow-on PR #169.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
mcp	Ready	Preview	Aug 29, 2026 12:36am

[cursor]

Bugbot request id: serverGenReqId_419a23ae-aea4-49c2-b30f-581f8ea90ad3

[cursor]

Bugbot rules debug

Bugbot rules included in this run

Rule	Type	Size	Status
CI failure handlers need brace groups with separators (learned-9f771265-bf35-4268-89a2-9e93f7cd319b)	learned	828 chars	included
Public-repo reusable workflows: public callee, explicit secrets, thin callers (learned-9686e6b6-c72a-4523-b1de-9bcf49bccf6e)	learned	909 chars	included
Centralize MCP protocol identifiers and capability parsing in client-capabilities.ts (learned-927c89cc-d35d-4410-b389-a60e72507ef6)	learned	1,000 chars	included
MCP contract tests must use real McpServer + Client + InMemoryTransport (learned-e81b0e99-accf-4ca0-a7ad-76e87ca1b04a)	learned	952 chars	included
Test suites using kernelClientMock must restore factory in afterEach (learned-78ed3fa1-73c0-4cfe-b825-0da52cd1b6bb)	learned	899 chars	included
Don't flag truthiness checks on binary (object-or-null) API response fields (learned-78a33748-9390-4f87-9227-25e53811484a)	learned	846 chars	included
MCP resources iterate all pages; tool list actions return single pages (learned-bc868ab8-13fa-44f6-8be5-00648f7dabe9)	learned	864 chars	included
MCP Zod schemas: match SDK parameter semantics, not response shapes (learned-8a00013b-1c05-4767-8ee1-c2cf035b4565)	learned	976 chars	included
Derive SDK-facing types from KernelClient, don't redeclare (learned-8affb170-c394-4faa-b176-3f1d14740be8)	learned	916 chars	included
Use z.number().int() for integer params in MCP tool schemas (learned-9ea319a3-b989-41c2-8e7d-cb258726b13a)	learned	892 chars	included
Do not flag missing SHA pinning on kernel-org reusable workflows (learned-04ffe74f-4f3b-4b88-a4d5-7a1594a563a6)	learned	607 chars	included

https://cursor.com/docs/bugbot#team-rules

Bugbot request id: serverGenReqId_419a23ae-aea4-49c2-b30f-581f8ea90ad3

[rgarcia]

@bmsaadat addressed the review in 17dc7a7, including every blocking item and the fast-follow list:

• comparisons now use the merge base, and failed/timed-out arms produce an explicit incomplete-result warning instead of a partial delta;
• errors are redacted before truncation, trial VMs receive only the selected provider's credentials, and PEM files are excluded from image contexts;
• reward selection and infra-rate semantics are shared, experiment metadata refreshes on re-publish, and the additional token/cookie/live-view redaction gaps are covered;
• workflow-level versions are pinned and observed per-arm configs are reported; ungraded counts, span IDs, score clamping, and fractional deltas are corrected.

Validation: 261 tests, production build, TypeScript, Prettier, Python/shell checks, Actionlint, Zizmor, and parsing an existing 129-task result. GitHub CI and the automated reviews are green. Cursor BugBot found two additional issues around Codex tool-name validation and env-file serialization; both are fixed in 685e66a, and BugBot is green on that commit.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 17dc7a7. Configure here.}

[bmsaadat]

Thanks for the thorough turnaround, this addressed everything. Verified locally at 685e66a: 261 tests pass, tsc clean, and I re-read each fix. Merge-base resolution on all three paths, the incomplete-run warning with suppressed deltas, redact-then-truncate on errors, per-agent credential split, and *.pem are all exactly what I was hoping for. Appreciate you also picking up the non-blocking items (shared reward selection, metadata PATCH on re-publish, workflow-level pins with the per-arm configuration column). Approving.