Add a ClawBench benchmark for Kernel MCP

.dockerignore

@@ -0,0 +1,9 @@
+.git
+.github
+.next
+node_modules
+coverage
+.env*
+*.log
+*.pem
+benchmarks/harbor/.image.env

.gitignore

@@ -1,5 +1,7 @@
 # Dependencies
 node_modules/
+__pycache__/
+*.py[cod]
 npm-debug.log*
 yarn-debug.log*
 yarn-error.log*
@@ -107,5 +109,9 @@ Makefile
 # private key
 mcp-key.pem
 
+# Harbor benchmark runtime data
+benchmarks/harbor/.image.env
+benchmarks/harbor/image/source-sha
+
 # TypeScript incremental build cache
 tsconfig.tsbuildinfo

README.md

@@ -298,7 +298,7 @@ Each Kernel feature has a single `manage_*` tool with an `action` parameter, kee
 
 One additional Managed Auth helper (`begin_auth_login`) is marked app-only (`_meta.ui.visibility: ["app"]`); it refuses to execute on hosts that do not declare MCP Apps support. The App forwards the server-issued signed flow checkpoint to the shared `manage_auth_connections` `wait` action, so flow identity and terminal-state decisions stay on the server.
 
-Self-hosted deployments can hide sensitive tool families by setting `KERNEL_MCP_DISABLED_TOOLSETS` to a comma-separated list. For example, `KERNEL_MCP_DISABLED_TOOLSETS=api_keys` prevents `manage_api_keys` from being registered.
+Self-hosted deployments can select tool families with `KERNEL_MCP_ENABLED_TOOLSETS` or hide them with `KERNEL_MCP_DISABLED_TOOLSETS`. Both accept comma- or space-separated toolset names and standalone aliases. For example, `KERNEL_MCP_ENABLED_TOOLSETS="playwright computer"` exposes browser-control tools without browser lifecycle or managed-auth tools, while `KERNEL_MCP_DISABLED_TOOLSETS=api_keys` only removes `manage_api_keys`. `get_connection_context` remains available in either mode.
 
 Call `get_connection_context` before deciding whether to create or select a project. Its canonical `connection_scope` reports whether the connection is organization-wide or fixed to a project. Project-scoped tools advertise an optional `project` (name or ID) and a deprecated `project_id`: organization-wide connections may omit them to preserve organization-wide reads and API default-project behavior, while fixed-project connections may omit them or pass the matching project. Project resources use project-qualified `kernel://orgs/{organizationId}/projects/{projectId}/...` URIs. Authorization remains enforced by the Kernel API; selecting a project never grants access to it.
 

benchmarks/harbor/README.md

@@ -0,0 +1,101 @@
+# Benchmark Kernel MCP with ClawBench
+
+[ClawBench](https://github.com/TIGER-AI-Lab/ClawBench) is a suite of browser tasks. Each task describes work to complete on a real website and an evaluator that watches for the network request representing completion. ClawBench then judges the submitted request parameters.
+
+[Harbor](https://github.com/laude-institute/harbor) runs those tasks as reproducible agent trials. For each trial, Harbor creates an isolated environment, installs a stock agent such as Codex or Claude Code, gives it the task's MCP tools and instruction, runs the verifier, and writes the reward and ATIF trajectory to a job directory.
+
+This benchmark uses `harbor_hypeman:HypemanEnvironment` as Harbor's execution backend. For every trial, Harbor asks Hypeman to start an isolated VM from this repository's benchmark image. Everything for that trial runs inside that VM:
+
+1. ClawBench creates one stealth Kernel browser and attaches its request evaluator.
+2. The task setup starts Redis and the locally built `kernel-mcp-server` on port 3002.
+3. Harbor starts the stock agent with a stdio MCP command that connects to that local server.
+4. The agent controls ClawBench's existing browser through `execute_playwright_code`; it cannot create browsers or use managed auth.
+5. ClawBench scores the intercepted request, downloads the replay, and deletes the browser.
+
+The image records the current Git SHA, and the generated task records the ClawBench SHA and browser session ID. The additional `kernel_mcp_valid` result confirms that the agent called the local server with the browser ClawBench created. Task reward still comes directly from ClawBench.
+
+## Requirements
+
+- `uv`, Harbor 0.21.0, and `harbor-hypeman` 0.1.1
+- Hypeman CLI credentials
+- a ClawBench checkout containing pinned commit `45a71c4`
+- `KERNEL_MCP_BENCHMARK_API_KEY` scoped to an isolated evaluation project, plus its `KERNEL_PROJECT` name
+- `PURELY_MAIL_API_KEY` and `PURELY_MAIL_DOMAIN` for ClawBench account tasks
+- `OPENAI_API_KEY` for Codex, or Anthropic credentials for Claude Code
+- the ClawBench judge variables when using a hosted judge: `CLAWBENCH_JUDGE_BASE_URL`, `CLAWBENCH_JUDGE_API_KEY`, `CLAWBENCH_JUDGE_MODEL`, and `CLAWBENCH_JUDGE_API_TYPE`
+- `BRAINTRUST_API_KEY` and `BRAINTRUST_PROJECT` when publishing results
+
+## Build the trial image
+
+From the `kernel-mcp-server` checkout:
+
+```bash
+./benchmarks/harbor/build-image.sh
+```
+
+This builds the current checkout with Bun and writes the image reference and Git SHA to the ignored `benchmarks/harbor/.image.env` file.
+
+## Run one task
+
+```bash
+export CLAWBENCH_REPO=../ClawBench
+./benchmarks/harbor/clawbench/run.sh codex \
+  v2-1134-chapter-finder-redcross
+```
+
+## Run the full suite
+
+```bash
+export CLAWBENCH_REPO=../ClawBench
+HARBOR_N_CONCURRENT=10 \
+  ./benchmarks/harbor/clawbench/run.sh codex all
+```
+
+Codex defaults to version `0.120.0` with `gpt-5.6-luna`. Claude Code defaults to version `2.1.238` with `claude-sonnet-5`. Override these with `CODEX_BENCHMARK_MODEL`, `CODEX_BENCHMARK_VERSION`, `CLAUDE_BENCHMARK_MODEL`, or `CLAUDE_BENCHMARK_VERSION`.
+
+Single-task runs have a 40-minute wall-clock limit. Full-suite runs default to six hours. Set `HARBOR_BENCHMARK_TIMEOUT` to override either limit. Set `HARBOR_JOBS_DIR` to choose where Harbor writes results.
+
+## GitHub Actions
+
+The `Benchmark ClawBench` workflow runs the complete suite weekly and on demand. Select it from the Actions tab and provide either a same-repository PR number or a ref. Comparison runs benchmark the candidate SHA against its merge base so unrelated changes on the target branch do not affect the delta. Harbor, Hypeman, agent, and model versions are pinned by the workflow and each arm's observed agent configuration appears in the report.
+
+An organization member or repository collaborator can also start the full PR comparison by commenting this exact command on a same-repository pull request:
+
+```text
+/benchmark clawbench
+```
+
+The command parser does not execute comment text. It accepts only the exact command, rejects fork pull requests and untrusted commenters, and resolves the candidate and merge-base SHAs through GitHub's API. The workflow uses the `benchmarks` environment for credentials, updates one benchmark comment on the pull request, and publishes the same results to Braintrust.
+
+## Results
+
+Harbor writes its normal job directory, including:
+
+- `trajectory.json`: the agent's ATIF messages and tool calls
+- `reward.json`: ClawBench's reward plus the `kernel_mcp_valid` diagnostic
+- `clawbench-result.json`: evaluator details
+- `kernel-mcp-result.json`: local-source and same-browser wiring details
+- `recording.mp4`: the finalized Kernel replay
+- `kernel-mcp/`: local server logs and the source/session manifest
+
+Generate a redacted summary from one or more completed jobs:
+
+```bash
+bun run benchmark:report -- \
+  --arm candidate=/path/to/candidate-job \
+  --arm baseline=/path/to/baseline-job \
+  --json /tmp/benchmark-summary.json \
+  --markdown /tmp/benchmark-summary.md
+```
+
+Publish those arms as one idempotent Braintrust experiment:
+
+```bash
+BRAINTRUST_PROJECT=kernel-mcp-server-benchmarks \
+  bun run benchmark:publish -- \
+    --experiment pr-162-a60c518-example \
+    --arm candidate=/path/to/candidate-job \
+    --arm baseline=/path/to/baseline-job
+```
+
+The experiment name and deterministic row/span IDs make it safe to publish the same job directories again. Re-publication replaces the rows and refreshes experiment metadata. Rows contain task identity, numeric rewards, provenance, bounded errors, timing, token, call, and cost metrics. ATIF agent/tool activity is attached as child spans after secret redaction. Task instructions, ground truth, browser session URLs, and recordings are not placed on experiment rows or public pull-request comments.

benchmarks/harbor/bin/kernel-mcp-local

@@ -0,0 +1,18 @@
+#!/bin/sh
+# Harbor launches MCP servers as stdio subprocesses of the benchmark agent.
+# This wrapper turns that stdio connection into an authenticated connection to
+# the kernel-mcp-server HTTP endpoint running locally in the same Hypeman task.
+# The setup script writes the project-scoped key to /run so it never appears in
+# the generated agent configuration.
+set -eu
+
+key_file=/run/kernel-mcp-benchmark/api-key
+if [ -z "${KERNEL_API_KEY:-}" ] && [ -r "$key_file" ]; then
+  KERNEL_API_KEY=$(cat "$key_file")
+  export KERNEL_API_KEY
+fi
+: "${KERNEL_API_KEY:?KERNEL_API_KEY is required}"
+
+exec mcp-remote \
+  http://127.0.0.1:3002/mcp \
+  --header "Authorization: Bearer ${KERNEL_API_KEY}"

benchmarks/harbor/bin/start-kernel-mcp-server

@@ -0,0 +1,69 @@
+#!/bin/bash
+# Start the services used by the source-pinned Kernel MCP build inside a Harbor
+# trial. ClawBench has already created the browser; this script starts Redis and
+# Next.js, stores the project-scoped key for the stdio wrapper, waits for MCP to
+# accept connections, and records which source build and browser the trial used.
+set -euo pipefail
+
+: "${KERNEL_API_KEY:?KERNEL_API_KEY is required}"
+
+log_dir=/logs/kernel-mcp
+key_dir=/run/kernel-mcp-benchmark
+mkdir -p "$log_dir" /logs/artifacts "$key_dir"
+chmod 0777 "$log_dir" /logs/artifacts
+chmod 0700 "$key_dir"
+printf '%s' "$KERNEL_API_KEY" >"$key_dir/api-key"
+chmod 0600 "$key_dir/api-key"
+
+redis-server --daemonize yes --bind 127.0.0.1 --port 6379 \
+  --logfile "$log_dir/redis.log" --dir /tmp
+
+export CLERK_SECRET_KEY=${CLERK_SECRET_KEY:-sk_test_kernel_mcp_benchmark_local_only}
+export NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=${NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY:-pk_test_YmVuY2htYXJrLmNsZXJrLmFjY291bnRzLmRldiQ}
+
+cd /opt/kernel-mcp-server
+nohup ./node_modules/.bin/next start -p 3002 \
+  >"$log_dir/server.stdout.log" \
+  2>"$log_dir/server.stderr.log" &
+echo $! >"$log_dir/server.pid"
+
+for _ in $(seq 1 90); do
+  if curl -fsS -X POST http://127.0.0.1:3002/mcp \
+    -H "Authorization: Bearer ${KERNEL_API_KEY}" \
+    -H "Content-Type: application/json" \
+    -H "Accept: application/json, text/event-stream" \
+    --data '{"jsonrpc":"2.0","id":"benchmark-healthcheck","method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"harbor-healthcheck","version":"1.0.0"}}}' \
+    >"$log_dir/initialize-response.txt"; then
+    break
+  fi
+  sleep 1
+done
+
+if [ ! -s "$log_dir/initialize-response.txt" ]; then
+  echo "Kernel MCP server did not become ready" >&2
+  tail -100 "$log_dir/server.stderr.log" >&2 || true
+  exit 1
+fi
+
+python3 - <<'PY'
+import json
+import os
+from pathlib import Path
+
+browser_path = Path("/my-info/kernel_browser.json")
+try:
+    browser = json.loads(browser_path.read_text())
+except (OSError, json.JSONDecodeError):
+    browser = {}
+
+manifest = {
+    "kernel_mcp_server_sha": Path("/opt/kernel-mcp-server/SOURCE_SHA").read_text().strip(),
+    "clawbench_source_sha": os.environ.get("CLAWBENCH_SOURCE_SHA", ""),
+    "browser_session_id": browser.get("session_id"),
+    "enabled_toolsets": os.environ.get("KERNEL_MCP_ENABLED_TOOLSETS", ""),
+    "image": os.environ.get("KERNEL_MCP_BENCHMARK_IMAGE", ""),
+}
+Path("/logs/kernel-mcp/run-manifest.json").write_text(json.dumps(manifest, indent=2))
+PY
+
+printf 'ready\n' >"$log_dir/ready"

benchmarks/harbor/build-image.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+set -euo pipefail
+
+repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
+cd "$repo_root"
+
+source_sha=$(git rev-parse HEAD)
+source_sha_file=benchmarks/harbor/image/source-sha
+build_log=$(mktemp)
+trap 'rm -f "$source_sha_file" "$build_log"' EXIT
+
+printf '%s\n' "$source_sha" >"$source_sha_file"
+
+set +e
+hypeman build \
+  --file benchmarks/harbor/image/Dockerfile \
+  --cpus 4 \
+  --memory 8192 \
+  --timeout 1800 \
+  . 2>&1 | tee "$build_log"
+build_status=${PIPESTATUS[0]}
+set -e
+
+build_id=$(sed -n -E 's/^Build (ID|started): //p' "$build_log" | tail -1)
+if [[ -z "$build_id" ]]; then
+  echo "Hypeman did not return a build ID" >&2
+  exit 1
+fi
+
+image_ref="docker.io/builds/$build_id:latest"
+if ((build_status != 0)); then
+  echo "Build record failed; checking for a delayed ready image for up to 5 minutes" >&2
+  image_ready=false
+  for _ in $(seq 1 30); do
+    if hypeman --format json image list | python3 -c '
+import json
+import sys
+
+image_ref = sys.argv[1]
+expected = {image_ref, image_ref.removeprefix("docker.io/")}
+images = json.load(sys.stdin)
+raise SystemExit(
+    0
+    if any(image.get("name") in expected and image.get("status") == "ready" for image in images)
+    else 1
+)
+' "$image_ref"
+    then
+      image_ready=true
+      break
+    fi
+    sleep 10
+  done
+  if [[ "$image_ready" != true ]]; then
+    exit "$build_status"
+  fi
+fi
+
+cat >benchmarks/harbor/.image.env <<EOF
+KERNEL_MCP_BENCHMARK_IMAGE=$image_ref
+KERNEL_MCP_SOURCE_SHA=$source_sha
+EOF
+chmod 0600 benchmarks/harbor/.image.env
+
+printf 'Wrote benchmarks/harbor/.image.env for %s\n' "$image_ref"