From 4ff44d2671a2e4cc1d3da1f61cb362e7e49bfdf4 Mon Sep 17 00:00:00 2001 From: Alex Lavaee Date: Wed, 4 Nov 2026 16:39:12 -0800 Subject: [PATCH 1/2] Classify and log why /research web search setup fails A research request whose web search could not be set up failed with the same opaque `web-search-unavailable` whatever the cause. In stage(), the `!web.ok` branch threw a fresh generic error and dropped the underlying cause. webSearchTool() never logged it. A missing `web_search` tool is returned from a path outside both catch blocks, so a console.error added to those blocks, as the reporter did, cannot show it. The failure line also carries no failure time. The only `elapsedMs` in the logs belongs to the `connecting-web` progress snapshot, which is when that phase began, so the failure read as instant. A local fake MCP server run through the real @ai-sdk/mcp HTTP transport shows this. HTTP 401, HTTP 403, HTTP 500, a JSON-RPC error while listing tools, and a tool list without `web_search` all produced the same `web-search-unavailable` line and card. Only the missing-tool case bypasses both catch blocks. With a 400 ms endpoint the attempt failed after 813 ms while the only `elapsedMs` logged was 0. Against the live endpoint, a fake `ghu_`-shaped token produced the same line the reporter posted (HTTP 401 on connect). The live endpoint answers HTTP 401 for fake `ghu_`, `gho_`, `ghp_` and `github_pat_` tokens alike, so these probes cannot show how it treats a real App user access token. webSearchTool() now classifies a setup failure as it happens and logs it once: the step (connect or tools), a token-safe classification with any HTTP status and JSON-RPC code, how long setup ran, the credential type by prefix (`ghu_` or `gho_`, never the token), and the client's error name and message collapsed to one line, with token-shaped text, `Bearer` values and the exact credential masked and clipped. The research stage records the classification and duration in the job diagnostic. An HTTP 401 or 403 now fails as `web-search-credential-rejected`, a connection without `web_search` as `web-search-not-offered`, and every other failure stays `web-search-unavailable`. The research card says which one happened and points at the server log. Self-hosting, background-jobs, authentication, conversation-research and the README now say what the endpoint is called with, how to read the log line, what an operator can check, and that whether it accepts a custom GitHub App's user access tokens for `web_search` is unconfirmed. This does not make web search work on such an instance. Whether GitHub's Copilot MCP endpoint refuses or withholds `web_search` for GitHub App user access tokens could not be established without one, so the cause behind the report stays unproven. Refs #699 Assistant-model: Claude Sonnet 5.5 Assistant-workflow: goal (run ca8f48c1-dfc2-4573-a32d-5037607e08ed) Assistant-verification: bun test apps/server/src/{harness,jobs,research} passed: 547 pass, 0 fail across 43 files, including web-search, research-workspace, request-view and runner tests Assistant-verification: bun run types passed: every workspace and e2e type-check exited 0 Assistant-verification: bun run ci passed: dprint, oxlint (0 errors, 72 existing warnings), token, design contract, design record and design checks Assistant-verification: red-green and mutation checks passed: the new credential-rejection test failed on the unfixed code, and removing the 403 case, exact-token masking, token-shape masking, clipping, or the not-offered reason failed a test each Assistant-verification: local fake-MCP reproduction through the real @ai-sdk/mcp transport passed: before, every setup failure logged an identical web-search-unavailable line; after, 401 and 403 are credential-rejected, a missing tool is not-offered, and the others stay unavailable with a logged duration Assistant-verification: live endpoint probe with fake tokens passed: HTTP 401 for every token shape, so it cannot show how a real ghu_ token is treated Assistant-verification: real ghu_ token check unavailable: no real GitHub App user access token was held, so the cause behind the report is unproven Assistant-verification: relative link and anchor check of the changed docs passed: every link resolves Assistant-duration: unmeasured Co-authored-by: Alex Lavaee --- README.md | 5 + apps/server/src/harness/web-search.test.ts | 172 +++++++++++++++++- apps/server/src/harness/web-search.ts | 105 +++++++++-- .../src/jobs/research-workspace.test.ts | 69 +++++++ apps/server/src/jobs/research-workspace.ts | 4 +- apps/server/src/research/request-view.test.ts | 9 + apps/server/src/research/request-view.ts | 5 + docs/authentication.md | 7 +- docs/background-jobs.md | 14 ++ docs/conversation-research.md | 7 + docs/self-hosting.md | 48 ++++- 11 files changed, 425 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 7886ebe82..7a58bbb67 100644 --- a/README.md +++ b/README.md @@ -185,6 +185,11 @@ Compose project. Set `AGENT=off` to prevent Planner turns and disable the background-job runner. This does not disable the `/mcp` endpoint used by external coding agents. +`/research` public web search calls GitHub's Copilot MCP endpoint with the signed-in +user's App user access token. Whether that endpoint accepts a self-registered App's +tokens for `web_search` is unconfirmed. If it refuses, the research card and server log +say so, and [Self-hosting](docs/self-hosting.md#troubleshooting) lists what to check. + ## Collaborate with the Planner The web composer treats `@chopin` as an instruction for the Planner: diff --git a/apps/server/src/harness/web-search.test.ts b/apps/server/src/harness/web-search.test.ts index 3bcc3ba0e..9b7712348 100644 --- a/apps/server/src/harness/web-search.test.ts +++ b/apps/server/src/harness/web-search.test.ts @@ -86,6 +86,7 @@ it("loads only the host GitHub MCP web_search and binds execution to the owner", it("fails closed when MCP does not offer web_search or connection fails", async () => { let closed = 0; let missing = await webSearchTool(credential(), { + log: () => {}, createClient: async () => ({ tools: async () => ({}), close: async () => { @@ -93,9 +94,10 @@ it("fails closed when MCP does not offer web_search or connection fails", async }, }), }); - expect(missing).toEqual({ ok: false, error: { kind: "MissingTool", name: "web_search" } }); + expect(missing).toMatchObject({ ok: false, error: { kind: "MissingTool", name: "web_search" } }); expect(closed).toBe(1); let unavailable = await webSearchTool(credential(), { + log: () => {}, createClient: async () => { throw new Error("offline"); }, @@ -103,6 +105,174 @@ it("fails closed when MCP does not offer web_search or connection fails", async expect(unavailable).toMatchObject({ ok: false, error: { kind: "Unavailable" } }); }); +it("names a credential the endpoint refuses at connect, with its type and a redacted message", async () => { + let secret = "ghu_abcdefghijklmnopqrstuvwxyz0123456789"; + let logs: string[] = []; + let result = await webSearchTool({ ...credential(), token: secret }, { + log: line => logs.push(line), + createClient: async () => { + throw new MCPClientError({ + message: "MCP HTTP Transport Error: POSTing to endpoint (HTTP 401): " + + `unauthorized: token authentication failed for Bearer ${secret}`, + statusCode: 401, + url: "https://api.githubcopilot.com/mcp/", + }); + }, + }); + expect(result).toMatchObject({ + ok: false, + error: { + kind: "Unavailable", + failure: { + reason: "web-search-credential-rejected", + classification: "local:connect:mcp-client:http:401", + durationMs: expect.any(Number), + }, + }, + }); + expect(logs).toHaveLength(1); + expect(logs[0]).toStartWith("chopin: web_search unavailable at connect - "); + expect(logs[0]).toContain("local:connect:mcp-client:http:401"); + expect(logs[0]).toContain("GitHub App user access token (ghu_)"); + expect(logs[0]).toContain("unauthorized: token authentication failed"); + expect(logs[0]).not.toContain(secret); + expect(logs[0]).not.toContain("abcdefghijkl"); + expect(logs[0]).not.toContain("\n"); +}); + +it("reads a refusal while listing tools as the same rejected credential", async () => { + let logs: string[] = []; + let closed = 0; + let result = await webSearchTool({ ...credential(), token: "gho_0123456789abcdefghijklmnop" }, { + log: line => logs.push(line), + createClient: async () => ({ + tools: async () => { + throw new MCPClientError({ + message: "MCP HTTP Transport Error: POSTing to endpoint (HTTP 403): forbidden", + statusCode: 403, + code: -32001, + }); + }, + close: async () => { + closed++; + }, + }), + }); + expect(result).toMatchObject({ + ok: false, + error: { + failure: { + reason: "web-search-credential-rejected", + classification: "local:tools:mcp-client:rpc:-32001:http:403", + }, + }, + }); + expect(closed).toBe(1); + expect(logs).toHaveLength(1); + expect(logs[0]).toStartWith("chopin: web_search unavailable at tools - "); + expect(logs[0]).toContain("OAuth app access token (gho_)"); +}); + +it("reports a connection that is accepted without web_search as not offered", async () => { + let logs: string[] = []; + let result = await webSearchTool({ + ...credential(), + token: "ghu_abcdefghijklmnopqrstuvwxyz0123456789", + }, { + log: line => logs.push(line), + createClient: async () => ({ + tools: async () => ({ + list_pull_requests: tool({ inputSchema: z.object({}), execute: async () => ({}) }), + }), + close: async () => {}, + }), + }); + expect(result).toMatchObject({ + ok: false, + error: { + kind: "MissingTool", + name: "web_search", + failure: { + reason: "web-search-not-offered", + classification: "local:tools:missing-tool", + }, + }, + }); + expect(logs).toEqual([ + expect.stringMatching( + /^chopin: web_search unavailable at tools - local:tools:missing-tool in \d+ms; credential: GitHub App user access token \(ghu_\); the endpoint answered but did not offer web_search; see Troubleshooting in docs\/self-hosting\.md$/, + ), + ]); +}); + +it("keeps other setup failures unavailable and says what kind of token was used", async () => { + let logs: string[] = []; + for ( + let cause of [ + new MCPClientError({ message: "bad gateway", statusCode: 502 }), + new MCPClientError({ message: "no such method", code: -32601 }), + Object.assign(new Error("Unable to connect"), { code: "ConnectionRefused" }), + "offline", + ] + ) { + let result = await webSearchTool({ ...credential(), token: "opaque-token-value" }, { + log: line => logs.push(line), + createClient: async () => { + throw cause; + }, + }); + expect(result).toMatchObject({ + ok: false, + error: { failure: { reason: "web-search-unavailable" } }, + }); + if (result.ok) throw new Error("fixture failed"); + expect(executionDiagnostic({ webFailure: result.error.failure.classification })) + .toBeDefined(); + } + expect(logs.map(line => /- (\S+) in /.exec(line)?.[1])).toEqual([ + "local:connect:mcp-client:http:502", + "local:connect:mcp-client:rpc:-32601", + "local:connect:error", + "local:connect:unknown", + ]); + for (let line of logs) { + expect(line).toContain("credential: token of an unrecognized type"); + expect(line).not.toContain("Troubleshooting"); + } +}); + +it("masks credentials and clips a setup failure before it is logged", async () => { + let secret = "opaque-credential-value"; + let message = [ + `MCP HTTP Transport Error: POSTing to endpoint (HTTP 401): rejected echo ${secret}`, + `Authorization: Bearer ${secret}`, + "retry with github_pat_11ABCDEFG0123456789_abcdefghijklmnop or ghs_abcdefghijklmnopqrstuvwxyz0123456789", + `\u001b[31mcolour\u001b[0m ${secret} ${"x".repeat(900)}`, + ].join("\n"); + for (let name of [`Bearer ${secret}`, "ghu_abcdefghijklmnopqrstuvwxyz0123456789"]) { + let logs: string[] = []; + await webSearchTool({ ...credential(), token: secret }, { + log: line => logs.push(line), + createClient: async () => { + throw Object.assign(new MCPClientError({ message, statusCode: 401 }), { name }); + }, + }); + expect(logs).toHaveLength(1); + let line = logs[0]!; + expect(line).not.toContain(secret); + expect(line).not.toContain("github_pat_"); + expect(line).not.toContain("ghs_"); + expect(line).not.toContain("ghu_abcdefghijkl"); + expect(line).not.toContain("abcdefghijkl"); + expect(line).not.toMatch(/\p{Cc}/u); + expect(line).toContain( + "rejected echo [redacted] Authorization: Bearer [redacted] retry with [redacted] or [redacted]", + ); + expect(line).toMatch(/; (?:Error|\[redacted\]): MCP HTTP Transport Error: POSTing/); + expect(line.length).toBeLessThan(500); + } +}); + it("does not count a resolved MCP error as a successful search", async () => { let results: unknown[] = []; let owner = credential(); diff --git a/apps/server/src/harness/web-search.ts b/apps/server/src/harness/web-search.ts index c8a3147c6..6c22f0e14 100644 --- a/apps/server/src/harness/web-search.ts +++ b/apps/server/src/harness/web-search.ts @@ -35,6 +35,87 @@ export type WebSearchFailureDiagnostic = { webFailureToolCallId?: string; }; +export type WebSearchSetupFailure = { + /** The reason the research request fails with, and the card explains. */ + reason: "web-search-credential-rejected" | "web-search-not-offered" | "web-search-unavailable"; + /** Token-safe, for example `local:connect:mcp-client:http:401`. */ + classification: string; + durationMs: number; +}; + +export type WebSearchSetupError = GitHubToolsError & { failure: WebSearchSetupFailure }; + +const CREDENTIAL_TYPES: Array<[prefix: string, name: string]> = [ + ["ghu_", "GitHub App user access token"], + ["gho_", "OAuth app access token"], +]; +const SECRET_TEXT = /\b(?:gh[pousr]_|github_pat_)[A-Za-z0-9_]+|\b(Bearer\s+)\S+/gi; + +function credentialType(token: string): string { + let known = CREDENTIAL_TYPES.find(([prefix]) => token.startsWith(prefix)); + return known ? `${known[1]} (${known[0]})` : "token of an unrecognized type"; +} + +function mcpCodes(error: unknown): string { + if (!MCPClientError.isInstance(error)) return ""; + let codes = ""; + if (Number.isSafeInteger(error.code)) codes += `:rpc:${error.code}`; + if ( + Number.isSafeInteger(error.statusCode) && error.statusCode! >= 100 && error.statusCode! <= 599 + ) codes += `:http:${error.statusCode}`; + return codes; +} + +/** One masked, clipped line of a setup error. No query has been sent when setup fails. */ +function setupMessage(error: unknown, token: string): string { + if (!(error instanceof Error)) return "a value that is not an Error was thrown"; + let mask = (text: string) => { + if (token.length >= 8) text = text.replaceAll(token, "[redacted]"); + return text.replace(SECRET_TEXT, (_match, bearer?: string) => `${bearer ?? ""}[redacted]`) + .replace(/[\s\p{Cc}]+/gu, " ") + .trim(); + }; + let name = /^[A-Za-z][A-Za-z0-9_.]{0,63}$/.test(error.name) ? mask(error.name) : "Error"; + return `${name}: ${mask(error.message).slice(0, 160)}`; +} + +function setupFailure( + step: "connect" | "tools", + error: GitHubToolsError, + context: { token: string; began: number; log: (line: string) => void }, +): WebSearchSetupError { + let cause = error.kind === "Unavailable" ? error.cause : undefined; + let refused = MCPClientError.isInstance(cause) + && (cause.statusCode === 401 || cause.statusCode === 403); + let kind = MCPClientError.isInstance(cause) + ? "mcp-client" + : cause instanceof Error + ? "error" + : "unknown"; + let failure: WebSearchSetupFailure = { + reason: error.kind === "MissingTool" + ? "web-search-not-offered" + : refused + ? "web-search-credential-rejected" + : "web-search-unavailable", + classification: error.kind === "MissingTool" + ? `local:${step}:missing-tool` + : `local:${step}:${kind}${mcpCodes(cause)}`, + durationMs: Math.round(performance.now() - context.began), + }; + context.log( + `chopin: web_search unavailable at ${step} - ${failure.classification} in ${failure.durationMs}ms; ` + + `credential: ${credentialType(context.token)}; ` + + (error.kind === "MissingTool" + ? "the endpoint answered but did not offer web_search" + : setupMessage(cause, context.token)) + + (failure.reason === "web-search-unavailable" + ? "" + : "; see Troubleshooting in docs/self-hosting.md"), + ); + return { ...error, failure }; +} + /** Calls one evidence attempt may start; the upstream tool takes about two minutes each. */ export const WEB_SEARCH_MAX_CALLS = 4; /** Below this much remaining attempt time a new call cannot finish usefully. */ @@ -81,7 +162,9 @@ export async function webSearchTool( now?: () => number; log?: (line: string) => void; } = {}, -): Promise Promise }, GitHubToolsError>> { +): Promise< + Result<{ tool: ToolSet["web_search"]; close: () => Promise }, WebSearchSetupError> +> { // The upstream tool runs an AI research agent, so a valid response can take over a minute. let timeoutMs = deps.timeoutMs ?? 180_000; let now = deps.now ?? Date.now; @@ -91,6 +174,7 @@ export async function webSearchTool( throw new Error("Invalid web search timeout"); } let client: Client; + let context = { token: credential.token, began: performance.now(), log }; try { client = await (deps.createClient ?? (createMCPClient as CreateClient))({ transport: { @@ -104,14 +188,14 @@ export async function webSearchTool( }, }); } catch (cause) { - return { ok: false, error: { kind: "Unavailable", cause } }; + return { ok: false, error: setupFailure("connect", { kind: "Unavailable", cause }, context) }; } try { let raw = await client.tools({ schemas: WEB_SEARCH_SCHEMA }); let picked = requireTools(raw, ["web_search"]); if (!picked.ok) { await client.close(); - return picked; + return { ok: false, error: setupFailure("tools", picked.error, context) }; } let source = picked.value.web_search!; if (!source.execute) throw new Error("MCP web_search has no execution handler"); @@ -209,20 +293,9 @@ export async function webSearchTool( ? "error" : "unknown"; let diagnostic: WebSearchFailureDiagnostic = { - webFailure: `local:${phase}:${kind}`, + webFailure: `local:${phase}:${kind}${mcpCodes(error)}`, webFailureDurationMs: Math.round(performance.now() - began), }; - if (MCPClientError.isInstance(error)) { - if (Number.isSafeInteger(error.code)) { - diagnostic.webFailure += `:rpc:${error.code}`; - } - if ( - Number.isSafeInteger(error.statusCode) && error.statusCode! >= 100 - && error.statusCode! <= 599 - ) { - diagnostic.webFailure += `:http:${error.statusCode}`; - } - } if ( /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test( options.toolCallId, @@ -246,6 +319,6 @@ export async function webSearchTool( }; } catch (cause) { await client.close().catch(() => {}); - return { ok: false, error: { kind: "Unavailable", cause } }; + return { ok: false, error: setupFailure("tools", { kind: "Unavailable", cause }, context) }; } } diff --git a/apps/server/src/jobs/research-workspace.test.ts b/apps/server/src/jobs/research-workspace.test.ts index 0a7a65e76..70663ea3c 100644 --- a/apps/server/src/jobs/research-workspace.test.ts +++ b/apps/server/src/jobs/research-workspace.test.ts @@ -510,6 +510,75 @@ describe("research workspace execution", () => { } }); + it.each([ + { + scenario: "a credential GitHub refuses", + createClient: async () => { + throw new MCPClientError({ + message: + "MCP HTTP Transport Error: POSTing to endpoint (HTTP 401): private provider payload", + statusCode: 401, + responseBody: "private provider payload", + }); + }, + reason: "web-search-credential-rejected", + webFailure: "local:connect:mcp-client:http:401", + }, + { + scenario: "a connection that does not offer web_search", + createClient: async () => ({ tools: async () => ({}), close: async () => {} }), + reason: "web-search-not-offered", + webFailure: "local:tools:missing-tool", + }, + { + scenario: "a service that cannot be reached", + createClient: async () => { + throw Object.assign(new Error("Unable to connect private provider payload"), { + code: "ConnectionRefused", + }); + }, + reason: "web-search-unavailable", + webFailure: "local:connect:error", + }, + ])("fails public research as $reason for $scenario and records why in its diagnostic", async ( + { createClient, reason, webFailure }, + ) => { + let originalWebSearch = webSearch.webSearchTool; + let snapshots: JobExecutionDiagnostic[] = []; + let web = spyOn(webSearch, "webSearchTool").mockImplementation((owner, deps) => + originalWebSearch(owner, { ...deps, createClient, log: () => {} }) + ); + let opening = spyOn(workerSession, "openWorkerSession"); + try { + let definition = researchEvidenceDefinition({ config: { agent: true, model: "test-model" } }); + let error = await definition.execute({ + ...execution("research-evidence", { + workspaceId: WORKSPACE_ID, + turnId: TURN_ID, + query: QUERY, + }), + diagnostic: value => snapshots.push(executionDiagnostic(value)!), + }).then(() => undefined, error => error); + expect(error).toBeInstanceOf(JobExecutionError); + expect(error.progressReason).toBe(reason); + expect(error.diagnostic).toMatchObject({ + stage: "public-web", + phase: "connecting-web", + webFailure, + webFailureDurationMs: expect.any(Number), + webCalls: 0, + webFailures: 0, + }); + expect(Object.keys(error.diagnostic).length).toBeLessThanOrEqual(20); + expect(JSON.stringify(error.diagnostic)).not.toContain("private provider payload"); + expect(error.message).toBe(`Background job execution failed: ${reason}.`); + expect(opening).not.toHaveBeenCalled(); + } finally { + opening.mockRestore(); + web.mockRestore(); + } + }); + it("keeps the public evidence worker input isolated and retries once", async () => { let observed: unknown; let progress: string[] = []; diff --git a/apps/server/src/jobs/research-workspace.ts b/apps/server/src/jobs/research-workspace.ts index 54583b6d3..c37692ae7 100644 --- a/apps/server/src/jobs/research-workspace.ts +++ b/apps/server/src/jobs/research-workspace.ts @@ -956,8 +956,10 @@ async function stage( throw err; } if (!web.ok) { + let { reason, classification, durationMs } = web.error.failure; + metrics.webFailure = { webFailure: classification, webFailureDurationMs: durationMs }; throw publicStageError( - "web-search-unavailable", + reason, metrics, new Error( web.error.kind === "MissingTool" diff --git a/apps/server/src/research/request-view.test.ts b/apps/server/src/research/request-view.test.ts index fe161ca02..5c92d4f1e 100644 --- a/apps/server/src/research/request-view.test.ts +++ b/apps/server/src/research/request-view.test.ts @@ -9,6 +9,15 @@ test.each([ "Authorization for public web search timed out.", ], ["web-search-unavailable", "The public web-search service was unavailable."], + [ + "attempts-exhausted:web-search-credential-rejected", + "GitHub refused the credential from this sign-in for public web search. " + + "The server log has the details.", + ], + [ + "attempts-exhausted:web-search-not-offered", + "GitHub did not offer public web search to this sign-in. The server log has the details.", + ], ])("maps the research failure %s to a safe explanation", (reason, expected) => { expect(researchFailureMessage(reason)).toBe(expected); }); diff --git a/apps/server/src/research/request-view.ts b/apps/server/src/research/request-view.ts index bfba5f92a..1557deed7 100644 --- a/apps/server/src/research/request-view.ts +++ b/apps/server/src/research/request-view.ts @@ -10,6 +10,11 @@ export function researchFailureMessage(reason: string | undefined): string { "web-search-timeout": "A public web-search request timed out.", "web-search-authorization-timeout": "Authorization for public web search timed out.", "web-search-unavailable": "The public web-search service was unavailable.", + "web-search-credential-rejected": + "GitHub refused the credential from this sign-in for public web search. " + + "The server log has the details.", + "web-search-not-offered": + "GitHub did not offer public web search to this sign-in. The server log has the details.", "web-search-failed": "Public web search did not complete successfully.", "web-search-not-invoked": "The research model did not perform a web search.", "public-session-failed": "The public research model could not complete its session.", diff --git a/docs/authentication.md b/docs/authentication.md index 3a456015f..1196fac6a 100644 --- a/docs/authentication.md +++ b/docs/authentication.md @@ -60,7 +60,12 @@ approve it. Chopin uses no App ID, private key, JWT, installation access token, or webhook secret. It acts as each signed-in user, with their App user access token, -so repository-role checks and Copilot entitlement stay theirs. +so repository-role checks and Copilot entitlement stay theirs. That token is also +what Chopin sends to GitHub's Copilot MCP endpoint for the Planner's pull request +tools and for public web search in `/research`. GitHub does not document which tokens +that endpoint accepts for `web_search`; see +[Troubleshooting](self-hosting.md#troubleshooting) if `/research` reports a refused +credential or no web search. Configure Chopin with the App's slug and OAuth client credentials. A minimal production environment contains: diff --git a/docs/background-jobs.md b/docs/background-jobs.md index bbc6bbbf7..9f134aa47 100644 --- a/docs/background-jobs.md +++ b/docs/background-jobs.md @@ -437,6 +437,20 @@ claim; heartbeats and lease renewal do not depend on it. Use the host wrapper's search counters for live progress: the harness can buffer tool lifecycle callbacks until the completed step is published. +Before any call, the stage connects to the endpoint and lists its tools. A failure there ends +the attempt with a reason that separates what Chopin can tell apart: `web-search-credential-rejected` +when the endpoint answered HTTP 401 or 403, `web-search-not-offered` when it answered without a +`web_search` tool, and `web-search-unavailable` for anything else. The diagnostic's `webFailure` +carries the same classification as a call failure, with `connect` or `tools` in place of the call +phase (`local:connect:mcp-client:http:401`, `local:tools:missing-tool`), and `webFailureDurationMs` +is how long setup ran before it failed. The `elapsedMs` of the `connecting-web` snapshot is when +that phase began, so it does not show how long the attempt took. Setup has sent no query, so its +failure log line, unlike a call failure's, also names the credential type (`ghu_` or `gho_` by +prefix, never the token) and the client's error name and message, collapsed to one line, with +token-shaped text and `Bearer` values masked and clipped to 160 characters. That line is +console-only; the durable reason and the diagnostic still carry no upstream text. See +[Self-hosting](self-hosting.md#troubleshooting) for what an operator can do about each reason. + ## Publication hooks A definition may delay completion with `publish()`: diff --git a/docs/conversation-research.md b/docs/conversation-research.md index a73966d7a..e1ba4ffd3 100644 --- a/docs/conversation-research.md +++ b/docs/conversation-research.md @@ -49,6 +49,13 @@ such as waiting for a web-search response or waiting for the model's findings. A failed request exposes its safe failure explanation and **Retry research** in the same card. Retrying keeps the accepted brief and request identity. +Public web search is reached through GitHub's Copilot MCP endpoint with the +signed-in user's App user access token. When connecting to it fails, the card +says whether GitHub refused the credential (HTTP 401 or 403), did not offer +public web search to this sign-in, or the service was unavailable, and the server +log names the cause. See [self-hosting troubleshooting](self-hosting.md#troubleshooting) +for what an operator can check. + ## Processing and persistence Both analysis queues commit with the saved message. Each has independent outcomes diff --git a/docs/self-hosting.md b/docs/self-hosting.md index 61f092aa1..3d4e60755 100644 --- a/docs/self-hosting.md +++ b/docs/self-hosting.md @@ -328,6 +328,10 @@ headers. - For `copilot-sdk`, an active Copilot entitlement for each user who may own a hosted agent session. For `atomic`, model credentials for the chosen `HARNESS_AUTH` mode instead. +- For public web research (`/research` with `WEB_RESEARCH` on), GitHub's Copilot + MCP endpoint must offer its `web_search` tool to each user's App user access + token. GitHub does not document that tool's requirements, so verify it on your + deployment; see [Troubleshooting](#troubleshooting). ## Register the GitHub App @@ -686,7 +690,10 @@ After the first deployment: entitlement for `copilot-sdk`, or the `HARNESS_AUTH` credentials for `atomic`) and the hosted agent runtime. -8. Connect a local coding agent and call `list_documents` if MCP is part of the +8. Run `/research` with a short brief to verify public web search. It reaches + GitHub's Copilot MCP endpoint with the signed-in user's App user access + token, a separate check from step 7. +9. Connect a local coding agent and call `list_documents` if MCP is part of the deployment's intended surface. The image health check calls `/api/session`. It confirms HTTP liveness but does @@ -787,6 +794,45 @@ entitlement, the model, App permissions, repository write access, and Copilot CLI startup logs. Planner and research request execution validate these dependencies lazily. +**Public web research fails before it searches.** `/research` finds public sources through +the `web_search` tool of GitHub's Copilot MCP endpoint (`https://api.githubcopilot.com/mcp/`). +It authorizes with the signed-in user's App user access token (`ghu_`), the same credential the +Planner's pull request tools send to that endpoint. When setup fails, the research card says +which of three things happened and the server log names the step and the cause: + +- `web-search-credential-rejected`: "GitHub refused the credential from this sign-in for public + web search." The endpoint answered HTTP 401 or 403 while connecting or listing tools. +- `web-search-not-offered`: "GitHub did not offer public web search to this sign-in." The endpoint + accepted the connection, but its tool list had no `web_search`. +- `web-search-unavailable`: "The public web-search service was unavailable." Any other setup + failure, such as a network error, an HTTP 5xx, or a JSON-RPC error. + +The log line names the step (`connect` or `tools`), a classification with any HTTP status and +JSON-RPC code, how long setup ran, the type of credential Chopin sent, and the client's error +with tokens and `Bearer` values masked and clipped to one short line. It never contains the token +or the `Authorization` header: + +```text +chopin: web_search unavailable at connect - local:connect:mcp-client:http:401 in 180ms; credential: GitHub App user access token (ghu_); MCPClientError: ... +``` + +The `elapsedMs` on the `connecting-web` progress line records when that phase began, not when it +failed; read the duration from the failure line. + +Chopin cannot tell from these responses why the endpoint refused the credential or withheld the +tool. GitHub's Copilot SDK documentation lists GitHub App user access tokens (`ghu_`) as supported +for Copilot requests, but GitHub documents no token requirement for this endpoint's `web_search` +tool, and the tool is not part of GitHub's open-source MCP server. A deployment that signs in +through its own GitHub App reported this step failing while a classic OAuth token (`gho_`) for the +same account was accepted by the endpoint's `initialize` request. Whether the endpoint refuses +App user access tokens for `web_search` is unconfirmed. + +Check the log line first, then the account's Copilot entitlement and the settings that gate web +search and MCP servers for the user and, in an organization, its Copilot policies. No Chopin +setting sends a different credential to this endpoint. If setup keeps failing, set +`WEB_RESEARCH=off` to stop offering public web research until you have verified it, and report +the log line, which holds no token. + **MCP returns 401 or 403.** A 401 indicates an invalid or expired bearer. A 403 indicates failed instance admission or a supplied Origin that differs from `APP_ORIGIN`. Repository authorization failures normally arrive as an MCP tool From 188ee17c8424eb14ad72c6af9a5e09de447c2812 Mon Sep 17 00:00:00 2001 From: Alex Lavaee Date: Wed, 4 Nov 2026 16:48:26 -0800 Subject: [PATCH 2/2] Keep a thrown string's reason in the web search setup log, and point every setup failure at it A transport that rejected with a string, such as a TLS handshake failure, logged only "a value that is not an Error was thrown", so the reason the setup failed was lost. The string is now logged, masked and clipped like an error's message. Any other non-Error value names its type. Only the credential-rejected and not-offered cards said the server log has the details. The unavailable card, which still covers an HTTP 500 or a 401 or 403 delivered as a JSON-RPC error, now says so too. The credential-rejected card names the token Chopin signs in with, a GitHub App user access token. Refs #699 Assistant-model: Claude Opus 5.5 Assistant-workflow: inline Assistant-verification: bun test passed: apps/server src/harness src/jobs src/research, 547 pass, 0 fail Assistant-verification: bun test failed without the logging change as intended: web-search.test.ts, the thrown-string test fails (21 pass, 1 fail) Assistant-verification: bun run types and bun run ci passed Co-authored-by: Alex Lavaee --- apps/server/src/harness/web-search.test.ts | 4 +++- apps/server/src/harness/web-search.ts | 5 ++++- apps/server/src/research/request-view.test.ts | 7 +++++-- apps/server/src/research/request-view.ts | 5 +++-- 4 files changed, 15 insertions(+), 6 deletions(-) diff --git a/apps/server/src/harness/web-search.test.ts b/apps/server/src/harness/web-search.test.ts index 9b7712348..c53255143 100644 --- a/apps/server/src/harness/web-search.test.ts +++ b/apps/server/src/harness/web-search.test.ts @@ -212,7 +212,7 @@ it("keeps other setup failures unavailable and says what kind of token was used" new MCPClientError({ message: "bad gateway", statusCode: 502 }), new MCPClientError({ message: "no such method", code: -32601 }), Object.assign(new Error("Unable to connect"), { code: "ConnectionRefused" }), - "offline", + "TLS handshake failed for opaque-token-value", ] ) { let result = await webSearchTool({ ...credential(), token: "opaque-token-value" }, { @@ -239,6 +239,8 @@ it("keeps other setup failures unavailable and says what kind of token was used" expect(line).toContain("credential: token of an unrecognized type"); expect(line).not.toContain("Troubleshooting"); } + expect(logs[3]).toContain("thrown string: TLS handshake failed for [redacted]"); + expect(logs[3]).not.toContain("opaque-token-value"); }); it("masks credentials and clips a setup failure before it is logged", async () => { diff --git a/apps/server/src/harness/web-search.ts b/apps/server/src/harness/web-search.ts index 6c22f0e14..117406291 100644 --- a/apps/server/src/harness/web-search.ts +++ b/apps/server/src/harness/web-search.ts @@ -68,13 +68,16 @@ function mcpCodes(error: unknown): string { /** One masked, clipped line of a setup error. No query has been sent when setup fails. */ function setupMessage(error: unknown, token: string): string { - if (!(error instanceof Error)) return "a value that is not an Error was thrown"; let mask = (text: string) => { if (token.length >= 8) text = text.replaceAll(token, "[redacted]"); return text.replace(SECRET_TEXT, (_match, bearer?: string) => `${bearer ?? ""}[redacted]`) .replace(/[\s\p{Cc}]+/gu, " ") .trim(); }; + if (typeof error === "string") return `thrown string: ${mask(error).slice(0, 160)}`; + if (!(error instanceof Error)) { + return `a ${error === null ? "null" : typeof error} value that is not an Error was thrown`; + } let name = /^[A-Za-z][A-Za-z0-9_.]{0,63}$/.test(error.name) ? mask(error.name) : "Error"; return `${name}: ${mask(error.message).slice(0, 160)}`; } diff --git a/apps/server/src/research/request-view.test.ts b/apps/server/src/research/request-view.test.ts index 5c92d4f1e..c13bf6de4 100644 --- a/apps/server/src/research/request-view.test.ts +++ b/apps/server/src/research/request-view.test.ts @@ -8,10 +8,13 @@ test.each([ "attempts-exhausted:web-search-authorization-timeout", "Authorization for public web search timed out.", ], - ["web-search-unavailable", "The public web-search service was unavailable."], + [ + "web-search-unavailable", + "The public web-search service was unavailable. The server log has the details.", + ], [ "attempts-exhausted:web-search-credential-rejected", - "GitHub refused the credential from this sign-in for public web search. " + "GitHub refused this sign-in's GitHub App user access token for public web search. " + "The server log has the details.", ], [ diff --git a/apps/server/src/research/request-view.ts b/apps/server/src/research/request-view.ts index 1557deed7..8b4de54bd 100644 --- a/apps/server/src/research/request-view.ts +++ b/apps/server/src/research/request-view.ts @@ -9,9 +9,10 @@ export function researchFailureMessage(reason: string | undefined): string { "attempt-timeout": "Research timed out before it returned a result.", "web-search-timeout": "A public web-search request timed out.", "web-search-authorization-timeout": "Authorization for public web search timed out.", - "web-search-unavailable": "The public web-search service was unavailable.", + "web-search-unavailable": + "The public web-search service was unavailable. The server log has the details.", "web-search-credential-rejected": - "GitHub refused the credential from this sign-in for public web search. " + "GitHub refused this sign-in's GitHub App user access token for public web search. " + "The server log has the details.", "web-search-not-offered": "GitHub did not offer public web search to this sign-in. The server log has the details.",