Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,12 +174,12 @@ Direct API access is an optional fallback. Copy the example environment file and
cp .env.example .env
```

Available Codex/OpenAI models:
Codex CLI uses the default model selected for the signed-in ChatGPT account. Wingdiff does not pass an API model ID to Codex, because API availability and ChatGPT-plan availability can differ. Codex CLI reviews offer low, medium, or high reasoning effort; if the account's default model does not accept the chosen effort, Codex reports the error and Wingdiff shows it. Named models are available through the direct OpenAI API provider:

- `gpt-6-sol` — default balance of review quality, speed, and cost
- `gpt-6-astra` — highest-capability option for difficult reviews
- `gpt-6-luna` — fast, cost-efficient investigations
- `gpt-5.3-codex` — Codex-tuned coding model
- `gpt-5.3-codex` — deprecated Codex-tuned model available only through the direct OpenAI API provider

Claude Sonnet 4.6 and Opus 4.6 remain available through the same provider interface. Select Codex CLI, OpenAI API, or Anthropic API plus the model and supported reasoning effort from the model control in the top bar.

Expand Down Expand Up @@ -229,6 +229,30 @@ npm run test:e2e

The tests enforce the product's grounding contract: claims must resolve to real evidence, generated tours must cover every included changed file, diff ranges must be internally consistent, revision coverage must partition the full fixture tour, findings must remain attached to known stops, and staged comments must match the pinned GitHub diff side and line fingerprint. The browser suite drives a mocked real-PR session through context approval, generation, investigation, visible-code comment drafting, persistence, and navigation home.

### Local live smoke tests

The opt-in live suite exercises the production build with your existing `gh` and
`codex` logins. It is separate from `npm run test:e2e`, refuses to run in CI,
uses an isolated temporary database, and never opens the publish flow.

```bash
# Fast GitHub acquisition and rendering check; infers the PR for this branch.
npm run test:e2e:live:acquire

# Full acquisition plus real Codex guided-review generation.
npm run test:e2e:live

# Target another PR or checkout explicitly.
WINGDIFF_LIVE_PR=https://github.com/owner/repository/pull/123 npm run test:e2e:live
WINGDIFF_LIVE_CHECKOUT=/path/to/checkout WINGDIFF_LIVE_PR=owner/repository#123 npm run test:e2e:live
```

Run `gh auth status --hostname github.com` and `codex login status` if the
preflight fails. Set `WINGDIFF_LIVE_KEEP_DATA=1` to retain the temporary
database for debugging; otherwise it is deleted even after a failure. Failure
traces and screenshots stay under `test-results/live/` and may contain source
from the tested pull request, so keep those artifacts local.

## Keyboard shortcuts

| Key | Action |
Expand Down
18 changes: 12 additions & 6 deletions apps/server/src/app.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ describe("generated tour API", () => {
const created = await fetch(url, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ selection: { provider: "codex", model: "gpt-6-sol", reasoningEffort: "medium" } }),
body: JSON.stringify({ selection: { provider: "codex", model: "codex-default", reasoningEffort: "medium" } }),
});
expect(created.status).toBe(202);
expect(await created.json()).toMatchObject({ generation: { state: "running", scope: "full" } });
Expand All @@ -103,7 +103,7 @@ describe("generated tour API", () => {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
selection: { provider: "codex", model: "gpt-6-sol", reasoningEffort: "medium" },
selection: { provider: "codex", model: "codex-default", reasoningEffort: "medium" },
scope: "full",
stopId: "atomic-counter",
question: "Can concurrent calls race?",
Expand Down Expand Up @@ -146,7 +146,7 @@ describe("generated tour API", () => {
await new Promise<void>((resolve) => server!.once("listening", resolve));
const port = (server.address() as AddressInfo).port;
const url = `http://127.0.0.1:${port}/api/sessions/${session.id}/tour`;
const input = { selection: { provider: "codex", model: "gpt-6-sol", reasoningEffort: "medium" } };
const input = { selection: { provider: "codex", model: "codex-default", reasoningEffort: "medium" } };

const started = await fetch(url, {
method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(input),
Expand All @@ -163,11 +163,17 @@ describe("generated tour API", () => {
});
expect(duplicate.status).toBe(202);

const conflicting = await fetch(url, {
const conflictingScope = await fetch(url, {
method: "POST", headers: { "Content-Type": "application/json" },
body: JSON.stringify({ ...input, scope: "update" }),
});
expect(conflictingScope.status).toBe(409);

const conflictingSelection = await fetch(url, {
method: "POST", headers: { "Content-Type": "application/json" },
body: JSON.stringify({ selection: { ...input.selection, reasoningEffort: "high" } }),
});
expect(conflicting.status).toBe(409);
expect(conflictingSelection.status).toBe(409);

release();
const completed = await waitForGeneration(url, "succeeded");
Expand Down Expand Up @@ -203,7 +209,7 @@ describe("generated tour API", () => {
const url = `http://127.0.0.1:${port}/api/sessions/${session.id}/tour`;
const created = await fetch(url, {
method: "POST", headers: { "Content-Type": "application/json" },
body: JSON.stringify({ selection: { provider: "codex", model: "gpt-6-sol", reasoningEffort: "medium" } }),
body: JSON.stringify({ selection: { provider: "codex", model: "codex-default", reasoningEffort: "medium" } }),
});
expect(created.status).toBe(202);
await waitForGeneration(url, "succeeded");
Expand Down
19 changes: 11 additions & 8 deletions apps/server/src/providers/catalog.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,7 @@ import { MODEL_CATALOG, publicProviders, validateSelection } from "./catalog.js"
describe("model catalog", () => {
it("offers OpenAI, Codex, and Anthropic choices", () => {
expect(MODEL_CATALOG.map((model) => model.id)).toEqual([
"gpt-6-sol",
"gpt-6-astra",
"gpt-6-luna",
"gpt-5.3-codex",
"codex-default",
"gpt-6-sol",
"gpt-6-astra",
"gpt-6-luna",
Expand All @@ -17,9 +14,10 @@ describe("model catalog", () => {
]);
});

it("defaults GPT-6 Sol to medium reasoning", () => {
const model = MODEL_CATALOG.find((candidate) => candidate.id === "gpt-6-sol");
it("lets Codex choose the signed-in account's default model", () => {
const model = MODEL_CATALOG.find((candidate) => candidate.id === "codex-default");
expect(model?.defaultEffort).toBe("medium");
expect(model?.reasoningEfforts).toEqual(["low", "medium", "high"]);
expect(model?.badge).toBe("Recommended");
});

Expand All @@ -28,15 +26,20 @@ describe("model catalog", () => {
expect(providers.find((provider) => provider.id === "codex")?.configured).toBe(true);
expect(providers.find((provider) => provider.id === "openai")?.configured).toBe(true);
expect(providers.find((provider) => provider.id === "anthropic")?.configured).toBe(false);
expect(providers.find((provider) => provider.id === "codex")?.models.map((model) => model.id)).toEqual(["codex-default"]);
expect(providers.find((provider) => provider.id === "openai")?.models.map((model) => model.id)).toContain("gpt-5.3-codex");
expect(JSON.stringify(providers)).not.toContain("secret-openai");
});

it("rejects cross-provider and unsupported reasoning selections", () => {
expect(() => validateSelection({ provider: "anthropic", model: "gpt-6-sol", reasoningEffort: "medium" })).toThrow(/Unknown model/);
expect(() => validateSelection({ provider: "openai", model: "gpt-6-astra", reasoningEffort: "none" })).toThrow(/does not support/);
expect(validateSelection({ provider: "codex", model: "gpt-6-sol", reasoningEffort: "medium" })).toEqual({
expect(() => validateSelection({ provider: "codex", model: "gpt-5.3-codex", reasoningEffort: "medium" })).toThrow(/must use your account's default model/);
expect(() => validateSelection({ provider: "codex", model: "gpt-6-sol", reasoningEffort: "medium" })).toThrow(/must use your account's default model/);
expect(validateSelection({ provider: "openai", model: "gpt-5.3-codex", reasoningEffort: "medium" }).provider).toBe("openai");
expect(validateSelection({ provider: "codex", model: "codex-default", reasoningEffort: "medium" })).toEqual({
provider: "codex",
model: "gpt-6-sol",
model: "codex-default",
reasoningEffort: "medium",
});
});
Expand Down
39 changes: 24 additions & 15 deletions apps/server/src/providers/catalog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,31 @@ import type {

const CODEX_MODELS: ModelDefinition[] = [
{
id: "gpt-6-sol",
id: "codex-default",
provider: "codex",
name: "Account default",
family: "Codex",
description: "Let Codex choose the default model supported by your signed-in account.",
badge: "Recommended",
reasoningEfforts: ["low", "medium", "high"],
defaultEffort: "medium",
},
];

const OPENAI_MODELS: ModelDefinition[] = [
{
id: "gpt-6-sol",
provider: "openai",
name: "GPT-6 Sol",
family: "OpenAI",
description: "Balanced reasoning, latency, and cost for everyday code review.",
badge: "Recommended",
badge: "API",
reasoningEfforts: ["none", "low", "medium", "high", "xhigh", "max"],
defaultEffort: "medium",
},
{
id: "gpt-6-astra",
provider: "codex",
provider: "openai",
name: "GPT-6 Astra",
family: "OpenAI",
description: "Highest capability for architectural and high-risk reviews.",
Expand All @@ -29,7 +42,7 @@ const CODEX_MODELS: ModelDefinition[] = [
},
{
id: "gpt-6-luna",
provider: "codex",
provider: "openai",
name: "GPT-6 Luna",
family: "OpenAI",
description: "Fast, cost-efficient investigation for routine questions.",
Expand All @@ -39,23 +52,16 @@ const CODEX_MODELS: ModelDefinition[] = [
},
{
id: "gpt-5.3-codex",
provider: "codex",
provider: "openai",
name: "GPT-5.3-Codex",
family: "Codex",
description: "Codex-tuned model for agentic coding and code investigation.",
badge: "Codex",
family: "OpenAI API",
description: "Deprecated Codex model available through direct API authentication only.",
badge: "Deprecated",
reasoningEfforts: ["low", "medium", "high", "xhigh"],
defaultEffort: "medium",
},
];

const OPENAI_MODELS: ModelDefinition[] = CODEX_MODELS.map((model) => ({
...model,
provider: "openai",
family: model.family === "Codex" ? "OpenAI API" : model.family,
badge: model.id === "gpt-6-sol" ? "API" : model.badge,
}));

const ANTHROPIC_MODELS: ModelDefinition[] = [
{
id: "claude-sonnet-4-6",
Expand Down Expand Up @@ -119,6 +125,9 @@ export function publicProviders(
export function validateSelection(input: unknown): ModelSelection {
if (!input || typeof input !== "object") throw new Error("A model selection is required.");
const value = input as Record<string, unknown>;
if (value.provider === "codex" && value.model !== "codex-default") {
throw new Error("Codex CLI must use your account's default model. Choose Account default, or use the OpenAI API provider for a named model.");
}
const model = MODEL_CATALOG.find((candidate) => (
candidate.id === value.model && candidate.provider === value.provider
));
Expand Down
90 changes: 89 additions & 1 deletion apps/server/src/providers/codex.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
import { describe, expect, it } from "vitest";
import { CODEX_SECURITY_OVERRIDES, codexInvestigationTimeoutMs, codexTourTimeoutMs } from "./codex.js";
import {
CODEX_SECURITY_OVERRIDES,
codexInvestigationTimeoutMs,
codexModelArguments,
codexTourTimeoutMs,
formatCodexCliError,
} from "./codex.js";

describe("Codex CLI isolation", () => {
it("prevents model-invoked tools from inheriting secrets or browsing", () => {
Expand All @@ -10,6 +16,11 @@ describe("Codex CLI isolation", () => {
"tools.web_search=false",
]);
});

it("lets the CLI choose the signed-in account's default model", () => {
expect(codexModelArguments("codex-default")).toEqual([]);
expect(codexModelArguments("gpt-6-sol")).toEqual(["--model", "gpt-6-sol"]);
});
});

describe("Codex CLI timeouts", () => {
Expand Down Expand Up @@ -37,3 +48,80 @@ describe("Codex CLI timeouts", () => {
expect(codexInvestigationTimeoutMs({ WINGDIFF_CODEX_INVESTIGATION_TIMEOUT_MS: "999" })).toBe(180_000);
});
});

describe("Codex CLI errors", () => {
it("turns a ChatGPT model rejection into an actionable provider choice", () => {
const stderr = 'ERROR request failed: {"status":400,"detail":"The gpt-5.3-codex model is not supported when using Codex with a ChatGPT account."}';
expect(formatCodexCliError(stderr, "codex-default")).toBe(
"Codex's account-default model is not available with this ChatGPT account. Update Codex CLI and run `codex login` again, or select OpenAI API and configure OPENAI_API_KEY.",
);
});

it("explains how to recover when Codex authentication expires", () => {
expect(formatCodexCliError("Error: 401 Unauthorized: authentication expired", "gpt-6-sol")).toBe(
"Codex authentication is missing or expired. Run `codex login`, then restart Wingdiff.",
);
});

it("extracts a structured error detail instead of exposing CLI log noise", () => {
const stderr = '2026-10-07T12:00:00Z ERROR codex_core::client: {"error":{"message":"The request was rejected."}}';
expect(formatCodexCliError(stderr, "gpt-6-sol")).toBe("The request was rejected.");
});

it("recognizes the unsupported wording of a ChatGPT model rejection", () => {
const stderr = "ERROR: The gpt-6-sol model is unsupported when using Codex with a ChatGPT account.";
expect(formatCodexCliError(stderr, "gpt-6-sol")).toMatch(/^gpt-6-sol is not available with this ChatGPT account\./);
});

it("recognizes a structured 401 status", () => {
expect(formatCodexCliError('ERROR: {"status":401,"detail":"Token rejected."}', "codex-default")).toBe(
"Codex authentication is missing or expired. Run `codex login`, then restart Wingdiff.",
);
});

it("explains a model that the current sign-in cannot use", () => {
const stderr = "ERROR: The model `gpt-6-astra` does not exist or you do not have access to it.";
expect(formatCodexCliError(stderr, "codex-default")).toBe(
"Codex's account-default model is not available with the current Codex sign-in. Update Codex CLI and run `codex login` again, or choose another provider.",
);
});

it("does not mistake echoed review evidence for an authentication failure", () => {
const stderr = [
'+ if (res.status === 401) throw new Error("Unauthorized");',
"ERROR: stream disconnected before completion: You have hit your usage limit.",
].join("\n");
expect(formatCodexCliError(stderr, "codex-default")).toBe(
"stream disconnected before completion: You have hit your usage limit.",
);
});

it("does not mistake model prose for an unavailable model", () => {
const stderr = "The data model for sessions is not supported by the old migration.\nERROR: stream error: connection reset";
expect(formatCodexCliError(stderr, "codex-default")).toBe("stream error: connection reset");
});

it("reports the error instead of a trailing warning", () => {
const stderr = [
"ERROR: unexpected status 429 Too Many Requests: rate limit reached",
"Warning: no last agent message; wrote empty content to /tmp/wingdiff-codex-abc/tour-output.json",
].join("\n");
expect(formatCodexCliError(stderr, "codex-default")).toBe("unexpected status 429 Too Many Requests: rate limit reached");
});

it("ignores structured fields that are not part of the error line", () => {
const stderr = 'ssage":"truncated"}\n{"path":"src/a.ts","message":"fix retry"}\nERROR: stream error: connection reset';
expect(formatCodexCliError(stderr, "codex-default")).toBe("stream error: connection reset");
});

it("keeps the whole stderr tail when no line reports an error", () => {
const stderr = "thread panicked at core/src/lib.rs\nstack backtrace omitted";
expect(formatCodexCliError(stderr, "codex-default")).toBe(stderr);
expect(formatCodexCliError("Not logged in. Run codex login.", "codex-default")).toMatch(/^Codex authentication is missing or expired\./);
});

it("returns nothing for empty stderr so the caller reports the exit status", () => {
expect(formatCodexCliError("", "codex-default")).toBe("");
expect(formatCodexCliError("WARNING: proceeding, even though we could not create PATH aliases: denied\n")).toBe("");
});
});
Loading
Loading