Feature/agent test set p1 UI - #478
Open
yuyixg wants to merge 4 commits into
Open
Conversation
Add the agent-test-service.js REST client, its api-endpoints.js URL group, and agentTestTypes.js JSDoc typedefs for suites/cases/runs/results, following the existing agent-service.js conventions. Wires the $agentTestTypes alias in svelte.config.js. No page consumes this yet (Tasks 3-6).
Adds the suite list page (page/agent-test): agent filter with Filter/Reset, table of suites (name/agent/enabled), create modal, Swal delete confirmation, and loading/empty/error states. Registers the new route in svelte.config.js's prerender entries, required for adapter-static's crawl:false + strict build to succeed.
Completes the four agent-test pages on top of the existing suite list, then adds the UI for multi-model runs and AI-assisted case extraction. New routes: the suite detail page (settings, case list with multi-select, run history), the case editor (turns, per-turn and case-level assertions, initial states, tool mocks with state writes; caseId=new creates), and the run detail page (case -> turn -> assertion, observed tool calls, 2s polling while non-terminal, cancel, re-run failures). Shared status/validation helpers live in $lib/helpers/utils/agent-test.js. Failed and Errored are deliberately styled and worded differently throughout. Collapsing them is the one mistake that makes "the harness broke" read as "the agent regressed", and the run page exists to tell those apart. Multi-model runs. The run button opens a modal that picks any number of chat-capable models and shows the resulting execution count up front, since each model multiplies the token cost. The run page grows a comparison grid: one row per case, one column per model, status and duration in each cell, per-model totals in the footer. Durations are summed rather than averaged -- "how long does the whole suite take on this model" is what decides whether an upgrade is affordable, and an average hides one pathological case. AI extraction. The record modal takes an optional extraction model, defaulting to off so the deterministic recorder (which never leaves the system) stays the default. Picking one warns which vendor the user messages and tool names go to, and states that tool arguments and results do not. recordCase became recordCases and returns a list. Judge provider/model became cascading dropdowns fed by /llm-configs and filtered to chat-capable models. Unlike the pattern they copy, a stored value the catalogue no longer offers stays selectable -- suite PUT is a full replace, so silently dropping it would blank the field on any unrelated save. i18n: the pages already called $_(), but no key existed in en.json or zh.json, so svelte-i18n fell back to the key and Chinese rendered as English. All 248 keys now resolve in both locales, including the strings that were previously hardcoded outside the markup (confirm dialogs, toasts, validation text, and the accessible names of icon-only buttons). Also fixes a relative goto in the suite list that resolved against the current URL. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
A run that stops before executing anything produces no case results, so the per-case error alert had nothing to render and the page showed status Error, 0/0/0/0, and "This run produced no case results." -- with the actual reason nowhere on screen. It now renders run.error above the (empty) results section, replaces that misleading empty-state line with a pointer to it, and repeats the reason inline in the suite's run history so it is visible without opening the run at all. The commonest way to reach that state was a trap this UI created: disabled cases were selectable and "Run Selected" happily queued them, which the executor then rejected wholesale. Selection counts now consider only enabled cases, a mixed selection warns how many will be skipped, an all-disabled selection is blocked with the reason, and only enabled ids are sent. Also translates the remaining Chinese in agentTestTypes.js to English. Only comments and JSDoc -- zh.json holds translations, which are data and untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.