feat(ui): compare runs across workspaces on two axes, by case, and score by score

README.md

@@ -150,7 +150,7 @@ const natural = defineScore('formula.natural', { grader: { id: 'judge', kind: 'm
 const pass = defineScore('formula.pass', {
   combine: { all: [correct, { atLeast: 0.5, score: natural }] },
   label: 'Pass',
-  primary: true,
+  rank: 1,
 })
 
 describeTask('read-formula', () => {

docs/content/en/guide/learn/assertions-scores-and-metrics.md

@@ -88,13 +88,13 @@ describeTask('retrieval', ({ caseOf }) => {
 - `recordScore` also accepts a bare name, for example `recordScore('retrieval.recall', 0.5)`. The score then has no label or description.
 - A definition is announced to the report the first time a case records it. List the task's scores in the `scores` option of `describeTask` to announce them when the task starts. A listed score is known before the first case, and it counts as missing for a passed case that never records it.
 
-## Choose the primary score
+## Choose the primary score and rank the core scores
 
 Pages, leaderboards, comparisons, and the CLI summary line show one score per task first, with its label next to its value. That is the **primary** score.
 
-- Mark it with `primary: true`. At most one score of a task is primary; a second one throws when it is announced.
-- A task with exactly one score uses that score as primary without the flag.
-- A task with several scores and none marked primary has no primary score. Its headline falls back to the share of passed cases.
+- Give it `rank: 1`. Rank the other core scores `2`, `3`, and so on, in order of importance: viewers list scores by rank and put the first ranks on the axes of comparison charts. Two scores of a task cannot share a rank; the second one throws when it is announced. A score without a rank is a supporting score, listed after the ranked ones.
+- A task with exactly one score uses that score as primary without a rank.
+- A task with several scores and none of rank 1 has no primary score. Its headline falls back to the share of passed cases.
 - When the tasks of a run have different primary scores, the run has no run-level headline score. Each task's primary score is listed with its task name, and the completion rate stays as the run headline.
 
 ## Combine scores
@@ -115,7 +115,7 @@ const pass = defineScore('tts.pass', {
   // A bare input means `atLeast: 1`, so a mixed reading (0.5) does not pass.
   combine: { all: [correct, natural] },
   label: { 'en': 'Pass', 'zh-Hans': '通过' },
-  primary: true,
+  rank: 1,
 })
 
 describeTask('read aloud', ({ caseOf }) => {
@@ -155,7 +155,7 @@ One rule decides how each case counts in each score. The task result, `run-summa
 Set `errors` per score. `exclude` follows the view that a provider outage is not a result of the test object; the completion rate then shows how many cases were left out. `zero` matches benchmarks that count every error as a wrong answer, for example Harbor and Terminal-Bench:
 
 ```ts
-const solved = defineScore('bench.solved', { errors: 'zero', primary: true })
+const solved = defineScore('bench.solved', { errors: 'zero', rank: 1 })
 ```
 
 Each score summary reports `mean` (over the counted cases, or `null`), `n` (cases that counted), and `total` (settled cases that were not skipped). An empty case selection, for example a `--case` filter that matches nothing, returns no scores.

docs/content/zh-hans/guide/learn/assertions-scores-and-metrics.md

@@ -92,13 +92,13 @@ describeTask('retrieval', ({ caseOf }) => {
 - `recordScore` 也可以直接传分数名，例如 `recordScore('retrieval.recall', 0.5)`，这时分数没有名称和说明。
 - 用例第一次记录某个定义时，Vieval 会把它写入报告。也可以在 `describeTask` 的 `scores` 选项中列出任务的分数，让它们在任务开始时就写入报告。列出的分数在第一个用例之前就是已知的；通过的用例如果没有记录它，就算作缺失。
 
-## 选择主分数
+## 选择主分数并给核心分数排序
 
 页面、排行榜、对比和 CLI 汇总行会优先显示每个任务的一个分数，并在数值旁边显示它的名称。这个分数就是**主分数**（primary）。
 
-- 用 `primary: true` 标记主分数。一个任务最多只有一个主分数，声明第二个时会抛出错误。
+- 用 `rank: 1` 标记主分数，其他核心分数按重要程度依次标 `2`、`3`……查看器按 rank 排列分数，并把排在前面的分数放到对比图的坐标轴上。同一个任务里两个分数不能用同一个 rank，声明第二个时会抛出错误。没有 rank 的分数是辅助分数，排在有 rank 的分数之后。
 - 只有一个分数的任务不需要标记，这个分数就是主分数。
-- 有多个分数但都没有标记的任务没有主分数，标题数值会回退为通过用例的比例。
+- 有多个分数但没有 `rank: 1` 的任务没有主分数，标题数值会回退为通过用例的比例。
 - 如果一次运行中各任务的主分数不同，运行级别就不显示标题分数，而是按任务名分别列出各自的主分数，运行的标题数值只保留执行成功率。
 
 ## 组合分数
@@ -120,7 +120,7 @@ const pass = defineScore('tts.pass', {
   // 不带阈值的输入相当于 `atLeast: 1`，所以 mixed（0.5）不算通过。
   combine: { all: [correct, natural] },
   label: { 'en': 'Pass', 'zh-Hans': '通过' },
-  primary: true,
+  rank: 1,
 })
 
 describeTask('read aloud', ({ caseOf }) => {
@@ -161,7 +161,7 @@ describeTask('read aloud', ({ caseOf }) => {
 `errors` 按分数分别设置。`exclude` 的出发点是：服务故障不是被测对象的结果，被排除的用例数量由执行成功率体现。`zero` 适合把每个错误都算作答错的基准测试，例如 Harbor 和 Terminal-Bench：
 
 ```ts
-const solved = defineScore('bench.solved', { errors: 'zero', primary: true })
+const solved = defineScore('bench.solved', { errors: 'zero', rank: 1 })
 ```
 
 每个分数的汇总包含 `mean`（计入用例的平均值，没有用例计入时为 `null`）、`n`（计入的用例数）和 `total`（已结束且未跳过的用例数）。如果没有选中任何用例，例如 `--case` 过滤后为空，任务不会返回分数。

packages/ui/API.md

@@ -160,10 +160,17 @@ storage. Spans of the live OpenTelemetry runtime are not collected.
 
 ### Comparisons
 
-All return `ComparisonResponse`: `columns` (each with the runs it covers), `rows` (one per case name; `cells[column]`
-= `{ state, scores, roundCount }`, `scores` being `{ values, unscored }`; `changed` when the state or the value of
-`primary` differs between columns), `primary` (the primary score the rows compare, when the columns share one; without
-it, `changed` uses the state only), `scoreDefinitions`, `summary`. Each column's runs carry `headline`.
+All return `ComparisonResponse`: `columns` (each with the runs it covers), `rows` (one per case name and `variant`;
+`cells[column]` = `{ state, scores, counted, metrics, durationMs, roundCount, pooled }`, `scores` being
+`{ values, unscored }`, `counted` the value the case counts with in each score (a failed case counts 0), `metrics` the
+numeric metrics of the case; `changed` when the state or the value of `primary` differs between columns), `primary`
+(the primary score the rows compare, when the columns share one; without it, `changed` uses the state only),
+`scoreDefinitions`, `metricDefinitions`, `summary`. Each column's runs carry `headline`, `workspaceId`, and `project`.
+
+When a column ran a case of one name in several tasks, rows are split by the matrix axes that tell those tasks apart
+and that every column has (`variant`, such as `context=none`), so the columns stay paired case by case. Tasks that
+differ only in an axis some column lacks are averaged into one cell (`pooled` = how many; scores, metrics, and duration
+are means, the state is the worst).
 
 | Route | Columns |
 |---|---|

packages/ui/README.md

@@ -76,7 +76,7 @@ every number on screen comes from a run's events, report files, or evidence dire
 | Run | `#/runs/<runKey>` | Exact score with its coverage (share of settled cases that produced a result) and the case bar in the header; a **No result** card that groups the cases without a result by error name and message (long numbers and ids masked), with one example case per group; progress bar and progress chart (solid: completed cases over time; dashed: projection at the current ETA to the estimated total, fainter when it is a lower bound; "now" marker; recent rate in cases/h), cases done / estimated, running cases, tasks done, elapsed time, ETA and expected finish, current position (running cases with round, retry, elapsed). Tabs: **Execution** (virtualized live tree `project → task → round → case → try`, state filters, search, *Current* jumps to the first running case), **Scores & metrics** (scores, **Case metrics** over every case of the run with the eval entry and the matrix axes as categories, opening on the parallel sets, matrix rows table and chart, scores per round, numeric metrics as line charts, other metrics collapsed to one line each, expandable to every value per case), **Console logs** (all `console.*` lines of the run, linked to their case), **OpenTelemetry** (trace waterfall, log records, gauges, raw `otlp/*.json`). |
 | Task | `#/runs/<runKey>/task?taskId=` | Matrix values, the exact score with how the cases' scores spread, rounds (auto attempts) with pass rates, **Case metrics**, the virtualized case list (sortable by score, duration, or a numeric column; metric columns, string metrics such as a case id included, scrolling sideways when they do not fit and shown under the name on phones; the charts' selection as a chip that clears it). Case metrics draws every case against its metrics, and picks the chart by the number of dimensions (`state` is the color, not a dimension): one dimension is a strip of dots, two a plane (metric × metric scatter, metric × category bands, category × category grid), three or more parallel coordinates with a scatterplot matrix of the first five picked metrics. A metric recorded as a list of numbers per case (the grade of each vote, every sample's reward) is drawn as two numeric axes, its mean and its length. The chart opens on the five metrics that best tell failed from passed cases (AUC, scores left out), or, when every case passed (a scored eval), cases below the mean exact score from the rest; the *Axes* button opens a searchable list of the axes on the chart, the others ranked by AUC, metrics that move with another one (\|r\| ≥ 0.95 on log values), and those hidden as constant or repeated. *Color by* recolors by state, a category, or a metric; it opens on the exact score when every case ended the same way. Scores are colored low to high with the low end taking the most ink, so the cases that lose the score stand out. A metric is colored in five equal steps; its bands sit in the toolbar with their case counts, *+ cut* splits the widest band with a cut line drawn across the metric's axis (drag it to move, drag it past an end of the axis to remove it, *steps* goes back to five steps). Clicking a legend entry or a band selects its cases. While the color or axes menu is open, the metric under the pointer or the keyboard is described beside the menu (on phones: one line of description per metric, details in a second drawer). Besides the chart, **Sets** draws the category axes as parallel sets (ribbons as wide as their cases, each value with the mean exact score of the selection, best on top; titles drag to reorder and show the axis card on hover, like the parallel coordinates), and **Groups** lists the values of one category as bars with their mean score, lowest first. Dragging on an axis or in a scatterplot cell, or clicking a category value, selects cases; the case list below shows the selection. Hovering a line or dot previews its case; clicking pins it (its line and its value on every axis stay highlighted) and opens a card with the case's categories on one line and its most unusual metrics (furthest from the median) with a small histogram each, the rest folded, and a link to the case. On phones a tap opens that card in a bottom drawer (vaul). |
 | Case | `#/runs/<runKey>/case?taskId=&caseId=` | **Overview**: error, timeline of rounds (auto attempts) and tries (retries), input and output as object trees, metrics. **Console log**, **Artifacts**, **Evidence**, **Inference**: calls, input and output tokens (per model when a case used several, plus their sum), model time, and one row per model call with the tool calls it asked for. |
-| Compare | `#/compare?view=experiments\|attempts\|runs\|matrix&…` | Per column: newest run, state, exact score, case counts; a bar chart of exact score and pass rate; a case × column table (state, exact score, rounds) with an "only differing" filter. `matrix` compares the matrix rows (for example `model` × `repeat`) of one eval entry inside one run. |
+| Compare | `#/compare?view=experiments\|attempts\|runs\|matrix&…` | Without a `view`: the experiments run in several workspaces as leaderboards (also on the home page), then only the comparisons that have two or more things to compare. With one: per column, newest run, state, exact score, case counts; Y / X / baseline pickers that preview each axis beside the menu; a **two-axis distribution** (one dot per column with 95% intervals, the Pareto front, a crosshair, linear or log scales, a click-pinned card to make a column the baseline or open its run); the **differences from the baseline** paired by case; **every score and metric** as compact heat cells (best per axis outlined, a beyond-noise tally per column); columns can be hidden from the charts; a case × column table (state, exact score, rounds) with an "only differing" filter. Rows split by the matrix axes every column has; an axis only one column runs is averaged. `matrix` compares the matrix rows (for example `model` × `repeat`) of one eval entry inside one run. |
 
 Ids are query parameters because task and case ids contain `/` and `%`. Breadcrumbs show the path
 `projects → workspace → project → experiment → attempt → run → task → case` up to the page shown. Ancestors are links;
@@ -152,7 +152,9 @@ A report path is `<workspace>/<project>/<experiment>/<attempt>/<run>/`. The four
 | try | `autoRetry` of a case | vieval retried one case at once inside a round. | — |
 
 *Compare → experiments* puts the newest run of each experiment of a project side by side. *Compare → attempts* puts
-the runs of one experiment side by side. Neither changes the data; both read the same report directories.
+the runs of one experiment side by side. *Same experiment across workspaces* puts the newest run of each workspace that
+ran one experiment side by side (the same eval against several products). None of them changes the data; all read the
+same report directories.
 
 ## Security of artifacts and evidence
 

packages/ui/client/src/caseMetrics.ts

@@ -9,7 +9,7 @@ import { parseTaskId } from '../../src/shared/reduce'
 import { formatCompact, formatScore } from './format'
 import { i18n } from './i18n'
 import { localize } from './localize'
-import { primaryScoreName, scoreColumn } from './scores'
+import { orderScores, primaryScoreName, scoreColumn } from './scores'
 
 /** One case of a task, flattened for the multi-metric charts (parallel coordinates, scatterplot matrix). */
 export interface CaseRow {
@@ -165,7 +165,7 @@ export function buildCaseTable(cases: readonly CaseNode[], metrics: MetricsRespo
     }
   }
   const primary = primaryScoreName(metrics?.scoreDefinitions ?? [], scoreNames)
-  const scores = primary == null ? scoreNames : [primary, ...scoreNames.filter(name => name !== primary)]
+  const scores = orderScores(primary == null ? scoreNames : [primary, ...scoreNames.filter(name => name !== primary)], metrics?.scoreDefinitions ?? [])
   const derivedNames: string[] = []
   for (const series of metrics?.nonNumeric ?? []) {
     for (const point of series.values) {