Fix remote eval reruns by preserving upsert IDs - #763
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8715337d74
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| if datum.upsert_id: | ||
| base_event["id"] = datum.upsert_id |
There was a problem hiding this comment.
When an eval case supplies upsert_id and its effective trial_count is greater than one, every concurrently scheduled trial receives this same root ID. Because each new SpanImpl initially emits a replacement rather than a merge, the trials race to overwrite one logical row instead of producing the distinct results promised by trial_count, potentially leaving a root and child spans from different trials. Derive an ID that is stable per trial (for example from upsert_id and trial_index) rather than applying the bare ID to every trial.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
do we have the same issue in the js sdk?
There was a problem hiding this comment.
from what I found no, the JS is not impacted, Python drops upsert_id; JS already preserves it
There was a problem hiding this comment.
(yes for the P1),JS SDK also assigns the same upsert_id to every trial, so with trialCount > 1, trials can overwrite each other’s root record. Let me make another PR to correct this is the JS SDK as well (fix here)
There was a problem hiding this comment.
### Why Related to [COR-84](https://linear.app/braintrustdata/issue/COR-84/remote-eval-playground-re-running-a-row-leaves-a-duplicate-grid-row). Addresses the JS counterpart of the [P1 raised on the Python SDK fix](braintrustdata/braintrust-sdk-python#763 (comment)). The intent is to keep one independently updatable result per evaluation trial. The JS SDK already preserves `upsert_id`, but currently reuses that exact record ID for every trial. With multiple trials, their root records overwrite each other, losing trial results and potentially leaving task spans attached to the wrong surviving root. Generating fresh random IDs would avoid collisions but break rerun upserts. This fixes an existing JS multi-trial bug, not Python's original dropped-`upsert_id` bug. It matches the per-trial ID scheme in [braintrust-sdk-python#763](braintrustdata/braintrust-sdk-python#763). ### Repro 1. Run an evaluator with a row containing `upsert_id: "eval-row"` and `trialCount: 3`, with a task returning a different output for each trial index. 2. Inspect the logged records after row merging: only one root record remains instead of three because all trials use `id: "eval-row"`. 3. Rerun with the same upsert ID: the trials collide again instead of each updating their own result. The regression reproduces this for both experiment-backed and remote-parent execution, and with global or per-row trial counts. Before the fix, four multi-trial cases fail with one logged root instead of three; the other 20 cases pass. Two remote eval columns are not required to trigger this P1. ### Fix - Keep the original `upsert_id` for trial zero, preserving single-trial behavior. - For later trials, derive a deterministic UUID v5 in the URL namespace from `braintrust:eval:<upsert_id>:trial:<trial_index>`. Each trial has a distinct ID that stays stable on rerun, matching Python. - Leave missing/empty upsert IDs on the existing fresh-ID path. Do not change dataset origins, trial scheduling, or public APIs; reuse the existing UUID dependency. - Include the required patch changeset for `braintrust`. The behavior change is limited to additional trials with a non-empty upsert ID. This prevents future trial collisions after upgrading; it does not restore overwritten results or clean up historical records. ### Test - `pnpm test src/framework.test.ts --reporter=dot`: all 104 tests pass, including 24 regression cases covering supplied/missing/empty IDs, global/per-row trial counts, and experiment/parent-context execution. - Regression assertions check separate roots, matching root/task outputs, stable IDs across reruns, and UUID values matching Python's implementation. - `pnpm run check:typings`: passes for production and test code. - Targeted ESLint: no errors; existing warnings remain and the repository config excludes test files. - `pnpm run fix:formatting` and `git diff --check`: pass. --------- Co-authored-by: Paul Tancre <paultancre@p-tancre.home> Co-authored-by: lforst <8118419+lforst@users.noreply.github.com>
2 participants
Why
Fixes COR-84.
The playground sends a stable
upsert_idfor each row/eval column so reruns replace the existing result. Python'sEvalCase.from_dict()drops this field, and the eval runner creates a fresh root record on every run. The resulting records appear as duplicate grid rows with stale outputs. The JS SDK already honorsupsert_id.This addresses the SDK cause identified during review of the earlier UI workaround, braintrust#20332.
Repro
upsert_id, Python writes a different root record ID, leaving both results in the grid.The regression test reproduces this by running an evaluator twice with the same
upsert_idand checking the logged root record IDs. It fails before the fix.Fix
Preserve optional
upsert_idinEvalCaseand its input TypedDicts. The first trial uses that value as its root record ID; additional trials derive deterministic UUIDs from the upsert ID and trial index. This keeps the single-trial playground behavior and lets each additional trial replace its own result on rerun without overwriting other trials. Both experiment-backed and parent-context evaluations use this logic. Missing or empty IDs continue to generate fresh record IDs; dataset row IDs and origins retain their existing meaning.This prevents future duplicates after SDK upgrade; it does not remove historical duplicate records. This PR changes Python only; the analogous JS multi-trial behavior needs a separate fix.
Test