Repository navigation
[Deps] Pin Jev on the direct TypeSafe path (it sends jev-latest) and re-run benchmarks on jev-1.14.0 #93
Description
Why
TypeSafe released Jev 1.14.0 on 2026-10-14 (blog). The post says jev-latest "will be updated to jev-1.14.0 in a week" (so around now), that jev-1.13.0 stays available "for a while", and that TypeSafe intends to give 60 days' warning before deprecating jev-1.13.0. 1.14 is described as "small, subtle improvements to everything", with the largest gain in multilingual support. It is also on Vercel AI Gateway and LiteLLM as jev-1.14.0.
System1-Agents currently uses two different Jev versions depending on the backend, and one of them moves silently:
| path | where | model sent |
|---|---|---|
direct TypeSafe (TYPESAFE_API_KEY) |
s1a/decision_models/wire.py TYPESAFE_DEFAULT_MODEL |
jev-latest (moving alias, TYPESAFE_MODEL is not read on this path) |
| OpenRouter proxy | wire.py DEFAULT_MODEL, .env.example, docs/configuration.md |
typesafe/jev-1.13 (pinned) |
So a direct-key user is now running 1.14 while every number in the README table, docs/benchmarks.md and the public231 benchmark was recorded on 1.13, and two runs a week apart can differ only because the alias moved. InfoQ's coverage of the launch gives the same advice: "Pin a version such as jev-1.13.0 rather than the moving jev-latest and jev-preview aliases" (InfoQ).
Proposed scope (one PR)
- Pin the direct path: default
TYPESAFE_MODEL-equivalent to an explicit version (jev-1.13.0until the re-run below says otherwise) and letTYPESAFE_MODELoverride it on the direct path too. Document both defaults indocs/configuration.mdand.env.example. - Record the model id the request actually used (and the one the response reports, if any) in step records / run summaries, so a saved run says which Jev answered it.
- Re-run, on both
jev-1.13.0andjev-1.14.0: the public231 decision benchmark ([Feat] Add a reproducible public231 decision benchmark #52/[Feat] Re-score saved public231 decisions #69 re-scorer),s1a probeon the shipped case files, and if [Eval] Option-order flip rate in the fit probe: s1a probe --order reverse|shuffle:N #92 lands, its--order reverseflip rate. Commit the results underdocs/results/with the date and model id. - Based on (3), either move the default to
jev-1.14.0(README/benchmarks note which version each row is) or keep 1.13.0 and note why. Either way add a line on the 1.13.0 deprecation window.
Acceptance criteria
- No code path sends
jev-latestorjev-previewunless the user sets it explicitly. - Offline tests cover the default and the override on both backends.
- Results for 1.13.0 vs 1.14.0 are committed with accuracy, median latency and cost per run, from the same case files.
- CHANGELOG notes the default model change.
Out of scope
- Jev 1.14's opt-in "danger zone" features (image input in
assets, theboxquestion type, long-context mode). They need their own issue; the image path overlaps [Feat] Use native JEV-VL for browser decisions #78 / [Feat] Add a served OneJev multimodal decision backend (System One + media) #88.
References
- Jev 1.14 release post: https://typesafe.ai/blog/jev-1-14
- LiteLLM day-0 notes (model names, headers): https://docs.litellm.ai/blog/typesafe_jev_images
- Vercel AI Gateway changelog: https://vercel.com/changelog/typesafe-ais-jev-1-14-is-now-available-on-ai-gateway
- InfoQ launch coverage (pinning advice): https://www.infoq.com/news/2026/10/typesafe-ai-jev-released/
- Related: [Feat] Add a reproducible public231 decision benchmark #52, [Feat] Re-score saved public231 decisions #69 (public231), [Eval] Measure option-order sensitivity of each decision backend (flip rate under reversed/shuffled options) #90 / [Eval] Option-order flip rate in the fit probe: s1a probe --order reverse|shuffle:N #92 (option-order flip rate)
Activity