Skip to content

[Deps] Pin Jev on the direct TypeSafe path (it sends jev-latest) and re-run benchmarks on jev-1.14.0 #93

Open

Description

@Birfy

Why

TypeSafe released Jev 1.14.0 on 2026-10-14 (blog). The post says jev-latest "will be updated to jev-1.14.0 in a week" (so around now), that jev-1.13.0 stays available "for a while", and that TypeSafe intends to give 60 days' warning before deprecating jev-1.13.0. 1.14 is described as "small, subtle improvements to everything", with the largest gain in multilingual support. It is also on Vercel AI Gateway and LiteLLM as jev-1.14.0.

System1-Agents currently uses two different Jev versions depending on the backend, and one of them moves silently:

path where model sent
direct TypeSafe (TYPESAFE_API_KEY) s1a/decision_models/wire.py TYPESAFE_DEFAULT_MODEL jev-latest (moving alias, TYPESAFE_MODEL is not read on this path)
OpenRouter proxy wire.py DEFAULT_MODEL, .env.example, docs/configuration.md typesafe/jev-1.13 (pinned)

So a direct-key user is now running 1.14 while every number in the README table, docs/benchmarks.md and the public231 benchmark was recorded on 1.13, and two runs a week apart can differ only because the alias moved. InfoQ's coverage of the launch gives the same advice: "Pin a version such as jev-1.13.0 rather than the moving jev-latest and jev-preview aliases" (InfoQ).

Proposed scope (one PR)

  1. Pin the direct path: default TYPESAFE_MODEL-equivalent to an explicit version (jev-1.13.0 until the re-run below says otherwise) and let TYPESAFE_MODEL override it on the direct path too. Document both defaults in docs/configuration.md and .env.example.
  2. Record the model id the request actually used (and the one the response reports, if any) in step records / run summaries, so a saved run says which Jev answered it.
  3. Re-run, on both jev-1.13.0 and jev-1.14.0: the public231 decision benchmark ([Feat] Add a reproducible public231 decision benchmark #52/[Feat] Re-score saved public231 decisions #69 re-scorer), s1a probe on the shipped case files, and if [Eval] Option-order flip rate in the fit probe: s1a probe --order reverse|shuffle:N #92 lands, its --order reverse flip rate. Commit the results under docs/results/ with the date and model id.
  4. Based on (3), either move the default to jev-1.14.0 (README/benchmarks note which version each row is) or keep 1.13.0 and note why. Either way add a line on the 1.13.0 deprecation window.

Acceptance criteria

  • No code path sends jev-latest or jev-preview unless the user sets it explicitly.
  • Offline tests cover the default and the override on both backends.
  • Results for 1.13.0 vs 1.14.0 are committed with accuracy, median latency and cost per run, from the same case files.
  • CHANGELOG notes the default model change.

Out of scope

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions