Skip to content

[Feat] Add a served OneJev multimodal decision backend (System One + media) #88

Open

Description

@Birfy

Why

system1-agents already has an in-process OmniJev path (--model omnijev, MSO1.system_one over a local clone) for screenshot + text choice/noul on the browser front. There is no served OmniJev/OneJev worker in System1-Omni today (capabilities matrix notes Omni vision support is Laya / Open-Jev text / Cua-S1 screenshots).

Separately, the OmniJev org published OneJev: open Apache-2.0 multimodal System One models (0.8B / 4B / 9B / 27B) that answer typed Choice / Noul / Score questions over screenshots (and text/video) in one forward pass, served by qev with a TypeSafe-compatible /v1/systemone API plus a media field.

This fits the existing served decision-model seam (laya-served, PR #86's served Cua-S1 client pattern): an HTTP client that posts observation + questions and maps answers into Decision, without cloning OmniJev into the Agents process.

Distinct from:

Proposed scope (one PR)

  1. Add --model onejev (name bikeshed OK) as a served client: env ONEJEV_URL (or reuse a documented System One URL), optional model id, timeout, and provenance fields (url, request_id, served_by) matching other served adapters.
  2. Map Agents choice / noul questions onto the OneJev/qev client; send the browser/desktop screenshot through the documented media / <image:N> state convention. Refuse cleanly when the observation has no image (same policy as OmniJev).
  3. Contract tests with a fake HTTP server (no GPU/weights in CI): happy path, missing image, transport error, malformed answers.
  4. Docs: docs/decision-models.md, docs/configuration.md, docs/capabilities.md — mark registration vs hardware validation separately; do not claim task success rates without a pinned run artifact.
  5. Optional smoke recipe: start qev serve --model OmniJev/OneJev-4B locally and run one browser tick or decide_many with a fixture screenshot (document only; do not gate CI on GPU).

Acceptance criteria

  • build_model("onejev") (or chosen name) works against a stub server in tests.
  • Env vars and failure modes documented; image-required path matches OmniJev semantics.
  • Capability matrix row added with validation marked unverified until a real run lands.
  • NOTICE / third-party attribution for OneJev / qev as needed.

Out of scope

  • Vendoring OneJev weights or training code.
  • Replacing in-process OmniJev in this PR.
  • Full Flights/OSWorld success-rate campaigns (follow-up eval issue after the adapter lands).

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions