Repository navigation
[Feat] Add a served OneJev multimodal decision backend (System One + media) #88
Description
Why
system1-agents already has an in-process OmniJev path (--model omnijev, MSO1.system_one over a local clone) for screenshot + text choice/noul on the browser front. There is no served OmniJev/OneJev worker in System1-Omni today (capabilities matrix notes Omni vision support is Laya / Open-Jev text / Cua-S1 screenshots).
Separately, the OmniJev org published OneJev: open Apache-2.0 multimodal System One models (0.8B / 4B / 9B / 27B) that answer typed Choice / Noul / Score questions over screenshots (and text/video) in one forward pass, served by qev with a TypeSafe-compatible /v1/systemone API plus a media field.
- GitHub: https://github.com/OmniJev/OneJev
- Hugging Face (4B): https://huggingface.co/OmniJev/OneJev-4B
- Overview write-up: https://modelsystem.one/news/onejev-open-multimodal-family/
This fits the existing served decision-model seam (laya-served, PR #86's served Cua-S1 client pattern): an HTTP client that posts observation + questions and maps answers into Decision, without cloning OmniJev into the Agents process.
Distinct from:
- [Feat] Integrate and evaluate Cua-S1 4B 0.2 #14 / [Feat] Connect desktop screenshot decisions to System1-Omni Cua-S1 4B #44 / [Feat] Served Cua-S1 4B screenshot decisions against system1-omni's multimodal worker #86 — Cua-S1 4B (element scoring / Omni multimodal worker), not OneJev.
- In-process
omnijev— different package and load path; keep that until a served path is proven. - [Feat] Integrate and evaluate CLM-v0.1-8B through clm-serve #24 / [Feat] Integrate and evaluate Julia-1 #25 — CLM / Julia text backends.
- [Feat] Use native JEV-VL for browser decisions #78 (draft) — native JEV-VL; different upstream.
Proposed scope (one PR)
- Add
--model onejev(name bikeshed OK) as a served client: envONEJEV_URL(or reuse a documented System One URL), optional model id, timeout, and provenance fields (url,request_id,served_by) matching other served adapters. - Map Agents
choice/noulquestions onto the OneJev/qev client; send the browser/desktop screenshot through the documentedmedia/<image:N>state convention. Refuse cleanly when the observation has no image (same policy as OmniJev). - Contract tests with a fake HTTP server (no GPU/weights in CI): happy path, missing image, transport error, malformed answers.
- Docs:
docs/decision-models.md,docs/configuration.md,docs/capabilities.md— mark registration vs hardware validation separately; do not claim task success rates without a pinned run artifact. - Optional smoke recipe: start
qev serve --model OmniJev/OneJev-4Blocally and run one browser tick ordecide_manywith a fixture screenshot (document only; do not gate CI on GPU).
Acceptance criteria
-
build_model("onejev")(or chosen name) works against a stub server in tests. - Env vars and failure modes documented; image-required path matches OmniJev semantics.
- Capability matrix row added with validation marked unverified until a real run lands.
- NOTICE / third-party attribution for OneJev / qev as needed.
Out of scope
- Vendoring OneJev weights or training code.
- Replacing in-process OmniJev in this PR.
- Full Flights/OSWorld success-rate campaigns (follow-up eval issue after the adapter lands).
References
- https://github.com/OmniJev/OneJev
- https://huggingface.co/OmniJev/OneJev-4B
- https://modelsystem.one/news/onejev-open-multimodal-family/
- https://pypi.org/project/onejev/ (related packaging; prefer the GitHub
qevinstall instructions from the model card) - Related: docs/decision-models.md OmniJev section; PR [Feat] Served Cua-S1 4B screenshot decisions against system1-omni's multimodal worker #86 served Cua pattern; issues [Feat] Integrate and evaluate Cua-S1 4B 0.2 #14, [Feat] Connect desktop screenshot decisions to System1-Omni Cua-S1 4B #44
Activity