diff --git a/.env.example b/.env.example index 58acac1..dd5ef65 100644 --- a/.env.example +++ b/.env.example @@ -39,6 +39,13 @@ MODEL_NAME=google/gemini-2.5-flash # CUA_S1_SUBFOLDER=text # the text-only checkpoint; the window is 256 bytes of state # CUA_S1_DEVICE=auto # cpu, cuda, mps +# ---- OmniJev (the in-process vision decision model behind --model omnijev; needs `uv sync --extra omnijev`) ---- +# OMNIJEV_REPO=/path/to/OmniJev # required: a clone of https://github.com/tinnel123666888/OmniJev +# OMNIJEV_CHECKPOINT=tinnel123/OmniJev-0.8B # or OmniJev-2B, OmniJev (4B); a Hugging Face id or a local directory +# OMNIJEV_REVISION=v1.1 # for a Hugging Face id +# OMNIJEV_BASE=Qwen/Qwen3.5-0.8B # the Qwen3.5 backbone of the same size +# OMNIJEV_PROMPT=full # full: text state + goal + rules; short: goal + ask over the screenshot + # ---- Cua Driver (the hands of the desktop agents, e.g. calculator; install: https://cua.ai/docs/cua-driver) ---- # CUA_DRIVER_BIN=/usr/local/bin/cua-driver # when cua-driver is not on PATH # CUA_DRIVER_BIN=C:/path/to/cua-driver.exe # Windows: full path to the installed executable diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b1eca6..3d9219a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,12 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver ### Added +- `--model omnijev` on the browser agents: [OmniJev](https://github.com/tinnel123666888/OmniJev) (Apache-2.0), a + Qwen3.5 vision-language decision model, in process behind `uv sync --extra omnijev` and a local clone named by + `OMNIJEV_REPO`. It decides over a screenshot; `docs/decision-models.md`, `docs/configuration.md`. +- Browser front: a decision model that reads images (`supports_images`) gets a viewport PNG with every tick's + observation, captured through the probe's run-code executor; the tick records its `screenshot_ms`. Jev, Laya and + Cua-S1 read text only and see no change. - The MCP `decide` tool accepts `model="jev"|"laya"|"cua"`, defaulting to `jev`. Local backends use their optional extras and need no Jev API key. - `docs/benchmarks.md`: the Google Flights driver comparison rerun on 2026-09-23 from Poland, every arm three times on diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9af4d53..93e2fa2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -71,9 +71,10 @@ Everything outside `openjiuwen` is an extra. An agent whose extra is missing say | `report` | pillow, playwright | `python -m evals.replay`, the showcase pages and GIFs; `--gif` also needs `uv run playwright install chromium` | | `laya` | laya (torch, transformers) | `--model laya` on every agent and on `decide` and `probe`: Laya in process, no Jev key; the checkpoint downloads into the Hugging Face cache (`HF_HOME`) on first use | | `cua` | cua-s1 (torch), huggingface-hub | `--model cua` on tool and browser agents and on `decide` and `probe`: Cua-S1 Nano in process; the 3 MB checkpoint downloads into the Hugging Face cache (`HF_HOME`) on first use | +| `omnijev` | torch, torchvision, transformers, peft, accelerate, safetensors | `--model omnijev` on the browser agents: OmniJev in process, no Jev key; the model code is a clone of the OmniJev repository named by `OMNIJEV_REPO`, the checkpoint and base model by `OMNIJEV_CHECKPOINT` and `OMNIJEV_BASE` (`docs/configuration.md`) | | `dev` | pytest, pytest-asyncio, ruff, ty | the test suite, `scripts/smoke.sh` and the lint and type checks | -`uv sync --all-extras` installs all seven. The CLI runs from a checkout; a wheel install (`uv tool install`, +`uv sync --all-extras` installs all eight. The CLI runs from a checkout; a wheel install (`uv tool install`, `pip install`) is unsupported, because the data folders (`evals/2048`, `evals/millionaire`, `evals/labelled`) sit next to the package; the command refuses to start outside a checkout with one line on stderr. Runs, logs and results go under the checkout (`runs/`, `evals/results`), or under `S1A_HOME` when that variable names another diff --git a/README.md b/README.md index 1df99dd..4f846aa 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,28 @@ wall clock. The other Allrecipes runs, longer games and the Google Flights drive +### OmniJev: a decision model that reads the screen + +[OmniJev](https://github.com/tinnel123666888/OmniJev) (Apache-2.0, Beijing Zhongguancun Academy, CASIA and Zevo) is a +System 1 decision model on Qwen3.5 vision-language backbones (0.8B, 2B, 4B): it answers the same typed questions as +Jev over a screenshot, a video or a robot camera. `--model omnijev` puts it in the slot of the browser agents, which +then send it a screenshot of the page at every step ([docs/decision-models.md](docs/decision-models.md)). + +The clips below are OmniJev's own v1.1 demos with its 4B model: replays of recorded trajectories with the model's +probabilities, not s1a runs and not live control. They are shown from the OmniJev repository and keep their upstream +terms. + + + + + + + + + + +
OmniJev on Mind2Web web tasks: a Central Park to JFK route and a nightstand comparison, with the model's probabilities per step
Web: Mind2Web test tasks (OmniJev v1.1 replay)		OmniJev on AndroidControl phone tasks: London weather, a 59-minute timer and a drawing tutorial
Phone: AndroidControl tasks (OmniJev v1.1 replay)
OmniJev on Atari recordings: Enduro, Skiing and Pong, next input and steering
Games: Enduro, Skiing and Pong (OmniJev v1.1 replay)		OmniJev on a two-view Bridge robot recording folding a cloth: jog direction, gripper and move size
Robotics: folding a cloth, two camera views (OmniJev v1.1 replay)
+ ## Choose your path ### From Claude Code or Codex @@ -124,8 +146,8 @@ The gates and the templates: [docs/skills.md](docs/skills.md#build-a-system-1-ag - `game2048`, `millionaire`, `blackjack`: games with a score per episode. - `injection_guard`: a rail that answers one question at a hook of a running agent and fails closed. -Every agent runs on `jev`, `laya` or `cua`, and on the chat model for the comparison. Flags, run commands and -extras: [docs/agents.md](docs/agents.md). +Every agent runs on `jev`, `laya` or `cua`, and on the chat model for the comparison; the browser agents also run on +`omnijev`, which decides over a screenshot. Flags, run commands and extras: [docs/agents.md](docs/agents.md). ## How it works diff --git a/docs/agents.md b/docs/agents.md index c7ff39a..9cd9033 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -18,7 +18,7 @@ hook of a running agent. The injection guard rail fails closed: a decision error Every tool agent takes `--model jev|laya|cua|llm|random|rule`, `--rethink on|off`, `--episodes N`, `--seed S`, `--max-steps` and `--timeout`, and writes a Harbor-shaped job folder under `evals/results//`. A browser agent -takes `--model jev|laya|cua|llm` and `--goal`. A rail takes `--model jev|laya`, the two models that answer `noul`. +takes `--model jev|laya|cua|omnijev|llm` and `--goal`. A rail takes `--model jev|laya`, the two models that answer `noul`. `uv run python -m evals.table evals/results` aggregates every job folder per eval and model into one table. Every `run` prints one JSON object on stdout and nothing else there; `s1a-mcp` serves the same agents over stdio diff --git a/docs/architecture.md b/docs/architecture.md index 5bfb690..6114a34 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,6 +1,6 @@ # Architecture: fronts, specs and the shared loop -A System 1 decision model, TypeSafe Jev over HTTP or Laya and Cua-S1 in process, fills the model slot of +A System 1 decision model, TypeSafe Jev over HTTP or Laya, Cua-S1 and OmniJev in process, fills the model slot of openJiuwen agents. The model slot is the `model` argument of an openJiuwen agent constructor. A slot model is a `Model` subclass: on a decision turn it asks the decision model one `choice` question over the options the environment enumerates (the browser policy asks one per head) and answers with exactly one tool call; every @@ -12,7 +12,7 @@ score, seconds, steps and dollars. sequenceDiagram participant A as openJiuwen agent participant E as page or environment - participant S1 as Jev, Laya or Cua-S1 Nano (System 1) + participant S1 as Jev, Laya, Cua-S1 Nano or OmniJev (System 1) participant C as chat model (System 2) A->>E: observe E-->>A: state + enumerated options @@ -35,7 +35,7 @@ the description, the rules text the model reads, a budget, and the front-specifi - Rail front (`RailSpec`, `s1a/rails.py`): `DecisionModelRail` asks one question at one DeepAgent callback hook, acts on it above a threshold, and is evaluated on a labelled set. Agent: `injection_guard`. -Shared: `s1a/decision_models/` (the decision-model interface, the Jev, Laya and Cua-S1 adapters, the baselines, the answer +Shared: `s1a/decision_models/` (the decision-model interface, the Jev, Laya, Cua-S1 and OmniJev adapters, the baselines, the answer validation; `docs/decision-models.md`), `s1a/decision_models/wire.py` (the HTTP transport and its two backends), `s1a/config.py` (env loading, the chat model, the browser launch flags), `s1a/tool/rethink.py` (a rail that blocks repeats and asks for a plan on a stall), `s1a/jobs.py` @@ -49,10 +49,11 @@ validation; `docs/decision-models.md`), `s1a/decision_models/wire.py` (the HTTP | `jev` | [TypeSafe Jev](https://typesafe.ai) | over HTTP with a key; 350 to 500 ms, $0.042 per million input tokens | up to 32K tokens of state | | `laya` | [Laya](https://huggingface.co/convaiinnovations/laya) (`convaiinnovations/laya`, 0.4B) | in process, `uv sync --extra laya`; no key | a 512 to 1024 token window | | `cua` | [Cua-S1 Nano](https://huggingface.co/cua-ai/cua-s1-nano-0.1) (`cua-ai/cua-s1-nano-0.1`, 855K) | in process, `uv sync --extra cua`; no key | a 256-byte context (header, goal, state, then rules), 96 bytes per option | +| `omnijev` | [OmniJev](https://github.com/tinnel123666888/OmniJev) (Qwen3.5 VL, 0.8B by default, 2B or 4B) | in process, browser agents only, `uv sync --extra omnijev` and a clone named by `OMNIJEV_REPO`; no key; a GPU for useful speed | a viewport screenshot per tick, plus the text state (recent actions first, page text capped at 2,000 characters) | | `random`, `rule` | the tool front's two baselines | in process | the candidates | | `llm` | the chat model | for the comparison columns | the transcript | -`decision_models.build_model(model_name)` builds the first five; `llm` is not a decision model. +`decision_models.build_model(model_name)` builds the first six; `llm` is not a decision model. ## Entry points @@ -62,7 +63,7 @@ the first import of both, routes the harness logs to files under `runs/logs` bef MCP stdio protocol only. Every `run` prints one JSON object on stdout: a tool agent's series summary with its `job_dir`, a browser agent's -answer, a rail's evaluation. A browser agent takes `--model jev|laya|cua|llm`; its policy switches are run-time flags: +answer, a rail's evaluation. A browser agent takes `--model jev|laya|cua|omnijev|llm`; its policy switches are run-time flags: `--batch on|off`, `--prefetch on|off`, `--goal-values on|off`. `s1a-mcp` serves the same agents to an MCP host over stdio, one Runner for the server's lifetime and one run at a time. `uv run python -m evals.table evals/results` aggregates every job folder per eval and model into one table. `scripts/showcase.sh` plays one visual episode per @@ -71,7 +72,7 @@ eval and model outside the matrix and `python -m evals.replay` renders a pair si Every tool agent, `desktop` included, takes `--model jev|laya|cua|llm|random|rule`, `--rethink on|off`, `--episodes N`, `--seed S`, `--max-steps`, `--timeout` and `--headed`, and writes a Harbor-shaped job folder under -`evals/results//`. Every browser agent takes `--model jev|laya|cua|llm` and `--goal`. A rail takes +`evals/results//`. Every browser agent takes `--model jev|laya|cua|omnijev|llm` and `--goal`. A rail takes `--model jev|laya`, the two models that answer `noul`. `decide` and `probe` take `--model jev|laya|cua`. On a browser agent `laya` needs `LAYA_MAX_LEN` raised to the page's size; `cua` reads a 256-byte context (header, goal, state, then rules) and 96 bytes per option, a baseline on any page. Exit codes: 0 for a finished run, including one whose diff --git a/docs/configuration.md b/docs/configuration.md index 7c71622..a6b575b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -35,6 +35,11 @@ Variables can be exported in your shell or placed in a `.env` file at the root o | `CUA_S1_CHECKPOINT` | `cua` model | `cua-ai/cua-s1-nano-0.1` | Hugging Face checkpoint ID or local directory for Cua-S1 Nano option scorer. | | `CUA_S1_SUBFOLDER` | `cua` model | `text` | Subfolder within checkpoint directory containing text option scoring weights. | | `CUA_S1_DEVICE` | `cua` model | `auto` | PyTorch device used for Cua-S1 Nano evaluation (`auto`, `cpu`, `cuda`, or `mps`). | +| `OMNIJEV_REPO` | `omnijev` model | *(unset, required)* | Local clone of the OmniJev repository; its `mso` package is imported from there. | +| `OMNIJEV_CHECKPOINT` | `omnijev` model | `tinnel123/OmniJev-0.8B` | Hugging Face ID or local directory of the OmniJev adapter and heads (`OmniJev-2B`, `OmniJev` for 4B). | +| `OMNIJEV_REVISION` | `omnijev` model | `v1.1` | Revision of a Hugging Face `OMNIJEV_CHECKPOINT`. | +| `OMNIJEV_BASE` | `omnijev` model | `Qwen/Qwen3.5-0.8B` | Hugging Face ID or local directory of the Qwen3.5 backbone of the same size as the checkpoint. | +| `OMNIJEV_PROMPT` | `omnijev` model | `full` | `full` sends the text state, goal and rules with each question; `short` sends the goal and the ask only, the screenshot carrying the page. | | `CUA_DRIVER_BIN` | `desktop` agent | `cua-driver` | Path to the `cua-driver` executable on Windows or macOS when not located on `PATH`. | | `CUA_DRIVER_PERMISSION_MODE` | `desktop` agent | `standard` | Permission mode passed to `cua-driver mcp` (`standard`, or `bounded` for restricted capability manifests). | | `HF_HOME` | Hugging Face runtime | `~/.cache/huggingface` | Cache directory where Laya and Cua-S1 checkpoints are downloaded on first run. | diff --git a/docs/decision-models.md b/docs/decision-models.md index 89b2e0d..b200c84 100644 --- a/docs/decision-models.md +++ b/docs/decision-models.md @@ -45,6 +45,7 @@ shorthands; `warm()` and `close()` open and release the backend. | `jev` | `JevModel(transport)` | `jev` | the request body every front sent before the layer existed, byte for byte; `from_env` picks TypeSafe or the OpenRouter proxy (see [configuration.md](configuration.md)) | | `laya` | `LayaModel(agent, model=)` | `laya` | one forward pass per call on a thread; `MODEL_SERVICE_CONFIG_ERROR` when `input_tokens` fills the window (Laya cuts the state silently; see `LAYA_MAX_LEN` and `LAYA_HEAD_MAX_LEN` in [configuration.md](configuration.md)); `ValueError` and `RuntimeError` from the library become `MODEL_CALL_FAILED` | | `cua` | `CuaS1Model(scorer, collator, model=, context_bytes=, option_bytes=)` | `cua` | Cua-S1 Nano, one `score_elements` pass per request on a thread; choice questions only, text only, deterministic; the context is header, state and rules; the checkpoint reads its first 256 bytes, and the first overflowing request logs one warning; `from_env` reads `CUA_S1_*` (see [configuration.md](configuration.md)) | +| `omnijev` | `OmniJevModel(agent, model=)` | `omnijev` | OmniJev (a Qwen3.5 vision-language decision model), one `MSO1.system_one` call per request on a thread; choice and noul questions over the observation's screenshot, deterministic; the text state, goal and rules go in front of each question (a browser state puts its recent actions first and caps the page text at 2,000 characters, so a long page cannot push the history out of the 6,000-character context), a browser element row becomes its label and value; a choice's probabilities, which OmniJev leaves `abstain` out of, are renormalised over the offered keys; an observation without an image is `MODEL_SERVICE_CONFIG_ERROR`; `from_env` reads `OMNIJEV_*` (see [configuration.md](configuration.md)) | | `random` | `RandomModel(seed)` | `random` | uniform over the offered keys, confidence 0, one seeded stream per episode; choice questions only | | `rule` | `RuleModel(name, rule)` | the rule's name | one-hot, confidence 1; a key outside the menu raises `RuntimeError`, a bug in the rule | diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-18-26/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-18-26/answer.json new file mode 100644 index 0000000..b716c37 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-18-26/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARoaEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZAAUgBcAGCAQsI____________AZgBAg&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":110.207,\"remaining_s\":129.793,\"invocation_remaining_s\":131.004},\"summary\":\"The form phase budget has been exhausted. I cannot perform any more actions on the current page. I have successfully set the origin to Zurich, the destination to London, the flight type to one-way, and the date to November 1, 2026. However, I was unable to confirm the number of adults or the economy class.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 111278, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 12621, + "chat_output_tokens": 118, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.004081 + }, + "report": { + "elapsed_ms": 110534, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1102, + "jev_input_tokens": 0, + "median_probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-18-26/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-18-26/decision_ticks.json new file mode 100644 index 0000000..c10b4bf --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-18-26/decision_ticks.json @@ -0,0 +1,1018 @@ +{ + "report": { + "elapsed_ms": 110534, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1102, + "jev_input_tokens": 0, + "median_probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5208, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3133, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.905, + "elements": 25, + "elapsed_ms": 10231, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 666, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1328, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 724, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.968, + "elements": 10, + "elapsed_ms": 13058, + "probabilities": { + "3": 0.963, + "5": 0.028, + "2": 0.007, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 664, + "text": null + }, + { + "tick": 3, + "probe_ms": 733, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1048, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.733, + "elements": 25, + "elapsed_ms": 16797, + "probabilities": { + "16": 0.984, + "18": 0.008, + "17": 0.006, + "15": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "17": "Departure", + "15": "Where from?" + }, + "screenshot_ms": 741, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1259, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 740, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.931, + "elements": 9, + "elapsed_ms": 19713, + "probabilities": { + "3": 0.763, + "5": 0.194, + "2": 0.019, + "7": 0.014, + "4": 0.006, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 754, + "text": null + }, + { + "tick": 5, + "probe_ms": 732, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1102, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.393, + "elements": 26, + "elapsed_ms": 23495, + "probabilities": { + "12": 0.696, + "18": 0.253, + "19": 0.027, + "16": 0.008, + "6": 0.003, + "20": 0.002 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 728, + "text": null + }, + { + "tick": 6, + "probe_ms": 722, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 485, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.998, + "elements": 3, + "elapsed_ms": 26626, + "probabilities": { + "2": 0.993, + "1": 0.007, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 745, + "text": null + }, + { + "tick": 7, + "probe_ms": 739, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1065, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.56, + "elements": 25, + "elapsed_ms": 30365, + "probabilities": { + "18": 0.999, + "19": 0.0, + "6": 0.0, + "9": 0.0, + "12": 0.0, + "14": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "6": "Flights", + "9": "Change appearance", + "12": "Change ticket type. One way", + "14": "Change seating class. Economy" + }, + "screenshot_ms": 731, + "text": null + }, + { + "tick": 8, + "probe_ms": 1710, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1789, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.789, + "elements": 50, + "elapsed_ms": 35504, + "probabilities": { + "SCROLL_DOWN": 0.825, + "CLICK": 0.143, + "WAIT": 0.028, + "BLOCKED": 0.005, + "TYPE_TEXT": 0.0, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 809, + "text": null + }, + { + "tick": 9, + "probe_ms": 1122, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1915, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.528, + "elements": 51, + "elapsed_ms": 39562, + "probabilities": { + "SCROLL_DOWN": 0.595, + "CLICK": 0.348, + "WAIT": 0.041, + "BLOCKED": 0.01, + "TYPE_TEXT": 0.004, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 795, + "text": null + }, + { + "tick": 10, + "probe_ms": 1110, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1912, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.403, + "elements": 51, + "elapsed_ms": 43610, + "probabilities": { + "SCROLL_DOWN": 0.488, + "CLICK": 0.439, + "WAIT": 0.05, + "BLOCKED": 0.016, + "TYPE_TEXT": 0.007, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 792, + "text": null + }, + { + "tick": 11, + "probe_ms": 1540, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3071, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.98, + "elements": 110, + "elapsed_ms": 49236, + "probabilities": { + "35": 0.796, + "1": 0.03, + "36": 0.03, + "65": 0.019, + "108": 0.016, + "16": 0.01 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "1": "Departure", + "36": "Monday, November 2, 2026", + "65": "Tuesday, December 1, 2026", + "108": "Previous", + "16": "Tuesday, October 13, 2026 , 55 euros" + }, + "screenshot_ms": 799, + "text": null + }, + { + "tick": 12, + "probe_ms": 1534, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2972, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.602, + "elements": 110, + "elapsed_ms": 55894, + "probabilities": { + "110": 0.833, + "1": 0.041, + "35": 0.039, + "65": 0.011, + "95": 0.007, + "70": 0.005 + }, + "candidates": { + "110": "Done. Search for one-way flights, departing on November 1, 2026", + "1": "Departure", + "35": "Sunday, November 1, 2026, departure date. , 72 euros", + "65": "Tuesday, December 1, 2026 , 22 euros, Cheapest price", + "95": "Thursday, December 31, 2026 , 36 euros", + "70": "Sunday, December 6, 2026 , 28 euros" + }, + "screenshot_ms": 817, + "text": null + }, + { + "tick": 13, + "probe_ms": 736, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1109, + "input_tokens": 0, + "output_tokens": 0, + "operation": "PRESS_ENTER", + "target": "Departure", + "confidence": 0.174, + "elements": 25, + "elapsed_ms": 59674, + "probabilities": { + "18": 0.894, + "17": 0.068, + "15": 0.039 + }, + "candidates": { + "18": "Departure", + "17": "Where to?", + "15": "Where from?" + }, + "screenshot_ms": 736, + "text": null + }, + { + "tick": 14, + "probe_ms": 4779, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1097, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.264, + "elements": 25, + "elapsed_ms": 66920, + "probabilities": { + "SCROLL_DOWN": 0.37, + "PRESS_ENTER": 0.231, + "CLICK": 0.197, + "DONE": 0.105, + "WAIT": 0.095, + "TYPE_TEXT": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 730, + "text": null + }, + { + "tick": 15, + "probe_ms": 761, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1034, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.673, + "elements": 21, + "elapsed_ms": 69638, + "probabilities": { + "13": 0.545, + "1": 0.273, + "6": 0.06, + "14": 0.024, + "21": 0.022, + "16": 0.021 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "16": "Düsseldorf" + }, + "screenshot_ms": 782, + "text": null + }, + { + "tick": 16, + "probe_ms": 3761, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1030, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.612, + "elements": 21, + "elapsed_ms": 75886, + "probabilities": { + "13": 0.557, + "1": 0.264, + "6": 0.059, + "14": 0.023, + "16": 0.021, + "21": 0.02 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 775, + "text": null + }, + { + "tick": 17, + "probe_ms": 3756, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1015, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.502, + "elements": 21, + "elapsed_ms": 82121, + "probabilities": { + "1": 0.621, + "6": 0.136, + "5": 0.051, + "14": 0.045, + "21": 0.041, + "16": 0.032 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "5": "Explore", + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "16": "Düsseldorf" + }, + "screenshot_ms": 785, + "text": null + }, + { + "tick": 18, + "probe_ms": 737, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1361, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.39, + "elements": 33, + "elapsed_ms": 85685, + "probabilities": { + "SCROLL_UP": 0.492, + "CLICK": 0.453, + "WAIT": 0.023, + "BLOCKED": 0.023, + "SCROLL_DOWN": 0.009, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 781, + "text": null + }, + { + "tick": 19, + "probe_ms": 756, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1333, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.195, + "elements": 37, + "elapsed_ms": 88667, + "probabilities": { + "SCROLL_DOWN": 0.31, + "PRESS_ENTER": 0.263, + "CLICK": 0.175, + "WAIT": 0.162, + "DONE": 0.085, + "TYPE_TEXT": 0.006 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 743, + "text": null + }, + { + "tick": 20, + "probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1360, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.571, + "elements": 33, + "elapsed_ms": 91732, + "probabilities": { + "13": 0.186, + "6": 0.137, + "1": 0.134, + "20": 0.11, + "26": 0.072, + "5": 0.071 + }, + "candidates": { + "13": "Selected Tab: Flights", + "6": "Flights", + "1": "Main menu", + "20": "Change location", + "26": "Munich", + "5": "Explore" + }, + "screenshot_ms": 803, + "text": null + }, + { + "tick": 21, + "probe_ms": 741, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1019, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.617, + "elements": 21, + "elapsed_ms": 95477, + "probabilities": { + "1": 0.477, + "6": 0.146, + "14": 0.083, + "16": 0.073, + "5": 0.047, + "21": 0.043 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 797, + "text": null + }, + { + "tick": 22, + "probe_ms": 735, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1355, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.681, + "elements": 33, + "elapsed_ms": 99015, + "probabilities": { + "13": 0.148, + "1": 0.137, + "6": 0.135, + "20": 0.092, + "26": 0.083, + "5": 0.066 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "26": "Munich", + "5": "Explore" + }, + "screenshot_ms": 778, + "text": null + }, + { + "tick": 23, + "probe_ms": 730, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1012, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.589, + "elements": 21, + "elapsed_ms": 102205, + "probabilities": { + "1": 0.387, + "6": 0.197, + "14": 0.079, + "16": 0.072, + "21": 0.055, + "5": 0.05 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 784, + "text": null + }, + { + "tick": 24, + "probe_ms": 732, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1352, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.61, + "elements": 33, + "elapsed_ms": 105762, + "probabilities": { + "13": 0.188, + "6": 0.174, + "20": 0.171, + "1": 0.108, + "5": 0.083, + "12": 0.065 + }, + "candidates": { + "13": "Selected Tab: Flights", + "6": "Flights", + "20": "Change location", + "1": "Main menu", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 797, + "text": null + }, + { + "tick": 25, + "probe_ms": 729, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1013, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.759, + "elements": 21, + "elapsed_ms": 108978, + "probabilities": { + "1": 0.391, + "6": 0.203, + "14": 0.073, + "16": 0.067, + "21": 0.06, + "5": 0.053 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 794, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 12621, + "chat_output_tokens": 118, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.004081 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-22-54/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-22-54/answer.json new file mode 100644 index 0000000..4707f33 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-22-54/answer.json @@ -0,0 +1,131 @@ +{ + "ok": true, + "final": "The page shows \"No results returned.\" for the specified flight search. Therefore, the flight options are not visible.", + "screenshot": null, + "error": null, + "elapsed_ms": 68380, + "usage": { + "decisions": 14, + "chat_calls": 12, + "chat_input_tokens": 4065, + "chat_output_tokens": 94, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.001454 + }, + "report": { + "elapsed_ms": 67712, + "decisions": 14, + "interactions": 13, + "waits": 1, + "median_decision_ms": 1128, + "jev_input_tokens": 0, + "median_probe_ms": 1195, + "settle_probes": 3, + "settle_ms": 2441, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "wait", + "kind": "wait", + "text": null, + "page_changed": null + } + ] + }, + "status": "BLOCKED", + "terminal": { + "status": "BLOCKED", + "reason": "waited without progress", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 13, + "elapsed_ms": 67385, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\nNo results returned.\nOops, something went wrong.\nReload", + "answer": "The page shows \"No results returned.\" for the specified flight search. Therefore, the flight options are not visible." + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-22-54/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-22-54/decision_ticks.json new file mode 100644 index 0000000..9dcfea8 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-22-54/decision_ticks.json @@ -0,0 +1,592 @@ +{ + "report": { + "elapsed_ms": 67712, + "decisions": 14, + "interactions": 13, + "waits": 1, + "median_decision_ms": 1128, + "jev_input_tokens": 0, + "median_probe_ms": 1195, + "settle_probes": 3, + "settle_ms": 2441, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "wait", + "kind": "wait", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 4990, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3017, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.891, + "elements": 20, + "elapsed_ms": 10232, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 654, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1402, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 701, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.979, + "elements": 10, + "elapsed_ms": 13105, + "probabilities": { + "3": 0.95, + "5": 0.04, + "2": 0.007, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 667, + "text": null + }, + { + "tick": 3, + "probe_ms": 721, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 938, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.576, + "elements": 20, + "elapsed_ms": 16694, + "probabilities": { + "16": 0.993, + "18": 0.003, + "15": 0.002, + "17": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "15": "Where from?", + "17": "Departure" + }, + "screenshot_ms": 722, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1279, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 736, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.957, + "elements": 9, + "elapsed_ms": 19615, + "probabilities": { + "3": 0.664, + "5": 0.286, + "7": 0.02, + "2": 0.02, + "4": 0.005, + "9": 0.003 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "7": "London Gatwick Airport (LGW)", + "2": "Destination, Select multiple airports", + "4": "Toggle nearby airports for London, United Kingdom", + "9": "London City Airport (LCY)" + }, + "screenshot_ms": 750, + "text": null + }, + { + "tick": 5, + "probe_ms": 721, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1085, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.477, + "elements": 21, + "elapsed_ms": 23375, + "probabilities": { + "12": 0.914, + "18": 0.068, + "19": 0.009, + "16": 0.004, + "6": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 737, + "text": null + }, + { + "tick": 6, + "probe_ms": 712, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 489, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.998, + "elements": 3, + "elapsed_ms": 26503, + "probabilities": { + "2": 0.987, + "1": 0.012, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 730, + "text": null + }, + { + "tick": 7, + "probe_ms": 722, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 943, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.512, + "elements": 20, + "elapsed_ms": 30079, + "probabilities": { + "18": 0.999, + "19": 0.001, + "6": 0.0, + "12": 0.0, + "14": 0.0, + "15": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "6": "Flights", + "12": "Change ticket type. One way", + "14": "Change seating class. Economy", + "15": "Where from?" + }, + "screenshot_ms": 716, + "text": null + }, + { + "tick": 8, + "probe_ms": 1654, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1789, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.789, + "elements": 50, + "elapsed_ms": 35141, + "probabilities": { + "SCROLL_DOWN": 0.824, + "CLICK": 0.144, + "WAIT": 0.027, + "BLOCKED": 0.005, + "TYPE_TEXT": 0.0, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 784, + "text": null + }, + { + "tick": 9, + "probe_ms": 1130, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1924, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.47, + "elements": 51, + "elapsed_ms": 39676, + "probabilities": { + "SCROLL_DOWN": 0.546, + "CLICK": 0.406, + "WAIT": 0.032, + "BLOCKED": 0.011, + "TYPE_TEXT": 0.005, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 1268, + "text": null + }, + { + "tick": 10, + "probe_ms": 1133, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1922, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.442, + "elements": 51, + "elapsed_ms": 43790, + "probabilities": { + "SCROLL_DOWN": 0.522, + "CLICK": 0.405, + "WAIT": 0.053, + "BLOCKED": 0.015, + "TYPE_TEXT": 0.005, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 812, + "text": null + }, + { + "tick": 11, + "probe_ms": 1590, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2939, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.98, + "elements": 110, + "elapsed_ms": 49343, + "probabilities": { + "35": 0.651, + "1": 0.04, + "16": 0.031, + "65": 0.029, + "108": 0.021, + "8": 0.014 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "1": "Departure", + "16": "Tuesday, October 13, 2026 , 55 euros", + "65": "Tuesday, December 1, 2026", + "108": "Previous", + "8": "Monday, October 5, 2026 , 113 euros" + }, + "screenshot_ms": 795, + "text": null + }, + { + "tick": 12, + "probe_ms": 1498, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2983, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.74, + "elements": 110, + "elapsed_ms": 55882, + "probabilities": { + "110": 0.847, + "35": 0.04, + "1": 0.037, + "65": 0.01, + "95": 0.007, + "3": 0.005 + }, + "candidates": { + "110": "Done. Search for one-way flights, departing on November 1, 2026", + "35": "Sunday, November 1, 2026, departure date. , 72 euros", + "1": "Departure", + "65": "Tuesday, December 1, 2026 , 22 euros, Cheapest price", + "95": "Thursday, December 31, 2026 , 36 euros", + "3": "Reset" + }, + "screenshot_ms": 781, + "text": null + }, + { + "tick": 13, + "probe_ms": 1240, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 994, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Search", + "confidence": 0.355, + "elements": 20, + "elapsed_ms": 59535, + "probabilities": { + "19": 0.997, + "18": 0.001, + "20": 0.001, + "2": 0.0, + "11": 0.0, + "6": 0.0 + }, + "candidates": { + "19": "Search", + "18": "Departure", + "20": "Explore deals with AI", + "2": "Google", + "11": "Sign in", + "6": "Flights" + }, + "screenshot_ms": 729, + "text": null + }, + { + "tick": 14, + "probe_ms": 1150, + "settle_probes": 3, + "settle_ms": 2441, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1171, + "input_tokens": 0, + "output_tokens": 0, + "operation": "WAIT", + "target": null, + "confidence": 0.792, + "elements": 28, + "elapsed_ms": 63659, + "probabilities": { + "WAIT": 0.821, + "SCROLL_DOWN": 0.082, + "CLICK": 0.057, + "PRESS_ENTER": 0.018, + "BLOCKED": 0.018, + "TYPE_TEXT": 0.003 + }, + "candidates": { + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 636 + } + ], + "final": { + "status": "BLOCKED", + "reason": "waited without progress", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 13, + "elapsed_ms": 67385, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\nNo results returned.\nOops, something went wrong.\nReload", + "answer": "The page shows \"No results returned.\" for the specified flight search. Therefore, the flight options are not visible." + }, + "usage": { + "decisions": 14, + "chat_calls": 12, + "chat_input_tokens": 4065, + "chat_output_tokens": 94, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.001454 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-54-07/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-54-07/answer.json new file mode 100644 index 0000000..0e42648 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-54-07/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARoaEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZAAUgBcAGCAQsI____________AZgBAg&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":115.321,\"remaining_s\":124.679,\"invocation_remaining_s\":125.977},\"summary\":\"The form phase budget has been exhausted. I have successfully entered the origin (Zurich), destination (London), and departure date (November 1, 2026) for a one-way flight. However, I was unable to confirm the number of adults and economy class.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 116294, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 17606, + "chat_output_tokens": 105, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.005544 + }, + "report": { + "elapsed_ms": 115648, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1101, + "jev_input_tokens": 0, + "median_probe_ms": 1121, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-54-07/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-54-07/decision_ticks.json new file mode 100644 index 0000000..ec5d5b5 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-54-07/decision_ticks.json @@ -0,0 +1,1018 @@ +{ + "report": { + "elapsed_ms": 115648, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1101, + "jev_input_tokens": 0, + "median_probe_ms": 1121, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 4973, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3111, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.905, + "elements": 25, + "elapsed_ms": 10089, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 672, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1321, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 715, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.968, + "elements": 10, + "elapsed_ms": 12921, + "probabilities": { + "3": 0.963, + "5": 0.028, + "2": 0.007, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 676, + "text": null + }, + { + "tick": 3, + "probe_ms": 741, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1050, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.733, + "elements": 25, + "elapsed_ms": 16680, + "probabilities": { + "16": 0.984, + "18": 0.008, + "17": 0.006, + "15": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "17": "Departure", + "15": "Where from?" + }, + "screenshot_ms": 748, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1281, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 740, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.931, + "elements": 9, + "elapsed_ms": 19631, + "probabilities": { + "3": 0.763, + "5": 0.194, + "2": 0.019, + "7": 0.014, + "4": 0.006, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 755, + "text": null + }, + { + "tick": 5, + "probe_ms": 747, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1096, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.393, + "elements": 26, + "elapsed_ms": 23477, + "probabilities": { + "12": 0.696, + "18": 0.253, + "19": 0.027, + "16": 0.008, + "6": 0.003, + "20": 0.002 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 756, + "text": null + }, + { + "tick": 6, + "probe_ms": 730, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 490, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.998, + "elements": 3, + "elapsed_ms": 26668, + "probabilities": { + "2": 0.993, + "1": 0.007, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 770, + "text": null + }, + { + "tick": 7, + "probe_ms": 743, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1067, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.561, + "elements": 25, + "elapsed_ms": 30445, + "probabilities": { + "18": 0.999, + "19": 0.0, + "6": 0.0, + "9": 0.0, + "11": 0.0, + "12": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "6": "Flights", + "9": "Change appearance", + "11": "Sign in", + "12": "Change ticket type. One way" + }, + "screenshot_ms": 747, + "text": null + }, + { + "tick": 8, + "probe_ms": 1659, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1794, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.789, + "elements": 50, + "elapsed_ms": 35556, + "probabilities": { + "SCROLL_DOWN": 0.825, + "CLICK": 0.143, + "WAIT": 0.028, + "BLOCKED": 0.005, + "TYPE_TEXT": 0.0, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 813, + "text": null + }, + { + "tick": 9, + "probe_ms": 6659, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1915, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.528, + "elements": 51, + "elapsed_ms": 45188, + "probabilities": { + "SCROLL_DOWN": 0.595, + "CLICK": 0.348, + "WAIT": 0.041, + "BLOCKED": 0.01, + "TYPE_TEXT": 0.004, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 812, + "text": null + }, + { + "tick": 10, + "probe_ms": 1121, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1918, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.403, + "elements": 51, + "elapsed_ms": 49258, + "probabilities": { + "SCROLL_DOWN": 0.488, + "CLICK": 0.439, + "WAIT": 0.05, + "BLOCKED": 0.016, + "TYPE_TEXT": 0.007, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 803, + "text": null + }, + { + "tick": 11, + "probe_ms": 1560, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2930, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.979, + "elements": 110, + "elapsed_ms": 54790, + "probabilities": { + "35": 0.864, + "36": 0.019, + "1": 0.015, + "65": 0.009, + "8": 0.009, + "37": 0.008 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "36": "Monday, November 2, 2026", + "1": "Departure", + "65": "Tuesday, December 1, 2026", + "8": "Monday, October 5, 2026 , 113 euros", + "37": "Tuesday, November 3, 2026" + }, + "screenshot_ms": 809, + "text": null + }, + { + "tick": 12, + "probe_ms": 1546, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2980, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.602, + "elements": 110, + "elapsed_ms": 60939, + "probabilities": { + "110": 0.833, + "1": 0.041, + "35": 0.039, + "65": 0.011, + "95": 0.007, + "70": 0.005 + }, + "candidates": { + "110": "Done. Search for one-way flights, departing on November 1, 2026", + "1": "Departure", + "35": "Sunday, November 1, 2026, departure date. , 72 euros", + "65": "Tuesday, December 1, 2026 , 22 euros, Cheapest price", + "95": "Thursday, December 31, 2026 , 36 euros", + "70": "Sunday, December 6, 2026 , 28 euros" + }, + "screenshot_ms": 817, + "text": null + }, + { + "tick": 13, + "probe_ms": 744, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1101, + "input_tokens": 0, + "output_tokens": 0, + "operation": "PRESS_ENTER", + "target": "Departure", + "confidence": 0.174, + "elements": 25, + "elapsed_ms": 64738, + "probabilities": { + "18": 0.894, + "17": 0.068, + "15": 0.039 + }, + "candidates": { + "18": "Departure", + "17": "Where to?", + "15": "Where from?" + }, + "screenshot_ms": 750, + "text": null + }, + { + "tick": 14, + "probe_ms": 3777, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1106, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.264, + "elements": 25, + "elapsed_ms": 71006, + "probabilities": { + "SCROLL_DOWN": 0.37, + "PRESS_ENTER": 0.231, + "CLICK": 0.197, + "DONE": 0.105, + "WAIT": 0.095, + "TYPE_TEXT": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 734, + "text": null + }, + { + "tick": 15, + "probe_ms": 1283, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1035, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.653, + "elements": 21, + "elapsed_ms": 74313, + "probabilities": { + "13": 0.546, + "1": 0.262, + "6": 0.057, + "14": 0.027, + "21": 0.026, + "16": 0.025 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "16": "Düsseldorf" + }, + "screenshot_ms": 839, + "text": null + }, + { + "tick": 16, + "probe_ms": 3786, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1033, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 80637, + "probabilities": { + "13": 0.62, + "1": 0.186, + "6": 0.052, + "14": 0.029, + "16": 0.028, + "21": 0.023 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 799, + "text": null + }, + { + "tick": 17, + "probe_ms": 3789, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1014, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.526, + "elements": 21, + "elapsed_ms": 86932, + "probabilities": { + "1": 0.537, + "6": 0.134, + "5": 0.064, + "21": 0.061, + "14": 0.061, + "16": 0.049 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "16": "Düsseldorf" + }, + "screenshot_ms": 810, + "text": null + }, + { + "tick": 18, + "probe_ms": 741, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1359, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.406, + "elements": 33, + "elapsed_ms": 90510, + "probabilities": { + "SCROLL_UP": 0.505, + "CLICK": 0.441, + "WAIT": 0.023, + "BLOCKED": 0.022, + "SCROLL_DOWN": 0.009, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 795, + "text": null + }, + { + "tick": 19, + "probe_ms": 1312, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1329, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.195, + "elements": 37, + "elapsed_ms": 94052, + "probabilities": { + "SCROLL_DOWN": 0.31, + "PRESS_ENTER": 0.263, + "CLICK": 0.175, + "WAIT": 0.162, + "DONE": 0.085, + "TYPE_TEXT": 0.006 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 741, + "text": null + }, + { + "tick": 20, + "probe_ms": 776, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1359, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.541, + "elements": 33, + "elapsed_ms": 97129, + "probabilities": { + "13": 0.174, + "1": 0.155, + "6": 0.133, + "20": 0.094, + "5": 0.076, + "12": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 788, + "text": null + }, + { + "tick": 21, + "probe_ms": 743, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1014, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.626, + "elements": 21, + "elapsed_ms": 100376, + "probabilities": { + "1": 0.491, + "6": 0.125, + "14": 0.089, + "16": 0.082, + "5": 0.046, + "21": 0.042 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 788, + "text": null + }, + { + "tick": 22, + "probe_ms": 740, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1358, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.671, + "elements": 33, + "elapsed_ms": 103947, + "probabilities": { + "13": 0.145, + "1": 0.143, + "6": 0.135, + "20": 0.078, + "5": 0.077, + "26": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "26": "Munich" + }, + "screenshot_ms": 786, + "text": null + }, + { + "tick": 23, + "probe_ms": 736, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1011, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 107189, + "probabilities": { + "1": 0.401, + "6": 0.185, + "14": 0.081, + "16": 0.076, + "21": 0.058, + "5": 0.048 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 802, + "text": null + }, + { + "tick": 24, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1350, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights", + "confidence": 0.624, + "elements": 33, + "elapsed_ms": 110753, + "probabilities": { + "6": 0.186, + "13": 0.181, + "20": 0.154, + "1": 0.103, + "5": 0.087, + "12": 0.07 + }, + "candidates": { + "6": "Flights", + "13": "Selected Tab: Flights", + "20": "Change location", + "1": "Main menu", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 784, + "text": null + }, + { + "tick": 25, + "probe_ms": 744, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1014, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.758, + "elements": 21, + "elapsed_ms": 114005, + "probabilities": { + "1": 0.43, + "6": 0.201, + "14": 0.076, + "16": 0.07, + "21": 0.046, + "5": 0.043 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 816, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 17606, + "chat_output_tokens": 105, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.005544 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-56-33/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-56-33/answer.json new file mode 100644 index 0000000..ec89f6d --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-56-33/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARoaEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZAAUgBcAGCAQsI____________AZgBAg&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":105.322,\"remaining_s\":134.678,\"invocation_remaining_s\":135.82},\"summary\":\"The form phase budget has been exhausted. I have filled in the origin, destination, and date, and set the flight type to one-way. However, I was unable to click the \\\"Done\\\" button to confirm the date selection and proceed to view the flight options.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 106333, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 18984, + "chat_output_tokens": 102, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.00595 + }, + "report": { + "elapsed_ms": 105651, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1100, + "jev_input_tokens": 0, + "median_probe_ms": 777, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-56-33/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-56-33/decision_ticks.json new file mode 100644 index 0000000..bea5a93 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-56-33/decision_ticks.json @@ -0,0 +1,1018 @@ +{ + "report": { + "elapsed_ms": 105651, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1100, + "jev_input_tokens": 0, + "median_probe_ms": 777, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 1191, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3119, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.905, + "elements": 25, + "elapsed_ms": 6413, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 679, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1314, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 713, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.968, + "elements": 10, + "elapsed_ms": 9239, + "probabilities": { + "3": 0.963, + "5": 0.028, + "2": 0.007, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 679, + "text": null + }, + { + "tick": 3, + "probe_ms": 736, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1054, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.733, + "elements": 25, + "elapsed_ms": 13025, + "probabilities": { + "16": 0.984, + "18": 0.008, + "17": 0.006, + "15": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "17": "Departure", + "15": "Where from?" + }, + "screenshot_ms": 744, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1287, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 742, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.931, + "elements": 9, + "elapsed_ms": 15979, + "probabilities": { + "3": 0.763, + "5": 0.194, + "2": 0.019, + "7": 0.014, + "4": 0.006, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 769, + "text": null + }, + { + "tick": 5, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1100, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.393, + "elements": 26, + "elapsed_ms": 19784, + "probabilities": { + "12": 0.696, + "18": 0.253, + "19": 0.027, + "16": 0.008, + "6": 0.003, + "20": 0.002 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 757, + "text": null + }, + { + "tick": 6, + "probe_ms": 729, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 493, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.998, + "elements": 3, + "elapsed_ms": 22957, + "probabilities": { + "2": 0.993, + "1": 0.007, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 752, + "text": null + }, + { + "tick": 7, + "probe_ms": 748, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1065, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.56, + "elements": 25, + "elapsed_ms": 26718, + "probabilities": { + "18": 0.999, + "19": 0.0, + "6": 0.0, + "9": 0.0, + "12": 0.0, + "14": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "6": "Flights", + "9": "Change appearance", + "12": "Change ticket type. One way", + "14": "Change seating class. Economy" + }, + "screenshot_ms": 728, + "text": null + }, + { + "tick": 8, + "probe_ms": 1653, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1790, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.789, + "elements": 50, + "elapsed_ms": 31810, + "probabilities": { + "SCROLL_DOWN": 0.825, + "CLICK": 0.143, + "WAIT": 0.028, + "BLOCKED": 0.005, + "TYPE_TEXT": 0.0, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 823, + "text": null + }, + { + "tick": 9, + "probe_ms": 1124, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1920, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.528, + "elements": 51, + "elapsed_ms": 35889, + "probabilities": { + "SCROLL_DOWN": 0.595, + "CLICK": 0.348, + "WAIT": 0.041, + "BLOCKED": 0.01, + "TYPE_TEXT": 0.004, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 807, + "text": null + }, + { + "tick": 10, + "probe_ms": 1116, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1915, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.403, + "elements": 51, + "elapsed_ms": 39959, + "probabilities": { + "SCROLL_DOWN": 0.488, + "CLICK": 0.439, + "WAIT": 0.05, + "BLOCKED": 0.016, + "TYPE_TEXT": 0.007, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 798, + "text": null + }, + { + "tick": 11, + "probe_ms": 1563, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2932, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.986, + "elements": 110, + "elapsed_ms": 45495, + "probabilities": { + "35": 0.83, + "1": 0.033, + "65": 0.017, + "108": 0.013, + "110": 0.008, + "39": 0.005 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "1": "Departure", + "65": "Tuesday, December 1, 2026", + "108": "Previous", + "110": "Done. Search for one-way flights, departing on November 29, 2026", + "39": "Thursday, November 5, 2026" + }, + "screenshot_ms": 802, + "text": null + }, + { + "tick": 12, + "probe_ms": 1515, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2973, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.84, + "elements": 110, + "elapsed_ms": 51561, + "probabilities": { + "110": 0.642, + "35": 0.221, + "1": 0.087, + "65": 0.006, + "36": 0.004, + "61": 0.003 + }, + "candidates": { + "110": "Done. Search for one-way flights, departing on November 1, 2026", + "35": "Sunday, November 1, 2026, departure date. , 72 euros", + "1": "Departure", + "65": "Tuesday, December 1, 2026 , 22 euros, Cheapest price", + "36": "Monday, November 2, 2026 , 45 euros", + "61": "Friday, November 27, 2026 , 27 euros" + }, + "screenshot_ms": 797, + "text": null + }, + { + "tick": 13, + "probe_ms": 737, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1099, + "input_tokens": 0, + "output_tokens": 0, + "operation": "PRESS_ENTER", + "target": "Departure", + "confidence": 0.174, + "elements": 25, + "elapsed_ms": 55317, + "probabilities": { + "18": 0.894, + "17": 0.068, + "15": 0.039 + }, + "candidates": { + "18": "Departure", + "17": "Where to?", + "15": "Where from?" + }, + "screenshot_ms": 723, + "text": null + }, + { + "tick": 14, + "probe_ms": 3786, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1100, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.264, + "elements": 25, + "elapsed_ms": 61600, + "probabilities": { + "SCROLL_DOWN": 0.37, + "PRESS_ENTER": 0.231, + "CLICK": 0.197, + "DONE": 0.105, + "WAIT": 0.095, + "TYPE_TEXT": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 756, + "text": null + }, + { + "tick": 15, + "probe_ms": 1258, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1032, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.653, + "elements": 21, + "elapsed_ms": 64826, + "probabilities": { + "13": 0.546, + "1": 0.262, + "6": 0.057, + "14": 0.027, + "21": 0.026, + "16": 0.025 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "16": "Düsseldorf" + }, + "screenshot_ms": 785, + "text": null + }, + { + "tick": 16, + "probe_ms": 3764, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1031, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 71087, + "probabilities": { + "13": 0.62, + "1": 0.186, + "6": 0.052, + "14": 0.029, + "16": 0.028, + "21": 0.023 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 784, + "text": null + }, + { + "tick": 17, + "probe_ms": 3779, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1014, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.526, + "elements": 21, + "elapsed_ms": 77334, + "probabilities": { + "1": 0.537, + "6": 0.134, + "5": 0.064, + "21": 0.061, + "14": 0.061, + "16": 0.049 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "16": "Düsseldorf" + }, + "screenshot_ms": 779, + "text": null + }, + { + "tick": 18, + "probe_ms": 744, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1359, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.406, + "elements": 33, + "elapsed_ms": 80901, + "probabilities": { + "SCROLL_UP": 0.505, + "CLICK": 0.441, + "WAIT": 0.023, + "BLOCKED": 0.022, + "SCROLL_DOWN": 0.009, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 785, + "text": null + }, + { + "tick": 19, + "probe_ms": 777, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1331, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.195, + "elements": 37, + "elapsed_ms": 83890, + "probabilities": { + "SCROLL_DOWN": 0.31, + "PRESS_ENTER": 0.263, + "CLICK": 0.175, + "WAIT": 0.162, + "DONE": 0.085, + "TYPE_TEXT": 0.006 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 737, + "text": null + }, + { + "tick": 20, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1358, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.541, + "elements": 33, + "elapsed_ms": 86945, + "probabilities": { + "13": 0.174, + "1": 0.155, + "6": 0.133, + "20": 0.094, + "5": 0.076, + "12": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 786, + "text": null + }, + { + "tick": 21, + "probe_ms": 739, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1012, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.626, + "elements": 21, + "elapsed_ms": 90661, + "probabilities": { + "1": 0.491, + "6": 0.125, + "14": 0.089, + "16": 0.082, + "5": 0.046, + "21": 0.042 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 775, + "text": null + }, + { + "tick": 22, + "probe_ms": 733, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1355, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.671, + "elements": 33, + "elapsed_ms": 94221, + "probabilities": { + "13": 0.145, + "1": 0.143, + "6": 0.135, + "20": 0.078, + "5": 0.077, + "26": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "26": "Munich" + }, + "screenshot_ms": 803, + "text": null + }, + { + "tick": 23, + "probe_ms": 730, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1011, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 97411, + "probabilities": { + "1": 0.401, + "6": 0.185, + "14": 0.081, + "16": 0.076, + "21": 0.058, + "5": 0.048 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 778, + "text": null + }, + { + "tick": 24, + "probe_ms": 740, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1345, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights", + "confidence": 0.624, + "elements": 33, + "elapsed_ms": 100942, + "probabilities": { + "6": 0.186, + "13": 0.181, + "20": 0.154, + "1": 0.103, + "5": 0.087, + "12": 0.07 + }, + "candidates": { + "6": "Flights", + "13": "Selected Tab: Flights", + "20": "Change location", + "1": "Main menu", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 782, + "text": null + }, + { + "tick": 25, + "probe_ms": 734, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1012, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.758, + "elements": 21, + "elapsed_ms": 104163, + "probabilities": { + "1": 0.43, + "6": 0.201, + "14": 0.076, + "16": 0.07, + "21": 0.046, + "5": 0.043 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 789, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 18984, + "chat_output_tokens": 102, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.00595 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-58-50/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-58-50/answer.json new file mode 100644 index 0000000..26d9940 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-58-50/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARoaEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZAAUgBcAGCAQsI____________AZgBAg&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":109.412,\"remaining_s\":130.588,\"invocation_remaining_s\":131.738},\"summary\":\"The form phase budget has been exhausted. I have successfully entered the origin, destination, and date for a one-way flight. However, I was unable to select the number of adults and economy class.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 110394, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 19544, + "chat_output_tokens": 88, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006083 + }, + "report": { + "elapsed_ms": 109744, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1105, + "jev_input_tokens": 0, + "median_probe_ms": 762, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-58-50/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-58-50/decision_ticks.json new file mode 100644 index 0000000..dcb43c9 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__14-58-50/decision_ticks.json @@ -0,0 +1,1018 @@ +{ + "report": { + "elapsed_ms": 109744, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1105, + "jev_input_tokens": 0, + "median_probe_ms": 762, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 4906, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3114, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.914, + "elements": 25, + "elapsed_ms": 10150, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 672, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1336, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 715, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.964, + "elements": 10, + "elapsed_ms": 12995, + "probabilities": { + "3": 0.949, + "5": 0.041, + "2": 0.007, + "4": 0.002, + "6": 0.001, + "9": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 672, + "text": null + }, + { + "tick": 3, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1055, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.71, + "elements": 25, + "elapsed_ms": 16840, + "probabilities": { + "16": 0.983, + "18": 0.009, + "17": 0.006, + "15": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "17": "Departure", + "15": "Where from?" + }, + "screenshot_ms": 748, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1295, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 738, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.935, + "elements": 9, + "elapsed_ms": 19805, + "probabilities": { + "3": 0.733, + "5": 0.218, + "2": 0.021, + "7": 0.017, + "4": 0.006, + "6": 0.002 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 766, + "text": null + }, + { + "tick": 5, + "probe_ms": 738, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1105, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.425, + "elements": 26, + "elapsed_ms": 23617, + "probabilities": { + "12": 0.658, + "18": 0.285, + "19": 0.035, + "16": 0.007, + "6": 0.003, + "20": 0.002 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 736, + "text": null + }, + { + "tick": 6, + "probe_ms": 730, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 483, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.998, + "elements": 3, + "elapsed_ms": 26814, + "probabilities": { + "2": 0.993, + "1": 0.007, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 746, + "text": null + }, + { + "tick": 7, + "probe_ms": 742, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1069, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.568, + "elements": 25, + "elapsed_ms": 30580, + "probabilities": { + "18": 0.999, + "19": 0.0, + "6": 0.0, + "9": 0.0, + "12": 0.0, + "14": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "6": "Flights", + "9": "Change appearance", + "12": "Change ticket type. One way", + "14": "Change seating class. Economy" + }, + "screenshot_ms": 736, + "text": null + }, + { + "tick": 8, + "probe_ms": 1656, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1793, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.773, + "elements": 50, + "elapsed_ms": 35649, + "probabilities": { + "SCROLL_DOWN": 0.811, + "CLICK": 0.158, + "WAIT": 0.027, + "BLOCKED": 0.005, + "TYPE_TEXT": 0.0, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 805, + "text": null + }, + { + "tick": 9, + "probe_ms": 1134, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1922, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.509, + "elements": 51, + "elapsed_ms": 39743, + "probabilities": { + "SCROLL_DOWN": 0.579, + "CLICK": 0.366, + "WAIT": 0.04, + "BLOCKED": 0.011, + "TYPE_TEXT": 0.004, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 808, + "text": null + }, + { + "tick": 10, + "probe_ms": 1145, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1914, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.432, + "elements": 51, + "elapsed_ms": 43836, + "probabilities": { + "SCROLL_DOWN": 0.513, + "CLICK": 0.411, + "WAIT": 0.053, + "BLOCKED": 0.017, + "TYPE_TEXT": 0.006, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 799, + "text": null + }, + { + "tick": 11, + "probe_ms": 1564, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2938, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.981, + "elements": 110, + "elapsed_ms": 49356, + "probabilities": { + "35": 0.895, + "36": 0.018, + "1": 0.013, + "65": 0.008, + "8": 0.006, + "108": 0.005 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "36": "Monday, November 2, 2026", + "1": "Departure", + "65": "Tuesday, December 1, 2026", + "8": "Monday, October 5, 2026 , 113 euros", + "108": "Previous" + }, + "screenshot_ms": 792, + "text": null + }, + { + "tick": 12, + "probe_ms": 1490, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2985, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.599, + "elements": 110, + "elapsed_ms": 55416, + "probabilities": { + "110": 0.827, + "35": 0.041, + "1": 0.038, + "65": 0.011, + "95": 0.006, + "61": 0.005 + }, + "candidates": { + "110": "Done. Search for one-way flights, departing on November 1, 2026", + "35": "Sunday, November 1, 2026, departure date. , 72 euros", + "1": "Departure", + "65": "Tuesday, December 1, 2026 , 22 euros, Cheapest price", + "95": "Thursday, December 31, 2026 , 36 euros", + "61": "Friday, November 27, 2026 , 27 euros" + }, + "screenshot_ms": 798, + "text": null + }, + { + "tick": 13, + "probe_ms": 738, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1103, + "input_tokens": 0, + "output_tokens": 0, + "operation": "PRESS_ENTER", + "target": "Departure", + "confidence": 0.174, + "elements": 25, + "elapsed_ms": 59233, + "probabilities": { + "18": 0.886, + "17": 0.073, + "15": 0.041 + }, + "candidates": { + "18": "Departure", + "17": "Where to?", + "15": "Where from?" + }, + "screenshot_ms": 761, + "text": null + }, + { + "tick": 14, + "probe_ms": 4278, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1105, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.231, + "elements": 25, + "elapsed_ms": 65997, + "probabilities": { + "SCROLL_DOWN": 0.341, + "PRESS_ENTER": 0.245, + "CLICK": 0.218, + "DONE": 0.101, + "WAIT": 0.095, + "TYPE_TEXT": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 736, + "text": null + }, + { + "tick": 15, + "probe_ms": 776, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1036, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.653, + "elements": 21, + "elapsed_ms": 68758, + "probabilities": { + "13": 0.546, + "1": 0.262, + "6": 0.057, + "14": 0.027, + "21": 0.026, + "16": 0.025 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "16": "Düsseldorf" + }, + "screenshot_ms": 808, + "text": null + }, + { + "tick": 16, + "probe_ms": 3821, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1032, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 75084, + "probabilities": { + "13": 0.62, + "1": 0.186, + "6": 0.052, + "14": 0.029, + "16": 0.028, + "21": 0.023 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 792, + "text": null + }, + { + "tick": 17, + "probe_ms": 3772, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1025, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.526, + "elements": 21, + "elapsed_ms": 81365, + "probabilities": { + "1": 0.537, + "6": 0.134, + "5": 0.064, + "21": 0.061, + "14": 0.061, + "16": 0.049 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "16": "Düsseldorf" + }, + "screenshot_ms": 804, + "text": null + }, + { + "tick": 18, + "probe_ms": 735, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1361, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.406, + "elements": 33, + "elapsed_ms": 84922, + "probabilities": { + "SCROLL_UP": 0.505, + "CLICK": 0.441, + "WAIT": 0.023, + "BLOCKED": 0.022, + "SCROLL_DOWN": 0.009, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 779, + "text": null + }, + { + "tick": 19, + "probe_ms": 762, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1335, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.195, + "elements": 37, + "elapsed_ms": 87930, + "probabilities": { + "SCROLL_DOWN": 0.31, + "PRESS_ENTER": 0.263, + "CLICK": 0.175, + "WAIT": 0.162, + "DONE": 0.085, + "TYPE_TEXT": 0.006 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 759, + "text": null + }, + { + "tick": 20, + "probe_ms": 761, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1362, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.541, + "elements": 33, + "elapsed_ms": 90987, + "probabilities": { + "13": 0.174, + "1": 0.155, + "6": 0.133, + "20": 0.094, + "5": 0.076, + "12": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 784, + "text": null + }, + { + "tick": 21, + "probe_ms": 738, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1016, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.626, + "elements": 21, + "elapsed_ms": 94715, + "probabilities": { + "1": 0.491, + "6": 0.125, + "14": 0.089, + "16": 0.082, + "5": 0.046, + "21": 0.042 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 778, + "text": null + }, + { + "tick": 22, + "probe_ms": 736, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1363, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.671, + "elements": 33, + "elapsed_ms": 98281, + "probabilities": { + "13": 0.145, + "1": 0.143, + "6": 0.135, + "20": 0.078, + "5": 0.077, + "26": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "26": "Munich" + }, + "screenshot_ms": 780, + "text": null + }, + { + "tick": 23, + "probe_ms": 732, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1016, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 101487, + "probabilities": { + "1": 0.401, + "6": 0.185, + "14": 0.081, + "16": 0.076, + "21": 0.058, + "5": 0.048 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 787, + "text": null + }, + { + "tick": 24, + "probe_ms": 732, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1354, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights", + "confidence": 0.624, + "elements": 33, + "elapsed_ms": 105027, + "probabilities": { + "6": 0.186, + "13": 0.181, + "20": 0.154, + "1": 0.103, + "5": 0.087, + "12": 0.07 + }, + "candidates": { + "6": "Flights", + "13": "Selected Tab: Flights", + "20": "Change location", + "1": "Main menu", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 780, + "text": null + }, + { + "tick": 25, + "probe_ms": 739, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1014, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.758, + "elements": 21, + "elapsed_ms": 108246, + "probabilities": { + "1": 0.43, + "6": 0.201, + "14": 0.076, + "16": 0.07, + "21": 0.046, + "5": 0.043 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 776, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 19544, + "chat_output_tokens": 88, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006083 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-01-10/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-01-10/answer.json new file mode 100644 index 0000000..8f0d330 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-01-10/answer.json @@ -0,0 +1,149 @@ +{ + "ok": true, + "final": "The page text is empty, so I cannot find the requested flight information.", + "screenshot": null, + "error": null, + "elapsed_ms": 84953, + "usage": { + "decisions": 17, + "chat_calls": 10, + "chat_input_tokens": 3088, + "chat_output_tokens": 68, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.001096 + }, + "report": { + "elapsed_ms": 84289, + "decisions": 17, + "interactions": 16, + "waits": 1, + "median_decision_ms": 1104, + "jev_input_tokens": 0, + "median_probe_ms": 1277, + "settle_probes": 3, + "settle_ms": 2345, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "wait", + "kind": "wait", + "text": null, + "page_changed": null + } + ] + }, + "status": "BLOCKED", + "terminal": { + "status": "BLOCKED", + "reason": "waited without progress", + "url": "https://www.google.com/travel/flights?tfs=CBwQARoOagwIAxIIL20vMDg5NjYaDnIMCAMSCC9tLzA4OTY2QAFIAXABggELCP___________wGYAQE&tfu=KgIIAw&hl=en", + "title": "Find Cheap Flights Worldwide & Book Your Ticket - Google Flights", + "steps": 16, + "elapsed_ms": 83943, + "page_text": "", + "answer": "The page text is empty, so I cannot find the requested flight information." + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-01-10/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-01-10/decision_ticks.json new file mode 100644 index 0000000..cd864ac --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-01-10/decision_ticks.json @@ -0,0 +1,701 @@ +{ + "report": { + "elapsed_ms": 84289, + "decisions": 17, + "interactions": 16, + "waits": 1, + "median_decision_ms": 1104, + "jev_input_tokens": 0, + "median_probe_ms": 1277, + "settle_probes": 3, + "settle_ms": 2345, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "wait", + "kind": "wait", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5562, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3115, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.914, + "elements": 25, + "elapsed_ms": 10599, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 682, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1293, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 721, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.964, + "elements": 10, + "elapsed_ms": 13406, + "probabilities": { + "3": 0.949, + "5": 0.041, + "2": 0.007, + "4": 0.002, + "6": 0.001, + "9": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 672, + "text": null + }, + { + "tick": 3, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1054, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.71, + "elements": 25, + "elapsed_ms": 17264, + "probabilities": { + "16": 0.983, + "18": 0.009, + "17": 0.006, + "15": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "17": "Departure", + "15": "Where from?" + }, + "screenshot_ms": 769, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1277, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 742, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.935, + "elements": 9, + "elapsed_ms": 20228, + "probabilities": { + "3": 0.733, + "5": 0.218, + "2": 0.021, + "7": 0.017, + "4": 0.006, + "6": 0.002 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 770, + "text": null + }, + { + "tick": 5, + "probe_ms": 742, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1104, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.425, + "elements": 26, + "elapsed_ms": 24061, + "probabilities": { + "12": 0.658, + "18": 0.285, + "19": 0.035, + "16": 0.007, + "6": 0.003, + "20": 0.002 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 750, + "text": null + }, + { + "tick": 6, + "probe_ms": 734, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 491, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.998, + "elements": 3, + "elapsed_ms": 27246, + "probabilities": { + "2": 0.993, + "1": 0.007, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 744, + "text": null + }, + { + "tick": 7, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1071, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.568, + "elements": 25, + "elapsed_ms": 31025, + "probabilities": { + "18": 0.999, + "19": 0.0, + "6": 0.0, + "9": 0.0, + "12": 0.0, + "14": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "6": "Flights", + "9": "Change appearance", + "12": "Change ticket type. One way", + "14": "Change seating class. Economy" + }, + "screenshot_ms": 734, + "text": null + }, + { + "tick": 8, + "probe_ms": 1650, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1792, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.773, + "elements": 50, + "elapsed_ms": 36123, + "probabilities": { + "SCROLL_DOWN": 0.811, + "CLICK": 0.158, + "WAIT": 0.027, + "BLOCKED": 0.005, + "TYPE_TEXT": 0.0, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 808, + "text": null + }, + { + "tick": 9, + "probe_ms": 4601, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1917, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.528, + "elements": 51, + "elapsed_ms": 43844, + "probabilities": { + "SCROLL_DOWN": 0.595, + "CLICK": 0.348, + "WAIT": 0.041, + "BLOCKED": 0.01, + "TYPE_TEXT": 0.004, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 837, + "text": null + }, + { + "tick": 10, + "probe_ms": 1138, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1959, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.419, + "elements": 51, + "elapsed_ms": 48035, + "probabilities": { + "SCROLL_DOWN": 0.502, + "CLICK": 0.42, + "WAIT": 0.058, + "BLOCKED": 0.014, + "TYPE_TEXT": 0.006, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 818, + "text": null + }, + { + "tick": 11, + "probe_ms": 1597, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2940, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.979, + "elements": 110, + "elapsed_ms": 53629, + "probabilities": { + "35": 0.864, + "36": 0.019, + "1": 0.015, + "65": 0.009, + "8": 0.009, + "37": 0.008 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "36": "Monday, November 2, 2026", + "1": "Departure", + "65": "Tuesday, December 1, 2026", + "8": "Monday, October 5, 2026 , 113 euros", + "37": "Tuesday, November 3, 2026" + }, + "screenshot_ms": 815, + "text": null + }, + { + "tick": 12, + "probe_ms": 1543, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2983, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.602, + "elements": 110, + "elapsed_ms": 59749, + "probabilities": { + "110": 0.833, + "1": 0.041, + "35": 0.039, + "65": 0.011, + "95": 0.007, + "70": 0.005 + }, + "candidates": { + "110": "Done. Search for one-way flights, departing on November 1, 2026", + "1": "Departure", + "35": "Sunday, November 1, 2026, departure date. , 72 euros", + "65": "Tuesday, December 1, 2026 , 22 euros, Cheapest price", + "95": "Thursday, December 31, 2026 , 36 euros", + "70": "Sunday, December 6, 2026 , 28 euros" + }, + "screenshot_ms": 804, + "text": null + }, + { + "tick": 13, + "probe_ms": 1249, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1139, + "input_tokens": 0, + "output_tokens": 0, + "operation": "PRESS_ENTER", + "target": "Departure", + "confidence": 0.174, + "elements": 25, + "elapsed_ms": 64150, + "probabilities": { + "18": 0.894, + "17": 0.068, + "15": 0.039 + }, + "candidates": { + "18": "Departure", + "17": "Where to?", + "15": "Where from?" + }, + "screenshot_ms": 777, + "text": null + }, + { + "tick": 14, + "probe_ms": 3215, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 1, + "action_settle_ms": 250, + "decision_ms": 951, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.641, + "elements": 20, + "elapsed_ms": 69676, + "probabilities": { + "15": 0.996, + "17": 0.002, + "16": 0.001, + "18": 0.001 + }, + "candidates": { + "15": "Where from?", + "17": "Departure", + "16": "Where to?", + "18": "Return" + }, + "screenshot_ms": 701, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 15, + "probe_ms": 1289, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 739, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.989, + "elements": 10, + "elapsed_ms": 72495, + "probabilities": { + "3": 0.919, + "5": 0.06, + "2": 0.014, + "4": 0.004, + "6": 0.002, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 673, + "text": null + }, + { + "tick": 16, + "probe_ms": 726, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1727, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.428, + "elements": 20, + "elapsed_ms": 76954, + "probabilities": { + "12": 0.497, + "16": 0.481, + "15": 0.009, + "17": 0.004, + "6": 0.002, + "18": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "16": "Where to?", + "15": "Where from?", + "17": "Departure", + "6": "Flights", + "18": "Return" + }, + "screenshot_ms": 738, + "text": null + }, + { + "tick": 17, + "probe_ms": 739, + "settle_probes": 3, + "settle_ms": 2345, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 571, + "input_tokens": 0, + "output_tokens": 0, + "operation": "WAIT", + "target": null, + "confidence": 0.685, + "elements": 0, + "elapsed_ms": 80223, + "probabilities": { + "WAIT": 0.764, + "SCROLL_DOWN": 0.172, + "DONE": 0.033, + "BLOCKED": 0.031 + }, + "candidates": { + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 746 + } + ], + "final": { + "status": "BLOCKED", + "reason": "waited without progress", + "url": "https://www.google.com/travel/flights?tfs=CBwQARoOagwIAxIIL20vMDg5NjYaDnIMCAMSCC9tLzA4OTY2QAFIAXABggELCP___________wGYAQE&tfu=KgIIAw&hl=en", + "title": "Find Cheap Flights Worldwide & Book Your Ticket - Google Flights", + "steps": 16, + "elapsed_ms": 83943, + "page_text": "", + "answer": "The page text is empty, so I cannot find the requested flight information." + }, + "usage": { + "decisions": 17, + "chat_calls": 10, + "chat_input_tokens": 3088, + "chat_output_tokens": 68, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.001096 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-02-15/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-02-15/answer.json new file mode 100644 index 0000000..a7aa9ab --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-02-15/answer.json @@ -0,0 +1,71 @@ +{ + "ok": true, + "final": "The page text is missing. I need the content of the webpage to find the flight information.", + "screenshot": null, + "error": null, + "elapsed_ms": 22019, + "usage": { + "decisions": 4, + "chat_calls": 8, + "chat_input_tokens": 2214, + "chat_output_tokens": 61, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.000817 + }, + "report": { + "elapsed_ms": 21347, + "decisions": 4, + "interactions": 3, + "waits": 1, + "median_decision_ms": 1221, + "jev_input_tokens": 0, + "median_probe_ms": 724, + "settle_probes": 3, + "settle_ms": 2546, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "wait", + "kind": "wait", + "text": null, + "page_changed": null + } + ] + }, + "status": "BLOCKED", + "terminal": { + "status": "BLOCKED", + "reason": "waited without progress", + "url": "https://www.google.com/travel/flights?tfs=CBwQARoOagwIAxIIL20vMDg5NjYaDnIMCAMSCC9tLzA4OTY2QAFIAXABggELCP___________wGYAQE&tfu=KgIIAw&hl=en", + "title": "Find Cheap Flights Worldwide & Book Your Ticket - Google Flights", + "steps": 3, + "elapsed_ms": 20989, + "page_text": "", + "answer": "The page text is missing. I need the content of the webpage to find the flight information." + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-02-15/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-02-15/decision_ticks.json new file mode 100644 index 0000000..589e20b --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-02-15/decision_ticks.json @@ -0,0 +1,194 @@ +{ + "report": { + "elapsed_ms": 21347, + "decisions": 4, + "interactions": 3, + "waits": 1, + "median_decision_ms": 1221, + "jev_input_tokens": 0, + "median_probe_ms": 724, + "settle_probes": 3, + "settle_ms": 2546, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "wait", + "kind": "wait", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 698, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3104, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.907, + "elements": 20, + "elapsed_ms": 6327, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 1201, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 674, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 701, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.982, + "elements": 10, + "elapsed_ms": 9108, + "probabilities": { + "3": 0.929, + "5": 0.059, + "2": 0.008, + "4": 0.002, + "6": 0.002, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 1283, + "text": null + }, + { + "tick": 3, + "probe_ms": 750, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1742, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.576, + "elements": 20, + "elapsed_ms": 12345, + "probabilities": { + "16": 0.993, + "18": 0.003, + "15": 0.002, + "17": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "15": "Where from?", + "17": "Departure" + }, + "screenshot_ms": 719, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 2230, + "settle_probes": 3, + "settle_ms": 2546, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 444, + "input_tokens": 0, + "output_tokens": 0, + "operation": "WAIT", + "target": null, + "confidence": 0.659, + "elements": 0, + "elapsed_ms": 15919, + "probabilities": { + "WAIT": 0.744, + "SCROLL_DOWN": 0.191, + "BLOCKED": 0.063, + "DONE": 0.002 + }, + "candidates": { + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 748 + } + ], + "final": { + "status": "BLOCKED", + "reason": "waited without progress", + "url": "https://www.google.com/travel/flights?tfs=CBwQARoOagwIAxIIL20vMDg5NjYaDnIMCAMSCC9tLzA4OTY2QAFIAXABggELCP___________wGYAQE&tfu=KgIIAw&hl=en", + "title": "Find Cheap Flights Worldwide & Book Your Ticket - Google Flights", + "steps": 3, + "elapsed_ms": 20989, + "page_text": "", + "answer": "The page text is missing. I need the content of the webpage to find the flight information." + }, + "usage": { + "decisions": 4, + "chat_calls": 8, + "chat_input_tokens": 2214, + "chat_output_tokens": 61, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.000817 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-06/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-06/answer.json new file mode 100644 index 0000000..9f12de9 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-06/answer.json @@ -0,0 +1,83 @@ +{ + "ok": true, + "final": "The page does not show any flight options for the specified criteria. It only displays example flight deals from Frankfurt am Main to other cities.", + "screenshot": null, + "error": null, + "elapsed_ms": 37153, + "usage": { + "decisions": 7, + "chat_calls": 10, + "chat_input_tokens": 3805, + "chat_output_tokens": 81, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.001344 + }, + "report": { + "elapsed_ms": 36468, + "decisions": 7, + "interactions": 6, + "waits": 0, + "median_decision_ms": 1441, + "jev_input_tokens": 0, + "median_probe_ms": 859, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 4, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + } + ] + }, + "status": "BLOCKED", + "terminal": { + "status": "BLOCKED", + "reason": "no value for field 'Return': value model returned no usable text", + "url": "https://www.google.com/travel/flights?tfs=CBwQARocagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbBocagwIAxIIL20vMDRqcGxyDAgDEggvbS8wODk2NkABSAFwAYIBCwj___________8BmAEB&tfu=KgIIAw&hl=en", + "title": "Find Cheap Flights Worldwide & Book Your Ticket - Google Flights", + "steps": 6, + "elapsed_ms": 36105, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlights\n1\nSearch\nFlexible? Discover the best flight deals with AI\nDescribe your ideal trip, and let Google Flights find the best deals for you\nExplore deals with AI\nFind and book cheap flights worldwide and track prices\nFind cheap flights from Germany to anywhere\nFrankfurt am Main\nMunich\nBerlin\nDüsseldorf", + "answer": "The page does not show any flight options for the specified criteria. It only displays example flight deals from Frankfurt am Main to other cities." + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-06/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-06/decision_ticks.json new file mode 100644 index 0000000..a55e06e --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-06/decision_ticks.json @@ -0,0 +1,310 @@ +{ + "report": { + "elapsed_ms": 36468, + "decisions": 7, + "interactions": 6, + "waits": 0, + "median_decision_ms": 1441, + "jev_input_tokens": 0, + "median_probe_ms": 859, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 4, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 4415, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3151, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.906, + "elements": 25, + "elapsed_ms": 10128, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 1251, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 689, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 930, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.975, + "elements": 10, + "elapsed_ms": 12547, + "probabilities": { + "3": 0.954, + "5": 0.038, + "2": 0.006, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 669, + "text": null + }, + { + "tick": 3, + "probe_ms": 859, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1304, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.733, + "elements": 25, + "elapsed_ms": 15600, + "probabilities": { + "16": 0.984, + "18": 0.008, + "17": 0.006, + "15": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "17": "Departure", + "15": "Where from?" + }, + "screenshot_ms": 865, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1292, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1258, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.931, + "elements": 9, + "elapsed_ms": 19095, + "probabilities": { + "3": 0.763, + "5": 0.194, + "2": 0.019, + "7": 0.014, + "4": 0.006, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 771, + "text": null + }, + { + "tick": 5, + "probe_ms": 840, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1441, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.393, + "elements": 26, + "elapsed_ms": 23393, + "probabilities": { + "12": 0.696, + "18": 0.253, + "19": 0.027, + "16": 0.008, + "6": 0.003, + "20": 0.002 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 783, + "text": null + }, + { + "tick": 6, + "probe_ms": 3855, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 2050, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Departure", + "confidence": 0.511, + "elements": 26, + "elapsed_ms": 30708, + "probabilities": { + "18": 0.603, + "19": 0.391, + "15": 0.003, + "17": 0.003 + }, + "candidates": { + "18": "Departure", + "19": "Return", + "15": "Where from?", + "17": "Where to?" + }, + "screenshot_ms": 719, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 7, + "probe_ms": 765, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2136, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Return", + "confidence": 0.473, + "elements": 26, + "elapsed_ms": 34489, + "probabilities": { + "19": 0.988, + "18": 0.005, + "17": 0.004, + "15": 0.003 + }, + "candidates": { + "19": "Return", + "18": "Departure", + "17": "Where to?", + "15": "Where from?" + }, + "screenshot_ms": 723, + "value_source": "prefetch", + "value_ms": 0, + "value_error": "value model returned no usable text" + } + ], + "final": { + "status": "BLOCKED", + "reason": "no value for field 'Return': value model returned no usable text", + "url": "https://www.google.com/travel/flights?tfs=CBwQARocagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbBocagwIAxIIL20vMDRqcGxyDAgDEggvbS8wODk2NkABSAFwAYIBCwj___________8BmAEB&tfu=KgIIAw&hl=en", + "title": "Find Cheap Flights Worldwide & Book Your Ticket - Google Flights", + "steps": 6, + "elapsed_ms": 36105, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlights\n1\nSearch\nFlexible? Discover the best flight deals with AI\nDescribe your ideal trip, and let Google Flights find the best deals for you\nExplore deals with AI\nFind and book cheap flights worldwide and track prices\nFind cheap flights from Germany to anywhere\nFrankfurt am Main\nMunich\nBerlin\nDüsseldorf", + "answer": "The page does not show any flight options for the specified criteria. It only displays example flight deals from Frankfurt am Main to other cities." + }, + "usage": { + "decisions": 7, + "chat_calls": 10, + "chat_input_tokens": 3805, + "chat_output_tokens": 81, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.001344 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-08/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-08/answer.json new file mode 100644 index 0000000..a1a2896 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-08/answer.json @@ -0,0 +1,83 @@ +{ + "ok": true, + "final": "The page does not show any flight options for the specified criteria. It only displays example flight deals from Frankfurt am Main to other cities.", + "screenshot": null, + "error": null, + "elapsed_ms": 34645, + "usage": { + "decisions": 7, + "chat_calls": 10, + "chat_input_tokens": 3805, + "chat_output_tokens": 81, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.001344 + }, + "report": { + "elapsed_ms": 33942, + "decisions": 7, + "interactions": 6, + "waits": 0, + "median_decision_ms": 1430, + "jev_input_tokens": 0, + "median_probe_ms": 877, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 4, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + } + ] + }, + "status": "BLOCKED", + "terminal": { + "status": "BLOCKED", + "reason": "no value for field 'Return': value model returned no usable text", + "url": "https://www.google.com/travel/flights?tfs=CBwQARocagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbBocagwIAxIIL20vMDRqcGxyDAgDEggvbS8wODk2NkABSAFwAYIBCwj___________8BmAEB&tfu=KgIIAw&hl=en", + "title": "Find Cheap Flights Worldwide & Book Your Ticket - Google Flights", + "steps": 6, + "elapsed_ms": 33597, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlights\n1\nSearch\nFlexible? Discover the best flight deals with AI\nDescribe your ideal trip, and let Google Flights find the best deals for you\nExplore deals with AI\nFind and book cheap flights worldwide and track prices\nFind cheap flights from Germany to anywhere\nFrankfurt am Main\nMunich\nBerlin\nDüsseldorf", + "answer": "The page does not show any flight options for the specified criteria. It only displays example flight deals from Frankfurt am Main to other cities." + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-08/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-08/decision_ticks.json new file mode 100644 index 0000000..66b530d --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-03-08/decision_ticks.json @@ -0,0 +1,310 @@ +{ + "report": { + "elapsed_ms": 33942, + "decisions": 7, + "interactions": 6, + "waits": 0, + "median_decision_ms": 1430, + "jev_input_tokens": 0, + "median_probe_ms": 877, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 4, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 1640, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3139, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.906, + "elements": 25, + "elapsed_ms": 6897, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 693, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1312, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 915, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.975, + "elements": 10, + "elapsed_ms": 9949, + "probabilities": { + "3": 0.954, + "5": 0.038, + "2": 0.006, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 682, + "text": null + }, + { + "tick": 3, + "probe_ms": 857, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1318, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.733, + "elements": 25, + "elapsed_ms": 14306, + "probabilities": { + "16": 0.984, + "18": 0.008, + "17": 0.006, + "15": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "17": "Departure", + "15": "Where from?" + }, + "screenshot_ms": 772, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 791, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1275, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.931, + "elements": 9, + "elapsed_ms": 17280, + "probabilities": { + "3": 0.763, + "5": 0.194, + "2": 0.019, + "7": 0.014, + "4": 0.006, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 736, + "text": null + }, + { + "tick": 5, + "probe_ms": 877, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1430, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.393, + "elements": 26, + "elapsed_ms": 20476, + "probabilities": { + "12": 0.696, + "18": 0.253, + "19": 0.027, + "16": 0.008, + "6": 0.003, + "20": 0.002 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 851, + "text": null + }, + { + "tick": 6, + "probe_ms": 3811, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 2007, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Departure", + "confidence": 0.511, + "elements": 26, + "elapsed_ms": 28247, + "probabilities": { + "18": 0.603, + "19": 0.391, + "15": 0.003, + "17": 0.003 + }, + "candidates": { + "18": "Departure", + "19": "Return", + "15": "Where from?", + "17": "Where to?" + }, + "screenshot_ms": 750, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 7, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2095, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Return", + "confidence": 0.473, + "elements": 26, + "elapsed_ms": 32030, + "probabilities": { + "19": 0.988, + "18": 0.005, + "17": 0.004, + "15": 0.003 + }, + "candidates": { + "19": "Return", + "18": "Departure", + "17": "Where to?", + "15": "Where from?" + }, + "screenshot_ms": 752, + "value_source": "prefetch", + "value_ms": 0, + "value_error": "value model returned no usable text" + } + ], + "final": { + "status": "BLOCKED", + "reason": "no value for field 'Return': value model returned no usable text", + "url": "https://www.google.com/travel/flights?tfs=CBwQARocagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbBocagwIAxIIL20vMDRqcGxyDAgDEggvbS8wODk2NkABSAFwAYIBCwj___________8BmAEB&tfu=KgIIAw&hl=en", + "title": "Find Cheap Flights Worldwide & Book Your Ticket - Google Flights", + "steps": 6, + "elapsed_ms": 33597, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlights\n1\nSearch\nFlexible? Discover the best flight deals with AI\nDescribe your ideal trip, and let Google Flights find the best deals for you\nExplore deals with AI\nFind and book cheap flights worldwide and track prices\nFind cheap flights from Germany to anywhere\nFrankfurt am Main\nMunich\nBerlin\nDüsseldorf", + "answer": "The page does not show any flight options for the specified criteria. It only displays example flight deals from Frankfurt am Main to other cities." + }, + "usage": { + "decisions": 7, + "chat_calls": 10, + "chat_input_tokens": 3805, + "chat_output_tokens": 81, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.001344 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-04-13/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-04-13/answer.json new file mode 100644 index 0000000..e661be0 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-04-13/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARoaEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZAAUgBcAGCAQsI____________AZgBAg&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":110.573,\"remaining_s\":129.427,\"invocation_remaining_s\":130.579},\"summary\":\"The form phase budget has been exhausted. I have successfully entered the origin, destination, and date for a one-way flight. However, I was unable to click the \\\"Done\\\" button to confirm the selections.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 111552, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 22343, + "chat_output_tokens": 90, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006928 + }, + "report": { + "elapsed_ms": 110914, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1104, + "jev_input_tokens": 0, + "median_probe_ms": 767, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-04-13/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-04-13/decision_ticks.json new file mode 100644 index 0000000..84f46b0 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-04-13/decision_ticks.json @@ -0,0 +1,1018 @@ +{ + "report": { + "elapsed_ms": 110914, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1104, + "jev_input_tokens": 0, + "median_probe_ms": 767, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 4959, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3125, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.906, + "elements": 25, + "elapsed_ms": 10097, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 685, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1303, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 719, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.975, + "elements": 10, + "elapsed_ms": 12933, + "probabilities": { + "3": 0.954, + "5": 0.038, + "2": 0.006, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 683, + "text": null + }, + { + "tick": 3, + "probe_ms": 751, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1052, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.733, + "elements": 25, + "elapsed_ms": 16807, + "probabilities": { + "16": 0.984, + "18": 0.008, + "17": 0.006, + "15": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "17": "Departure", + "15": "Where from?" + }, + "screenshot_ms": 757, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1298, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 742, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.931, + "elements": 9, + "elapsed_ms": 19804, + "probabilities": { + "3": 0.763, + "5": 0.194, + "2": 0.019, + "7": 0.014, + "4": 0.006, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 774, + "text": null + }, + { + "tick": 5, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1102, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.393, + "elements": 26, + "elapsed_ms": 23668, + "probabilities": { + "12": 0.696, + "18": 0.253, + "19": 0.027, + "16": 0.008, + "6": 0.003, + "20": 0.002 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 758, + "text": null + }, + { + "tick": 6, + "probe_ms": 736, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 486, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.998, + "elements": 3, + "elapsed_ms": 26868, + "probabilities": { + "2": 0.993, + "1": 0.007, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 753, + "text": null + }, + { + "tick": 7, + "probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1068, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.56, + "elements": 25, + "elapsed_ms": 30667, + "probabilities": { + "18": 0.999, + "19": 0.0, + "6": 0.0, + "9": 0.0, + "12": 0.0, + "14": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "6": "Flights", + "9": "Change appearance", + "12": "Change ticket type. One way", + "14": "Change seating class. Economy" + }, + "screenshot_ms": 747, + "text": null + }, + { + "tick": 8, + "probe_ms": 1676, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1793, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.789, + "elements": 50, + "elapsed_ms": 35791, + "probabilities": { + "SCROLL_DOWN": 0.825, + "CLICK": 0.143, + "WAIT": 0.028, + "BLOCKED": 0.005, + "TYPE_TEXT": 0.0, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 812, + "text": null + }, + { + "tick": 9, + "probe_ms": 1650, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1918, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.528, + "elements": 51, + "elapsed_ms": 40413, + "probabilities": { + "SCROLL_DOWN": 0.595, + "CLICK": 0.348, + "WAIT": 0.041, + "BLOCKED": 0.01, + "TYPE_TEXT": 0.004, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 813, + "text": null + }, + { + "tick": 10, + "probe_ms": 1145, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1911, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.403, + "elements": 51, + "elapsed_ms": 44514, + "probabilities": { + "SCROLL_DOWN": 0.488, + "CLICK": 0.439, + "WAIT": 0.05, + "BLOCKED": 0.016, + "TYPE_TEXT": 0.007, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 795, + "text": null + }, + { + "tick": 11, + "probe_ms": 1574, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2934, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.979, + "elements": 110, + "elapsed_ms": 50075, + "probabilities": { + "35": 0.864, + "36": 0.019, + "1": 0.015, + "65": 0.009, + "8": 0.009, + "37": 0.008 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "36": "Monday, November 2, 2026", + "1": "Departure", + "65": "Tuesday, December 1, 2026", + "8": "Monday, October 5, 2026 , 113 euros", + "37": "Tuesday, November 3, 2026" + }, + "screenshot_ms": 815, + "text": null + }, + { + "tick": 12, + "probe_ms": 1529, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2982, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.602, + "elements": 110, + "elapsed_ms": 56173, + "probabilities": { + "110": 0.833, + "1": 0.041, + "35": 0.039, + "65": 0.011, + "95": 0.007, + "70": 0.005 + }, + "candidates": { + "110": "Done. Search for one-way flights, departing on November 1, 2026", + "1": "Departure", + "35": "Sunday, November 1, 2026, departure date. , 72 euros", + "65": "Tuesday, December 1, 2026 , 22 euros, Cheapest price", + "95": "Thursday, December 31, 2026 , 36 euros", + "70": "Sunday, December 6, 2026 , 28 euros" + }, + "screenshot_ms": 803, + "text": null + }, + { + "tick": 13, + "probe_ms": 742, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1105, + "input_tokens": 0, + "output_tokens": 0, + "operation": "PRESS_ENTER", + "target": "Departure", + "confidence": 0.174, + "elements": 25, + "elapsed_ms": 60002, + "probabilities": { + "18": 0.894, + "17": 0.068, + "15": 0.039 + }, + "candidates": { + "18": "Departure", + "17": "Where to?", + "15": "Where from?" + }, + "screenshot_ms": 748, + "text": null + }, + { + "tick": 14, + "probe_ms": 4300, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1104, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.264, + "elements": 25, + "elapsed_ms": 66802, + "probabilities": { + "SCROLL_DOWN": 0.37, + "PRESS_ENTER": 0.231, + "CLICK": 0.197, + "DONE": 0.105, + "WAIT": 0.095, + "TYPE_TEXT": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 742, + "text": null + }, + { + "tick": 15, + "probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1034, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.653, + "elements": 21, + "elapsed_ms": 69565, + "probabilities": { + "13": 0.546, + "1": 0.262, + "6": 0.057, + "14": 0.027, + "21": 0.026, + "16": 0.025 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "16": "Düsseldorf" + }, + "screenshot_ms": 811, + "text": null + }, + { + "tick": 16, + "probe_ms": 3822, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1033, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 75948, + "probabilities": { + "13": 0.62, + "1": 0.186, + "6": 0.052, + "14": 0.029, + "16": 0.028, + "21": 0.023 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 810, + "text": null + }, + { + "tick": 17, + "probe_ms": 3808, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1018, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.526, + "elements": 21, + "elapsed_ms": 82270, + "probabilities": { + "1": 0.537, + "6": 0.134, + "5": 0.064, + "21": 0.061, + "14": 0.061, + "16": 0.049 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "16": "Düsseldorf" + }, + "screenshot_ms": 809, + "text": null + }, + { + "tick": 18, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1360, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.406, + "elements": 33, + "elapsed_ms": 85858, + "probabilities": { + "SCROLL_UP": 0.505, + "CLICK": 0.441, + "WAIT": 0.023, + "BLOCKED": 0.022, + "SCROLL_DOWN": 0.009, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 790, + "text": null + }, + { + "tick": 19, + "probe_ms": 767, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1333, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.195, + "elements": 37, + "elapsed_ms": 88869, + "probabilities": { + "SCROLL_DOWN": 0.31, + "PRESS_ENTER": 0.263, + "CLICK": 0.175, + "WAIT": 0.162, + "DONE": 0.085, + "TYPE_TEXT": 0.006 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 758, + "text": null + }, + { + "tick": 20, + "probe_ms": 773, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1359, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.541, + "elements": 33, + "elapsed_ms": 91953, + "probabilities": { + "13": 0.174, + "1": 0.155, + "6": 0.133, + "20": 0.094, + "5": 0.076, + "12": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 787, + "text": null + }, + { + "tick": 21, + "probe_ms": 743, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1017, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.626, + "elements": 21, + "elapsed_ms": 95749, + "probabilities": { + "1": 0.491, + "6": 0.125, + "14": 0.089, + "16": 0.082, + "5": 0.046, + "21": 0.042 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 830, + "text": null + }, + { + "tick": 22, + "probe_ms": 748, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1359, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.671, + "elements": 33, + "elapsed_ms": 99352, + "probabilities": { + "13": 0.145, + "1": 0.143, + "6": 0.135, + "20": 0.078, + "5": 0.077, + "26": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "26": "Munich" + }, + "screenshot_ms": 807, + "text": null + }, + { + "tick": 23, + "probe_ms": 742, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1015, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 102586, + "probabilities": { + "1": 0.401, + "6": 0.185, + "14": 0.081, + "16": 0.076, + "21": 0.058, + "5": 0.048 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 793, + "text": null + }, + { + "tick": 24, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1352, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights", + "confidence": 0.624, + "elements": 33, + "elapsed_ms": 106175, + "probabilities": { + "6": 0.186, + "13": 0.181, + "20": 0.154, + "1": 0.103, + "5": 0.087, + "12": 0.07 + }, + "candidates": { + "6": "Flights", + "13": "Selected Tab: Flights", + "20": "Change location", + "1": "Main menu", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 799, + "text": null + }, + { + "tick": 25, + "probe_ms": 743, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1013, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.758, + "elements": 21, + "elapsed_ms": 109403, + "probabilities": { + "1": 0.43, + "6": 0.201, + "14": 0.076, + "16": 0.07, + "21": 0.046, + "5": 0.043 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 789, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 22343, + "chat_output_tokens": 90, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006928 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-06-35/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-06-35/answer.json new file mode 100644 index 0000000..960f542 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-06-35/answer.json @@ -0,0 +1,131 @@ +{ + "ok": true, + "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €72 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", + "screenshot": null, + "error": null, + "elapsed_ms": 73612, + "usage": { + "decisions": 15, + "chat_calls": 12, + "chat_input_tokens": 6017, + "chat_output_tokens": 124, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.002115 + }, + "report": { + "elapsed_ms": 72951, + "decisions": 15, + "interactions": 14, + "waits": 0, + "median_decision_ms": 1742, + "jev_input_tokens": 0, + "median_probe_ms": 1157, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flight details. Leaves Zurich Airport at 9:30 PM on Sunday, November 1 and arrives at London Gatwick Airport at 10:10 PM on Sunday, November 1.", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "status": "DONE", + "terminal": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 14, + "elapsed_ms": 72602, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€72\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\nDeparture\nSun, Nov 1\nSelect flight\n€77\n9:30 PM\nZurich Airport\n(\nZRH\n)\nTravel time: 1 hr 40 min\n10:10 PM\nLondon Gatwick Airport\n(\nLGW\n)\neasyJet\nEconomy\nAirbus A319\nU2\n8474\nOften delayed by 30+ min\nEmissions estimate:\n86\nkg CO2e\nContrail warming potential: Medium\n9:10 PM\n–\n9:55 PM\n1 hr 45 min\nNonstop\n€79", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €72 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-06-35/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-06-35/decision_ticks.json new file mode 100644 index 0000000..496aae5 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-06-35/decision_ticks.json @@ -0,0 +1,626 @@ +{ + "report": { + "elapsed_ms": 72951, + "decisions": 15, + "interactions": 14, + "waits": 0, + "median_decision_ms": 1742, + "jev_input_tokens": 0, + "median_probe_ms": 1157, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flight details. Leaves Zurich Airport at 9:30 PM on Sunday, November 1 and arrives at London Gatwick Airport at 10:10 PM on Sunday, November 1.", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5102, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3045, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.888, + "elements": 20, + "elapsed_ms": 10064, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 680, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1286, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 715, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.978, + "elements": 10, + "elapsed_ms": 12856, + "probabilities": { + "3": 0.947, + "5": 0.045, + "2": 0.006, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 671, + "text": null + }, + { + "tick": 3, + "probe_ms": 735, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 937, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.566, + "elements": 20, + "elapsed_ms": 16502, + "probabilities": { + "16": 0.992, + "18": 0.005, + "15": 0.002, + "17": 0.001 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "15": "Where from?", + "17": "Departure" + }, + "screenshot_ms": 743, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1288, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 738, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.947, + "elements": 9, + "elapsed_ms": 19450, + "probabilities": { + "3": 0.643, + "5": 0.308, + "7": 0.02, + "2": 0.02, + "4": 0.005, + "6": 0.002 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "7": "London Gatwick Airport (LGW)", + "2": "Destination, Select multiple airports", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 758, + "text": null + }, + { + "tick": 5, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 993, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.466, + "elements": 21, + "elapsed_ms": 23172, + "probabilities": { + "12": 0.905, + "18": 0.078, + "19": 0.01, + "16": 0.003, + "6": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 743, + "text": null + }, + { + "tick": 6, + "probe_ms": 731, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 491, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.998, + "elements": 3, + "elapsed_ms": 26328, + "probabilities": { + "2": 0.985, + "1": 0.014, + "3": 0.001 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 742, + "text": null + }, + { + "tick": 7, + "probe_ms": 736, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 943, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.456, + "elements": 20, + "elapsed_ms": 29954, + "probabilities": { + "18": 0.999, + "19": 0.001, + "6": 0.0, + "12": 0.0, + "14": 0.0, + "15": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "6": "Flights", + "12": "Change ticket type. One way", + "14": "Change seating class. Economy", + "15": "Where from?" + }, + "screenshot_ms": 743, + "text": null + }, + { + "tick": 8, + "probe_ms": 1666, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1799, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.787, + "elements": 50, + "elapsed_ms": 35069, + "probabilities": { + "SCROLL_DOWN": 0.822, + "CLICK": 0.146, + "WAIT": 0.026, + "BLOCKED": 0.005, + "TYPE_TEXT": 0.0, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 798, + "text": null + }, + { + "tick": 9, + "probe_ms": 1121, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1923, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.466, + "elements": 51, + "elapsed_ms": 39124, + "probabilities": { + "SCROLL_DOWN": 0.542, + "CLICK": 0.413, + "WAIT": 0.03, + "BLOCKED": 0.011, + "TYPE_TEXT": 0.004, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 781, + "text": null + }, + { + "tick": 10, + "probe_ms": 1157, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1926, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.44, + "elements": 51, + "elapsed_ms": 43276, + "probabilities": { + "SCROLL_DOWN": 0.52, + "CLICK": 0.409, + "WAIT": 0.051, + "BLOCKED": 0.015, + "TYPE_TEXT": 0.005, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 809, + "text": null + }, + { + "tick": 11, + "probe_ms": 1578, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2954, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.976, + "elements": 110, + "elapsed_ms": 48827, + "probabilities": { + "35": 0.643, + "1": 0.04, + "65": 0.028, + "16": 0.028, + "8": 0.019, + "108": 0.019 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "1": "Departure", + "65": "Tuesday, December 1, 2026", + "16": "Tuesday, October 13, 2026 , 55 euros", + "8": "Monday, October 5, 2026 , 113 euros", + "108": "Previous" + }, + "screenshot_ms": 780, + "text": null + }, + { + "tick": 12, + "probe_ms": 2037, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2995, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.711, + "elements": 110, + "elapsed_ms": 55458, + "probabilities": { + "110": 0.815, + "1": 0.046, + "35": 0.042, + "65": 0.012, + "95": 0.008, + "3": 0.005 + }, + "candidates": { + "110": "Done. Search for one-way flights, departing on November 1, 2026", + "1": "Departure", + "35": "Sunday, November 1, 2026, departure date. , 72 euros", + "65": "Tuesday, December 1, 2026 , 22 euros, Cheapest price", + "95": "Thursday, December 31, 2026 , 36 euros", + "3": "Reset" + }, + "screenshot_ms": 805, + "text": null + }, + { + "tick": 13, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 992, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Search", + "confidence": 0.331, + "elements": 20, + "elapsed_ms": 58670, + "probabilities": { + "19": 0.997, + "18": 0.001, + "20": 0.001, + "2": 0.0, + "11": 0.0, + "6": 0.0 + }, + "candidates": { + "19": "Search", + "18": "Departure", + "20": "Explore deals with AI", + "2": "Google", + "11": "Sign in", + "6": "Flights" + }, + "screenshot_ms": 754, + "text": null + }, + { + "tick": 14, + "probe_ms": 1465, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1842, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flight details. Leaves Zurich Airport at 9:30 PM on Sunday, November 1 and arrives at London Gatwick Airport at 10:10 PM on Sunday, November 1.", + "confidence": 0.121, + "elements": 48, + "elapsed_ms": 65737, + "probabilities": { + "39": 0.46, + "35": 0.084, + "18": 0.064, + "30": 0.053, + "42": 0.045, + "48": 0.043 + }, + "candidates": { + "39": "Flight details. Leaves Zurich Airport at 9:30 PM on Sunday, November 1 and arrives at London Gatwick Airport at 10:10 PM on Sunday, November 1.", + "35": "Sorted by top flights, Change sort order.", + "18": "Departure", + "30": "Cheapest from 72 euros Learn more about ranking Looking for the lowest fares? These additional budget picks help you save but may involve trade-offs, such as flying back to a different airport in the ", + "42": "Flight details. Leaves Zurich Airport at 9:10 PM on Sunday, November 1 and arrives at Heathrow Airport at 9:55 PM on Sunday, November 1.", + "48": "See more train options" + }, + "screenshot_ms": 1523, + "text": null + }, + { + "tick": 15, + "probe_ms": 956, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1742, + "input_tokens": 0, + "output_tokens": 0, + "operation": "DONE", + "target": null, + "confidence": 0.16, + "elements": 41, + "elapsed_ms": 70956, + "probabilities": { + "DONE": 0.28, + "CLICK": 0.205, + "TYPE_TEXT": 0.167, + "PRESS_ENTER": 0.159, + "SCROLL_DOWN": 0.104, + "WAIT": 0.08 + }, + "candidates": { + "DONE": "The page shows every requirement of the task met.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "SCROLL_DOWN": "Move the viewport down the page.", + "WAIT": "Give the page time to finish updating." + }, + "screenshot_ms": 991 + } + ], + "final": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 14, + "elapsed_ms": 72602, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€72\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\nDeparture\nSun, Nov 1\nSelect flight\n€77\n9:30 PM\nZurich Airport\n(\nZRH\n)\nTravel time: 1 hr 40 min\n10:10 PM\nLondon Gatwick Airport\n(\nLGW\n)\neasyJet\nEconomy\nAirbus A319\nU2\n8474\nOften delayed by 30+ min\nEmissions estimate:\n86\nkg CO2e\nContrail warming potential: Medium\n9:10 PM\n–\n9:55 PM\n1 hr 45 min\nNonstop\n€79", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €72 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "usage": { + "decisions": 15, + "chat_calls": 12, + "chat_input_tokens": 6017, + "chat_output_tokens": 124, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.002115 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-08-19/answer.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-08-19/answer.json new file mode 100644 index 0000000..29ed135 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-08-19/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARoaEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZAAUgBcAGCAQsI____________AZgBAg&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":111.989,\"remaining_s\":128.011,\"invocation_remaining_s\":129.52},\"summary\":\"The form phase budget has been exhausted. I have successfully entered the origin, destination, and date for a one-way flight. However, I was unable to specify the number of adults and economy class.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 112984, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 23462, + "chat_output_tokens": 89, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.007261 + }, + "report": { + "elapsed_ms": 112334, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1107, + "jev_input_tokens": 0, + "median_probe_ms": 775, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-08-19/decision_ticks.json b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-08-19/decision_ticks.json new file mode 100644 index 0000000..c1c927a --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-29/2026-09-29__15-08-19/decision_ticks.json @@ -0,0 +1,1018 @@ +{ + "report": { + "elapsed_ms": 112334, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1107, + "jev_input_tokens": 0, + "median_probe_ms": 775, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "submit", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Selected Tab: Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5128, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3143, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.905, + "elements": 25, + "elapsed_ms": 10256, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 684, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1309, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 723, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.968, + "elements": 10, + "elapsed_ms": 13099, + "probabilities": { + "3": 0.963, + "5": 0.028, + "2": 0.007, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 687, + "text": null + }, + { + "tick": 3, + "probe_ms": 756, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1063, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.733, + "elements": 25, + "elapsed_ms": 16981, + "probabilities": { + "16": 0.984, + "18": 0.008, + "17": 0.006, + "15": 0.002 + }, + "candidates": { + "16": "Where to?", + "18": "Return", + "17": "Departure", + "15": "Where from?" + }, + "screenshot_ms": 752, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1269, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 751, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.931, + "elements": 9, + "elapsed_ms": 19953, + "probabilities": { + "3": 0.763, + "5": 0.194, + "2": 0.019, + "7": 0.014, + "4": 0.006, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 765, + "text": null + }, + { + "tick": 5, + "probe_ms": 762, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1107, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.393, + "elements": 26, + "elapsed_ms": 23829, + "probabilities": { + "12": 0.696, + "18": 0.253, + "19": 0.027, + "16": 0.008, + "6": 0.003, + "20": 0.002 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "6": "Flights", + "20": "Search" + }, + "screenshot_ms": 757, + "text": null + }, + { + "tick": 6, + "probe_ms": 751, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 495, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.998, + "elements": 3, + "elapsed_ms": 27072, + "probabilities": { + "2": 0.993, + "1": 0.007, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 763, + "text": null + }, + { + "tick": 7, + "probe_ms": 750, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1080, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.56, + "elements": 25, + "elapsed_ms": 30901, + "probabilities": { + "18": 0.999, + "19": 0.0, + "6": 0.0, + "9": 0.0, + "12": 0.0, + "14": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "6": "Flights", + "9": "Change appearance", + "12": "Change ticket type. One way", + "14": "Change seating class. Economy" + }, + "screenshot_ms": 772, + "text": null + }, + { + "tick": 8, + "probe_ms": 1676, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1800, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.789, + "elements": 50, + "elapsed_ms": 36029, + "probabilities": { + "SCROLL_DOWN": 0.825, + "CLICK": 0.143, + "WAIT": 0.028, + "BLOCKED": 0.005, + "TYPE_TEXT": 0.0, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 809, + "text": null + }, + { + "tick": 9, + "probe_ms": 1646, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1930, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.528, + "elements": 51, + "elapsed_ms": 40663, + "probabilities": { + "SCROLL_DOWN": 0.595, + "CLICK": 0.348, + "WAIT": 0.041, + "BLOCKED": 0.01, + "TYPE_TEXT": 0.004, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 818, + "text": null + }, + { + "tick": 10, + "probe_ms": 1125, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1928, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.403, + "elements": 51, + "elapsed_ms": 44751, + "probabilities": { + "SCROLL_DOWN": 0.488, + "CLICK": 0.439, + "WAIT": 0.05, + "BLOCKED": 0.016, + "TYPE_TEXT": 0.007, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + }, + "screenshot_ms": 802, + "text": null + }, + { + "tick": 11, + "probe_ms": 1580, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2957, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.972, + "elements": 110, + "elapsed_ms": 50352, + "probabilities": { + "35": 0.677, + "1": 0.051, + "65": 0.031, + "108": 0.028, + "16": 0.023, + "36": 0.014 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "1": "Departure", + "65": "Tuesday, December 1, 2026", + "108": "Previous", + "16": "Tuesday, October 13, 2026 , 55 euros", + "36": "Monday, November 2, 2026" + }, + "screenshot_ms": 820, + "text": null + }, + { + "tick": 12, + "probe_ms": 1528, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2999, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.602, + "elements": 110, + "elapsed_ms": 56493, + "probabilities": { + "110": 0.833, + "1": 0.041, + "35": 0.039, + "65": 0.011, + "95": 0.007, + "70": 0.005 + }, + "candidates": { + "110": "Done. Search for one-way flights, departing on November 1, 2026", + "1": "Departure", + "35": "Sunday, November 1, 2026, departure date. , 72 euros", + "65": "Tuesday, December 1, 2026 , 22 euros, Cheapest price", + "95": "Thursday, December 31, 2026 , 36 euros", + "70": "Sunday, December 6, 2026 , 28 euros" + }, + "screenshot_ms": 818, + "text": null + }, + { + "tick": 13, + "probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1115, + "input_tokens": 0, + "output_tokens": 0, + "operation": "PRESS_ENTER", + "target": "Departure", + "confidence": 0.174, + "elements": 25, + "elapsed_ms": 60372, + "probabilities": { + "18": 0.894, + "17": 0.068, + "15": 0.039 + }, + "candidates": { + "18": "Departure", + "17": "Where to?", + "15": "Where from?" + }, + "screenshot_ms": 792, + "text": null + }, + { + "tick": 14, + "probe_ms": 4825, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1105, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.264, + "elements": 25, + "elapsed_ms": 67719, + "probabilities": { + "SCROLL_DOWN": 0.37, + "PRESS_ENTER": 0.231, + "CLICK": 0.197, + "DONE": 0.105, + "WAIT": 0.095, + "TYPE_TEXT": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 759, + "text": null + }, + { + "tick": 15, + "probe_ms": 765, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1040, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.653, + "elements": 21, + "elapsed_ms": 70488, + "probabilities": { + "13": 0.546, + "1": 0.262, + "6": 0.057, + "14": 0.027, + "21": 0.026, + "16": 0.025 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "16": "Düsseldorf" + }, + "screenshot_ms": 788, + "text": null + }, + { + "tick": 16, + "probe_ms": 3805, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1037, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 76826, + "probabilities": { + "13": 0.62, + "1": 0.186, + "6": 0.052, + "14": 0.029, + "16": 0.028, + "21": 0.023 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 808, + "text": null + }, + { + "tick": 17, + "probe_ms": 3834, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1021, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.526, + "elements": 21, + "elapsed_ms": 83176, + "probabilities": { + "1": 0.537, + "6": 0.134, + "5": 0.064, + "21": 0.061, + "14": 0.061, + "16": 0.049 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "16": "Düsseldorf" + }, + "screenshot_ms": 817, + "text": null + }, + { + "tick": 18, + "probe_ms": 759, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1370, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.406, + "elements": 33, + "elapsed_ms": 86787, + "probabilities": { + "SCROLL_UP": 0.505, + "CLICK": 0.441, + "WAIT": 0.023, + "BLOCKED": 0.022, + "SCROLL_DOWN": 0.009, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 796, + "text": null + }, + { + "tick": 19, + "probe_ms": 784, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1346, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.195, + "elements": 37, + "elapsed_ms": 89841, + "probabilities": { + "SCROLL_DOWN": 0.31, + "PRESS_ENTER": 0.263, + "CLICK": 0.175, + "WAIT": 0.162, + "DONE": 0.085, + "TYPE_TEXT": 0.006 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 770, + "text": null + }, + { + "tick": 20, + "probe_ms": 775, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1366, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.541, + "elements": 33, + "elapsed_ms": 93443, + "probabilities": { + "13": 0.174, + "1": 0.155, + "6": 0.133, + "20": 0.094, + "5": 0.076, + "12": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 1302, + "text": null + }, + { + "tick": 21, + "probe_ms": 742, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1022, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.626, + "elements": 21, + "elapsed_ms": 96694, + "probabilities": { + "1": 0.491, + "6": 0.125, + "14": 0.089, + "16": 0.082, + "5": 0.046, + "21": 0.042 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "5": "Explore", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals" + }, + "screenshot_ms": 791, + "text": null + }, + { + "tick": 22, + "probe_ms": 750, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1368, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Selected Tab: Flights", + "confidence": 0.671, + "elements": 33, + "elapsed_ms": 100278, + "probabilities": { + "13": 0.145, + "1": 0.143, + "6": 0.135, + "20": 0.078, + "5": 0.077, + "26": 0.069 + }, + "candidates": { + "13": "Selected Tab: Flights", + "1": "Main menu", + "6": "Flights", + "20": "Change location", + "5": "Explore", + "26": "Munich" + }, + "screenshot_ms": 784, + "text": null + }, + { + "tick": 23, + "probe_ms": 762, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1019, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.59, + "elements": 21, + "elapsed_ms": 103570, + "probabilities": { + "1": 0.401, + "6": 0.185, + "14": 0.081, + "16": 0.076, + "21": 0.058, + "5": 0.048 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 807, + "text": null + }, + { + "tick": 24, + "probe_ms": 757, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1357, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights", + "confidence": 0.624, + "elements": 33, + "elapsed_ms": 107191, + "probabilities": { + "6": 0.186, + "13": 0.181, + "20": 0.154, + "1": 0.103, + "5": 0.087, + "12": 0.07 + }, + "candidates": { + "6": "Flights", + "13": "Selected Tab: Flights", + "20": "Change location", + "1": "Main menu", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 806, + "text": null + }, + { + "tick": 25, + "probe_ms": 765, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1020, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.758, + "elements": 21, + "elapsed_ms": 110464, + "probabilities": { + "1": 0.43, + "6": 0.201, + "14": 0.076, + "16": 0.07, + "21": 0.046, + "5": 0.043 + }, + "candidates": { + "1": "Main menu", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "5": "Explore" + }, + "screenshot_ms": 790, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 23462, + "chat_output_tokens": 89, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.007261 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-37-51/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-37-51/answer.json new file mode 100644 index 0000000..ece3d7b --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-37-51/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":113.516,\"remaining_s\":126.484,\"invocation_remaining_s\":128.1},\"summary\":\"The form phase budget has been exhausted. I cannot perform any more actions on the current page. I have successfully navigated to Google Flights, entered Zurich as the origin, London as the destination, selected one-way, and set the date to November 1, 2026. However, I was unable to confirm the number of adults or the economy class.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 114531, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 27121, + "chat_output_tokens": 121, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.008439 + }, + "report": { + "elapsed_ms": 113866, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1053, + "jev_input_tokens": 0, + "median_probe_ms": 776, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-37-51/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-37-51/decision_ticks.json new file mode 100644 index 0000000..2c591d1 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-37-51/decision_ticks.json @@ -0,0 +1,1016 @@ +{ + "report": { + "elapsed_ms": 113866, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1053, + "jev_input_tokens": 0, + "median_probe_ms": 776, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5602, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3115, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.901, + "elements": 25, + "elapsed_ms": 10708, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 689, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1296, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 722, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.982, + "elements": 10, + "elapsed_ms": 13540, + "probabilities": { + "3": 0.969, + "5": 0.023, + "2": 0.004, + "4": 0.002, + "6": 0.001, + "9": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 686, + "text": null + }, + { + "tick": 3, + "probe_ms": 756, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1053, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.753, + "elements": 25, + "elapsed_ms": 17364, + "probabilities": { + "16": 0.994, + "17": 0.003, + "18": 0.002, + "15": 0.001 + }, + "candidates": { + "16": "Where to?", + "17": "Departure", + "18": "Return", + "15": "Where from?" + }, + "screenshot_ms": 783, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1275, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 745, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.93, + "elements": 9, + "elapsed_ms": 20338, + "probabilities": { + "3": 0.787, + "5": 0.163, + "2": 0.027, + "7": 0.012, + "4": 0.009, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 775, + "text": null + }, + { + "tick": 5, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1101, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.594, + "elements": 26, + "elapsed_ms": 24185, + "probabilities": { + "12": 0.778, + "18": 0.181, + "19": 0.024, + "16": 0.007, + "13": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "13": "1 passenger", + "20": "Search" + }, + "screenshot_ms": 751, + "text": null + }, + { + "tick": 6, + "probe_ms": 741, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 496, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 27389, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 767, + "text": null + }, + { + "tick": 7, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1072, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.7, + "elements": 25, + "elapsed_ms": 31187, + "probabilities": { + "18": 0.999, + "19": 0.0, + "14": 0.0, + "9": 0.0, + "12": 0.0, + "13": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "14": "Change seating class. Economy", + "9": "Change appearance", + "12": "Change ticket type. One way", + "13": "1 passenger" + }, + "screenshot_ms": 749, + "text": null + }, + { + "tick": 8, + "probe_ms": 1658, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1772, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.545, + "elements": 49, + "elapsed_ms": 36292, + "probabilities": { + "SCROLL_DOWN": 0.621, + "CLICK": 0.294, + "WAIT": 0.077, + "BLOCKED": 0.007, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 813, + "text": null + }, + { + "tick": 9, + "probe_ms": 1655, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1902, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.421, + "elements": 50, + "elapsed_ms": 40943, + "probabilities": { + "49": 0.537, + "36": 0.126, + "35": 0.108, + "38": 0.043, + "39": 0.036, + "42": 0.025 + }, + "candidates": { + "49": "Next", + "36": "Monday, November 2, 2026", + "35": "Saturday, October 31, 2026 , 84 euros", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 839, + "text": null + }, + { + "tick": 10, + "probe_ms": 1868, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2638, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.991, + "elements": 82, + "elapsed_ms": 47007, + "probabilities": { + "36": 0.942, + "66": 0.013, + "81": 0.007, + "37": 0.006, + "40": 0.003, + "67": 0.003 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "37": "Monday, November 2, 2026", + "40": "Thursday, November 5, 2026", + "67": "Wednesday, December 2, 2026" + }, + "screenshot_ms": 798, + "text": null + }, + { + "tick": 11, + "probe_ms": 1320, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2809, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.45, + "elements": 82, + "elapsed_ms": 52737, + "probabilities": { + "82": 0.975, + "36": 0.021, + "1": 0.001, + "66": 0.0, + "62": 0.0, + "81": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "66": "Tuesday, December 1, 2026", + "62": "Friday, November 27, 2026 , 27 euros, Cheapest price", + "81": "Next" + }, + "screenshot_ms": 810, + "text": null + }, + { + "tick": 12, + "probe_ms": 746, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1105, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.707, + "elements": 25, + "elapsed_ms": 56067, + "probabilities": { + "SCROLL_DOWN": 0.749, + "PRESS_ENTER": 0.134, + "CLICK": 0.074, + "WAIT": 0.021, + "DONE": 0.021, + "BLOCKED": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 763, + "text": null + }, + { + "tick": 13, + "probe_ms": 773, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1036, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.38, + "elements": 21, + "elapsed_ms": 58831, + "probabilities": { + "13": 0.82, + "21": 0.05, + "1": 0.05, + "6": 0.021, + "19": 0.014, + "16": 0.008 + }, + "candidates": { + "13": "Frankfurt am Main", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "1": "Main menu", + "6": "Flights", + "19": "Find flights from Frankfurt am Main (FRA) to London (LGW) from €134. Operated by Condor. Oct 29 to Nov 4. Nonstop", + "16": "Düsseldorf" + }, + "screenshot_ms": 795, + "text": null + }, + { + "tick": 14, + "probe_ms": 4324, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1037, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.42, + "elements": 21, + "elapsed_ms": 66186, + "probabilities": { + "13": 0.729, + "1": 0.113, + "21": 0.043, + "6": 0.031, + "16": 0.016, + "2": 0.013 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 1298, + "text": null + }, + { + "tick": 15, + "probe_ms": 3821, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1018, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.611, + "elements": 21, + "elapsed_ms": 72516, + "probabilities": { + "1": 0.363, + "21": 0.176, + "14": 0.102, + "6": 0.099, + "16": 0.078, + "2": 0.036 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 811, + "text": null + }, + { + "tick": 16, + "probe_ms": 748, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1359, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.717, + "elements": 33, + "elapsed_ms": 76113, + "probabilities": { + "1": 0.292, + "26": 0.229, + "33": 0.054, + "6": 0.045, + "20": 0.041, + "12": 0.038 + }, + "candidates": { + "1": "Main menu", + "26": "Munich", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "20": "Change location", + "12": "Explore" + }, + "screenshot_ms": 793, + "text": null + }, + { + "tick": 17, + "probe_ms": 743, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1017, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.546, + "elements": 21, + "elapsed_ms": 79357, + "probabilities": { + "1": 0.273, + "21": 0.262, + "6": 0.145, + "14": 0.111, + "16": 0.058, + "2": 0.03 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 794, + "text": null + }, + { + "tick": 18, + "probe_ms": 762, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1359, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.697, + "elements": 33, + "elapsed_ms": 82951, + "probabilities": { + "26": 0.189, + "1": 0.157, + "33": 0.097, + "6": 0.07, + "5": 0.062, + "12": 0.061 + }, + "candidates": { + "26": "Munich", + "1": "Main menu", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 793, + "text": null + }, + { + "tick": 19, + "probe_ms": 744, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1012, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.198, + "elements": 21, + "elapsed_ms": 86713, + "probabilities": { + "14": 0.461, + "21": 0.201, + "1": 0.144, + "6": 0.045, + "2": 0.029, + "10": 0.022 + }, + "candidates": { + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "1": "Main menu", + "6": "Flights", + "2": "Google", + "10": "Google apps" + }, + "screenshot_ms": 800, + "text": null + }, + { + "tick": 20, + "probe_ms": 3796, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1011, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.43, + "elements": 21, + "elapsed_ms": 93025, + "probabilities": { + "SCROLL_UP": 0.525, + "CLICK": 0.285, + "SCROLL_DOWN": 0.08, + "WAIT": 0.065, + "BLOCKED": 0.044, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_DOWN": "Move the viewport down the page.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 811, + "text": null + }, + { + "tick": 21, + "probe_ms": 776, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1093, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.235, + "elements": 25, + "elapsed_ms": 95810, + "probabilities": { + "SCROLL_DOWN": 0.344, + "PRESS_ENTER": 0.261, + "CLICK": 0.245, + "WAIT": 0.096, + "DONE": 0.05, + "TYPE_TEXT": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 760, + "text": null + }, + { + "tick": 22, + "probe_ms": 765, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1007, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.631, + "elements": 21, + "elapsed_ms": 98542, + "probabilities": { + "14": 0.424, + "1": 0.213, + "21": 0.094, + "6": 0.072, + "10": 0.041, + "2": 0.035 + }, + "candidates": { + "14": "Munich", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "10": "Google apps", + "2": "Google" + }, + "screenshot_ms": 803, + "text": null + }, + { + "tick": 23, + "probe_ms": 3800, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 981, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.624, + "elements": 21, + "elapsed_ms": 104838, + "probabilities": { + "1": 0.25, + "21": 0.17, + "6": 0.154, + "15": 0.112, + "2": 0.073, + "16": 0.067 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "15": "Berlin", + "2": "Google", + "16": "Düsseldorf" + }, + "screenshot_ms": 824, + "text": null + }, + { + "tick": 24, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1304, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change location", + "confidence": 0.748, + "elements": 33, + "elapsed_ms": 108403, + "probabilities": { + "20": 0.24, + "6": 0.132, + "1": 0.103, + "13": 0.075, + "5": 0.072, + "28": 0.066 + }, + "candidates": { + "20": "Change location", + "6": "Flights", + "1": "Main menu", + "13": "Selected Tab: Flights", + "5": "Explore", + "28": "Düsseldorf" + }, + "screenshot_ms": 815, + "text": null + }, + { + "tick": 25, + "probe_ms": 802, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 473, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "OK", + "confidence": 0.902, + "elements": 2, + "elapsed_ms": 111882, + "probabilities": { + "2": 0.96, + "1": 0.04 + }, + "candidates": { + "2": "OK", + "1": "Cancel" + }, + "screenshot_ms": 875, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 27121, + "chat_output_tokens": 121, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.008439 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-40-16/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-40-16/answer.json new file mode 100644 index 0000000..720ca8d --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-40-16/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights/deals?hl=en&authuser\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":102.924,\"remaining_s\":137.076,\"invocation_remaining_s\":138.714},\"summary\":\"The form phase budget has been exhausted. I have entered \\\"Zurich\\\" as the departure city, \\\"London\\\" as the destination city, and selected November 1, 2026, as the one-way travel date. I also tried to click on \\\"Flight Deals\\\" but it seems to have reset the form. I am unable to proceed further due to the budget exhaustion.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 103910, + "usage": { + "decisions": 25, + "chat_calls": 11, + "chat_input_tokens": 28367, + "chat_output_tokens": 159, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.008908 + }, + "report": { + "elapsed_ms": 103264, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1173, + "jev_input_tokens": 0, + "median_probe_ms": 804, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Location ​ Germany", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights from popular destinations", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flight Deals", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Got it", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "City or airport", + "kind": "fill", + "text": "Zurich", + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-40-16/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-40-16/decision_ticks.json new file mode 100644 index 0000000..0d0394b --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-40-16/decision_ticks.json @@ -0,0 +1,995 @@ +{ + "report": { + "elapsed_ms": 103264, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1173, + "jev_input_tokens": 0, + "median_probe_ms": 804, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Location ​ Germany", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights from popular destinations", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flight Deals", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Got it", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "City or airport", + "kind": "fill", + "text": "Zurich", + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5853, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3036, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.915, + "elements": 20, + "elapsed_ms": 10903, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 684, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1321, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 704, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.988, + "elements": 10, + "elapsed_ms": 13723, + "probabilities": { + "3": 0.954, + "5": 0.036, + "2": 0.005, + "4": 0.003, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 670, + "text": null + }, + { + "tick": 3, + "probe_ms": 740, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 940, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.478, + "elements": 20, + "elapsed_ms": 17409, + "probabilities": { + "16": 0.996, + "15": 0.001, + "17": 0.001, + "18": 0.001 + }, + "candidates": { + "16": "Where to?", + "15": "Where from?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 765, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1277, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 738, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.95, + "elements": 9, + "elapsed_ms": 20363, + "probabilities": { + "3": 0.754, + "5": 0.201, + "2": 0.023, + "7": 0.012, + "4": 0.006, + "9": 0.002 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "9": "London City Airport (LCY)" + }, + "screenshot_ms": 768, + "text": null + }, + { + "tick": 5, + "probe_ms": 741, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 992, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.685, + "elements": 21, + "elapsed_ms": 24081, + "probabilities": { + "12": 0.924, + "18": 0.059, + "16": 0.008, + "19": 0.006, + "13": 0.001, + "15": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "16": "Swap origin and destination.", + "19": "Return", + "13": "1 passenger", + "15": "Where from?" + }, + "screenshot_ms": 754, + "text": null + }, + { + "tick": 6, + "probe_ms": 737, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 491, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 27246, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 752, + "text": null + }, + { + "tick": 7, + "probe_ms": 738, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 954, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.704, + "elements": 20, + "elapsed_ms": 30890, + "probabilities": { + "18": 0.999, + "14": 0.0, + "19": 0.0, + "12": 0.0, + "1": 0.0, + "2": 0.0 + }, + "candidates": { + "18": "Departure", + "14": "Change seating class. Economy", + "19": "Search", + "12": "Change ticket type. One way", + "1": "Main menu", + "2": "Google" + }, + "screenshot_ms": 742, + "text": null + }, + { + "tick": 8, + "probe_ms": 1663, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1772, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.501, + "elements": 49, + "elapsed_ms": 35971, + "probabilities": { + "SCROLL_DOWN": 0.584, + "CLICK": 0.33, + "WAIT": 0.077, + "BLOCKED": 0.008, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 801, + "text": null + }, + { + "tick": 9, + "probe_ms": 1107, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1911, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.44, + "elements": 50, + "elapsed_ms": 40018, + "probabilities": { + "49": 0.555, + "36": 0.126, + "35": 0.115, + "38": 0.036, + "39": 0.031, + "42": 0.025 + }, + "candidates": { + "49": "Next", + "36": "Monday, November 2, 2026", + "35": "Saturday, October 31, 2026 , 84 euros", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 798, + "text": null + }, + { + "tick": 10, + "probe_ms": 1851, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2651, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.993, + "elements": 82, + "elapsed_ms": 46060, + "probabilities": { + "36": 0.931, + "66": 0.021, + "81": 0.008, + "37": 0.006, + "67": 0.004, + "40": 0.003 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "37": "Monday, November 2, 2026", + "67": "Wednesday, December 2, 2026", + "40": "Thursday, November 5, 2026" + }, + "screenshot_ms": 801, + "text": null + }, + { + "tick": 11, + "probe_ms": 1308, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2821, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.413, + "elements": 82, + "elapsed_ms": 51764, + "probabilities": { + "82": 0.967, + "36": 0.027, + "1": 0.002, + "62": 0.001, + "66": 0.001, + "81": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "62": "Friday, November 27, 2026 , 27 euros, Cheapest price", + "66": "Tuesday, December 1, 2026", + "81": "Next" + }, + "screenshot_ms": 792, + "text": null + }, + { + "tick": 12, + "probe_ms": 741, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 994, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.532, + "elements": 20, + "elapsed_ms": 54955, + "probabilities": { + "SCROLL_DOWN": 0.599, + "PRESS_ENTER": 0.236, + "CLICK": 0.106, + "WAIT": 0.036, + "DONE": 0.022, + "BLOCKED": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 743, + "text": null + }, + { + "tick": 13, + "probe_ms": 1265, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 840, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.386, + "elements": 15, + "elapsed_ms": 58011, + "probabilities": { + "SCROLL_DOWN": 0.488, + "SCROLL_UP": 0.227, + "WAIT": 0.204, + "CLICK": 0.065, + "DONE": 0.014, + "BLOCKED": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "SCROLL_UP": "Move the viewport up the page.", + "WAIT": "Give the page time to finish updating.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 781, + "text": null + }, + { + "tick": 14, + "probe_ms": 750, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1173, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.342, + "elements": 27, + "elapsed_ms": 60823, + "probabilities": { + "SCROLL_DOWN": 0.452, + "WAIT": 0.259, + "CLICK": 0.2, + "SCROLL_UP": 0.077, + "DONE": 0.01, + "BLOCKED": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "WAIT": "Give the page time to finish updating.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_UP": "Move the viewport up the page.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 740, + "text": null + }, + { + "tick": 15, + "probe_ms": 1247, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1336, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.644, + "elements": 43, + "elapsed_ms": 64288, + "probabilities": { + "1": 0.388, + "32": 0.332, + "13": 0.063, + "14": 0.027, + "16": 0.018, + "25": 0.018 + }, + "candidates": { + "1": "Main menu", + "32": "Location ​ Germany", + "13": "Flights from popular destinations", + "14": "Flights to popular destinations", + "16": "Flights from New York", + "25": "Flights from Berlin" + }, + "screenshot_ms": 740, + "text": null + }, + { + "tick": 16, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1528, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change location", + "confidence": 0.388, + "elements": 55, + "elapsed_ms": 68107, + "probabilities": { + "20": 0.209, + "44": 0.183, + "25": 0.078, + "30": 0.06, + "27": 0.059, + "28": 0.059 + }, + "candidates": { + "20": "Change location", + "44": "Location ​ Germany", + "25": "Flights from popular destinations", + "30": "Flights from London", + "27": "Popular flights routes", + "28": "Flights from New York" + }, + "screenshot_ms": 759, + "text": null + }, + { + "tick": 17, + "probe_ms": 804, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 489, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "OK", + "confidence": 0.883, + "elements": 2, + "elapsed_ms": 71573, + "probabilities": { + "2": 0.982, + "1": 0.018 + }, + "candidates": { + "2": "OK", + "1": "Cancel" + }, + "screenshot_ms": 836, + "text": null + }, + { + "tick": 18, + "probe_ms": 742, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1332, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Location ​ Germany", + "confidence": 0.481, + "elements": 43, + "elapsed_ms": 75139, + "probabilities": { + "32": 0.285, + "1": 0.224, + "13": 0.136, + "14": 0.053, + "25": 0.033, + "19": 0.026 + }, + "candidates": { + "32": "Location ​ Germany", + "1": "Main menu", + "13": "Flights from popular destinations", + "14": "Flights to popular destinations", + "25": "Flights from Berlin", + "19": "Flights from Toronto" + }, + "screenshot_ms": 742, + "text": null + }, + { + "tick": 19, + "probe_ms": 804, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 484, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "OK", + "confidence": 0.664, + "elements": 2, + "elapsed_ms": 78675, + "probabilities": { + "2": 0.991, + "1": 0.009 + }, + "candidates": { + "2": "OK", + "1": "Cancel" + }, + "screenshot_ms": 841, + "text": null + }, + { + "tick": 20, + "probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1334, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.28, + "elements": 43, + "elapsed_ms": 82239, + "probabilities": { + "1": 0.326, + "32": 0.139, + "13": 0.088, + "25": 0.042, + "14": 0.038, + "6": 0.036 + }, + "candidates": { + "1": "Main menu", + "32": "Location ​ Germany", + "13": "Flights from popular destinations", + "25": "Flights from Berlin", + "14": "Flights to popular destinations", + "6": "Flights" + }, + "screenshot_ms": 737, + "text": null + }, + { + "tick": 21, + "probe_ms": 744, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1532, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights from popular destinations", + "confidence": 0.351, + "elements": 55, + "elapsed_ms": 85992, + "probabilities": { + "25": 0.143, + "28": 0.114, + "27": 0.104, + "44": 0.067, + "37": 0.064, + "26": 0.061 + }, + "candidates": { + "25": "Flights from popular destinations", + "28": "Flights from New York", + "27": "Popular flights routes", + "44": "Location ​ Germany", + "37": "Flights from Berlin", + "26": "Flights to popular destinations" + }, + "screenshot_ms": 794, + "text": null + }, + { + "tick": 22, + "probe_ms": 773, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1327, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.564, + "elements": 43, + "elapsed_ms": 89529, + "probabilities": { + "1": 0.154, + "32": 0.124, + "16": 0.081, + "25": 0.077, + "6": 0.061, + "14": 0.051 + }, + "candidates": { + "1": "Main menu", + "32": "Location ​ Germany", + "16": "Flights from New York", + "25": "Flights from Berlin", + "6": "Flights", + "14": "Flights to popular destinations" + }, + "screenshot_ms": 742, + "text": null + }, + { + "tick": 23, + "probe_ms": 1245, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1525, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flight Deals", + "confidence": 0.465, + "elements": 55, + "elapsed_ms": 94227, + "probabilities": { + "16": 0.111, + "27": 0.083, + "20": 0.081, + "1": 0.081, + "6": 0.081, + "28": 0.065 + }, + "candidates": { + "16": "Flight Deals", + "27": "Popular flights routes", + "20": "Change location", + "1": "Main menu", + "6": "Flights", + "28": "Flights from New York" + }, + "screenshot_ms": 746, + "text": null + }, + { + "tick": 24, + "probe_ms": 1155, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 459, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Got it", + "confidence": 0.627, + "elements": 2, + "elapsed_ms": 97737, + "probabilities": { + "2": 0.989, + "1": 0.011 + }, + "candidates": { + "2": "Got it", + "1": "Learn more about Flight Deals" + }, + "screenshot_ms": 660, + "text": null + }, + { + "tick": 25, + "probe_ms": 654, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1090, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "City or airport", + "confidence": 0.507, + "elements": 21, + "elapsed_ms": 101268, + "probabilities": { + "12": 0.891, + "13": 0.109 + }, + "candidates": { + "12": "City or airport", + "13": "When, where, and how do you want to travel?" + }, + "screenshot_ms": 658, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 11, + "chat_input_tokens": 28367, + "chat_output_tokens": 159, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.008908 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-42-30/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-42-30/answer.json new file mode 100644 index 0000000..e673c0e --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-42-30/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":114.194,\"remaining_s\":125.806,\"invocation_remaining_s\":127.312},\"summary\":\"The form phase budget has been exhausted. I have successfully entered the origin (Zurich), destination (London), and selected a one-way trip for November 1, 2026. However, I was unable to confirm the number of adults or the economy class.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 115193, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 28386, + "chat_output_tokens": 103, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.008773 + }, + "report": { + "elapsed_ms": 114541, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1059, + "jev_input_tokens": 0, + "median_probe_ms": 771, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-42-30/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-42-30/decision_ticks.json new file mode 100644 index 0000000..e37b422 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-42-30/decision_ticks.json @@ -0,0 +1,1016 @@ +{ + "report": { + "elapsed_ms": 114541, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1059, + "jev_input_tokens": 0, + "median_probe_ms": 771, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5860, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3133, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.901, + "elements": 25, + "elapsed_ms": 11003, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 692, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1304, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 719, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.982, + "elements": 10, + "elapsed_ms": 13837, + "probabilities": { + "3": 0.969, + "5": 0.023, + "2": 0.004, + "4": 0.002, + "6": 0.001, + "9": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 680, + "text": null + }, + { + "tick": 3, + "probe_ms": 748, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1059, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.753, + "elements": 25, + "elapsed_ms": 17722, + "probabilities": { + "16": 0.994, + "17": 0.003, + "18": 0.002, + "15": 0.001 + }, + "candidates": { + "16": "Where to?", + "17": "Departure", + "18": "Return", + "15": "Where from?" + }, + "screenshot_ms": 771, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1317, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 741, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.93, + "elements": 9, + "elapsed_ms": 20760, + "probabilities": { + "3": 0.787, + "5": 0.163, + "2": 0.027, + "7": 0.012, + "4": 0.009, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 802, + "text": null + }, + { + "tick": 5, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1108, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.594, + "elements": 26, + "elapsed_ms": 24613, + "probabilities": { + "12": 0.778, + "18": 0.181, + "19": 0.024, + "16": 0.007, + "13": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "13": "1 passenger", + "20": "Search" + }, + "screenshot_ms": 757, + "text": null + }, + { + "tick": 6, + "probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 490, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 27825, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 752, + "text": null + }, + { + "tick": 7, + "probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1073, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.7, + "elements": 25, + "elapsed_ms": 31624, + "probabilities": { + "18": 0.999, + "19": 0.0, + "14": 0.0, + "9": 0.0, + "12": 0.0, + "13": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "14": "Change seating class. Economy", + "9": "Change appearance", + "12": "Change ticket type. One way", + "13": "1 passenger" + }, + "screenshot_ms": 741, + "text": null + }, + { + "tick": 8, + "probe_ms": 1657, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1778, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.545, + "elements": 49, + "elapsed_ms": 36729, + "probabilities": { + "SCROLL_DOWN": 0.621, + "CLICK": 0.294, + "WAIT": 0.077, + "BLOCKED": 0.007, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 823, + "text": null + }, + { + "tick": 9, + "probe_ms": 1636, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1915, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.421, + "elements": 50, + "elapsed_ms": 41325, + "probabilities": { + "49": 0.537, + "36": 0.126, + "35": 0.108, + "38": 0.043, + "39": 0.036, + "42": 0.025 + }, + "candidates": { + "49": "Next", + "36": "Monday, November 2, 2026", + "35": "Saturday, October 31, 2026 , 84 euros", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 806, + "text": null + }, + { + "tick": 10, + "probe_ms": 1857, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2649, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.991, + "elements": 82, + "elapsed_ms": 47380, + "probabilities": { + "36": 0.942, + "66": 0.013, + "81": 0.007, + "37": 0.006, + "40": 0.003, + "67": 0.003 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "37": "Monday, November 2, 2026", + "40": "Thursday, November 5, 2026", + "67": "Wednesday, December 2, 2026" + }, + "screenshot_ms": 806, + "text": null + }, + { + "tick": 11, + "probe_ms": 1327, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2819, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.45, + "elements": 82, + "elapsed_ms": 53150, + "probabilities": { + "82": 0.975, + "36": 0.021, + "1": 0.001, + "66": 0.0, + "62": 0.0, + "81": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "66": "Tuesday, December 1, 2026", + "62": "Friday, November 27, 2026 , 27 euros, Cheapest price", + "81": "Next" + }, + "screenshot_ms": 821, + "text": null + }, + { + "tick": 12, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1115, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.707, + "elements": 25, + "elapsed_ms": 56482, + "probabilities": { + "SCROLL_DOWN": 0.749, + "PRESS_ENTER": 0.134, + "CLICK": 0.074, + "WAIT": 0.021, + "DONE": 0.021, + "BLOCKED": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 744, + "text": null + }, + { + "tick": 13, + "probe_ms": 766, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1036, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.38, + "elements": 21, + "elapsed_ms": 59229, + "probabilities": { + "13": 0.82, + "21": 0.05, + "1": 0.05, + "6": 0.021, + "19": 0.014, + "16": 0.008 + }, + "candidates": { + "13": "Frankfurt am Main", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "1": "Main menu", + "6": "Flights", + "19": "Find flights from Frankfurt am Main (FRA) to London (LGW) from €134. Operated by Condor. Oct 29 to Nov 4. Nonstop", + "16": "Düsseldorf" + }, + "screenshot_ms": 785, + "text": null + }, + { + "tick": 14, + "probe_ms": 4811, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1037, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.42, + "elements": 21, + "elapsed_ms": 66562, + "probabilities": { + "13": 0.729, + "1": 0.113, + "21": 0.043, + "6": 0.031, + "16": 0.016, + "2": 0.013 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 800, + "text": null + }, + { + "tick": 15, + "probe_ms": 3799, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1018, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.611, + "elements": 21, + "elapsed_ms": 72862, + "probabilities": { + "1": 0.363, + "21": 0.176, + "14": 0.102, + "6": 0.099, + "16": 0.078, + "2": 0.036 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 790, + "text": null + }, + { + "tick": 16, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1360, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.717, + "elements": 33, + "elapsed_ms": 76440, + "probabilities": { + "1": 0.292, + "26": 0.229, + "33": 0.054, + "6": 0.045, + "20": 0.041, + "12": 0.038 + }, + "candidates": { + "1": "Main menu", + "26": "Munich", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "20": "Change location", + "12": "Explore" + }, + "screenshot_ms": 786, + "text": null + }, + { + "tick": 17, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1018, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.546, + "elements": 21, + "elapsed_ms": 79677, + "probabilities": { + "1": 0.273, + "21": 0.262, + "6": 0.145, + "14": 0.111, + "16": 0.058, + "2": 0.03 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 786, + "text": null + }, + { + "tick": 18, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1360, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.697, + "elements": 33, + "elapsed_ms": 83262, + "probabilities": { + "26": 0.189, + "1": 0.157, + "33": 0.097, + "6": 0.07, + "5": 0.062, + "12": 0.061 + }, + "candidates": { + "26": "Munich", + "1": "Main menu", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 794, + "text": null + }, + { + "tick": 19, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1012, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.198, + "elements": 21, + "elapsed_ms": 87033, + "probabilities": { + "14": 0.461, + "21": 0.201, + "1": 0.144, + "6": 0.045, + "2": 0.029, + "10": 0.022 + }, + "candidates": { + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "1": "Main menu", + "6": "Flights", + "2": "Google", + "10": "Google apps" + }, + "screenshot_ms": 800, + "text": null + }, + { + "tick": 20, + "probe_ms": 4303, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1011, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.43, + "elements": 21, + "elapsed_ms": 93842, + "probabilities": { + "SCROLL_UP": 0.525, + "CLICK": 0.285, + "SCROLL_DOWN": 0.08, + "WAIT": 0.065, + "BLOCKED": 0.044, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_DOWN": "Move the viewport down the page.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 805, + "text": null + }, + { + "tick": 21, + "probe_ms": 771, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1096, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.235, + "elements": 25, + "elapsed_ms": 96607, + "probabilities": { + "SCROLL_DOWN": 0.344, + "PRESS_ENTER": 0.261, + "CLICK": 0.245, + "WAIT": 0.096, + "DONE": 0.05, + "TYPE_TEXT": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 739, + "text": null + }, + { + "tick": 22, + "probe_ms": 771, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1011, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.631, + "elements": 21, + "elapsed_ms": 99348, + "probabilities": { + "14": 0.424, + "1": 0.213, + "21": 0.094, + "6": 0.072, + "10": 0.041, + "2": 0.035 + }, + "candidates": { + "14": "Munich", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "10": "Google apps", + "2": "Google" + }, + "screenshot_ms": 793, + "text": null + }, + { + "tick": 23, + "probe_ms": 3809, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 981, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.624, + "elements": 21, + "elapsed_ms": 105641, + "probabilities": { + "1": 0.25, + "21": 0.17, + "6": 0.154, + "15": 0.112, + "2": 0.073, + "16": 0.067 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "15": "Berlin", + "2": "Google", + "16": "Düsseldorf" + }, + "screenshot_ms": 817, + "text": null + }, + { + "tick": 24, + "probe_ms": 747, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1306, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change location", + "confidence": 0.748, + "elements": 33, + "elapsed_ms": 109210, + "probabilities": { + "20": 0.24, + "6": 0.132, + "1": 0.103, + "13": 0.075, + "5": 0.072, + "28": 0.066 + }, + "candidates": { + "20": "Change location", + "6": "Flights", + "1": "Main menu", + "13": "Selected Tab: Flights", + "5": "Explore", + "28": "Düsseldorf" + }, + "screenshot_ms": 816, + "text": null + }, + { + "tick": 25, + "probe_ms": 800, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 468, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "OK", + "confidence": 0.902, + "elements": 2, + "elapsed_ms": 112671, + "probabilities": { + "2": 0.96, + "1": 0.04 + }, + "candidates": { + "2": "OK", + "1": "Cancel" + }, + "screenshot_ms": 871, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 28386, + "chat_output_tokens": 103, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.008773 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-44-55/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-44-55/answer.json new file mode 100644 index 0000000..af7ea95 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-44-55/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":113.776,\"remaining_s\":126.224,\"invocation_remaining_s\":127.961},\"summary\":\"The form phase budget has been exhausted. I have successfully entered the origin (Zurich), destination (London), and selected a one-way trip for November 1, 2026. However, I was unable to confirm the number of adults or the economy class.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 114801, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 29013, + "chat_output_tokens": 103, + "chat_cache_tokens": 26439, + "jev_input_tokens": 0, + "cost_usd": 0.001823 + }, + "report": { + "elapsed_ms": 114120, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1055, + "jev_input_tokens": 0, + "median_probe_ms": 784, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-44-55/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-44-55/decision_ticks.json new file mode 100644 index 0000000..ad59205 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-44-55/decision_ticks.json @@ -0,0 +1,1016 @@ +{ + "report": { + "elapsed_ms": 114120, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1055, + "jev_input_tokens": 0, + "median_probe_ms": 784, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5365, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3135, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.901, + "elements": 25, + "elapsed_ms": 10611, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 697, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1290, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 716, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.982, + "elements": 10, + "elapsed_ms": 13445, + "probabilities": { + "3": 0.969, + "5": 0.023, + "2": 0.004, + "4": 0.002, + "6": 0.001, + "9": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 692, + "text": null + }, + { + "tick": 3, + "probe_ms": 761, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1055, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.753, + "elements": 25, + "elapsed_ms": 17336, + "probabilities": { + "16": 0.994, + "17": 0.003, + "18": 0.002, + "15": 0.001 + }, + "candidates": { + "16": "Where to?", + "17": "Departure", + "18": "Return", + "15": "Where from?" + }, + "screenshot_ms": 762, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1282, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 746, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.93, + "elements": 9, + "elapsed_ms": 20314, + "probabilities": { + "3": 0.787, + "5": 0.163, + "2": 0.027, + "7": 0.012, + "4": 0.009, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 775, + "text": null + }, + { + "tick": 5, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1103, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.594, + "elements": 26, + "elapsed_ms": 24168, + "probabilities": { + "12": 0.778, + "18": 0.181, + "19": 0.024, + "16": 0.007, + "13": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "13": "1 passenger", + "20": "Search" + }, + "screenshot_ms": 750, + "text": null + }, + { + "tick": 6, + "probe_ms": 740, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 491, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 27370, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 749, + "text": null + }, + { + "tick": 7, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1072, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.7, + "elements": 25, + "elapsed_ms": 31182, + "probabilities": { + "18": 0.999, + "19": 0.0, + "14": 0.0, + "9": 0.0, + "12": 0.0, + "13": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "14": "Change seating class. Economy", + "9": "Change appearance", + "12": "Change ticket type. One way", + "13": "1 passenger" + }, + "screenshot_ms": 749, + "text": null + }, + { + "tick": 8, + "probe_ms": 1663, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1782, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.545, + "elements": 49, + "elapsed_ms": 36290, + "probabilities": { + "SCROLL_DOWN": 0.621, + "CLICK": 0.294, + "WAIT": 0.077, + "BLOCKED": 0.007, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 814, + "text": null + }, + { + "tick": 9, + "probe_ms": 1634, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1910, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.421, + "elements": 50, + "elapsed_ms": 40886, + "probabilities": { + "49": 0.537, + "36": 0.126, + "35": 0.108, + "38": 0.043, + "39": 0.036, + "42": 0.025 + }, + "candidates": { + "49": "Next", + "36": "Monday, November 2, 2026", + "35": "Saturday, October 31, 2026 , 84 euros", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 808, + "text": null + }, + { + "tick": 10, + "probe_ms": 1872, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2647, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.991, + "elements": 82, + "elapsed_ms": 46962, + "probabilities": { + "36": 0.942, + "66": 0.013, + "81": 0.007, + "37": 0.006, + "40": 0.003, + "67": 0.003 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "37": "Monday, November 2, 2026", + "40": "Thursday, November 5, 2026", + "67": "Wednesday, December 2, 2026" + }, + "screenshot_ms": 802, + "text": null + }, + { + "tick": 11, + "probe_ms": 1317, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2834, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.45, + "elements": 82, + "elapsed_ms": 52726, + "probabilities": { + "82": 0.975, + "36": 0.021, + "1": 0.001, + "66": 0.0, + "62": 0.0, + "81": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "66": "Tuesday, December 1, 2026", + "62": "Friday, November 27, 2026 , 27 euros, Cheapest price", + "81": "Next" + }, + "screenshot_ms": 810, + "text": null + }, + { + "tick": 12, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1114, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.707, + "elements": 25, + "elapsed_ms": 56072, + "probabilities": { + "SCROLL_DOWN": 0.749, + "PRESS_ENTER": 0.134, + "CLICK": 0.074, + "WAIT": 0.021, + "DONE": 0.021, + "BLOCKED": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 762, + "text": null + }, + { + "tick": 13, + "probe_ms": 784, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1038, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.38, + "elements": 21, + "elapsed_ms": 58851, + "probabilities": { + "13": 0.82, + "21": 0.05, + "1": 0.05, + "6": 0.021, + "19": 0.014, + "16": 0.008 + }, + "candidates": { + "13": "Frankfurt am Main", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "1": "Main menu", + "6": "Flights", + "19": "Find flights from Frankfurt am Main (FRA) to London (LGW) from €134. Operated by Condor. Oct 29 to Nov 4. Nonstop", + "16": "Düsseldorf" + }, + "screenshot_ms": 795, + "text": null + }, + { + "tick": 14, + "probe_ms": 4333, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1036, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.42, + "elements": 21, + "elapsed_ms": 66229, + "probabilities": { + "13": 0.729, + "1": 0.113, + "21": 0.043, + "6": 0.031, + "16": 0.016, + "2": 0.013 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 1310, + "text": null + }, + { + "tick": 15, + "probe_ms": 3822, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1018, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.611, + "elements": 21, + "elapsed_ms": 72544, + "probabilities": { + "1": 0.363, + "21": 0.176, + "14": 0.102, + "6": 0.099, + "16": 0.078, + "2": 0.036 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 787, + "text": null + }, + { + "tick": 16, + "probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1361, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.717, + "elements": 33, + "elapsed_ms": 76160, + "probabilities": { + "1": 0.292, + "26": 0.229, + "33": 0.054, + "6": 0.045, + "20": 0.041, + "12": 0.038 + }, + "candidates": { + "1": "Main menu", + "26": "Munich", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "20": "Change location", + "12": "Explore" + }, + "screenshot_ms": 806, + "text": null + }, + { + "tick": 17, + "probe_ms": 747, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1018, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.546, + "elements": 21, + "elapsed_ms": 79408, + "probabilities": { + "1": 0.273, + "21": 0.262, + "6": 0.145, + "14": 0.111, + "16": 0.058, + "2": 0.03 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 786, + "text": null + }, + { + "tick": 18, + "probe_ms": 756, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1359, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.697, + "elements": 33, + "elapsed_ms": 83019, + "probabilities": { + "26": 0.189, + "1": 0.157, + "33": 0.097, + "6": 0.07, + "5": 0.062, + "12": 0.061 + }, + "candidates": { + "26": "Munich", + "1": "Main menu", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 793, + "text": null + }, + { + "tick": 19, + "probe_ms": 746, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1011, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.198, + "elements": 21, + "elapsed_ms": 86792, + "probabilities": { + "14": 0.461, + "21": 0.201, + "1": 0.144, + "6": 0.045, + "2": 0.029, + "10": 0.022 + }, + "candidates": { + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "1": "Main menu", + "6": "Flights", + "2": "Google", + "10": "Google apps" + }, + "screenshot_ms": 813, + "text": null + }, + { + "tick": 20, + "probe_ms": 3807, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1010, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.43, + "elements": 21, + "elapsed_ms": 93114, + "probabilities": { + "SCROLL_UP": 0.525, + "CLICK": 0.285, + "SCROLL_DOWN": 0.08, + "WAIT": 0.065, + "BLOCKED": 0.044, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_DOWN": "Move the viewport down the page.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 816, + "text": null + }, + { + "tick": 21, + "probe_ms": 768, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1095, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.235, + "elements": 25, + "elapsed_ms": 95892, + "probabilities": { + "SCROLL_DOWN": 0.344, + "PRESS_ENTER": 0.261, + "CLICK": 0.245, + "WAIT": 0.096, + "DONE": 0.05, + "TYPE_TEXT": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 756, + "text": null + }, + { + "tick": 22, + "probe_ms": 770, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1010, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.631, + "elements": 21, + "elapsed_ms": 98645, + "probabilities": { + "14": 0.424, + "1": 0.213, + "21": 0.094, + "6": 0.072, + "10": 0.041, + "2": 0.035 + }, + "candidates": { + "14": "Munich", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "10": "Google apps", + "2": "Google" + }, + "screenshot_ms": 808, + "text": null + }, + { + "tick": 23, + "probe_ms": 3814, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 981, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.624, + "elements": 21, + "elapsed_ms": 104949, + "probabilities": { + "1": 0.25, + "21": 0.17, + "6": 0.154, + "15": 0.112, + "2": 0.073, + "16": 0.067 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "15": "Berlin", + "2": "Google", + "16": "Düsseldorf" + }, + "screenshot_ms": 823, + "text": null + }, + { + "tick": 24, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1305, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change location", + "confidence": 0.748, + "elements": 33, + "elapsed_ms": 108525, + "probabilities": { + "20": 0.24, + "6": 0.132, + "1": 0.103, + "13": 0.075, + "5": 0.072, + "28": 0.066 + }, + "candidates": { + "20": "Change location", + "6": "Flights", + "1": "Main menu", + "13": "Selected Tab: Flights", + "5": "Explore", + "28": "Düsseldorf" + }, + "screenshot_ms": 812, + "text": null + }, + { + "tick": 25, + "probe_ms": 800, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 461, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "OK", + "confidence": 0.902, + "elements": 2, + "elapsed_ms": 112022, + "probabilities": { + "2": 0.96, + "1": 0.04 + }, + "candidates": { + "2": "OK", + "1": "Cancel" + }, + "screenshot_ms": 890, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 29013, + "chat_output_tokens": 103, + "chat_cache_tokens": 26439, + "jev_input_tokens": 0, + "cost_usd": 0.001823 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-47-20/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-47-20/answer.json new file mode 100644 index 0000000..e1d9800 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-47-20/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":115.533,\"remaining_s\":124.467,\"invocation_remaining_s\":126.25},\"summary\":\"The form phase budget has been exhausted. I have filled in the origin, destination, and date, and selected one-way. I am unable to proceed further.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 116529, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 29644, + "chat_output_tokens": 80, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.009093 + }, + "report": { + "elapsed_ms": 115876, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1059, + "jev_input_tokens": 0, + "median_probe_ms": 782, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-47-20/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-47-20/decision_ticks.json new file mode 100644 index 0000000..ea5131f --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__09-47-20/decision_ticks.json @@ -0,0 +1,1016 @@ +{ + "report": { + "elapsed_ms": 115876, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1059, + "jev_input_tokens": 0, + "median_probe_ms": 782, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5581, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3115, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.901, + "elements": 25, + "elapsed_ms": 10730, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 685, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1297, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 718, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.982, + "elements": 10, + "elapsed_ms": 13563, + "probabilities": { + "3": 0.969, + "5": 0.023, + "2": 0.004, + "4": 0.002, + "6": 0.001, + "9": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 686, + "text": null + }, + { + "tick": 3, + "probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1059, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.753, + "elements": 25, + "elapsed_ms": 17457, + "probabilities": { + "16": 0.994, + "17": 0.003, + "18": 0.002, + "15": 0.001 + }, + "candidates": { + "16": "Where to?", + "17": "Departure", + "18": "Return", + "15": "Where from?" + }, + "screenshot_ms": 762, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1268, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 744, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.93, + "elements": 9, + "elapsed_ms": 20420, + "probabilities": { + "3": 0.787, + "5": 0.163, + "2": 0.027, + "7": 0.012, + "4": 0.009, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 777, + "text": null + }, + { + "tick": 5, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1108, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.594, + "elements": 26, + "elapsed_ms": 24280, + "probabilities": { + "12": 0.778, + "18": 0.181, + "19": 0.024, + "16": 0.007, + "13": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "13": "1 passenger", + "20": "Search" + }, + "screenshot_ms": 754, + "text": null + }, + { + "tick": 6, + "probe_ms": 743, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 491, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 27487, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 767, + "text": null + }, + { + "tick": 7, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1072, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.7, + "elements": 25, + "elapsed_ms": 31308, + "probabilities": { + "18": 0.999, + "19": 0.0, + "14": 0.0, + "9": 0.0, + "12": 0.0, + "13": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "14": "Change seating class. Economy", + "9": "Change appearance", + "12": "Change ticket type. One way", + "13": "1 passenger" + }, + "screenshot_ms": 759, + "text": null + }, + { + "tick": 8, + "probe_ms": 1669, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1781, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.545, + "elements": 49, + "elapsed_ms": 36430, + "probabilities": { + "SCROLL_DOWN": 0.621, + "CLICK": 0.294, + "WAIT": 0.077, + "BLOCKED": 0.007, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 824, + "text": null + }, + { + "tick": 9, + "probe_ms": 1648, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1912, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.421, + "elements": 50, + "elapsed_ms": 41053, + "probabilities": { + "49": 0.537, + "36": 0.126, + "35": 0.108, + "38": 0.043, + "39": 0.036, + "42": 0.025 + }, + "candidates": { + "49": "Next", + "36": "Monday, November 2, 2026", + "35": "Saturday, October 31, 2026 , 84 euros", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 819, + "text": null + }, + { + "tick": 10, + "probe_ms": 1858, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2658, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.991, + "elements": 82, + "elapsed_ms": 47130, + "probabilities": { + "36": 0.942, + "66": 0.013, + "81": 0.007, + "37": 0.006, + "40": 0.003, + "67": 0.003 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "37": "Monday, November 2, 2026", + "40": "Thursday, November 5, 2026", + "67": "Wednesday, December 2, 2026" + }, + "screenshot_ms": 818, + "text": null + }, + { + "tick": 11, + "probe_ms": 1402, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2830, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.45, + "elements": 82, + "elapsed_ms": 53005, + "probabilities": { + "82": 0.975, + "36": 0.021, + "1": 0.001, + "66": 0.0, + "62": 0.0, + "81": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "66": "Tuesday, December 1, 2026", + "62": "Friday, November 27, 2026 , 27 euros, Cheapest price", + "81": "Next" + }, + "screenshot_ms": 819, + "text": null + }, + { + "tick": 12, + "probe_ms": 762, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1120, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.707, + "elements": 25, + "elapsed_ms": 56390, + "probabilities": { + "SCROLL_DOWN": 0.749, + "PRESS_ENTER": 0.134, + "CLICK": 0.074, + "WAIT": 0.021, + "DONE": 0.021, + "BLOCKED": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 747, + "text": null + }, + { + "tick": 13, + "probe_ms": 772, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1045, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.38, + "elements": 21, + "elapsed_ms": 59198, + "probabilities": { + "13": 0.82, + "21": 0.05, + "1": 0.05, + "6": 0.021, + "19": 0.014, + "16": 0.008 + }, + "candidates": { + "13": "Frankfurt am Main", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "1": "Main menu", + "6": "Flights", + "19": "Find flights from Frankfurt am Main (FRA) to London (LGW) from €134. Operated by Condor. Oct 29 to Nov 4. Nonstop", + "16": "Düsseldorf" + }, + "screenshot_ms": 821, + "text": null + }, + { + "tick": 14, + "probe_ms": 5347, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1039, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.42, + "elements": 21, + "elapsed_ms": 67119, + "probabilities": { + "13": 0.729, + "1": 0.113, + "21": 0.043, + "6": 0.031, + "16": 0.016, + "2": 0.013 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 836, + "text": null + }, + { + "tick": 15, + "probe_ms": 3833, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1021, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.611, + "elements": 21, + "elapsed_ms": 73496, + "probabilities": { + "1": 0.363, + "21": 0.176, + "14": 0.102, + "6": 0.099, + "16": 0.078, + "2": 0.036 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 833, + "text": null + }, + { + "tick": 16, + "probe_ms": 759, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1366, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.717, + "elements": 33, + "elapsed_ms": 77124, + "probabilities": { + "1": 0.292, + "26": 0.229, + "33": 0.054, + "6": 0.045, + "20": 0.041, + "12": 0.038 + }, + "candidates": { + "1": "Main menu", + "26": "Munich", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "20": "Change location", + "12": "Explore" + }, + "screenshot_ms": 804, + "text": null + }, + { + "tick": 17, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1020, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.546, + "elements": 21, + "elapsed_ms": 80373, + "probabilities": { + "1": 0.273, + "21": 0.262, + "6": 0.145, + "14": 0.111, + "16": 0.058, + "2": 0.03 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 798, + "text": null + }, + { + "tick": 18, + "probe_ms": 772, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1364, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.697, + "elements": 33, + "elapsed_ms": 84010, + "probabilities": { + "26": 0.189, + "1": 0.157, + "33": 0.097, + "6": 0.07, + "5": 0.062, + "12": 0.061 + }, + "candidates": { + "26": "Munich", + "1": "Main menu", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "5": "Explore", + "12": "Explore" + }, + "screenshot_ms": 813, + "text": null + }, + { + "tick": 19, + "probe_ms": 765, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1018, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.198, + "elements": 21, + "elapsed_ms": 87830, + "probabilities": { + "14": 0.461, + "21": 0.201, + "1": 0.144, + "6": 0.045, + "2": 0.029, + "10": 0.022 + }, + "candidates": { + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "1": "Main menu", + "6": "Flights", + "2": "Google", + "10": "Google apps" + }, + "screenshot_ms": 813, + "text": null + }, + { + "tick": 20, + "probe_ms": 4359, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1013, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.43, + "elements": 21, + "elapsed_ms": 94745, + "probabilities": { + "SCROLL_UP": 0.525, + "CLICK": 0.285, + "SCROLL_DOWN": 0.08, + "WAIT": 0.065, + "BLOCKED": 0.044, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_DOWN": "Move the viewport down the page.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 846, + "text": null + }, + { + "tick": 21, + "probe_ms": 782, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1095, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.235, + "elements": 25, + "elapsed_ms": 97540, + "probabilities": { + "SCROLL_DOWN": 0.344, + "PRESS_ENTER": 0.261, + "CLICK": 0.245, + "WAIT": 0.096, + "DONE": 0.05, + "TYPE_TEXT": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 756, + "text": null + }, + { + "tick": 22, + "probe_ms": 775, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1013, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.631, + "elements": 21, + "elapsed_ms": 100303, + "probabilities": { + "14": 0.424, + "1": 0.213, + "21": 0.094, + "6": 0.072, + "10": 0.041, + "2": 0.035 + }, + "candidates": { + "14": "Munich", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "10": "Google apps", + "2": "Google" + }, + "screenshot_ms": 816, + "text": null + }, + { + "tick": 23, + "probe_ms": 3840, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 984, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.624, + "elements": 21, + "elapsed_ms": 106642, + "probabilities": { + "1": 0.25, + "21": 0.17, + "6": 0.154, + "15": 0.112, + "2": 0.073, + "16": 0.067 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "15": "Berlin", + "2": "Google", + "16": "Düsseldorf" + }, + "screenshot_ms": 829, + "text": null + }, + { + "tick": 24, + "probe_ms": 758, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1311, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change location", + "confidence": 0.748, + "elements": 33, + "elapsed_ms": 110214, + "probabilities": { + "20": 0.24, + "6": 0.132, + "1": 0.103, + "13": 0.075, + "5": 0.072, + "28": 0.066 + }, + "candidates": { + "20": "Change location", + "6": "Flights", + "1": "Main menu", + "13": "Selected Tab: Flights", + "5": "Explore", + "28": "Düsseldorf" + }, + "screenshot_ms": 818, + "text": null + }, + { + "tick": 25, + "probe_ms": 832, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 467, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "OK", + "confidence": 0.902, + "elements": 2, + "elapsed_ms": 113733, + "probabilities": { + "2": 0.96, + "1": 0.04 + }, + "candidates": { + "2": "OK", + "1": "Cancel" + }, + "screenshot_ms": 880, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 29644, + "chat_output_tokens": 80, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.009093 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-32-35/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-32-35/answer.json new file mode 100644 index 0000000..126e912 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-32-35/answer.json @@ -0,0 +1,119 @@ +{ + "ok": true, + "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", + "screenshot": null, + "error": null, + "elapsed_ms": 39528, + "usage": { + "decisions": 13, + "chat_calls": 13, + "chat_input_tokens": 6173, + "chat_output_tokens": 119, + "chat_cache_tokens": 0, + "jev_input_tokens": 95564, + "cost_usd": 0.006163 + }, + "report": { + "elapsed_ms": 38862, + "decisions": 13, + "interactions": 12, + "waits": 0, + "median_decision_ms": 299, + "jev_input_tokens": 95564, + "median_probe_ms": 941, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "status": "DONE", + "terminal": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 12, + "elapsed_ms": 38515, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "agent": "flights", + "model": "jev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-32-35/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-32-35/decision_ticks.json new file mode 100644 index 0000000..0ec833b --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-32-35/decision_ticks.json @@ -0,0 +1,526 @@ +{ + "report": { + "elapsed_ms": 38862, + "decisions": 13, + "interactions": 12, + "waits": 0, + "median_decision_ms": 299, + "jev_input_tokens": 95564, + "median_probe_ms": 941, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 5320, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 304, + "input_tokens": 4802, + "output_tokens": 306, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.88, + "elements": 25, + "elapsed_ms": 7394, + "probabilities": { + "12": 0.83, + "15": 0.17, + "1": 0.0, + "2": 0.0, + "5": 0.0, + "6": 0.0 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "15": "Where from?", + "1": "Main menu", + "2": "Google", + "5": "Explore", + "6": "Flights" + }, + "text": null + }, + { + "tick": 2, + "probe_ms": 681, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 258, + "input_tokens": 2022, + "output_tokens": 92, + "operation": "CLICK", + "target": "One way", + "confidence": 0.77, + "elements": 3, + "elapsed_ms": 9511, + "probabilities": { + "2": 0.98, + "1": 0.02, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "text": null + }, + { + "tick": 3, + "probe_ms": 764, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 271, + "input_tokens": 4756, + "output_tokens": 291, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.92, + "elements": 24, + "elapsed_ms": 11780, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 4, + "probe_ms": 1269, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 245, + "input_tokens": 4471, + "output_tokens": 197, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.92, + "elements": 10, + "elapsed_ms": 13481, + "probabilities": { + "3": 0.85, + "5": 0.1, + "2": 0.05, + "4": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "text": null + }, + { + "tick": 5, + "probe_ms": 757, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 299, + "input_tokens": 5566, + "output_tokens": 324, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.95, + "elements": 24, + "elapsed_ms": 15775, + "probabilities": { + "16": 1.0, + "15": 0.0, + "17": 0.0 + }, + "candidates": { + "16": "Where to?", + "15": "Where from?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 6, + "probe_ms": 1267, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 325, + "input_tokens": 4451, + "output_tokens": 189, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.91, + "elements": 9, + "elapsed_ms": 17540, + "probabilities": { + "3": 0.88, + "2": 0.07, + "5": 0.05, + "4": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "London, United Kingdom", + "2": "Destination, Select multiple airports", + "5": "Heathrow Airport (LHR)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)", + "7": "London Gatwick Airport (LGW)" + }, + "text": null + }, + { + "tick": 7, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 274, + "input_tokens": 5795, + "output_tokens": 340, + "operation": "TYPE_TEXT", + "target": "Departure", + "confidence": 0.41, + "elements": 25, + "elapsed_ms": 19786, + "probabilities": { + "18": 1.0, + "15": 0.0, + "17": 0.0 + }, + "candidates": { + "18": "Departure", + "15": "Where from?", + "17": "Where to?" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 8, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 275, + "input_tokens": 5890, + "output_tokens": 347, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.58, + "elements": 25, + "elapsed_ms": 20982, + "probabilities": { + "18": 0.64, + "19": 0.3, + "12": 0.02, + "13": 0.02, + "15": 0.01, + "17": 0.01 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "12": "Change ticket type. One way", + "13": "1 passenger", + "15": "Where from?", + "17": "Where to?" + }, + "text": null + }, + { + "tick": 9, + "probe_ms": 1661, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 303, + "input_tokens": 10302, + "output_tokens": 486, + "operation": "CLICK", + "target": "Next", + "confidence": 0.77, + "elements": 49, + "elapsed_ms": 23792, + "probabilities": { + "48": 0.37, + "35": 0.25, + "34": 0.15, + "1": 0.08, + "3": 0.07, + "49": 0.03 + }, + "candidates": { + "48": "Next", + "35": "Monday, November 2, 2026", + "34": "Saturday, October 31, 2026 ????", + "1": "Departure", + "3": "Wednesday, September 30, 2026 ????", + "49": "Done." + }, + "text": null + }, + { + "tick": 10, + "probe_ms": 2286, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 441, + "input_tokens": 15640, + "output_tokens": 742, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.97, + "elements": 81, + "elapsed_ms": 27267, + "probabilities": { + "35": 0.98, + "1": 0.01, + "81": 0.01, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "1": "Departure", + "81": "Done.", + "2": "Select your ticket type. One way", + "3": "Wednesday, September 30, 2026 , 279 euros", + "4": "Thursday, October 1, 2026 , 182 euros" + }, + "text": null + }, + { + "tick": 11, + "probe_ms": 1874, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1275, + "input_tokens": 17012, + "output_tokens": 781, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.9, + "elements": 82, + "elapsed_ms": 31728, + "probabilities": { + "82": 0.94, + "36": 0.05, + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "2": "Select your ticket type. One way", + "3": "Reset", + "4": "Wednesday, September 30, 2026 , 279 euros" + }, + "text": null + }, + { + "tick": 12, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 289, + "input_tokens": 6026, + "output_tokens": 347, + "operation": "CLICK", + "target": "Search", + "confidence": 0.93, + "elements": 25, + "elapsed_ms": 33484, + "probabilities": { + "19": 0.96, + "6": 0.01, + "15": 0.01, + "17": 0.01, + "18": 0.01, + "1": 0.0 + }, + "candidates": { + "19": "Search", + "6": "Flights", + "15": "Where from?", + "17": "Where to?", + "18": "Departure", + "1": "Main menu" + }, + "text": null + }, + { + "tick": 13, + "probe_ms": 941, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 330, + "input_tokens": 8831, + "output_tokens": 498, + "operation": "DONE", + "target": null, + "confidence": 0.95, + "elements": 48, + "elapsed_ms": 36754, + "probabilities": { + "DONE": 0.96, + "TYPE_TEXT": 0.02, + "CLICK": 0.01, + "WAIT": 0.01, + "BLOCKED": 0.0, + "PRESS_ENTER": 0.0 + }, + "candidates": { + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search." + } + } + ], + "final": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 12, + "elapsed_ms": 38515, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "usage": { + "decisions": 13, + "chat_calls": 13, + "chat_input_tokens": 6173, + "chat_output_tokens": 119, + "chat_cache_tokens": 0, + "jev_input_tokens": 95564, + "cost_usd": 0.006163 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-33-21/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-33-21/answer.json new file mode 100644 index 0000000..66b2f00 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-33-21/answer.json @@ -0,0 +1,119 @@ +{ + "ok": true, + "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", + "screenshot": null, + "error": null, + "elapsed_ms": 36109, + "usage": { + "decisions": 13, + "chat_calls": 13, + "chat_input_tokens": 6497, + "chat_output_tokens": 128, + "chat_cache_tokens": 0, + "jev_input_tokens": 95564, + "cost_usd": 0.006283 + }, + "report": { + "elapsed_ms": 35443, + "decisions": 13, + "interactions": 12, + "waits": 0, + "median_decision_ms": 367, + "jev_input_tokens": 95564, + "median_probe_ms": 1204, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "status": "DONE", + "terminal": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 12, + "elapsed_ms": 35092, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "agent": "flights", + "model": "jev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-33-21/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-33-21/decision_ticks.json new file mode 100644 index 0000000..990cd8f --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-33-21/decision_ticks.json @@ -0,0 +1,526 @@ +{ + "report": { + "elapsed_ms": 35443, + "decisions": 13, + "interactions": 12, + "waits": 0, + "median_decision_ms": 367, + "jev_input_tokens": 95564, + "median_probe_ms": 1204, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 1204, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 345, + "input_tokens": 4802, + "output_tokens": 306, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.89, + "elements": 25, + "elapsed_ms": 3387, + "probabilities": { + "12": 0.83, + "15": 0.17, + "1": 0.0, + "2": 0.0, + "5": 0.0, + "6": 0.0 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "15": "Where from?", + "1": "Main menu", + "2": "Google", + "5": "Explore", + "6": "Flights" + }, + "text": null + }, + { + "tick": 2, + "probe_ms": 683, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 310, + "input_tokens": 2022, + "output_tokens": 92, + "operation": "CLICK", + "target": "One way", + "confidence": 0.76, + "elements": 3, + "elapsed_ms": 5533, + "probabilities": { + "2": 0.98, + "1": 0.02, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "text": null + }, + { + "tick": 3, + "probe_ms": 758, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 351, + "input_tokens": 4756, + "output_tokens": 291, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.94, + "elements": 24, + "elapsed_ms": 7881, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 4, + "probe_ms": 1269, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 507, + "input_tokens": 4471, + "output_tokens": 197, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.95, + "elements": 10, + "elapsed_ms": 9840, + "probabilities": { + "3": 0.86, + "5": 0.11, + "2": 0.03, + "4": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "text": null + }, + { + "tick": 5, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 310, + "input_tokens": 5566, + "output_tokens": 324, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.95, + "elements": 24, + "elapsed_ms": 12123, + "probabilities": { + "16": 1.0, + "15": 0.0, + "17": 0.0 + }, + "candidates": { + "16": "Where to?", + "15": "Where from?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 6, + "probe_ms": 1278, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 839, + "input_tokens": 4451, + "output_tokens": 189, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.92, + "elements": 9, + "elapsed_ms": 14416, + "probabilities": { + "3": 0.88, + "2": 0.07, + "5": 0.05, + "4": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "London, United Kingdom", + "2": "Destination, Select multiple airports", + "5": "Heathrow Airport (LHR)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)", + "7": "London Gatwick Airport (LGW)" + }, + "text": null + }, + { + "tick": 7, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 298, + "input_tokens": 5795, + "output_tokens": 340, + "operation": "TYPE_TEXT", + "target": "Departure", + "confidence": 0.43, + "elements": 25, + "elapsed_ms": 16674, + "probabilities": { + "18": 1.0, + "15": 0.0, + "17": 0.0 + }, + "candidates": { + "18": "Departure", + "15": "Where from?", + "17": "Where to?" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 8, + "probe_ms": 763, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1390, + "input_tokens": 5890, + "output_tokens": 347, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.6, + "elements": 25, + "elapsed_ms": 18991, + "probabilities": { + "18": 0.55, + "19": 0.34, + "12": 0.04, + "13": 0.04, + "15": 0.01, + "17": 0.01 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "12": "Change ticket type. One way", + "13": "1 passenger", + "15": "Where from?", + "17": "Where to?" + }, + "text": null + }, + { + "tick": 9, + "probe_ms": 1651, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 371, + "input_tokens": 10302, + "output_tokens": 486, + "operation": "CLICK", + "target": "Next", + "confidence": 0.8, + "elements": 49, + "elapsed_ms": 21848, + "probabilities": { + "48": 0.5, + "35": 0.17, + "34": 0.13, + "1": 0.08, + "3": 0.06, + "2": 0.02 + }, + "candidates": { + "48": "Next", + "35": "Monday, November 2, 2026", + "34": "Saturday, October 31, 2026 ????", + "1": "Departure", + "3": "Wednesday, September 30, 2026 ????", + "2": "Select your ticket type. One way" + }, + "text": null + }, + { + "tick": 10, + "probe_ms": 1854, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 395, + "input_tokens": 15640, + "output_tokens": 742, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.97, + "elements": 81, + "elapsed_ms": 24850, + "probabilities": { + "35": 0.98, + "81": 0.02, + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "81": "Done.", + "1": "Departure", + "2": "Select your ticket type. One way", + "3": "Wednesday, September 30, 2026 , 279 euros", + "4": "Thursday, October 1, 2026 , 182 euros" + }, + "text": null + }, + { + "tick": 11, + "probe_ms": 1834, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 370, + "input_tokens": 17012, + "output_tokens": 781, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.88, + "elements": 82, + "elapsed_ms": 27845, + "probabilities": { + "82": 0.95, + "36": 0.05, + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "2": "Select your ticket type. One way", + "3": "Reset", + "4": "Wednesday, September 30, 2026 , 279 euros" + }, + "text": null + }, + { + "tick": 12, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 334, + "input_tokens": 6026, + "output_tokens": 347, + "operation": "CLICK", + "target": "Search", + "confidence": 0.91, + "elements": 25, + "elapsed_ms": 29649, + "probabilities": { + "19": 0.96, + "13": 0.01, + "15": 0.01, + "17": 0.01, + "18": 0.01, + "1": 0.0 + }, + "candidates": { + "19": "Search", + "13": "1 passenger", + "15": "Where from?", + "17": "Where to?", + "18": "Departure", + "1": "Main menu" + }, + "text": null + }, + { + "tick": 13, + "probe_ms": 1443, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 367, + "input_tokens": 8831, + "output_tokens": 498, + "operation": "DONE", + "target": null, + "confidence": 0.95, + "elements": 48, + "elapsed_ms": 33390, + "probabilities": { + "DONE": 0.96, + "TYPE_TEXT": 0.02, + "PRESS_ENTER": 0.01, + "WAIT": 0.01, + "CLICK": 0.0, + "SCROLL_DOWN": 0.0 + }, + "candidates": { + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "WAIT": "Give the page time to finish updating.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_DOWN": "Move the viewport down the page." + } + } + ], + "final": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 12, + "elapsed_ms": 35092, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "usage": { + "decisions": 13, + "chat_calls": 13, + "chat_input_tokens": 6497, + "chat_output_tokens": 128, + "chat_cache_tokens": 0, + "jev_input_tokens": 95564, + "cost_usd": 0.006283 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-04/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-04/answer.json new file mode 100644 index 0000000..9a3e146 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-04/answer.json @@ -0,0 +1,119 @@ +{ + "ok": true, + "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", + "screenshot": null, + "error": null, + "elapsed_ms": 34605, + "usage": { + "decisions": 13, + "chat_calls": 13, + "chat_input_tokens": 6497, + "chat_output_tokens": 128, + "chat_cache_tokens": 0, + "jev_input_tokens": 95564, + "cost_usd": 0.006283 + }, + "report": { + "elapsed_ms": 33966, + "decisions": 13, + "interactions": 12, + "waits": 0, + "median_decision_ms": 321, + "jev_input_tokens": 95564, + "median_probe_ms": 928, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "status": "DONE", + "terminal": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 12, + "elapsed_ms": 33614, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "agent": "flights", + "model": "jev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-04/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-04/decision_ticks.json new file mode 100644 index 0000000..fe75549 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-04/decision_ticks.json @@ -0,0 +1,526 @@ +{ + "report": { + "elapsed_ms": 33966, + "decisions": 13, + "interactions": 12, + "waits": 0, + "median_decision_ms": 321, + "jev_input_tokens": 95564, + "median_probe_ms": 928, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 701, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 314, + "input_tokens": 4802, + "output_tokens": 306, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.87, + "elements": 25, + "elapsed_ms": 2689, + "probabilities": { + "12": 0.85, + "15": 0.15, + "1": 0.0, + "2": 0.0, + "5": 0.0, + "6": 0.0 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "15": "Where from?", + "1": "Main menu", + "2": "Google", + "5": "Explore", + "6": "Flights" + }, + "text": null + }, + { + "tick": 2, + "probe_ms": 1185, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 264, + "input_tokens": 2022, + "output_tokens": 92, + "operation": "CLICK", + "target": "One way", + "confidence": 0.74, + "elements": 3, + "elapsed_ms": 5298, + "probabilities": { + "2": 0.98, + "1": 0.02, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "text": null + }, + { + "tick": 3, + "probe_ms": 771, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 290, + "input_tokens": 4756, + "output_tokens": 291, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.92, + "elements": 24, + "elapsed_ms": 7601, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 4, + "probe_ms": 1257, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 320, + "input_tokens": 4471, + "output_tokens": 197, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.9, + "elements": 10, + "elapsed_ms": 9361, + "probabilities": { + "3": 0.87, + "5": 0.09, + "2": 0.04, + "4": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "text": null + }, + { + "tick": 5, + "probe_ms": 759, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1048, + "input_tokens": 5566, + "output_tokens": 324, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.95, + "elements": 24, + "elapsed_ms": 12387, + "probabilities": { + "16": 1.0, + "15": 0.0, + "17": 0.0 + }, + "candidates": { + "16": "Where to?", + "15": "Where from?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 6, + "probe_ms": 1278, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 339, + "input_tokens": 4451, + "output_tokens": 189, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.93, + "elements": 9, + "elapsed_ms": 14187, + "probabilities": { + "3": 0.86, + "2": 0.07, + "5": 0.07, + "4": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "London, United Kingdom", + "2": "Destination, Select multiple airports", + "5": "Heathrow Airport (LHR)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)", + "7": "London Gatwick Airport (LGW)" + }, + "text": null + }, + { + "tick": 7, + "probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 294, + "input_tokens": 5795, + "output_tokens": 340, + "operation": "TYPE_TEXT", + "target": "Departure", + "confidence": 0.43, + "elements": 25, + "elapsed_ms": 16441, + "probabilities": { + "18": 1.0, + "15": 0.0, + "17": 0.0 + }, + "candidates": { + "18": "Departure", + "15": "Where from?", + "17": "Where to?" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 8, + "probe_ms": 764, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 362, + "input_tokens": 5890, + "output_tokens": 347, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.57, + "elements": 25, + "elapsed_ms": 17754, + "probabilities": { + "18": 0.64, + "19": 0.29, + "13": 0.03, + "12": 0.02, + "15": 0.01, + "17": 0.01 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "13": "1 passenger", + "12": "Change ticket type. One way", + "15": "Where from?", + "17": "Where to?" + }, + "text": null + }, + { + "tick": 9, + "probe_ms": 1667, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 321, + "input_tokens": 10302, + "output_tokens": 486, + "operation": "CLICK", + "target": "Next", + "confidence": 0.76, + "elements": 49, + "elapsed_ms": 20565, + "probabilities": { + "48": 0.51, + "34": 0.19, + "35": 0.15, + "1": 0.06, + "3": 0.03, + "2": 0.02 + }, + "candidates": { + "48": "Next", + "34": "Saturday, October 31, 2026 ????", + "35": "Monday, November 2, 2026", + "1": "Departure", + "3": "Wednesday, September 30, 2026 ????", + "2": "Select your ticket type. One way" + }, + "text": null + }, + { + "tick": 10, + "probe_ms": 1864, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 334, + "input_tokens": 15640, + "output_tokens": 742, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.96, + "elements": 81, + "elapsed_ms": 23509, + "probabilities": { + "35": 0.97, + "81": 0.02, + "1": 0.01, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "81": "Done.", + "1": "Departure", + "2": "Select your ticket type. One way", + "3": "Wednesday, September 30, 2026 , 279 euros", + "4": "Thursday, October 1, 2026 , 182 euros" + }, + "text": null + }, + { + "tick": 11, + "probe_ms": 1830, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 360, + "input_tokens": 17012, + "output_tokens": 781, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.88, + "elements": 82, + "elapsed_ms": 26514, + "probabilities": { + "82": 0.94, + "36": 0.05, + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "2": "Select your ticket type. One way", + "3": "Reset", + "4": "Wednesday, September 30, 2026 , 279 euros" + }, + "text": null + }, + { + "tick": 12, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 311, + "input_tokens": 6026, + "output_tokens": 347, + "operation": "CLICK", + "target": "Search", + "confidence": 0.91, + "elements": 25, + "elapsed_ms": 28299, + "probabilities": { + "19": 0.97, + "13": 0.01, + "15": 0.01, + "18": 0.01, + "1": 0.0, + "2": 0.0 + }, + "candidates": { + "19": "Search", + "13": "1 passenger", + "15": "Where from?", + "18": "Departure", + "1": "Main menu", + "2": "Google" + }, + "text": null + }, + { + "tick": 13, + "probe_ms": 928, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 333, + "input_tokens": 8831, + "output_tokens": 498, + "operation": "DONE", + "target": null, + "confidence": 0.95, + "elements": 48, + "elapsed_ms": 31729, + "probabilities": { + "DONE": 0.96, + "TYPE_TEXT": 0.02, + "WAIT": 0.01, + "CLICK": 0.01, + "SCROLL_DOWN": 0.0, + "BLOCKED": 0.0 + }, + "candidates": { + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "WAIT": "Give the page time to finish updating.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward." + } + } + ], + "final": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 12, + "elapsed_ms": 33614, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "usage": { + "decisions": 13, + "chat_calls": 13, + "chat_input_tokens": 6497, + "chat_output_tokens": 128, + "chat_cache_tokens": 0, + "jev_input_tokens": 95564, + "cost_usd": 0.006283 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-46/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-46/answer.json new file mode 100644 index 0000000..e3a6e78 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-46/answer.json @@ -0,0 +1,113 @@ +{ + "ok": true, + "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", + "screenshot": null, + "error": null, + "elapsed_ms": 34502, + "usage": { + "decisions": 12, + "chat_calls": 12, + "chat_input_tokens": 6099, + "chat_output_tokens": 123, + "chat_cache_tokens": 0, + "jev_input_tokens": 89557, + "cost_usd": 0.005899 + }, + "report": { + "elapsed_ms": 33868, + "decisions": 12, + "interactions": 11, + "waits": 0, + "median_decision_ms": 347, + "jev_input_tokens": 89557, + "median_probe_ms": 1061, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "status": "DONE", + "terminal": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 11, + "elapsed_ms": 33513, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "agent": "flights", + "model": "jev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-46/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-46/decision_ticks.json new file mode 100644 index 0000000..1312a84 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-34-46/decision_ticks.json @@ -0,0 +1,490 @@ +{ + "report": { + "elapsed_ms": 33868, + "decisions": 12, + "interactions": 11, + "waits": 0, + "median_decision_ms": 347, + "jev_input_tokens": 89557, + "median_probe_ms": 1061, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 724, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 646, + "input_tokens": 4802, + "output_tokens": 306, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.92, + "elements": 25, + "elapsed_ms": 2906, + "probabilities": { + "12": 0.83, + "15": 0.17, + "1": 0.0, + "2": 0.0, + "5": 0.0, + "6": 0.0 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "15": "Where from?", + "1": "Main menu", + "2": "Google", + "5": "Explore", + "6": "Flights" + }, + "text": null + }, + { + "tick": 2, + "probe_ms": 1185, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 237, + "input_tokens": 2022, + "output_tokens": 92, + "operation": "CLICK", + "target": "One way", + "confidence": 0.77, + "elements": 3, + "elapsed_ms": 5494, + "probabilities": { + "2": 0.98, + "1": 0.02, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "text": null + }, + { + "tick": 3, + "probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 269, + "input_tokens": 4756, + "output_tokens": 291, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.93, + "elements": 24, + "elapsed_ms": 7882, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 4, + "probe_ms": 1265, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 320, + "input_tokens": 4471, + "output_tokens": 197, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.94, + "elements": 10, + "elapsed_ms": 9647, + "probabilities": { + "3": 0.84, + "5": 0.13, + "2": 0.03, + "4": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "text": null + }, + { + "tick": 5, + "probe_ms": 758, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 360, + "input_tokens": 5566, + "output_tokens": 324, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.96, + "elements": 24, + "elapsed_ms": 11998, + "probabilities": { + "16": 1.0, + "15": 0.0, + "17": 0.0 + }, + "candidates": { + "16": "Where to?", + "15": "Where from?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 6, + "probe_ms": 1301, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1259, + "input_tokens": 4451, + "output_tokens": 189, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.94, + "elements": 9, + "elapsed_ms": 14729, + "probabilities": { + "3": 0.87, + "5": 0.07, + "2": 0.06, + "4": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)", + "7": "London Gatwick Airport (LGW)" + }, + "text": null + }, + { + "tick": 7, + "probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 370, + "input_tokens": 5795, + "output_tokens": 339, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.47, + "elements": 25, + "elapsed_ms": 17087, + "probabilities": { + "18": 0.95, + "19": 0.03, + "12": 0.02, + "1": 0.0, + "2": 0.0, + "5": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "12": "Change ticket type. One way", + "1": "Main menu", + "2": "Google", + "5": "Explore" + }, + "text": null + }, + { + "tick": 8, + "probe_ms": 1673, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 298, + "input_tokens": 10265, + "output_tokens": 486, + "operation": "CLICK", + "target": "Next", + "confidence": 0.76, + "elements": 49, + "elapsed_ms": 19902, + "probabilities": { + "48": 0.52, + "35": 0.23, + "34": 0.13, + "1": 0.05, + "49": 0.03, + "2": 0.01 + }, + "candidates": { + "48": "Next", + "35": "Monday, November 2, 2026", + "34": "Saturday, October 31, 2026 ????", + "1": "Departure", + "49": "Done.", + "2": "Select your ticket type. One way" + }, + "text": null + }, + { + "tick": 9, + "probe_ms": 1886, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 383, + "input_tokens": 15603, + "output_tokens": 742, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.97, + "elements": 81, + "elapsed_ms": 22920, + "probabilities": { + "35": 0.99, + "81": 0.01, + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "81": "Done.", + "1": "Departure", + "2": "Select your ticket type. One way", + "3": "Wednesday, September 30, 2026 , 279 euros", + "4": "Thursday, October 1, 2026 , 182 euros" + }, + "text": null + }, + { + "tick": 10, + "probe_ms": 1861, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 352, + "input_tokens": 16975, + "output_tokens": 781, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.88, + "elements": 82, + "elapsed_ms": 25959, + "probabilities": { + "82": 0.96, + "36": 0.04, + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "2": "Select your ticket type. One way", + "3": "Reset", + "4": "Wednesday, September 30, 2026 , 279 euros" + }, + "text": null + }, + { + "tick": 11, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 306, + "input_tokens": 6025, + "output_tokens": 347, + "operation": "CLICK", + "target": "Search", + "confidence": 0.93, + "elements": 25, + "elapsed_ms": 27734, + "probabilities": { + "19": 1.0, + "1": 0.0, + "2": 0.0, + "5": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "19": "Search", + "1": "Main menu", + "2": "Google", + "5": "Explore", + "6": "Flights", + "7": "Hotels" + }, + "text": null + }, + { + "tick": 12, + "probe_ms": 937, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 343, + "input_tokens": 8826, + "output_tokens": 499, + "operation": "DONE", + "target": null, + "confidence": 0.98, + "elements": 48, + "elapsed_ms": 31160, + "probabilities": { + "DONE": 0.99, + "WAIT": 0.01, + "CLICK": 0.0, + "BLOCKED": 0.0, + "SCROLL_DOWN": 0.0, + "TYPE_TEXT": 0.0 + }, + "candidates": { + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "BLOCKED": "None of the offered operations can move the task forward.", + "SCROLL_DOWN": "Move the viewport down the page.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + } + } + ], + "final": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 11, + "elapsed_ms": 33513, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "usage": { + "decisions": 12, + "chat_calls": 12, + "chat_input_tokens": 6099, + "chat_output_tokens": 123, + "chat_cache_tokens": 0, + "jev_input_tokens": 89557, + "cost_usd": 0.005899 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-35-27/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-35-27/answer.json new file mode 100644 index 0000000..600874a --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-35-27/answer.json @@ -0,0 +1,119 @@ +{ + "ok": true, + "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", + "screenshot": null, + "error": null, + "elapsed_ms": 33679, + "usage": { + "decisions": 13, + "chat_calls": 13, + "chat_input_tokens": 6497, + "chat_output_tokens": 129, + "chat_cache_tokens": 0, + "jev_input_tokens": 95564, + "cost_usd": 0.006285 + }, + "report": { + "elapsed_ms": 33033, + "decisions": 13, + "interactions": 12, + "waits": 0, + "median_decision_ms": 283, + "jev_input_tokens": 95564, + "median_probe_ms": 932, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "status": "DONE", + "terminal": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 12, + "elapsed_ms": 32673, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "agent": "flights", + "model": "jev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-35-27/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-35-27/decision_ticks.json new file mode 100644 index 0000000..5dc882d --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-35-27/decision_ticks.json @@ -0,0 +1,526 @@ +{ + "report": { + "elapsed_ms": 33033, + "decisions": 13, + "interactions": 12, + "waits": 0, + "median_decision_ms": 283, + "jev_input_tokens": 95564, + "median_probe_ms": 932, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 1201, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 309, + "input_tokens": 4802, + "output_tokens": 306, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.88, + "elements": 25, + "elapsed_ms": 3370, + "probabilities": { + "12": 0.85, + "15": 0.15, + "1": 0.0, + "2": 0.0, + "5": 0.0, + "6": 0.0 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "15": "Where from?", + "1": "Main menu", + "2": "Google", + "5": "Explore", + "6": "Flights" + }, + "text": null + }, + { + "tick": 2, + "probe_ms": 682, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 322, + "input_tokens": 2022, + "output_tokens": 92, + "operation": "CLICK", + "target": "One way", + "confidence": 0.79, + "elements": 3, + "elapsed_ms": 5534, + "probabilities": { + "2": 0.97, + "1": 0.03, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "text": null + }, + { + "tick": 3, + "probe_ms": 769, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 275, + "input_tokens": 4756, + "output_tokens": 291, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.94, + "elements": 24, + "elapsed_ms": 7820, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 4, + "probe_ms": 1259, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 250, + "input_tokens": 4471, + "output_tokens": 197, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.92, + "elements": 10, + "elapsed_ms": 9511, + "probabilities": { + "3": 0.87, + "5": 0.09, + "2": 0.04, + "4": 0.0, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "text": null + }, + { + "tick": 5, + "probe_ms": 758, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 270, + "input_tokens": 5566, + "output_tokens": 324, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.95, + "elements": 24, + "elapsed_ms": 11759, + "probabilities": { + "16": 1.0, + "15": 0.0, + "17": 0.0 + }, + "candidates": { + "16": "Where to?", + "15": "Where from?", + "17": "Departure" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 6, + "probe_ms": 1277, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 265, + "input_tokens": 4451, + "output_tokens": 189, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.92, + "elements": 9, + "elapsed_ms": 13477, + "probabilities": { + "3": 0.87, + "2": 0.07, + "5": 0.05, + "4": 0.01, + "6": 0.0, + "7": 0.0 + }, + "candidates": { + "3": "London, United Kingdom", + "2": "Destination, Select multiple airports", + "5": "Heathrow Airport (LHR)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)", + "7": "London Gatwick Airport (LGW)" + }, + "text": null + }, + { + "tick": 7, + "probe_ms": 757, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 283, + "input_tokens": 5795, + "output_tokens": 340, + "operation": "TYPE_TEXT", + "target": "Departure", + "confidence": 0.46, + "elements": 25, + "elapsed_ms": 15721, + "probabilities": { + "18": 1.0, + "15": 0.0, + "17": 0.0 + }, + "candidates": { + "18": "Departure", + "15": "Where from?", + "17": "Where to?" + }, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 8, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 282, + "input_tokens": 5890, + "output_tokens": 347, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.59, + "elements": 25, + "elapsed_ms": 16930, + "probabilities": { + "18": 0.63, + "19": 0.29, + "12": 0.04, + "13": 0.02, + "15": 0.01, + "17": 0.01 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "12": "Change ticket type. One way", + "13": "1 passenger", + "15": "Where from?", + "17": "Where to?" + }, + "text": null + }, + { + "tick": 9, + "probe_ms": 1686, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 314, + "input_tokens": 10302, + "output_tokens": 486, + "operation": "CLICK", + "target": "Next", + "confidence": 0.72, + "elements": 49, + "elapsed_ms": 19757, + "probabilities": { + "48": 0.48, + "35": 0.18, + "34": 0.17, + "1": 0.08, + "49": 0.03, + "2": 0.02 + }, + "candidates": { + "48": "Next", + "35": "Monday, November 2, 2026", + "34": "Saturday, October 31, 2026 ????", + "1": "Departure", + "49": "Done.", + "2": "Select your ticket type. One way" + }, + "text": null + }, + { + "tick": 10, + "probe_ms": 1858, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 346, + "input_tokens": 15640, + "output_tokens": 742, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.96, + "elements": 81, + "elapsed_ms": 22710, + "probabilities": { + "35": 0.97, + "81": 0.02, + "1": 0.01, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "35": "Sunday, November 1, 2026", + "81": "Done.", + "1": "Departure", + "2": "Select your ticket type. One way", + "3": "Wednesday, September 30, 2026 , 279 euros", + "4": "Thursday, October 1, 2026 , 182 euros" + }, + "text": null + }, + { + "tick": 11, + "probe_ms": 1850, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 370, + "input_tokens": 17012, + "output_tokens": 781, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.87, + "elements": 82, + "elapsed_ms": 25733, + "probabilities": { + "82": 0.95, + "36": 0.05, + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "2": "Select your ticket type. One way", + "3": "Reset", + "4": "Wednesday, September 30, 2026 , 279 euros" + }, + "text": null + }, + { + "tick": 12, + "probe_ms": 751, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 274, + "input_tokens": 6026, + "output_tokens": 347, + "operation": "CLICK", + "target": "Search", + "confidence": 0.91, + "elements": 25, + "elapsed_ms": 27481, + "probabilities": { + "19": 0.97, + "6": 0.01, + "13": 0.01, + "15": 0.01, + "1": 0.0, + "2": 0.0 + }, + "candidates": { + "19": "Search", + "6": "Flights", + "13": "1 passenger", + "15": "Where from?", + "1": "Main menu", + "2": "Google" + }, + "text": null + }, + { + "tick": 13, + "probe_ms": 932, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 373, + "input_tokens": 8831, + "output_tokens": 498, + "operation": "DONE", + "target": null, + "confidence": 0.93, + "elements": 48, + "elapsed_ms": 30889, + "probabilities": { + "DONE": 0.94, + "TYPE_TEXT": 0.03, + "WAIT": 0.01, + "CLICK": 0.01, + "PRESS_ENTER": 0.01, + "BLOCKED": 0.0 + }, + "candidates": { + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "WAIT": "Give the page time to finish updating.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "BLOCKED": "None of the offered operations can move the task forward." + } + } + ], + "final": { + "status": "DONE", + "reason": "", + "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", + "title": "Zürich to London | Google Flights", + "steps": 12, + "elapsed_ms": 32673, + "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", + "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM." + }, + "usage": { + "decisions": 13, + "chat_calls": 13, + "chat_input_tokens": 6497, + "chat_output_tokens": 129, + "chat_cache_tokens": 0, + "jev_input_tokens": 95564, + "cost_usd": 0.006285 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-36-07/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-36-07/answer.json new file mode 100644 index 0000000..6599071 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-36-07/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":96.699,\"remaining_s\":143.301,\"invocation_remaining_s\":145.246},\"summary\":\"The form phase budget has been exhausted. I have successfully navigated to the Google Flights page, entered Zurich as the origin, London as the destination, selected one-way, and set the date to November 1, 2026. The flight options should now be visible on the page.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 97696, + "usage": { + "decisions": 25, + "chat_calls": 12, + "chat_input_tokens": 19526, + "chat_output_tokens": 126, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006173 + }, + "report": { + "elapsed_ms": 97060, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1440, + "jev_input_tokens": 0, + "median_probe_ms": 761, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-36-07/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-36-07/decision_ticks.json new file mode 100644 index 0000000..b36169b --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-36-07/decision_ticks.json @@ -0,0 +1,1014 @@ +{ + "report": { + "elapsed_ms": 97060, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1440, + "jev_input_tokens": 0, + "median_probe_ms": 761, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 713, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3111, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.9, + "elements": 25, + "elapsed_ms": 6481, + "probabilities": { + "15": 1.0, + "17": 0.0, + "16": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "17": "Departure", + "16": "Where to?", + "18": "Return" + }, + "screenshot_ms": 1189, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1289, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 717, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.98, + "elements": 10, + "elapsed_ms": 9301, + "probabilities": { + "3": 0.948, + "5": 0.04, + "2": 0.006, + "4": 0.004, + "6": 0.001, + "9": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 680, + "text": null + }, + { + "tick": 3, + "probe_ms": 758, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1052, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.753, + "elements": 25, + "elapsed_ms": 13137, + "probabilities": { + "16": 0.994, + "17": 0.003, + "18": 0.002, + "15": 0.001 + }, + "candidates": { + "16": "Where to?", + "17": "Departure", + "18": "Return", + "15": "Where from?" + }, + "screenshot_ms": 783, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1264, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 741, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.93, + "elements": 9, + "elapsed_ms": 16100, + "probabilities": { + "3": 0.787, + "5": 0.163, + "2": 0.027, + "7": 0.012, + "4": 0.009, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 780, + "text": null + }, + { + "tick": 5, + "probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1100, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.594, + "elements": 26, + "elapsed_ms": 19938, + "probabilities": { + "12": 0.778, + "18": 0.181, + "19": 0.024, + "16": 0.007, + "13": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "13": "1 passenger", + "20": "Search" + }, + "screenshot_ms": 758, + "text": null + }, + { + "tick": 6, + "probe_ms": 762, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 496, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 23150, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 754, + "text": null + }, + { + "tick": 7, + "probe_ms": 761, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1068, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.7, + "elements": 25, + "elapsed_ms": 26956, + "probabilities": { + "18": 0.999, + "19": 0.0, + "14": 0.0, + "9": 0.0, + "12": 0.0, + "13": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "14": "Change seating class. Economy", + "9": "Change appearance", + "12": "Change ticket type. One way", + "13": "1 passenger" + }, + "screenshot_ms": 745, + "text": null + }, + { + "tick": 8, + "probe_ms": 1679, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1770, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.575, + "elements": 49, + "elapsed_ms": 32069, + "probabilities": { + "SCROLL_DOWN": 0.646, + "CLICK": 0.259, + "WAIT": 0.086, + "BLOCKED": 0.008, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 814, + "text": null + }, + { + "tick": 9, + "probe_ms": 1647, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1901, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.352, + "elements": 50, + "elapsed_ms": 36683, + "probabilities": { + "49": 0.513, + "35": 0.139, + "36": 0.102, + "38": 0.048, + "39": 0.038, + "42": 0.026 + }, + "candidates": { + "49": "Next", + "35": "Saturday, October 31, 2026 , 89 euros", + "36": "Monday, November 2, 2026", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 821, + "text": null + }, + { + "tick": 10, + "probe_ms": 1874, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2639, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.989, + "elements": 82, + "elapsed_ms": 42763, + "probabilities": { + "36": 0.936, + "66": 0.014, + "81": 0.011, + "9": 0.006, + "37": 0.004, + "40": 0.003 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "9": "Monday, October 5, 2026 , 138 euros", + "37": "Monday, November 2, 2026", + "40": "Thursday, November 5, 2026" + }, + "screenshot_ms": 809, + "text": null + }, + { + "tick": 11, + "probe_ms": 1334, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2806, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.397, + "elements": 82, + "elapsed_ms": 48527, + "probabilities": { + "SCROLL_DOWN": 0.483, + "CLICK": 0.395, + "PRESS_ENTER": 0.076, + "DONE": 0.023, + "WAIT": 0.021, + "BLOCKED": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 823, + "text": null + }, + { + "tick": 12, + "probe_ms": 827, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 518, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.591, + "elements": 2, + "elapsed_ms": 50979, + "probabilities": { + "2": 1.0 + }, + "candidates": { + "2": "Done. Search for one-way flights, departing on November 1, 2026" + }, + "screenshot_ms": 878, + "text": null + }, + { + "tick": 13, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1440, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.424, + "elements": 28, + "elapsed_ms": 54644, + "probabilities": { + "SCROLL_DOWN": 0.496, + "PRESS_ENTER": 0.176, + "DONE": 0.174, + "WAIT": 0.071, + "CLICK": 0.067, + "SCROLL_UP": 0.014 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 746, + "text": null + }, + { + "tick": 14, + "probe_ms": 756, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 974, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.534, + "elements": 21, + "elapsed_ms": 57347, + "probabilities": { + "SCROLL_UP": 0.612, + "CLICK": 0.161, + "WAIT": 0.157, + "SCROLL_DOWN": 0.052, + "BLOCKED": 0.009, + "DONE": 0.009 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 796, + "text": null + }, + { + "tick": 15, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1442, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.352, + "elements": 28, + "elapsed_ms": 60451, + "probabilities": { + "SCROLL_DOWN": 0.433, + "DONE": 0.165, + "PRESS_ENTER": 0.158, + "CLICK": 0.123, + "WAIT": 0.084, + "SCROLL_UP": 0.036 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 752, + "text": null + }, + { + "tick": 16, + "probe_ms": 1255, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 973, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.47, + "elements": 21, + "elapsed_ms": 63657, + "probabilities": { + "SCROLL_UP": 0.558, + "CLICK": 0.255, + "WAIT": 0.12, + "SCROLL_DOWN": 0.046, + "BLOCKED": 0.019, + "DONE": 0.002 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 815, + "text": null + }, + { + "tick": 17, + "probe_ms": 1259, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1442, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.23, + "elements": 28, + "elapsed_ms": 67258, + "probabilities": { + "SCROLL_DOWN": 0.326, + "DONE": 0.218, + "PRESS_ENTER": 0.193, + "CLICK": 0.115, + "WAIT": 0.088, + "SCROLL_UP": 0.058 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 748, + "text": null + }, + { + "tick": 18, + "probe_ms": 750, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 971, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.366, + "elements": 21, + "elapsed_ms": 69938, + "probabilities": { + "SCROLL_UP": 0.471, + "CLICK": 0.275, + "WAIT": 0.136, + "SCROLL_DOWN": 0.084, + "BLOCKED": 0.032, + "DONE": 0.001 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 800, + "text": null + }, + { + "tick": 19, + "probe_ms": 757, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1442, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.25, + "elements": 28, + "elapsed_ms": 73061, + "probabilities": { + "SCROLL_DOWN": 0.344, + "PRESS_ENTER": 0.191, + "DONE": 0.182, + "CLICK": 0.151, + "WAIT": 0.081, + "SCROLL_UP": 0.05 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 750, + "text": null + }, + { + "tick": 20, + "probe_ms": 746, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 970, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.495, + "elements": 21, + "elapsed_ms": 75755, + "probabilities": { + "SCROLL_UP": 0.58, + "CLICK": 0.26, + "WAIT": 0.082, + "SCROLL_DOWN": 0.045, + "BLOCKED": 0.032, + "DONE": 0.001 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 821, + "text": null + }, + { + "tick": 21, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1443, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.236, + "elements": 28, + "elapsed_ms": 78859, + "probabilities": { + "SCROLL_DOWN": 0.332, + "CLICK": 0.229, + "PRESS_ENTER": 0.172, + "DONE": 0.146, + "WAIT": 0.084, + "SCROLL_UP": 0.035 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 747, + "text": null + }, + { + "tick": 22, + "probe_ms": 757, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 970, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.364, + "elements": 21, + "elapsed_ms": 81533, + "probabilities": { + "SCROLL_UP": 0.47, + "CLICK": 0.319, + "WAIT": 0.092, + "SCROLL_DOWN": 0.061, + "BLOCKED": 0.057, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 793, + "text": null + }, + { + "tick": 23, + "probe_ms": 748, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1443, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Search", + "confidence": 0.293, + "elements": 28, + "elapsed_ms": 84630, + "probabilities": { + "19": 0.7, + "18": 0.271, + "28": 0.008, + "2": 0.003, + "20": 0.002, + "22": 0.002 + }, + "candidates": { + "19": "Search", + "18": "Departure", + "28": "Find flights from Frankfurt am Main (FRA) to London (LGW) from €134. Operated by Condor. Oct 29 to Nov 4. Nonstop", + "2": "Google", + "20": "Explore deals with AI", + "22": "Frankfurt am Main" + }, + "screenshot_ms": 746, + "text": null + }, + { + "tick": 24, + "probe_ms": 939, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1858, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.6, + "elements": 48, + "elapsed_ms": 90988, + "probabilities": { + "SCROLL_DOWN": 0.657, + "DONE": 0.154, + "CLICK": 0.082, + "PRESS_ENTER": 0.057, + "WAIT": 0.047, + "TYPE_TEXT": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "WAIT": "Give the page time to finish updating.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 1474, + "text": null + }, + { + "tick": 25, + "probe_ms": 975, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1478, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.123, + "elements": 44, + "elapsed_ms": 94737, + "probabilities": { + "SCROLL_DOWN": 0.269, + "SCROLL_UP": 0.253, + "CLICK": 0.25, + "WAIT": 0.148, + "DONE": 0.066, + "BLOCKED": 0.014 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 968, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 12, + "chat_input_tokens": 19526, + "chat_output_tokens": 126, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006173 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-38-08/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-38-08/answer.json new file mode 100644 index 0000000..5ea20a7 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-38-08/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":95.371,\"remaining_s\":144.629,\"invocation_remaining_s\":146.025},\"summary\":\"The form phase budget has been exhausted. I have successfully navigated to the Google Flights page and entered the origin, destination, and date for a one-way flight from Zurich to London on November 1, 2026. However, I was unable to confirm the number of adults or the economy class, and the search results are not yet visible.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 96391, + "usage": { + "decisions": 25, + "chat_calls": 12, + "chat_input_tokens": 19527, + "chat_output_tokens": 137, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006201 + }, + "report": { + "elapsed_ms": 95720, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1446, + "jev_input_tokens": 0, + "median_probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-38-08/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-38-08/decision_ticks.json new file mode 100644 index 0000000..c381129 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-38-08/decision_ticks.json @@ -0,0 +1,1014 @@ +{ + "report": { + "elapsed_ms": 95720, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1446, + "jev_input_tokens": 0, + "median_probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 701, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3108, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.898, + "elements": 25, + "elapsed_ms": 6031, + "probabilities": { + "15": 1.0, + "17": 0.0, + "16": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "17": "Departure", + "16": "Where to?", + "18": "Return" + }, + "screenshot_ms": 694, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1284, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 720, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.985, + "elements": 10, + "elapsed_ms": 8860, + "probabilities": { + "3": 0.971, + "5": 0.023, + "2": 0.003, + "4": 0.002, + "6": 0.001, + "9": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 693, + "text": null + }, + { + "tick": 3, + "probe_ms": 758, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1061, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.753, + "elements": 25, + "elapsed_ms": 12761, + "probabilities": { + "16": 0.994, + "17": 0.003, + "18": 0.002, + "15": 0.001 + }, + "candidates": { + "16": "Where to?", + "17": "Departure", + "18": "Return", + "15": "Where from?" + }, + "screenshot_ms": 776, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1275, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 744, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.93, + "elements": 9, + "elapsed_ms": 15744, + "probabilities": { + "3": 0.787, + "5": 0.163, + "2": 0.027, + "7": 0.012, + "4": 0.009, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 788, + "text": null + }, + { + "tick": 5, + "probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1105, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.594, + "elements": 26, + "elapsed_ms": 19582, + "probabilities": { + "12": 0.778, + "18": 0.181, + "19": 0.024, + "16": 0.007, + "13": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "13": "1 passenger", + "20": "Search" + }, + "screenshot_ms": 758, + "text": null + }, + { + "tick": 6, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 491, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 22766, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 753, + "text": null + }, + { + "tick": 7, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1072, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.7, + "elements": 25, + "elapsed_ms": 26600, + "probabilities": { + "18": 0.999, + "19": 0.0, + "14": 0.0, + "9": 0.0, + "12": 0.0, + "13": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "14": "Change seating class. Economy", + "9": "Change appearance", + "12": "Change ticket type. One way", + "13": "1 passenger" + }, + "screenshot_ms": 779, + "text": null + }, + { + "tick": 8, + "probe_ms": 1651, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1781, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.575, + "elements": 49, + "elapsed_ms": 31723, + "probabilities": { + "SCROLL_DOWN": 0.646, + "CLICK": 0.259, + "WAIT": 0.086, + "BLOCKED": 0.008, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 829, + "text": null + }, + { + "tick": 9, + "probe_ms": 1647, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1905, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.352, + "elements": 50, + "elapsed_ms": 36340, + "probabilities": { + "49": 0.513, + "35": 0.139, + "36": 0.102, + "38": 0.048, + "39": 0.038, + "42": 0.026 + }, + "candidates": { + "49": "Next", + "35": "Saturday, October 31, 2026 , 89 euros", + "36": "Monday, November 2, 2026", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 820, + "text": null + }, + { + "tick": 10, + "probe_ms": 1859, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2648, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.989, + "elements": 82, + "elapsed_ms": 42430, + "probabilities": { + "36": 0.936, + "66": 0.014, + "81": 0.011, + "9": 0.006, + "37": 0.004, + "40": 0.003 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "9": "Monday, October 5, 2026 , 138 euros", + "37": "Monday, November 2, 2026", + "40": "Thursday, November 5, 2026" + }, + "screenshot_ms": 806, + "text": null + }, + { + "tick": 11, + "probe_ms": 1319, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2822, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.397, + "elements": 82, + "elapsed_ms": 48189, + "probabilities": { + "SCROLL_DOWN": 0.483, + "CLICK": 0.395, + "PRESS_ENTER": 0.076, + "DONE": 0.023, + "WAIT": 0.021, + "BLOCKED": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 828, + "text": null + }, + { + "tick": 12, + "probe_ms": 822, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 518, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.591, + "elements": 2, + "elapsed_ms": 50609, + "probabilities": { + "2": 1.0 + }, + "candidates": { + "2": "Done. Search for one-way flights, departing on November 1, 2026" + }, + "screenshot_ms": 852, + "text": null + }, + { + "tick": 13, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1448, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.424, + "elements": 28, + "elapsed_ms": 54273, + "probabilities": { + "SCROLL_DOWN": 0.496, + "PRESS_ENTER": 0.176, + "DONE": 0.174, + "WAIT": 0.071, + "CLICK": 0.067, + "SCROLL_UP": 0.014 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 754, + "text": null + }, + { + "tick": 14, + "probe_ms": 746, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 978, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.534, + "elements": 21, + "elapsed_ms": 56975, + "probabilities": { + "SCROLL_UP": 0.612, + "CLICK": 0.161, + "WAIT": 0.157, + "SCROLL_DOWN": 0.052, + "BLOCKED": 0.009, + "DONE": 0.009 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 821, + "text": null + }, + { + "tick": 15, + "probe_ms": 751, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1447, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.352, + "elements": 28, + "elapsed_ms": 60089, + "probabilities": { + "SCROLL_DOWN": 0.433, + "DONE": 0.165, + "PRESS_ENTER": 0.158, + "CLICK": 0.123, + "WAIT": 0.084, + "SCROLL_UP": 0.036 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 761, + "text": null + }, + { + "tick": 16, + "probe_ms": 1258, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 976, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.47, + "elements": 21, + "elapsed_ms": 63293, + "probabilities": { + "SCROLL_UP": 0.558, + "CLICK": 0.255, + "WAIT": 0.12, + "SCROLL_DOWN": 0.046, + "BLOCKED": 0.019, + "DONE": 0.002 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 812, + "text": null + }, + { + "tick": 17, + "probe_ms": 751, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1446, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.23, + "elements": 28, + "elapsed_ms": 66404, + "probabilities": { + "SCROLL_DOWN": 0.326, + "DONE": 0.218, + "PRESS_ENTER": 0.193, + "CLICK": 0.115, + "WAIT": 0.088, + "SCROLL_UP": 0.058 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 758, + "text": null + }, + { + "tick": 18, + "probe_ms": 760, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 978, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.366, + "elements": 21, + "elapsed_ms": 69115, + "probabilities": { + "SCROLL_UP": 0.471, + "CLICK": 0.275, + "WAIT": 0.136, + "SCROLL_DOWN": 0.084, + "BLOCKED": 0.032, + "DONE": 0.001 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 808, + "text": null + }, + { + "tick": 19, + "probe_ms": 751, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1450, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.25, + "elements": 28, + "elapsed_ms": 72233, + "probabilities": { + "SCROLL_DOWN": 0.344, + "PRESS_ENTER": 0.191, + "DONE": 0.182, + "CLICK": 0.151, + "WAIT": 0.081, + "SCROLL_UP": 0.05 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 763, + "text": null + }, + { + "tick": 20, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 972, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.495, + "elements": 21, + "elapsed_ms": 74909, + "probabilities": { + "SCROLL_UP": 0.58, + "CLICK": 0.26, + "WAIT": 0.082, + "SCROLL_DOWN": 0.045, + "BLOCKED": 0.032, + "DONE": 0.001 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 793, + "text": null + }, + { + "tick": 21, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1451, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.236, + "elements": 28, + "elapsed_ms": 78030, + "probabilities": { + "SCROLL_DOWN": 0.332, + "CLICK": 0.229, + "PRESS_ENTER": 0.172, + "DONE": 0.146, + "WAIT": 0.084, + "SCROLL_UP": 0.035 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 756, + "text": null + }, + { + "tick": 22, + "probe_ms": 747, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 972, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.364, + "elements": 21, + "elapsed_ms": 80703, + "probabilities": { + "SCROLL_UP": 0.47, + "CLICK": 0.319, + "WAIT": 0.092, + "SCROLL_DOWN": 0.061, + "BLOCKED": 0.057, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 789, + "text": null + }, + { + "tick": 23, + "probe_ms": 752, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1446, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Search", + "confidence": 0.293, + "elements": 28, + "elapsed_ms": 83818, + "probabilities": { + "19": 0.7, + "18": 0.271, + "28": 0.008, + "2": 0.003, + "20": 0.002, + "22": 0.002 + }, + "candidates": { + "19": "Search", + "18": "Departure", + "28": "Find flights from Frankfurt am Main (FRA) to London (LGW) from €134. Operated by Condor. Oct 29 to Nov 4. Nonstop", + "2": "Google", + "20": "Explore deals with AI", + "22": "Frankfurt am Main" + }, + "screenshot_ms": 751, + "text": null + }, + { + "tick": 24, + "probe_ms": 935, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1855, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.6, + "elements": 48, + "elapsed_ms": 90202, + "probabilities": { + "SCROLL_DOWN": 0.657, + "DONE": 0.154, + "CLICK": 0.082, + "PRESS_ENTER": 0.057, + "WAIT": 0.047, + "TYPE_TEXT": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "WAIT": "Give the page time to finish updating.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 1492, + "text": null + }, + { + "tick": 25, + "probe_ms": 989, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1488, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.123, + "elements": 44, + "elapsed_ms": 93959, + "probabilities": { + "SCROLL_DOWN": 0.269, + "SCROLL_UP": 0.253, + "CLICK": 0.25, + "WAIT": 0.148, + "DONE": 0.066, + "BLOCKED": 0.014 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 968, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 12, + "chat_input_tokens": 19527, + "chat_output_tokens": 137, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006201 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-40-07/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-40-07/answer.json new file mode 100644 index 0000000..bf9026f --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-40-07/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARocagwIAxIIL20vMDg5NjZyDAgCEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&tfu=KgIIBA\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":101.382,\"remaining_s\":138.618,\"invocation_remaining_s\":140.162},\"summary\":\"The form phase budget has been exhausted. I have successfully entered the origin and destination, selected one-way, and opened the date picker. However, I was unable to select the date and complete the search due to the budget exhaustion.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 102364, + "usage": { + "decisions": 25, + "chat_calls": 18, + "chat_input_tokens": 32627, + "chat_output_tokens": 154, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.010173 + }, + "report": { + "elapsed_ms": 101730, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1016, + "jev_input_tokens": 0, + "median_probe_ms": 777, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Flights from popular destinations", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Popular flights routes", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights from popular destinations", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights to popular destinations", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights to London", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-40-07/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-40-07/decision_ticks.json new file mode 100644 index 0000000..ddcb916 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-40-07/decision_ticks.json @@ -0,0 +1,1007 @@ +{ + "report": { + "elapsed_ms": 101730, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1016, + "jev_input_tokens": 0, + "median_probe_ms": 777, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 3, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Flights from popular destinations", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Popular flights routes", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights from popular destinations", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights to popular destinations", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Flights to London", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 1200, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3019, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.899, + "elements": 20, + "elapsed_ms": 6492, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 690, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1413, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 704, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.984, + "elements": 10, + "elapsed_ms": 9414, + "probabilities": { + "3": 0.957, + "5": 0.033, + "2": 0.006, + "4": 0.003, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 682, + "text": null + }, + { + "tick": 3, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 940, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.478, + "elements": 20, + "elapsed_ms": 13091, + "probabilities": { + "16": 0.996, + "15": 0.001, + "17": 0.001, + "18": 0.001 + }, + "candidates": { + "16": "Where to?", + "15": "Where from?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 761, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1270, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 738, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.95, + "elements": 9, + "elapsed_ms": 16031, + "probabilities": { + "3": 0.754, + "5": 0.201, + "2": 0.023, + "7": 0.012, + "4": 0.006, + "9": 0.002 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "9": "London City Airport (LCY)" + }, + "screenshot_ms": 768, + "text": null + }, + { + "tick": 5, + "probe_ms": 742, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 992, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.685, + "elements": 21, + "elapsed_ms": 19714, + "probabilities": { + "12": 0.924, + "18": 0.059, + "16": 0.008, + "19": 0.006, + "13": 0.001, + "15": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "16": "Swap origin and destination.", + "19": "Return", + "13": "1 passenger", + "15": "Where from?" + }, + "screenshot_ms": 739, + "text": null + }, + { + "tick": 6, + "probe_ms": 733, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 490, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 22893, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 755, + "text": null + }, + { + "tick": 7, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 947, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.704, + "elements": 20, + "elapsed_ms": 26555, + "probabilities": { + "18": 0.999, + "14": 0.0, + "19": 0.0, + "12": 0.0, + "1": 0.0, + "2": 0.0 + }, + "candidates": { + "18": "Departure", + "14": "Change seating class. Economy", + "19": "Search", + "12": "Change ticket type. One way", + "1": "Main menu", + "2": "Google" + }, + "screenshot_ms": 749, + "text": null + }, + { + "tick": 8, + "probe_ms": 1658, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1783, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.546, + "elements": 49, + "elapsed_ms": 31668, + "probabilities": { + "SCROLL_DOWN": 0.622, + "CLICK": 0.286, + "WAIT": 0.082, + "BLOCKED": 0.009, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 809, + "text": null + }, + { + "tick": 9, + "probe_ms": 1128, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1906, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.395, + "elements": 50, + "elapsed_ms": 35757, + "probabilities": { + "49": 0.527, + "35": 0.149, + "36": 0.1, + "38": 0.042, + "39": 0.033, + "42": 0.024 + }, + "candidates": { + "49": "Next", + "35": "Saturday, October 31, 2026 , 89 euros", + "36": "Monday, November 2, 2026", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 809, + "text": null + }, + { + "tick": 10, + "probe_ms": 1855, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2654, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.992, + "elements": 82, + "elapsed_ms": 41812, + "probabilities": { + "36": 0.925, + "66": 0.02, + "81": 0.012, + "9": 0.005, + "37": 0.005, + "67": 0.004 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "9": "Monday, October 5, 2026 , 138 euros", + "37": "Monday, November 2, 2026", + "67": "Wednesday, December 2, 2026" + }, + "screenshot_ms": 793, + "text": null + }, + { + "tick": 11, + "probe_ms": 1304, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2836, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.391, + "elements": 82, + "elapsed_ms": 47549, + "probabilities": { + "SCROLL_DOWN": 0.478, + "CLICK": 0.399, + "PRESS_ENTER": 0.073, + "DONE": 0.026, + "WAIT": 0.023, + "BLOCKED": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 815, + "text": null + }, + { + "tick": 12, + "probe_ms": 818, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 516, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.36, + "elements": 2, + "elapsed_ms": 49934, + "probabilities": { + "2": 1.0 + }, + "candidates": { + "2": "Done. Search for one-way flights, departing on November 1, 2026" + }, + "screenshot_ms": 839, + "text": null + }, + { + "tick": 13, + "probe_ms": 746, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1009, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.552, + "elements": 20, + "elapsed_ms": 53183, + "probabilities": { + "SCROLL_DOWN": 0.608, + "PRESS_ENTER": 0.189, + "CLICK": 0.107, + "WAIT": 0.046, + "DONE": 0.033, + "SCROLL_UP": 0.015 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 791, + "text": null + }, + { + "tick": 14, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 838, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.469, + "elements": 15, + "elapsed_ms": 55681, + "probabilities": { + "SCROLL_DOWN": 0.557, + "WAIT": 0.253, + "SCROLL_UP": 0.11, + "CLICK": 0.058, + "DONE": 0.021, + "BLOCKED": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 761, + "text": null + }, + { + "tick": 15, + "probe_ms": 741, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1286, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights from popular destinations", + "confidence": 0.245, + "elements": 33, + "elapsed_ms": 58601, + "probabilities": { + "19": 0.149, + "20": 0.137, + "1": 0.097, + "17": 0.057, + "21": 0.056, + "23": 0.054 + }, + "candidates": { + "19": "Flights from popular destinations", + "20": "Flights to popular destinations", + "1": "Main menu", + "17": "How can I find cheap flights to anywhere?", + "21": "Popular flights routes", + "23": "Flights from Paris" + }, + "screenshot_ms": 730, + "text": null + }, + { + "tick": 16, + "probe_ms": 4305, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1285, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Popular flights routes", + "confidence": 0.288, + "elements": 33, + "elapsed_ms": 66121, + "probabilities": { + "21": 0.241, + "20": 0.203, + "24": 0.078, + "19": 0.066, + "11": 0.049, + "31": 0.047 + }, + "candidates": { + "21": "Popular flights routes", + "20": "Flights to popular destinations", + "24": "Flights from London", + "19": "Flights from popular destinations", + "11": "Sign in", + "31": "Flights from Berlin" + }, + "screenshot_ms": 1246, + "text": null + }, + { + "tick": 17, + "probe_ms": 761, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1294, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights from popular destinations", + "confidence": 0.401, + "elements": 33, + "elapsed_ms": 69632, + "probabilities": { + "19": 0.207, + "1": 0.114, + "22": 0.103, + "28": 0.088, + "17": 0.087, + "31": 0.065 + }, + "candidates": { + "19": "Flights from popular destinations", + "1": "Main menu", + "22": "Flights from New York to London", + "28": "Flights from Toronto to London", + "17": "How can I find cheap flights to anywhere?", + "31": "Flights from London to Tokyo" + }, + "screenshot_ms": 775, + "text": null + }, + { + "tick": 18, + "probe_ms": 744, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1288, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights to popular destinations", + "confidence": 0.417, + "elements": 33, + "elapsed_ms": 73081, + "probabilities": { + "20": 0.122, + "1": 0.116, + "17": 0.093, + "31": 0.07, + "23": 0.061, + "13": 0.061 + }, + "candidates": { + "20": "Flights to popular destinations", + "1": "Main menu", + "17": "How can I find cheap flights to anywhere?", + "31": "Flights from Berlin", + "23": "Flights from Paris", + "13": "Explore the best flight deals with AI Describe your ideal trip and let Google Flights find you the best deals" + }, + "screenshot_ms": 736, + "text": null + }, + { + "tick": 19, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1283, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Flights to London", + "confidence": 0.371, + "elements": 33, + "elapsed_ms": 76541, + "probabilities": { + "24": 0.828, + "21": 0.032, + "11": 0.02, + "19": 0.017, + "20": 0.013, + "22": 0.011 + }, + "candidates": { + "24": "Flights to London", + "21": "Popular flights routes", + "11": "Sign in", + "19": "Flights from popular destinations", + "20": "Flights to popular destinations", + "22": "Flights to Paris" + }, + "screenshot_ms": 729, + "text": null + }, + { + "tick": 20, + "probe_ms": 1198, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1016, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.582, + "elements": 22, + "elapsed_ms": 80944, + "probabilities": { + "15": 0.999, + "17": 0.0, + "18": 0.0, + "16": 0.0 + }, + "candidates": { + "15": "Where from?", + "17": "Departure", + "18": "Return", + "16": "Where to?" + }, + "screenshot_ms": 685, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 21, + "probe_ms": 1290, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 735, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.95, + "elements": 10, + "elapsed_ms": 83813, + "probabilities": { + "3": 0.786, + "5": 0.175, + "2": 0.018, + "4": 0.016, + "6": 0.004, + "9": 0.001 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 683, + "text": null + }, + { + "tick": 22, + "probe_ms": 758, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1092, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.427, + "elements": 23, + "elapsed_ms": 87672, + "probabilities": { + "12": 0.591, + "18": 0.299, + "19": 0.029, + "20": 0.021, + "16": 0.011, + "6": 0.011 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "20": "Search", + "16": "Swap origin and destination.", + "6": "Flights" + }, + "screenshot_ms": 764, + "text": null + }, + { + "tick": 23, + "probe_ms": 747, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 503, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.997, + "elements": 3, + "elapsed_ms": 90908, + "probabilities": { + "2": 0.994, + "1": 0.006, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 768, + "text": null + }, + { + "tick": 24, + "probe_ms": 777, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1016, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.578, + "elements": 22, + "elapsed_ms": 94696, + "probabilities": { + "18": 0.997, + "19": 0.001, + "12": 0.001, + "15": 0.0, + "14": 0.0, + "21": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "12": "Change ticket type. One way", + "15": "Where from?", + "14": "Change seating class. Economy", + "21": "To United Kingdom" + }, + "screenshot_ms": 760, + "text": null + }, + { + "tick": 25, + "probe_ms": 1673, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1780, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.362, + "elements": 49, + "elapsed_ms": 99822, + "probabilities": { + "SCROLL_DOWN": 0.468, + "CLICK": 0.404, + "WAIT": 0.119, + "BLOCKED": 0.008, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 828, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 18, + "chat_input_tokens": 32627, + "chat_output_tokens": 154, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.010173 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-42-13/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-42-13/answer.json new file mode 100644 index 0000000..f0d42f7 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-42-13/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights?tfs=CBwQARooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&tfu=KgIIAw&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":121.594,\"remaining_s\":118.406,\"invocation_remaining_s\":120.591},\"summary\":\"The form phase budget has been exhausted. I have successfully entered the origin (Zurich), destination (London), and selected a one-way trip for November 1, 2026. However, I was unable to confirm the number of adults or the economy class. The runtime has indicated that the form phase budget is exhausted, so I cannot proceed with further interactions on the form.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 122595, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 30274, + "chat_output_tokens": 126, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.009397 + }, + "report": { + "elapsed_ms": 121937, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1061, + "jev_input_tokens": 0, + "median_probe_ms": 777, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-42-13/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-42-13/decision_ticks.json new file mode 100644 index 0000000..0726931 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-42-13/decision_ticks.json @@ -0,0 +1,1016 @@ +{ + "report": { + "elapsed_ms": 121937, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1061, + "jev_input_tokens": 0, + "median_probe_ms": 777, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Frankfurt am Main", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Munich", + "kind": "click", + "text": null, + "page_changed": false + }, + { + "action": "Main menu", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change location", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "OK", + "kind": "click", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 701, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3134, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.901, + "elements": 25, + "elapsed_ms": 6490, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 1225, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1279, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 718, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.986, + "elements": 10, + "elapsed_ms": 9300, + "probabilities": { + "3": 0.976, + "5": 0.017, + "2": 0.004, + "4": 0.002, + "6": 0.001, + "7": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "7": "Lake Zurich, Illinois, USA" + }, + "screenshot_ms": 680, + "text": null + }, + { + "tick": 3, + "probe_ms": 777, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1061, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.753, + "elements": 25, + "elapsed_ms": 13219, + "probabilities": { + "16": 0.994, + "17": 0.003, + "18": 0.002, + "15": 0.001 + }, + "candidates": { + "16": "Where to?", + "17": "Departure", + "18": "Return", + "15": "Where from?" + }, + "screenshot_ms": 765, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1274, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 745, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.93, + "elements": 9, + "elapsed_ms": 16197, + "probabilities": { + "3": 0.787, + "5": 0.163, + "2": 0.027, + "7": 0.012, + "4": 0.009, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 778, + "text": null + }, + { + "tick": 5, + "probe_ms": 759, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1108, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.594, + "elements": 26, + "elapsed_ms": 20062, + "probabilities": { + "12": 0.778, + "18": 0.181, + "19": 0.024, + "16": 0.007, + "13": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "13": "1 passenger", + "20": "Search" + }, + "screenshot_ms": 766, + "text": null + }, + { + "tick": 6, + "probe_ms": 745, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 489, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 23260, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 758, + "text": null + }, + { + "tick": 7, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1074, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.7, + "elements": 25, + "elapsed_ms": 27086, + "probabilities": { + "18": 0.999, + "19": 0.0, + "14": 0.0, + "9": 0.0, + "12": 0.0, + "13": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "14": "Change seating class. Economy", + "9": "Change appearance", + "12": "Change ticket type. One way", + "13": "1 passenger" + }, + "screenshot_ms": 765, + "text": null + }, + { + "tick": 8, + "probe_ms": 1680, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1782, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.575, + "elements": 49, + "elapsed_ms": 32263, + "probabilities": { + "SCROLL_DOWN": 0.646, + "CLICK": 0.259, + "WAIT": 0.086, + "BLOCKED": 0.008, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 857, + "text": null + }, + { + "tick": 9, + "probe_ms": 1643, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1909, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.352, + "elements": 50, + "elapsed_ms": 36888, + "probabilities": { + "49": 0.513, + "35": 0.139, + "36": 0.102, + "38": 0.048, + "39": 0.038, + "42": 0.026 + }, + "candidates": { + "49": "Next", + "35": "Saturday, October 31, 2026 , 89 euros", + "36": "Monday, November 2, 2026", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 823, + "text": null + }, + { + "tick": 10, + "probe_ms": 6882, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2656, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.99, + "elements": 82, + "elapsed_ms": 48002, + "probabilities": { + "36": 0.914, + "66": 0.015, + "81": 0.013, + "9": 0.009, + "40": 0.008, + "37": 0.007 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "9": "Monday, October 5, 2026 , 138 euros", + "40": "Thursday, November 5, 2026", + "37": "Monday, November 2, 2026" + }, + "screenshot_ms": 812, + "text": null + }, + { + "tick": 11, + "probe_ms": 1357, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2874, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.329, + "elements": 82, + "elapsed_ms": 59367, + "probabilities": { + "82": 0.969, + "36": 0.022, + "1": 0.002, + "61": 0.001, + "62": 0.001, + "38": 0.0 + }, + "candidates": { + "82": "Done. Search for one-way flights, departing on November 1, 2026", + "36": "Sunday, November 1, 2026, departure date. , 74 euros", + "1": "Departure", + "61": "Thursday, November 26, 2026 ????", + "62": "Friday, November 27, 2026 , 27 euros, Cheapest price", + "38": "Tuesday, November 3, 2026 , 38 euros" + }, + "screenshot_ms": 814, + "text": null + }, + { + "tick": 12, + "probe_ms": 767, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1116, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.707, + "elements": 25, + "elapsed_ms": 63241, + "probabilities": { + "SCROLL_DOWN": 0.749, + "PRESS_ENTER": 0.134, + "CLICK": 0.074, + "WAIT": 0.021, + "DONE": 0.021, + "BLOCKED": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 1266, + "text": null + }, + { + "tick": 13, + "probe_ms": 773, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1036, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.449, + "elements": 21, + "elapsed_ms": 66038, + "probabilities": { + "13": 0.804, + "1": 0.059, + "21": 0.054, + "6": 0.021, + "19": 0.014, + "16": 0.009 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "19": "Find flights from Frankfurt am Main (FRA) to London (LGW) from €134. Operated by Condor. Oct 29 to Nov 4. Nonstop", + "16": "Düsseldorf" + }, + "screenshot_ms": 822, + "text": null + }, + { + "tick": 14, + "probe_ms": 4330, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1033, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Frankfurt am Main", + "confidence": 0.518, + "elements": 21, + "elapsed_ms": 72925, + "probabilities": { + "13": 0.748, + "1": 0.102, + "21": 0.049, + "6": 0.028, + "16": 0.014, + "2": 0.01 + }, + "candidates": { + "13": "Frankfurt am Main", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 817, + "text": null + }, + { + "tick": 15, + "probe_ms": 3835, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1016, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.642, + "elements": 21, + "elapsed_ms": 79276, + "probabilities": { + "1": 0.369, + "21": 0.208, + "6": 0.096, + "14": 0.093, + "16": 0.072, + "10": 0.03 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "14": "Munich", + "16": "Düsseldorf", + "10": "Google apps" + }, + "screenshot_ms": 804, + "text": null + }, + { + "tick": 16, + "probe_ms": 750, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1358, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.745, + "elements": 33, + "elapsed_ms": 82882, + "probabilities": { + "1": 0.294, + "26": 0.207, + "33": 0.058, + "2": 0.052, + "6": 0.05, + "13": 0.045 + }, + "candidates": { + "1": "Main menu", + "26": "Munich", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "2": "Google", + "6": "Flights", + "13": "Selected Tab: Flights" + }, + "screenshot_ms": 801, + "text": null + }, + { + "tick": 17, + "probe_ms": 748, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1015, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.572, + "elements": 21, + "elapsed_ms": 86134, + "probabilities": { + "1": 0.329, + "21": 0.227, + "14": 0.116, + "6": 0.102, + "16": 0.068, + "2": 0.03 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "14": "Munich", + "6": "Flights", + "16": "Düsseldorf", + "2": "Google" + }, + "screenshot_ms": 799, + "text": null + }, + { + "tick": 18, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1358, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.679, + "elements": 33, + "elapsed_ms": 89755, + "probabilities": { + "26": 0.183, + "1": 0.15, + "33": 0.092, + "6": 0.074, + "2": 0.061, + "12": 0.059 + }, + "candidates": { + "26": "Munich", + "1": "Main menu", + "33": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "2": "Google", + "12": "Explore" + }, + "screenshot_ms": 816, + "text": null + }, + { + "tick": 19, + "probe_ms": 1264, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1017, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.198, + "elements": 21, + "elapsed_ms": 94054, + "probabilities": { + "14": 0.461, + "21": 0.201, + "1": 0.144, + "6": 0.045, + "2": 0.029, + "10": 0.022 + }, + "candidates": { + "14": "Munich", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "1": "Main menu", + "6": "Flights", + "2": "Google", + "10": "Google apps" + }, + "screenshot_ms": 811, + "text": null + }, + { + "tick": 20, + "probe_ms": 3848, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 1014, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.43, + "elements": 21, + "elapsed_ms": 100402, + "probabilities": { + "SCROLL_UP": 0.525, + "CLICK": 0.285, + "SCROLL_DOWN": 0.08, + "WAIT": 0.065, + "BLOCKED": 0.044, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_DOWN": "Move the viewport down the page.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 802, + "text": null + }, + { + "tick": 21, + "probe_ms": 773, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1100, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.235, + "elements": 25, + "elapsed_ms": 103214, + "probabilities": { + "SCROLL_DOWN": 0.344, + "PRESS_ENTER": 0.261, + "CLICK": 0.245, + "WAIT": 0.096, + "DONE": 0.05, + "TYPE_TEXT": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 766, + "text": null + }, + { + "tick": 22, + "probe_ms": 772, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1014, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Munich", + "confidence": 0.631, + "elements": 21, + "elapsed_ms": 105981, + "probabilities": { + "14": 0.424, + "1": 0.213, + "21": 0.094, + "6": 0.072, + "10": 0.041, + "2": 0.035 + }, + "candidates": { + "14": "Munich", + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "10": "Google apps", + "2": "Google" + }, + "screenshot_ms": 820, + "text": null + }, + { + "tick": 23, + "probe_ms": 3845, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 3, + "action_settle_ms": 1000, + "decision_ms": 984, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Main menu", + "confidence": 0.624, + "elements": 21, + "elapsed_ms": 112309, + "probabilities": { + "1": 0.25, + "21": 0.17, + "6": 0.154, + "15": 0.112, + "2": 0.073, + "16": 0.067 + }, + "candidates": { + "1": "Main menu", + "21": "Find the cheapest days to fly The Date grid and Price graph make it easy to find the best flight deals", + "6": "Flights", + "15": "Berlin", + "2": "Google", + "16": "Düsseldorf" + }, + "screenshot_ms": 812, + "text": null + }, + { + "tick": 24, + "probe_ms": 748, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1311, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change location", + "confidence": 0.748, + "elements": 33, + "elapsed_ms": 115885, + "probabilities": { + "20": 0.24, + "6": 0.132, + "1": 0.103, + "13": 0.075, + "5": 0.072, + "28": 0.066 + }, + "candidates": { + "20": "Change location", + "6": "Flights", + "1": "Main menu", + "13": "Selected Tab: Flights", + "5": "Explore", + "28": "Düsseldorf" + }, + "screenshot_ms": 821, + "text": null + }, + { + "tick": 25, + "probe_ms": 822, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 470, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "OK", + "confidence": 0.902, + "elements": 2, + "elapsed_ms": 119392, + "probabilities": { + "2": 0.96, + "1": 0.04 + }, + "candidates": { + "2": "OK", + "1": "Cancel" + }, + "screenshot_ms": 888, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 9, + "chat_input_tokens": 30274, + "chat_output_tokens": 126, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.009397 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-44-39/answer.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-44-39/answer.json new file mode 100644 index 0000000..b0e5432 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-44-39/answer.json @@ -0,0 +1,186 @@ +{ + "ok": false, + "final": "{\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"requested_slots\":[],\"blockers\":[\"form_phase_budget_exhausted\"],\"field_coverage\":[],\"evidence\":[],\"observations\":[],\"current_page\":{\"url\":\"https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en\",\"title\":\"Find Cheap Flights Worldwide & Book Your Ticket - Google Flights\"},\"requested_result_count\":1,\"observed_result_count\":0,\"terminal_reason\":null,\"retryable\":false,\"recommended_recovery\":\"collect_missing_evidence_from_current_page\",\"resume_count\":0,\"deadline\":{\"budget_s\":240.0,\"elapsed_s\":97.03,\"remaining_s\":142.97,\"invocation_remaining_s\":144.853},\"summary\":\"The form phase budget has been exhausted. I have successfully navigated to the Google Flights page, entered the origin and destination, selected one-way, and set the date to November 1, 2026. The search results for flights are now visible.\"}}", + "screenshot": null, + "error": "result_type='error': {\"browser_result\":{\"status\":\"partial\",\"task_id\":\"0538807d19be0b84\",\"current_phase\":\"form\",\"missing_fields\":[\"result_count:0/1\"],\"unverified_fields\":[],\"missing_slots\":[],\"unavailable_slots\":[],\"reques", + "elapsed_ms": 98041, + "usage": { + "decisions": 25, + "chat_calls": 12, + "chat_input_tokens": 19528, + "chat_output_tokens": 116, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006148 + }, + "report": { + "elapsed_ms": 97373, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1451, + "jev_input_tokens": 0, + "median_probe_ms": 763, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": null + } + ] + }, + "agent": "flights", + "model": "omnijev" +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-44-39/decision_ticks.json b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-44-39/decision_ticks.json new file mode 100644 index 0000000..07b70dc --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/2026-09-30__15-44-39/decision_ticks.json @@ -0,0 +1,1014 @@ +{ + "report": { + "elapsed_ms": 97373, + "decisions": 25, + "interactions": 25, + "waits": 0, + "median_decision_ms": 1451, + "jev_input_tokens": 0, + "median_probe_ms": 763, + "settle_probes": 0, + "settle_ms": 0, + "values": { + "cache": 0, + "prefetch": 2, + "llm": 0 + }, + "history": [ + { + "action": "Where from?", + "kind": "fill", + "text": "Zurich", + "page_changed": true + }, + { + "action": "Zürich, Switzerland", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Where to?", + "kind": "fill", + "text": "London", + "page_changed": true + }, + { + "action": "London, United Kingdom", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Change ticket type. Round trip", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "One way", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Departure", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Next", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "Sunday, November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Done. Search for one-way flights, departing on November 1, 2026", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_UP", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "Search", + "kind": "click", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": true + }, + { + "action": "SCROLL_DOWN", + "kind": "scroll", + "text": null, + "page_changed": null + } + ] + }, + "ticks": [ + { + "tick": 1, + "probe_ms": 700, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 3129, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where from?", + "confidence": 0.901, + "elements": 25, + "elapsed_ms": 6487, + "probabilities": { + "15": 1.0, + "16": 0.0, + "17": 0.0, + "18": 0.0 + }, + "candidates": { + "15": "Where from?", + "16": "Where to?", + "17": "Departure", + "18": "Return" + }, + "screenshot_ms": 1194, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "Zurich" + }, + { + "tick": 2, + "probe_ms": 1279, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 722, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Zürich, Switzerland", + "confidence": 0.982, + "elements": 10, + "elapsed_ms": 9311, + "probabilities": { + "3": 0.969, + "5": 0.023, + "2": 0.004, + "4": 0.002, + "6": 0.001, + "9": 0.0 + }, + "candidates": { + "3": "Zürich, Switzerland", + "5": "Zurich Airport (ZRH)", + "2": "Origin, Select multiple airports", + "4": "Toggle nearby airports for Zürich, Switzerland", + "6": "Zürich HB", + "9": "Zurich, Netherlands" + }, + "screenshot_ms": 691, + "text": null + }, + { + "tick": 3, + "probe_ms": 763, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1061, + "input_tokens": 0, + "output_tokens": 0, + "operation": "TYPE_TEXT", + "target": "Where to?", + "confidence": 0.753, + "elements": 25, + "elapsed_ms": 13212, + "probabilities": { + "16": 0.994, + "17": 0.003, + "18": 0.002, + "15": 0.001 + }, + "candidates": { + "16": "Where to?", + "17": "Departure", + "18": "Return", + "15": "Where from?" + }, + "screenshot_ms": 767, + "value_source": "prefetch", + "value_ms": 0, + "value_error": null, + "text": "London" + }, + { + "tick": 4, + "probe_ms": 1273, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 745, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "London, United Kingdom", + "confidence": 0.93, + "elements": 9, + "elapsed_ms": 16196, + "probabilities": { + "3": 0.787, + "5": 0.163, + "2": 0.027, + "7": 0.012, + "4": 0.009, + "6": 0.001 + }, + "candidates": { + "3": "London, United Kingdom", + "5": "Heathrow Airport (LHR)", + "2": "Destination, Select multiple airports", + "7": "London Gatwick Airport (LGW)", + "4": "Toggle nearby airports for London, United Kingdom", + "6": "London Stansted Airport (STN)" + }, + "screenshot_ms": 783, + "text": null + }, + { + "tick": 5, + "probe_ms": 755, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1109, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Change ticket type. Round trip", + "confidence": 0.594, + "elements": 26, + "elapsed_ms": 20053, + "probabilities": { + "12": 0.778, + "18": 0.181, + "19": 0.024, + "16": 0.007, + "13": 0.001, + "20": 0.001 + }, + "candidates": { + "12": "Change ticket type. Round trip", + "18": "Departure", + "19": "Return", + "16": "Swap origin and destination.", + "13": "1 passenger", + "20": "Search" + }, + "screenshot_ms": 759, + "text": null + }, + { + "tick": 6, + "probe_ms": 742, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 494, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "One way", + "confidence": 0.996, + "elements": 3, + "elapsed_ms": 23250, + "probabilities": { + "2": 0.995, + "1": 0.005, + "3": 0.0 + }, + "candidates": { + "2": "One way", + "1": "Round trip", + "3": "Multi-city" + }, + "screenshot_ms": 743, + "text": null + }, + { + "tick": 7, + "probe_ms": 753, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1076, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Departure", + "confidence": 0.7, + "elements": 25, + "elapsed_ms": 27072, + "probabilities": { + "18": 0.999, + "19": 0.0, + "14": 0.0, + "9": 0.0, + "12": 0.0, + "13": 0.0 + }, + "candidates": { + "18": "Departure", + "19": "Search", + "14": "Change seating class. Economy", + "9": "Change appearance", + "12": "Change ticket type. One way", + "13": "1 passenger" + }, + "screenshot_ms": 758, + "text": null + }, + { + "tick": 8, + "probe_ms": 1673, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1780, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.575, + "elements": 49, + "elapsed_ms": 32203, + "probabilities": { + "SCROLL_DOWN": 0.646, + "CLICK": 0.259, + "WAIT": 0.086, + "BLOCKED": 0.008, + "TYPE_TEXT": 0.001, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 824, + "text": null + }, + { + "tick": 9, + "probe_ms": 1668, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1915, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Next", + "confidence": 0.352, + "elements": 50, + "elapsed_ms": 36907, + "probabilities": { + "49": 0.513, + "35": 0.139, + "36": 0.102, + "38": 0.048, + "39": 0.038, + "42": 0.026 + }, + "candidates": { + "49": "Next", + "35": "Saturday, October 31, 2026 , 89 euros", + "36": "Monday, November 2, 2026", + "38": "Wednesday, November 4, 2026", + "39": "Monday, November 9, 2026", + "42": "Monday, November 16, 2026" + }, + "screenshot_ms": 842, + "text": null + }, + { + "tick": 10, + "probe_ms": 1891, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2654, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Sunday, November 1, 2026", + "confidence": 0.989, + "elements": 82, + "elapsed_ms": 43051, + "probabilities": { + "36": 0.936, + "66": 0.014, + "81": 0.011, + "9": 0.006, + "37": 0.004, + "40": 0.003 + }, + "candidates": { + "36": "Sunday, November 1, 2026", + "66": "Tuesday, December 1, 2026", + "81": "Next", + "9": "Monday, October 5, 2026 , 138 euros", + "37": "Monday, November 2, 2026", + "40": "Thursday, November 5, 2026" + }, + "screenshot_ms": 828, + "text": null + }, + { + "tick": 11, + "probe_ms": 1333, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 2831, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.397, + "elements": 82, + "elapsed_ms": 48848, + "probabilities": { + "SCROLL_DOWN": 0.483, + "CLICK": 0.395, + "PRESS_ENTER": 0.076, + "DONE": 0.023, + "WAIT": 0.021, + "BLOCKED": 0.001 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 809, + "text": null + }, + { + "tick": 12, + "probe_ms": 839, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 524, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Done. Search for one-way flights, departing on November 1, 2026", + "confidence": 0.591, + "elements": 2, + "elapsed_ms": 51338, + "probabilities": { + "2": 1.0 + }, + "candidates": { + "2": "Done. Search for one-way flights, departing on November 1, 2026" + }, + "screenshot_ms": 885, + "text": null + }, + { + "tick": 13, + "probe_ms": 757, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1453, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.424, + "elements": 28, + "elapsed_ms": 55012, + "probabilities": { + "SCROLL_DOWN": 0.496, + "PRESS_ENTER": 0.176, + "DONE": 0.174, + "WAIT": 0.071, + "CLICK": 0.067, + "SCROLL_UP": 0.014 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 745, + "text": null + }, + { + "tick": 14, + "probe_ms": 775, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 973, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.534, + "elements": 21, + "elapsed_ms": 57732, + "probabilities": { + "SCROLL_UP": 0.612, + "CLICK": 0.161, + "WAIT": 0.157, + "SCROLL_DOWN": 0.052, + "BLOCKED": 0.009, + "DONE": 0.009 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 810, + "text": null + }, + { + "tick": 15, + "probe_ms": 751, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1452, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.352, + "elements": 28, + "elapsed_ms": 60846, + "probabilities": { + "SCROLL_DOWN": 0.433, + "DONE": 0.165, + "PRESS_ENTER": 0.158, + "CLICK": 0.123, + "WAIT": 0.084, + "SCROLL_UP": 0.036 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 751, + "text": null + }, + { + "tick": 16, + "probe_ms": 1259, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 978, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.47, + "elements": 21, + "elapsed_ms": 64081, + "probabilities": { + "SCROLL_UP": 0.558, + "CLICK": 0.255, + "WAIT": 0.12, + "SCROLL_DOWN": 0.046, + "BLOCKED": 0.019, + "DONE": 0.002 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 811, + "text": null + }, + { + "tick": 17, + "probe_ms": 751, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1452, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.23, + "elements": 28, + "elapsed_ms": 67225, + "probabilities": { + "SCROLL_DOWN": 0.326, + "DONE": 0.218, + "PRESS_ENTER": 0.193, + "CLICK": 0.115, + "WAIT": 0.088, + "SCROLL_UP": 0.058 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 787, + "text": null + }, + { + "tick": 18, + "probe_ms": 759, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 980, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.366, + "elements": 21, + "elapsed_ms": 69945, + "probabilities": { + "SCROLL_UP": 0.471, + "CLICK": 0.275, + "WAIT": 0.136, + "SCROLL_DOWN": 0.084, + "BLOCKED": 0.032, + "DONE": 0.001 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 814, + "text": null + }, + { + "tick": 19, + "probe_ms": 765, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1452, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.25, + "elements": 28, + "elapsed_ms": 73089, + "probabilities": { + "SCROLL_DOWN": 0.344, + "PRESS_ENTER": 0.191, + "DONE": 0.182, + "CLICK": 0.151, + "WAIT": 0.081, + "SCROLL_UP": 0.05 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 761, + "text": null + }, + { + "tick": 20, + "probe_ms": 748, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 976, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.495, + "elements": 21, + "elapsed_ms": 75771, + "probabilities": { + "SCROLL_UP": 0.58, + "CLICK": 0.26, + "WAIT": 0.082, + "SCROLL_DOWN": 0.045, + "BLOCKED": 0.032, + "DONE": 0.001 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 797, + "text": null + }, + { + "tick": 21, + "probe_ms": 758, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1452, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.236, + "elements": 28, + "elapsed_ms": 78912, + "probabilities": { + "SCROLL_DOWN": 0.332, + "CLICK": 0.229, + "PRESS_ENTER": 0.172, + "DONE": 0.146, + "WAIT": 0.084, + "SCROLL_UP": 0.035 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "DONE": "The page shows every requirement of the task met.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_UP": "Move the viewport up the page." + }, + "screenshot_ms": 764, + "text": null + }, + { + "tick": 22, + "probe_ms": 749, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 978, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_UP", + "target": null, + "confidence": 0.364, + "elements": 21, + "elapsed_ms": 81613, + "probabilities": { + "SCROLL_UP": 0.47, + "CLICK": 0.319, + "WAIT": 0.092, + "SCROLL_DOWN": 0.061, + "BLOCKED": 0.057, + "DONE": 0.0 + }, + "candidates": { + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "SCROLL_DOWN": "Move the viewport down the page.", + "BLOCKED": "None of the offered operations can move the task forward.", + "DONE": "The page shows every requirement of the task met." + }, + "screenshot_ms": 813, + "text": null + }, + { + "tick": 23, + "probe_ms": 754, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1451, + "input_tokens": 0, + "output_tokens": 0, + "operation": "CLICK", + "target": "Search", + "confidence": 0.293, + "elements": 28, + "elapsed_ms": 84739, + "probabilities": { + "19": 0.7, + "18": 0.271, + "28": 0.008, + "2": 0.003, + "20": 0.002, + "22": 0.002 + }, + "candidates": { + "19": "Search", + "18": "Departure", + "28": "Find flights from Frankfurt am Main (FRA) to London (LGW) from €134. Operated by Condor. Oct 29 to Nov 4. Nonstop", + "2": "Google", + "20": "Explore deals with AI", + "22": "Frankfurt am Main" + }, + "screenshot_ms": 757, + "text": null + }, + { + "tick": 24, + "probe_ms": 946, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1866, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.6, + "elements": 48, + "elapsed_ms": 90829, + "probabilities": { + "SCROLL_DOWN": 0.657, + "DONE": 0.154, + "CLICK": 0.082, + "PRESS_ENTER": 0.057, + "WAIT": 0.047, + "TYPE_TEXT": 0.002 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "DONE": "The page shows every requirement of the task met.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "PRESS_ENTER": "Submit a filled field by pressing Enter in it, the way a person submits a search.", + "WAIT": "Give the page time to finish updating.", + "TYPE_TEXT": "Type a value into an editable field, replacing what it holds." + }, + "screenshot_ms": 975, + "text": null + }, + { + "tick": 25, + "probe_ms": 1489, + "settle_probes": 0, + "settle_ms": 0, + "action_settle_probes": 0, + "action_settle_ms": 0, + "decision_ms": 1483, + "input_tokens": 0, + "output_tokens": 0, + "operation": "SCROLL_DOWN", + "target": null, + "confidence": 0.123, + "elements": 44, + "elapsed_ms": 95130, + "probabilities": { + "SCROLL_DOWN": 0.269, + "SCROLL_UP": 0.253, + "CLICK": 0.25, + "WAIT": 0.148, + "DONE": 0.066, + "BLOCKED": 0.014 + }, + "candidates": { + "SCROLL_DOWN": "Move the viewport down the page.", + "SCROLL_UP": "Move the viewport up the page.", + "CLICK": "Press a control on the page: a button, a link, a menu entry, an autocomplete suggestion or a calendar day.", + "WAIT": "Give the page time to finish updating.", + "DONE": "The page shows every requirement of the task met.", + "BLOCKED": "None of the offered operations can move the task forward." + }, + "screenshot_ms": 992, + "text": null + } + ], + "final": null, + "usage": { + "decisions": 25, + "chat_calls": 12, + "chat_input_tokens": 19528, + "chat_output_tokens": 116, + "chat_cache_tokens": 0, + "jev_input_tokens": 0, + "cost_usd": 0.006148 + } +} \ No newline at end of file diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench.csv b/docs/results/omnijev-flights/2026-09-30/benchmark/bench.csv new file mode 100644 index 0000000..4a4027f --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/benchmark/bench.csv @@ -0,0 +1,12 @@ +time,model,run,success,total_s,decisions,median_decision_ms,reason +15:33:18,jev,1,1,38.9,13,299,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +15:34:01,jev,2,1,35.4,13,367,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +15:34:42,jev,3,1,34.0,13,321,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +15:35:24,jev,4,1,33.9,12,347,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +15:36:04,jev,5,1,33.0,13,283,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +15:38:05,omnijev,1,0,97.1,25,1440,The task was to find flights from Zurich to London; but the current results are for train travel from Zurich to St Pancras International; and the date is November 1; not November 1; 2026. +15:40:04,omnijev,2,0,95.7,25,1446,The task was to find flights from Zurich to London; but the current results are for train travel from Zurich to St Pancras International; and the date is November 1; not November 1; 2026. +15:42:10,omnijev,3,0,101.7,25,1016,The agent has not yet selected the correct date (November 1; 2026) and has not displayed flight options. +15:44:36,omnijev,4,0,121.9,25,1061,The agent is stuck on a location selection pop-up and has not yet started searching for flights. +15:46:37,omnijev,5,0,97.4,25,1451,The task was to find flights from Zurich to London; but the current results are for train travel from Zurich to St Pancras International; and the date is November 1; not November 1; 2026. +15:47:21,clm,1,0,36.5,21,3,The agent stopped on the initial flight search page; not on a page displaying flight results. diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench.sh b/docs/results/omnijev-flights/2026-09-30/benchmark/bench.sh new file mode 100644 index 0000000..4bc507c --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/benchmark/bench.sh @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# Google Flights benchmark: success rate (vision judge on the final screenshot) and time, per decision model. +# Usage: bash bench.sh "jev omnijev clm" 5 (models, runs per model); results in results/bench.csv +W=~/chaimae; K=$W/gpu_kit; E=$W/clm-env +MODELS=${1:-"jev omnijev clm"}; RUNS=${2:-5} +export PATH=$HOME/.local/bin:$W/node-v22.12.0-linux-x64/bin:$PATH +export LD_LIBRARY_PATH=$W/libs/usr/lib/x86_64-linux-gnu +export PLAYWRIGHT_MCP_ARGS="-y @playwright/mcp@0.0.78 --cdp-endpoint=http://localhost:9222" HF_HUB_OFFLINE=1 +mkdir -p $K/results +CSV=$K/results/bench2.csv +[ -s $CSV ] || echo "time,model,run,success,total_s,decisions,median_decision_ms,model_s,page_s,browser_s,reason" > $CSV + +# CLM needs its two servers (Qwen3-8B embeddings + clm-serve); started once, left running +if [[ " $MODELS " == *" clm "* ]]; then + if [ ! -x $E/bin/clm-serve ]; then echo "== installing vLLM + CLM (~5 min)"; uv venv -q --python 3.12 $E && uv pip install -q --python $E/bin/python vllm contrastive-lm || exit 1; fi + curl -s localhost:8090/v1/models >/dev/null || { echo "== starting Qwen3-8B (download ~16 GB the first time)"; HF_HUB_OFFLINE=0 nohup $E/bin/vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --runner pooling --max-model-len 2048 --gpu-memory-utilization 0.5 --port 8090 > $K/results/vllm.log 2>&1 & } + until curl -s localhost:8090/v1/models >/dev/null; do sleep 15; echo " waiting for Qwen3-8B..."; done + curl -s localhost:8700/health >/dev/null || { echo "== starting CLM"; HF_HUB_OFFLINE=0 nohup $E/bin/clm-serve --port 8700 --emb-url http://127.0.0.1:8090/v1/embeddings > $K/results/clm.log 2>&1 & } + until curl -s localhost:8700/health >/dev/null; do sleep 5; echo " waiting for CLM..."; done +fi + +# CLM with a window that holds the whole page (8,192 tokens instead of 2,048): both servers restarted with it +if [[ " $MODELS " == *" clm8k "* ]]; then + pkill -f "vllm serve" ; pkill -f clm-serve ; pkill -f "qev serve" ; sleep 5 + [ -x $E/bin/clm-serve ] || { echo "== installing vLLM + CLM (~5 min)"; uv venv -q --python 3.12 $E && uv pip install -q --python $E/bin/python vllm contrastive-lm || exit 1; } + echo "== starting Qwen3-8B with an 8,192-token window" + HF_HUB_OFFLINE=0 nohup $E/bin/vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --runner pooling --max-model-len 8192 --gpu-memory-utilization 0.5 --port 8090 > $K/results/vllm8k.log 2>&1 & + until curl -s localhost:8090/v1/models >/dev/null; do sleep 15; echo " waiting for Qwen3-8B..."; done + echo "== starting CLM reading up to 8,192 tokens" + HF_HUB_OFFLINE=0 nohup $E/bin/clm-serve --port 8700 --emb-url http://127.0.0.1:8090/v1/embeddings --max-tokens 8192 > $K/results/clm8k.log 2>&1 & + until curl -s localhost:8700/health >/dev/null; do sleep 5; echo " waiting for CLM..."; done +fi + +# OneJev (27B or 9B): served by its own qev server on port 8000, which speaks Jev's API; the GPU cannot hold it next +# to CLM's servers, so those are stopped first. Only one OneJev size per call. +Q=$W/qev-env +for size in 27B 9B 4B 0.8B; do + if [[ " $MODELS " == *" onejev$size "* || " $MODELS " == *" onejev${size}c "* ]]; then + pkill -f "vllm serve" ; pkill -f clm-serve ; pkill -f "qev serve" ; sleep 5 + if [ ! -x $Q/bin/qev ]; then echo "== installing OneJev's server (qev)"; uv venv -q --python 3.12 $Q && uv pip install -q --python $Q/bin/python "qev[torch] @ git+https://github.com/OmniJev/OneJev.git" || exit 1; fi + echo "== starting OneJev-$size (download the first time: 55 GB for 27B, 19 GB for 9B)" + HF_HUB_OFFLINE=0 nohup $Q/bin/qev serve --model OmniJev/OneJev-$size --multimodal --port 8000 > $K/results/onejev_$size.log 2>&1 & + until curl -s localhost:8000/health >/dev/null; do sleep 15; echo " waiting for OneJev-$size... ($(tail -c 100 $K/results/onejev_$size.log | tr '\n' ' '))"; done + curl -s localhost:8000/health; echo + fi +done + +# One headless Chrome with its own profile, cookies refused once +C=$(ls -d $HOME/.cache/ms-playwright/chromium-*/chrome-linux*/chrome | head -1) +pgrep -f "remote-debugging-port=9222" >/dev/null || { "$C" --headless=new --no-sandbox --remote-debugging-port=9222 --user-data-dir="$K/chrome-profile" --window-size=1280,900 --no-first-run about:blank >/dev/null 2>&1 & sleep 5; } +cd $K/system1-agents +uv run --with websocket-client python $K/reject_cookies.py >/dev/null +TASK=$(uv run python -c "from s1a.agents.flights import SPEC; print(SPEC.goal)" 2>/dev/null | tail -1) +echo "== task: $TASK" + +for model in $MODELS; do + for i in $(seq 1 $RUNS); do + log=$K/results/bench_${model}_$(date +%H%M%S).log + prof=${log%.log}.json # the team's profiler: where the seconds go (model, page, browser) + case $model in + jev) env -u TYPESAFE_API_URL -u TYPESAFE_MODEL uv run s1a run flights --model jev --timeout 600 --profile-out "$prof" > "$log" 2>&1 ;; + clm|clm8k) TYPESAFE_API_URL=http://127.0.0.1:8700/v1/systemone TYPESAFE_MODEL=clm-latest uv run python $K/slow_ok.py run flights --model jev --timeout 600 --profile-out "$prof" > "$log" 2>&1 ;; + onejev27B|onejev9B|onejev4B|onejev0.8B)TYPESAFE_API_URL=http://127.0.0.1:8000/v1/systemone TYPESAFE_MODEL=jev-latest uv run python $K/slow_ok.py run flights --model jev --timeout 600 --profile-out "$prof" > "$log" 2>&1 ;; + onejev27Bc|onejev9Bc) TYPESAFE_API_URL=http://127.0.0.1:8000/v1/systemone TYPESAFE_MODEL=jev-latest uv run python $K/compact_ok.py run flights --model jev --timeout 600 --profile-out "$prof" > "$log" 2>&1 ;; + omnijev) OMNIJEV_REPO=$K/OmniJev OMNIJEV_CHECKPOINT=$K/ckpt-4b OMNIJEV_BASE=$K/base-4b uv run s1a run flights --model omnijev --timeout 600 --profile-out "$prof" > "$log" 2>&1 ;; + esac + shot=${log%.log}.png + uv run --with websocket-client python $K/final_shot.py "$shot" >/dev/null 2>&1 + verdict=$(uv run python $K/judge.py "$shot" "$TASK" 2>/dev/null | tail -1) + line=$(uv run python - "$log" "$verdict" "$model" "$i" <<'PY' +import json, sys, time +log, verdict, model, run = sys.argv[1:5] +text = open(log, encoding="utf-8", errors="replace").read() +k = text.rfind('{"ok"') +try: + answer = json.loads(text[k:text.index("\n", k)] if "\n" in text[k:] else text[k:]) if k >= 0 else {} +except json.JSONDecodeError: + answer = {} +report = answer.get("report") or {} +try: + judged = json.loads(verdict) +except json.JSONDecodeError: + judged = {"success": False, "reason": "judge failed"} +reason = str(judged.get("reason", "")).replace(",", ";").replace("\n", " ") +try: # the team's profiler (docs/benchmarks.md): decision = waiting on the model, probe = page, tool = browser + phases = json.load(open(log[:-4] + ".json", encoding="utf-8")).get("phases") or {} +except (OSError, ValueError): + phases = {} +model_s = phases.get("decision", phases.get("jev", 0)) / 1000 +print(f"{time.strftime('%H:%M:%S')},{model},{run},{int(bool(judged.get('success')))}," + f"{report.get('elapsed_ms', 0) / 1000:.1f},{report.get('decisions', 0)},{report.get('median_decision_ms', 0)}," + f"{model_s:.1f},{phases.get('probe', 0) / 1000:.1f},{phases.get('tool', 0) / 1000:.1f},{reason}") +PY +) + echo "$line" >> $CSV + echo "$line" + # the actions this run took, in order, to see where it goes wrong + uv run python - "$log" <<'PY' +import json, sys +text = open(sys.argv[1], encoding="utf-8", errors="replace").read() +k = text.rfind('{"ok"') +try: + history = (json.loads(text[k:].split("\n", 1)[0]).get("report") or {}).get("history") or [] +except (json.JSONDecodeError, ValueError): + history = [] +steps = [(h.get("text") and f"type {h['text']}") or f"{h.get('kind')} {str(h.get('action'))[:30]}" for h in history] +print(" actions: " + " > ".join(steps[:16]) + (" > ..." if len(steps) > 16 else "")) +PY + done +done + +echo "== summary (all rows in results/bench2.csv)" +uv run python - "$CSV" <<'PY' +import csv, statistics, sys +rows = list(csv.DictReader(open(sys.argv[1], encoding="utf-8"))) +print(f"{'model':<10} {'runs':>4} {'success':>8} {'rate':>6} {'total s':>8} {'model s':>8} {'page s':>7} {'browser s':>10} {'per decision ms':>16}") +for model in dict.fromkeys(r["model"] for r in rows): + rs = [r for r in rows if r["model"] == model] + ok = sum(int(r["success"]) for r in rs) + med = lambda key: statistics.median(float(r[key]) for r in rs) + print(f"{model:<10} {len(rs):>4} {ok:>8} {100 * ok / len(rs):>5.0f}% {med('total_s'):>8.1f} {med('model_s'):>8.1f} " + f"{med('page_s'):>7.1f} {med('browser_s'):>10.1f} {med('median_decision_ms'):>16.0f}") +PY diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153232.log b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153232.log new file mode 100644 index 0000000..c535826 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153232.log @@ -0,0 +1 @@ +{"ok": true, "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", "screenshot": null, "error": null, "elapsed_ms": 39528, "usage": {"decisions": 13, "chat_calls": 13, "chat_input_tokens": 6173, "chat_output_tokens": 119, "chat_cache_tokens": 0, "jev_input_tokens": 95564, "cost_usd": 0.006163}, "report": {"elapsed_ms": 38862, "decisions": 13, "interactions": 12, "waits": 0, "median_decision_ms": 299, "jev_input_tokens": 95564, "median_probe_ms": 941, "settle_probes": 0, "settle_ms": 0, "values": {"cache": 0, "prefetch": 3, "llm": 0}, "history": [{"action": "Change ticket type. Round trip", "kind": "click", "text": null, "page_changed": true}, {"action": "One way", "kind": "click", "text": null, "page_changed": true}, {"action": "Where from?", "kind": "fill", "text": "Zurich", "page_changed": true}, {"action": "Zürich, Switzerland", "kind": "click", "text": null, "page_changed": true}, {"action": "Where to?", "kind": "fill", "text": "London", "page_changed": true}, {"action": "London, United Kingdom", "kind": "click", "text": null, "page_changed": true}, {"action": "Departure", "kind": "fill", "text": "Zurich", "page_changed": true}, {"action": "Departure", "kind": "click", "text": null, "page_changed": true}, {"action": "Next", "kind": "click", "text": null, "page_changed": true}, {"action": "Sunday, November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Done. Search for one-way flights, departing on November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Search", "kind": "click", "text": null, "page_changed": true}]}, "status": "DONE", "terminal": {"status": "DONE", "reason": "", "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", "title": "Zürich to London | Google Flights", "steps": 12, "elapsed_ms": 38515, "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM."}} diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153232.png b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153232.png new file mode 100644 index 0000000..0e623bc Binary files /dev/null and b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153232.png differ diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153318.log b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153318.log new file mode 100644 index 0000000..201fe3c --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153318.log @@ -0,0 +1 @@ +{"ok": true, "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", "screenshot": null, "error": null, "elapsed_ms": 36109, "usage": {"decisions": 13, "chat_calls": 13, "chat_input_tokens": 6497, "chat_output_tokens": 128, "chat_cache_tokens": 0, "jev_input_tokens": 95564, "cost_usd": 0.006283}, "report": {"elapsed_ms": 35443, "decisions": 13, "interactions": 12, "waits": 0, "median_decision_ms": 367, "jev_input_tokens": 95564, "median_probe_ms": 1204, "settle_probes": 0, "settle_ms": 0, "values": {"cache": 0, "prefetch": 3, "llm": 0}, "history": [{"action": "Change ticket type. Round trip", "kind": "click", "text": null, "page_changed": true}, {"action": "One way", "kind": "click", "text": null, "page_changed": true}, {"action": "Where from?", "kind": "fill", "text": "Zurich", "page_changed": true}, {"action": "Zürich, Switzerland", "kind": "click", "text": null, "page_changed": true}, {"action": "Where to?", "kind": "fill", "text": "London", "page_changed": true}, {"action": "London, United Kingdom", "kind": "click", "text": null, "page_changed": true}, {"action": "Departure", "kind": "fill", "text": "Zurich", "page_changed": true}, {"action": "Departure", "kind": "click", "text": null, "page_changed": true}, {"action": "Next", "kind": "click", "text": null, "page_changed": true}, {"action": "Sunday, November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Done. Search for one-way flights, departing on November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Search", "kind": "click", "text": null, "page_changed": true}]}, "status": "DONE", "terminal": {"status": "DONE", "reason": "", "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", "title": "Zürich to London | Google Flights", "steps": 12, "elapsed_ms": 35092, "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM."}} diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153318.png b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153318.png new file mode 100644 index 0000000..0e623bc Binary files /dev/null and b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153318.png differ diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153401.log b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153401.log new file mode 100644 index 0000000..741dc53 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153401.log @@ -0,0 +1 @@ +{"ok": true, "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", "screenshot": null, "error": null, "elapsed_ms": 34605, "usage": {"decisions": 13, "chat_calls": 13, "chat_input_tokens": 6497, "chat_output_tokens": 128, "chat_cache_tokens": 0, "jev_input_tokens": 95564, "cost_usd": 0.006283}, "report": {"elapsed_ms": 33966, "decisions": 13, "interactions": 12, "waits": 0, "median_decision_ms": 321, "jev_input_tokens": 95564, "median_probe_ms": 928, "settle_probes": 0, "settle_ms": 0, "values": {"cache": 0, "prefetch": 3, "llm": 0}, "history": [{"action": "Change ticket type. Round trip", "kind": "click", "text": null, "page_changed": true}, {"action": "One way", "kind": "click", "text": null, "page_changed": true}, {"action": "Where from?", "kind": "fill", "text": "Zurich", "page_changed": true}, {"action": "Zürich, Switzerland", "kind": "click", "text": null, "page_changed": true}, {"action": "Where to?", "kind": "fill", "text": "London", "page_changed": true}, {"action": "London, United Kingdom", "kind": "click", "text": null, "page_changed": true}, {"action": "Departure", "kind": "fill", "text": "Zurich", "page_changed": true}, {"action": "Departure", "kind": "click", "text": null, "page_changed": true}, {"action": "Next", "kind": "click", "text": null, "page_changed": true}, {"action": "Sunday, November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Done. Search for one-way flights, departing on November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Search", "kind": "click", "text": null, "page_changed": true}]}, "status": "DONE", "terminal": {"status": "DONE", "reason": "", "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", "title": "Zürich to London | Google Flights", "steps": 12, "elapsed_ms": 33614, "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM."}} diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153401.png b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153401.png new file mode 100644 index 0000000..0e623bc Binary files /dev/null and b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153401.png differ diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153442.log b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153442.log new file mode 100644 index 0000000..ffe4b31 --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153442.log @@ -0,0 +1 @@ +{"ok": true, "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", "screenshot": null, "error": null, "elapsed_ms": 34502, "usage": {"decisions": 12, "chat_calls": 12, "chat_input_tokens": 6099, "chat_output_tokens": 123, "chat_cache_tokens": 0, "jev_input_tokens": 89557, "cost_usd": 0.005899}, "report": {"elapsed_ms": 33868, "decisions": 12, "interactions": 11, "waits": 0, "median_decision_ms": 347, "jev_input_tokens": 89557, "median_probe_ms": 1061, "settle_probes": 0, "settle_ms": 0, "values": {"cache": 0, "prefetch": 2, "llm": 0}, "history": [{"action": "Change ticket type. Round trip", "kind": "click", "text": null, "page_changed": true}, {"action": "One way", "kind": "click", "text": null, "page_changed": true}, {"action": "Where from?", "kind": "fill", "text": "Zurich", "page_changed": true}, {"action": "Zürich, Switzerland", "kind": "click", "text": null, "page_changed": true}, {"action": "Where to?", "kind": "fill", "text": "London", "page_changed": true}, {"action": "London, United Kingdom", "kind": "click", "text": null, "page_changed": true}, {"action": "Departure", "kind": "click", "text": null, "page_changed": true}, {"action": "Next", "kind": "click", "text": null, "page_changed": true}, {"action": "Sunday, November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Done. Search for one-way flights, departing on November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Search", "kind": "click", "text": null, "page_changed": true}]}, "status": "DONE", "terminal": {"status": "DONE", "reason": "", "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", "title": "Zürich to London | Google Flights", "steps": 11, "elapsed_ms": 33513, "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM."}} diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153442.png b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153442.png new file mode 100644 index 0000000..0e623bc Binary files /dev/null and b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153442.png differ diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153524.log b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153524.log new file mode 100644 index 0000000..814f63f --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153524.log @@ -0,0 +1 @@ +{"ok": true, "final": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM.", "screenshot": null, "error": null, "elapsed_ms": 33679, "usage": {"decisions": 13, "chat_calls": 13, "chat_input_tokens": 6497, "chat_output_tokens": 129, "chat_cache_tokens": 0, "jev_input_tokens": 95564, "cost_usd": 0.006285}, "report": {"elapsed_ms": 33033, "decisions": 13, "interactions": 12, "waits": 0, "median_decision_ms": 283, "jev_input_tokens": 95564, "median_probe_ms": 932, "settle_probes": 0, "settle_ms": 0, "values": {"cache": 0, "prefetch": 3, "llm": 0}, "history": [{"action": "Change ticket type. Round trip", "kind": "click", "text": null, "page_changed": true}, {"action": "One way", "kind": "click", "text": null, "page_changed": true}, {"action": "Where from?", "kind": "fill", "text": "Zurich", "page_changed": true}, {"action": "Zürich, Switzerland", "kind": "click", "text": null, "page_changed": true}, {"action": "Where to?", "kind": "fill", "text": "London", "page_changed": true}, {"action": "London, United Kingdom", "kind": "click", "text": null, "page_changed": true}, {"action": "Departure", "kind": "fill", "text": "Zurich", "page_changed": true}, {"action": "Departure", "kind": "click", "text": null, "page_changed": true}, {"action": "Next", "kind": "click", "text": null, "page_changed": true}, {"action": "Sunday, November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Done. Search for one-way flights, departing on November 1, 2026", "kind": "click", "text": null, "page_changed": true}, {"action": "Search", "kind": "click", "text": null, "page_changed": true}]}, "status": "DONE", "terminal": {"status": "DONE", "reason": "", "url": "https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTExLTAxagwIAxIIL20vMDg5NjZyDAgDEggvbS8wNGpwbEABSAFwAYIBCwj___________8BmAEC&hl=en", "title": "Zürich to London | Google Flights", "steps": 12, "elapsed_ms": 32673, "page_text": "Skip to main content\nAccessibility feedback\nExplore\nFlights\nHotels\nVacation rentals\nSign in\nFlight search\n1\nFilters\nAll filters\nStops\nAirlines\nBags\nPrice\nTimes\nEmissions\nConnecting airports\nDuration\nSearch results\n23 results returned.\nTravel update\nAir traffic disruptions may affect flights.\nBest\nCheapest\nfrom\n€74\nTop options\nRanked based on price and convenience\nPrices include required taxes + fees for 1 adult.\nOptional charges and\nbag fees\nmay apply.\nPassenger assistance\ninfo.\nSorted by top flights\n9:30 PM\n–\n10:10 PM\neasyJet\n1 hr 40 min\nNonstop\n€76\n9:10 PM\n–\n9:55 PM\nBritish Airways\n1 hr 45 min\nNonstop\n€79\n5:00 PM\n–\n5:50 PM\nSWISS\n1 hr 50 min\nNonstop\n€113\nTrains to consider\nTo arrive closer to your destination\n1:34 PM\n–\n8:00 PM\n7 hr 26 min\nSee more train options", "answer": "The cheapest one-way flight from Zurich to London on November 1, 2026, for one adult in economy is €74 on easyJet, departing at 9:30 PM and arriving at 10:20 PM."}} diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153524.png b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153524.png new file mode 100644 index 0000000..0e623bc Binary files /dev/null and b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_jev_153524.png differ diff --git a/docs/results/omnijev-flights/2026-09-30/benchmark/bench_omnijev_153604.log b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_omnijev_153604.log new file mode 100644 index 0000000..d3a6a6d --- /dev/null +++ b/docs/results/omnijev-flights/2026-09-30/benchmark/bench_omnijev_153604.log @@ -0,0 +1,6 @@ + Loading weights: 0%| | 0/723 [00:00" +Prints one JSON line: {"success": true|false, "reason": "...", "judge": ""}. +Reads OPENROUTER_API_KEY from the environment or from system1-agents/.env; JUDGE_MODEL picks the model. +""" + +import base64 +import json +import os +import pathlib +import re +import sys +import urllib.request + +JUDGE_MODEL = os.getenv("JUDGE_MODEL") or "google/gemini-2.5-flash" +PROMPT = ( + "You are a strict judge of a web task. Task given to a browser agent:\n{task}\n\n" + "Look only at the screenshot of the page where the agent stopped. The task counts as done ONLY if the page " + "shows a list of flight results that match the task: the right origin, the right destination, the right date, " + "one way. A filled form without results, an error page, 'No results', or results for another route or date " + "is NOT done.\n" + 'Answer with JSON only: {{"success": true or false, "reason": ""}}' +) + + +def api_key() -> str: + key = os.getenv("OPENROUTER_API_KEY") + if key: + return key + env = pathlib.Path(__file__).with_name("system1-agents") / ".env" + for line in env.read_text(encoding="utf-8").splitlines(): + if line.startswith("OPENROUTER_API_KEY="): + return line.split("=", 1)[1].strip() + sys.exit("no OPENROUTER_API_KEY") + + +def main() -> None: + image = base64.b64encode(pathlib.Path(sys.argv[1]).read_bytes()).decode() + body = { + "model": JUDGE_MODEL, + "temperature": 0, + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": PROMPT.format(task=sys.argv[2])}, + {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}}, + ], + } + ], + } + request = urllib.request.Request( + "https://openrouter.ai/api/v1/chat/completions", + data=json.dumps(body).encode(), + headers={"Authorization": f"Bearer {api_key()}", "Content-Type": "application/json"}, + ) + reply = json.load(urllib.request.urlopen(request, timeout=60))["choices"][0]["message"]["content"] + found = re.search(r"\{.*\}", reply, re.S) + try: + verdict = json.loads(found.group(0)) if found else {} + except json.JSONDecodeError: + verdict = {} + print( + json.dumps( + { + "success": bool(verdict.get("success")), + "reason": str(verdict.get("reason") or reply.strip())[:200], + "judge": JUDGE_MODEL, + } + ) + ) + + +if __name__ == "__main__": + main() diff --git a/docs/results/omnijev-flights/README.md b/docs/results/omnijev-flights/README.md new file mode 100644 index 0000000..766555d --- /dev/null +++ b/docs/results/omnijev-flights/README.md @@ -0,0 +1,77 @@ +# OmniJev-4B on Google Flights: every s1a run + +Every `s1a run flights --model omnijev` run on the A100 server that wrote a record, and the matched Jev runs. These +are s1a runs; the four OmniJev GIFs in the README are OmniJev's own recorded-trajectory replays, not these. + +Task: `flights`, one-way Zurich to London on November 1, 2026, one adult, economy; stop when matching flights show. + +## What each run did + +| run | code | decisions | end | final page | outcome | +|---|---|---|---|---|---| +| `2026-09-29__14-18-26` | 1d01e69 | 25 | form budget | form | form filled, then Enter in Departure instead of Search; wandered (Frankfurt card, main menu) | +| `2026-09-29__14-22-54` | 1d01e69 | 14 | BLOCKED | results | every step, Search included; Google answered "No results returned. Oops, something went wrong." | +| `2026-09-29__14-54-07` | 1d01e69 | 25 | form budget | form | form filled, lost after it | +| `2026-09-29__14-56-33` | 1d01e69 | 25 | form budget | form | form filled, lost after it | +| `2026-09-29__14-58-50` | 1d01e69 | 25 | form budget | form | form filled, lost after it | +| `2026-09-29__15-01-10` | 1d01e69 | 17 | BLOCKED | form | lost, blocked | +| `2026-09-29__15-02-15` | 1d01e69 | 4 | BLOCKED | form | WAIT with no progress after 3 actions | +| `2026-09-29__15-03-06` | 1d01e69 | 7 | BLOCKED | form | started 2 s before the next one on the same Chrome: not a valid run | +| `2026-09-29__15-03-08` | 1d01e69 | 7 | BLOCKED | form | the other half of that pair, not valid either; both typed into "Return" and had no value for it | +| `2026-09-29__15-04-13` | 1d01e69 | 25 | form budget | form | form filled, lost after it | +| **`2026-09-29__15-06-35`** | 1d01e69 | 15 | **DONE** | results | **every step, Search included; answer "easyJet, €72, 9:30 PM to 10:20 PM, November 1", the flight Jev found** | +| `2026-09-29__15-08-19` | 1d01e69 | 25 | form budget | form | form filled, lost after it | +| `2026-09-30__09-37-51` to `09-47-20` (5 runs) | e2bf4c5 | 25 each | form budget | form | form filled, then lost (location menu, deal pages) | +| `2026-09-30__15-36-07`, `15-38-08`, `15-44-39` | e2bf4c5 | 25 each | form budget | **results** | form filled, scrolled, clicked Search at decision 23; the flights show, but the budget ended before DONE | +| `2026-09-30__15-40-07` | e2bf4c5 | 25 | form budget | form | opened "Flights from popular destinations" and started over | +| `2026-09-30__15-42-13` | e2bf4c5 | 25 | form budget | form | stuck on the location pop-up | +| `2026-09-30__15-32-35` to `15-35-27` (5 runs) | e2bf4c5 | 12 to 13 | DONE | results | **Jev 1.13**, the matched runs: same server, Chrome and profile | + +Counted: 20 OmniJev runs (22 records, the concurrent pair left out). 1 completed (DONE on the right flight), 4 more +reached the results page (one Google error, three without DONE before the budget), 15 did not. Before the e2bf4c5 +history fix (1d01e69, 2026-09-29): 1 of 10 completed. After it (2026-09-30): 0 of 10 completed, 3 reached the results. Jev: 5 of +5 completed, 12 to 13 decisions, 283 to 367 ms median per decision. + +The PR's first results table listed seven of the 2026-09-29 runs (14-18-26, 14-22-54, 15-02-15, 15-03-0x, 15-04-13, +15-06-35, 15-08-19). The other four that day (14-54-07, 14-56-33, 14-58-50, 15-01-10) were not in it; they are here. +Four earlier folders that day (14-13-24, 14-17-35, 14-46-20, 14-51-28) wrote no record and are not counted. + +## The vision judge's verdicts (2026-09-30 afternoon) + +`2026-09-30/benchmark/` holds that batch's final screenshots, logs and `bench.csv` with Gemini 2.5 Flash's verdicts +on each screenshot. It marked all five OmniJev runs as failed. For `15-36-07`, `15-38-08` and `15-44-39` the +screenshot does show Zurich to London flights on November 1 (the results URL's search date is 2026-11-01); the judge +read the train suggestion and the "Nov 1" chip as a wrong match. Those three are counted above as "reached the +results, no DONE", not as completions: the task asks the agent to stop there, and it did not. + +## Files + +| file | what it is | +|---|---| +| `2026-09-29//`, `2026-09-30//` | each run's `answer.json` (status, final page, the chat model's answer) and `decision_ticks.json` (every decision: operation, target, confidence, decision and screenshot time) | +| `actions.txt` | every run's final page and actions, from `summarize.py` over those records | +| `2026-09-29/logs/` | the terminal output of the 15:02 to 15:08 batch (`run1.log` to `run5.log`) and the launcher `run_flights_gpu.sh` | +| `2026-09-30/benchmark/` | the afternoon batch: final screenshots (`bench__