feat: reins do — delegate a browsing goal to Jev, with benchmark (3/3)

[karngyan]

Part 3 of 3 for reins do (#41 and #42 are merged): the command itself, the benchmark, and every fix from review and live testing.

Why

Driving a site one command at a time costs an agent one turn per click. reins do "<goal>" runs the whole loop inside the daemon, with Jev (TypeSafe's action model) picking each action. It returns only when the goal is met or a decision is needed.

What

• reins do '<goal>' [--fill name=value]... [--confirm '<label>']... [--continue] [--max-steps 30] [--timeout 60] [--tab] [--browser] [--json]

  • The loop:
    1. Read the page.
    2. Ask Jev once. The request includes speculative heads for the operation, the target, and the text to fill.
    3. Apply the stop rules.
    4. Act and record.
  • Text only ever comes from --fill.

• Bounded autonomy. Every run ends with a status and, where one applies, a next: line to run:

  • done
  • risky_action
  • needs_text
  • left_site
  • dialog
  • interrupted
  • blocked
  • stuck
  • budget
  • error

  Exit codes: 0 done, 2 stopped, 1 error.

• Self-check. Every done, and every stuck run, gets a yes/no question to Jev. A low score prints done (unsure: self-check 0.34). In the benchmark, all 17 wrong done results scored below 0.5; 13% of right ones did too.

• Safety. Risky clicks stop the run. The rule is whole-word, Unicode-aware, and fires on unlabelled buttons. "Accept" on a cookie or consent banner is allowed. Leaving the site stops the run (a leading www. is ignored). Password and hidden values are never sent.

• Coverage of real pages:

  • SUBMIT_SEARCH (Enter, only in search-like fields)
  • following a new tab
  • open shadow DOM, including names through slots
  • off-screen pagers, and controls whose names match words in the goal
  • waiting for navigation and in-flight data requests
  • typing key by key
  • naming unlabelled controls by their row
  • bounded stale retries
  • retargeting a text field when Jev picks one that fits none of the fills
  • rounded probability ties accepted

• Browser-wide fixes found along the way:

  • Page focus emulation. Tabs that reins opens had document.hasFocus() false.
  • Links that open a new tab are opened by the extension. A native click made Chrome jump in front of the user's terminal on macOS.
  • reins do --help works.
  • Shell-quoted next: lines.
  • A corrupt key file is reported instead of being treated as missing.

• Skill. A rewritten reins do section covering when to use it, how to call it, a table of statuses and what to do for each, and its limits.

• Benchmark. 38 tasks: local fixture pages for tricky widgets plus live sites. A dev set, and an unseen holdout that stayed frozen until its one run. Traces are optional (REINS_JEV_TRACE). There is a report and a /docs/benchmarks page with every limitation listed.

• CI. Runs the real-browser extension suite against CI's Chrome. REINS_TEST_CHROME is now declared in turbo.json; turbo's strict env had been dropping it.

Benchmark (38 tasks; reins do 5 runs each, Claude Opus 5.5 step by step 3 runs each)

	reins do	Claude step by step
Dev set (30 tasks, tuned on for five rounds)	124/150 (82.6%)	81/90 (90%)
Unseen holdout (8 tasks, first and only run)	35/40 (87.5%)	21/24 (87.5%)
Model spend	≈ $0.32 for 190 runs	$21.83 for 114 runs

On tasks both arms passed, the median was 4.6× faster (range 2.4–22.6×) and 111× cheaper (range 44–535×). The cost figure counts Jev only; the calling agent still spends a turn to issue the command and verify.

Tasks reins do still fails:

• fx-filters, arxiv, fx-datepicker, fx-orders. Jev says DONE or BLOCKED while the control it needs is visible. The self-check flags every one of these.
• huggingface, github. Two search boxes, and the choice is a near-tie.
• xe. The site labels its input "Receiving amount", the same label as its output.

The full report is docs/benchmarks/2026-09-reins-do.md.

The report also records two bugs in the step-by-step commands. They are out of scope here and will be fixed in a follow-up:

• reins snapshot doesn't read shadow DOM.
• Some custom controls report "zero size" on click.

Testing

• pnpm build, pnpm lint and pnpm typecheck are clean.
• Tests pass: protocol 95, CLI 509, extension 312 (including the real-Chrome suite).
• Live on the maintainer's Chrome:
  • the benchmark above;
  • new-tab follow;
  • a Finder-frontmost check showing a new-tab link no longer raises Chrome;
  • a Delete-account page stops at risky_action.

🤖 Generated with Claude Code