feat(verify): independent verification pass and escalation ladder - #20
Conversation
…rdict (S5) Add VerifyResultDict + Job.verify_result (schema v3, None = not requested, distinct from a ran-and-failed verify per D7) and refactor delegation_verdict into a gate combiner: any failed gate blocks the green path, a passing gate greens it, an inconclusive/absent gate is neutral. A structured verify 'fail' therefore blocks green even when the shell check passed (D6); a prose-only or errored verifier degrades to inconclusive, never a pass (D4). Surface verify_result in runtime_status.
…dation (S5) Spawn a FRESH peer session that grades the delegate's artifact supplied as user-turn input (D6) — build_verifier_command adds no resume/fork/name flag and never reads the session registry, removing the implicit-authorship channel that weakens self-grading. Prefer a machine-checkable contract where the advisor exposes one (Claude --json-schema -> structured_output; new Advisor.json_schema_flag), else parse a JSON verdict out of the answer; free prose -> unverified, never a pass. The verifier is a delegate too: credentials scrubbed and CROSSAGENT_* lineage stripped. Never raises — a broken verifier degrades to error/unverified. Scope-honest: removes the implicit-authorship channel; does NOT claim to eliminate self-preference bias (a separate documented effect), and the source paper did not study many-turn agentic settings — crossagent's own setting.
…ace children (S5) On a failed delegation (failing check, scope violation, or failing verification -> delegation_verdict == failed), re-dispatch the task to the next larger peer. Recorded as option (a): a same-trace child job with parent_job_id set and the trace_id preserved, resolved through resolve_lineage — exactly what the shipped analytics.py escalation-rate definition counts, so no analytics change is needed. Ladder = ordered advisor[:model] rungs; each hop hands the remainder down, and the check/verify/scope/pass_env posture is propagated so the larger peer is held to the same write boundary and credential withholding. Bounded twice over: the rung list shrinks each hop, and MAX_NESTING_DEPTH is enforced (a LineageError is caught and audited as a skipped escalation, never a crash or a runaway loop).
…d the worker (S5) Add --verify-with/--verify-model/--escalate-to to 'start', persist them in command.json, and load them in the worker. The worker runs the fresh verification pass after the check/scope gates and persists verify_result on the SAME terminal transition (the only writer of terminal state); after that transition it re-dispatches up the escalation ladder when the delegation failed. Refine the result-command verdict line to name the actual failing gate. E2E worker tests drive real jobs through worker_main and reload from disk: a structured verify 'fail' blocks green while the delegate exited 0; a pass yields verified; the verifier argv carries no session flag and the artifact is its final user-turn arg; a prose verifier stays unverified; a failed delegation spawns a same-trace child with parent_job_id set at depth+1.
Three HIGH review findings fixedFixes for all three HIGH items from the review on #19, pushed to
Verification: 513 passed (was 505), ruff Not addressed here, deliberately, to keep the diff reviewable: the MEDIUM items ( |
Summary
Completes the delegation loop: a delegate's output can now be checked by an independent
peer agent, and a failed delegation can escalate up a ladder to a stronger one. This is
where second-opinion stops being a sibling feature and becomes the verification step inside
delegation.
Why an independent verifier, specifically
Two measured findings drive the design:
100% of runs but resolved only 44% (Claude Sonnet 4.5 97%/65%, Llama 4 Maverick 99%/18%,
Gemini 3.1 Pro 70%/50%).
falls 0.99 → 0.89 when a model grades work sitting in its own prior turn — same patches,
only the authorship framing varies — and the trigger is implicit authorship, i.e. the
artifact being present in the model's own context.
So the verifier must be a fresh session receiving the artifact as user-turn input.
A resumed session, or one where the artifact arrives as prior assistant context,
reintroduces exactly the channel the design removes.
Scope of that claim, stated honestly: this removes the implicit-authorship channel. It
does not "eliminate self-preference bias" — the source paper does not validate the
mitigation, residual self-recognition preference is a separate documented effect, and the
authors note they did not study many-turn agentic settings, which is precisely crossagent's
setting. The code comments and docstrings are worded accordingly.
Freshness is guaranteed by construction, not convention
The verifier command builder deliberately omits every session-attachment flag (resume,
fork, session-name) and never consults the session registry. There is no code path that
can accidentally attach the verifier to an existing conversation. Pinned by
test_verifier_command_has_no_session_flagsandtest_artifact_is_delivered_as_the_prompt_argument.Lineage environment variables are also stripped from the verifier
(
test_verifier_env_strips_lineage_vars) — otherwise it could be treated as part of the sameorchestration, which is a quieter way independence erodes.
Verdict composition
delegation_verdict()now composes three gates without adding a fifth state. Vetoesshort-circuit; reaching
verifiedrequires at least one affirmative gate:!= ok→failed(scope is a veto;okalone does not grant green)failed; exit 0 → counts as an affirmative passfail→failed;pass→ affirmativeunverified/error→ inconclusive: neither green nor a hard failThat last row matters: an advisor that can't produce a structured verdict yields
unverified, never a false pass. Free prose is not treated as a passing verification.Escalation ladder
Re-dispatch is recorded as a same-trace child job —
trace_idpreserved,parent_job_idset to the failed job, depth via the existingresolve_lineage, capped byMAX_NESTING_DEPTH.This was a deliberate choice over inventing a new representation:
analytics.py(merged in#18) already defines escalation as a failed delegation with a same-trace child, and that
definition shipped before this slice existed. Matching it means the escalation column
starts reporting real numbers with no change to
analytics.py. Had this slice recordedretries some other way, that column would have read 0% forever while escalations were
happening, and no test would have failed.
An escalation ladder is a recursion source, so a rung that would exceed the depth cap is
refused rather than attempted, and escalation never raises — a failure to escalate degrades
visibly instead of taking the job down.
Security
The verifier and any escalated child are themselves delegates, so they inherit the
protections from #19 rather than bypassing them: credential scrubbing via
credentials.pywith an explicit
pass_envopt-in, and the diff-scope assertion. Covered bytest_verifier_env_scrubs_credentials,test_verifier_env_honours_pass_env_opt_in, andtest_run_verification_secret_does_not_reach_verifier.Subprocess invocation uses argument lists with
shell=False, matchingcheck.py.Test plan
python3 -m pytest -q→ 505 passed (453 at this branch point, +52 here).python3 -m ruff check .andruff format --check .→ both green, no unrelated filestouched.
prompt argument; structured pass/fail; prose degrading to
unverified; a missing advisorexecutable not raising; secrets not reaching the verifier; lineage vars stripped; artifact
construction tolerating a non-git cwd; escalated job appearing as a same-trace child;
depth-cap refusal; and
schema_version1/2 records still loading.worker_mainand reloading statefrom disk, not just direct state-transition calls — an earlier slice in this series had
321 green unit tests while runtime persistence was entirely broken.
Known gaps
here and flows into feat(dashboard): analytics rollups, delegation history, and trace_mismatch cue #18's rollups through
delegation_verdict().other advisors relies on parsing a JSON answer out of the response and degrades to
unverifiedwhen that isn't present.