Skip to content

feat(verify): independent verification pass and escalation ladder - #20

Merged
datj9 merged 4 commits into
feat/delegation-scopefrom
feat/delegation-verify
Jul 26, 2026
Merged

datj9 merged 4 commits into
feat/delegation-scopefrom
feat/delegation-verify

Conversation

@datj9

@datj9 datj9 commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Stacked on #19 (feat/delegation-scope). Review that first; this branch's base is
feat/delegation-scope, not main.

Summary

Completes the delegation loop: a delegate's output can now be checked by an independent
peer agent
, and a failed delegation can escalate up a ladder to a stronger one. This is
where second-opinion stops being a sibling feature and becomes the verification step inside
delegation.

Why an independent verifier, specifically

Two measured findings drive the design:

  • "Done" is not evidence. On 50 SWE-bench Verified tasks, GPT-5 submitted a patch on
    100% of runs but resolved only 44% (Claude Sonnet 4.5 97%/65%, Llama 4 Maverick 99%/18%,
    Gemini 3.1 Pro 70%/50%).
  • Self-grading is measurably weaker than independent grading. Monitor correctness AUROC
    falls 0.99 → 0.89 when a model grades work sitting in its own prior turn — same patches,
    only the authorship framing varies — and the trigger is implicit authorship, i.e. the
    artifact being present in the model's own context.

So the verifier must be a fresh session receiving the artifact as user-turn input.
A resumed session, or one where the artifact arrives as prior assistant context,
reintroduces exactly the channel the design removes.

Scope of that claim, stated honestly: this removes the implicit-authorship channel. It
does not "eliminate self-preference bias" — the source paper does not validate the
mitigation, residual self-recognition preference is a separate documented effect, and the
authors note they did not study many-turn agentic settings, which is precisely crossagent's
setting. The code comments and docstrings are worded accordingly.

Freshness is guaranteed by construction, not convention

The verifier command builder deliberately omits every session-attachment flag (resume,
fork, session-name) and never consults the session registry. There is no code path that
can accidentally attach the verifier to an existing conversation. Pinned by
test_verifier_command_has_no_session_flags and
test_artifact_is_delivered_as_the_prompt_argument.

Lineage environment variables are also stripped from the verifier
(test_verifier_env_strips_lineage_vars) — otherwise it could be treated as part of the same
orchestration, which is a quieter way independence erodes.

Verdict composition

delegation_verdict() now composes three gates without adding a fifth state. Vetoes
short-circuit; reaching verified requires at least one affirmative gate:

  • scope != okfailed (scope is a veto; ok alone does not grant green)
  • check exit non-zero → failed; exit 0 → counts as an affirmative pass
  • verification failfailed; pass → affirmative
  • verification unverified / errorinconclusive: neither green nor a hard fail

That last row matters: an advisor that can't produce a structured verdict yields
unverified, never a false pass. Free prose is not treated as a passing verification.

Escalation ladder

Re-dispatch is recorded as a same-trace child jobtrace_id preserved,
parent_job_id set to the failed job, depth via the existing resolve_lineage, capped by
MAX_NESTING_DEPTH.

This was a deliberate choice over inventing a new representation: analytics.py (merged in
#18) already defines escalation as a failed delegation with a same-trace child, and that
definition shipped before this slice existed. Matching it means the escalation column
starts reporting real numbers with no change to analytics.py
. Had this slice recorded
retries some other way, that column would have read 0% forever while escalations were
happening, and no test would have failed.

An escalation ladder is a recursion source, so a rung that would exceed the depth cap is
refused rather than attempted, and escalation never raises — a failure to escalate degrades
visibly instead of taking the job down.

Security

The verifier and any escalated child are themselves delegates, so they inherit the
protections from #19 rather than bypassing them: credential scrubbing via credentials.py
with an explicit pass_env opt-in, and the diff-scope assertion. Covered by
test_verifier_env_scrubs_credentials, test_verifier_env_honours_pass_env_opt_in, and
test_run_verification_secret_does_not_reach_verifier.

Subprocess invocation uses argument lists with shell=False, matching check.py.

Test plan

  • python3 -m pytest -q505 passed (453 at this branch point, +52 here).
  • python3 -m ruff check . and ruff format --check . → both green, no unrelated files
    touched.
  • Coverage includes: no session flags in the verifier command; artifact delivered as the
    prompt argument; structured pass/fail; prose degrading to unverified; a missing advisor
    executable not raising; secrets not reaching the verifier; lineage vars stripped; artifact
    construction tolerating a non-git cwd; escalated job appearing as a same-trace child;
    depth-cap refusal; and schema_version 1/2 records still loading.
  • Includes end-to-end tests driving a real job through worker_main and reloading state
    from disk, not just direct state-transition calls — an earlier slice in this series had
    321 green unit tests while runtime persistence was entirely broken.

Known gaps

  • No dashboard UI for verification results or the escalation ladder; the data layer lands
    here and flows into feat(dashboard): analytics rollups, delegation history, and trace_mismatch cue #18's rollups through delegation_verdict().
  • Only Claude currently supports the machine-checkable output contract, so verification via
    other advisors relies on parsing a JSON answer out of the response and degrades to
    unverified when that isn't present.

datj9 added 4 commits July 26, 2026 17:07
…rdict (S5)

Add VerifyResultDict + Job.verify_result (schema v3, None = not requested,
distinct from a ran-and-failed verify per D7) and refactor delegation_verdict
into a gate combiner: any failed gate blocks the green path, a passing gate
greens it, an inconclusive/absent gate is neutral. A structured verify 'fail'
therefore blocks green even when the shell check passed (D6); a prose-only or
errored verifier degrades to inconclusive, never a pass (D4). Surface
verify_result in runtime_status.
…dation (S5)

Spawn a FRESH peer session that grades the delegate's artifact supplied as
user-turn input (D6) — build_verifier_command adds no resume/fork/name flag and
never reads the session registry, removing the implicit-authorship channel that
weakens self-grading. Prefer a machine-checkable contract where the advisor
exposes one (Claude --json-schema -> structured_output; new Advisor.json_schema_flag),
else parse a JSON verdict out of the answer; free prose -> unverified, never a
pass. The verifier is a delegate too: credentials scrubbed and CROSSAGENT_*
lineage stripped. Never raises — a broken verifier degrades to error/unverified.

Scope-honest: removes the implicit-authorship channel; does NOT claim to
eliminate self-preference bias (a separate documented effect), and the source
paper did not study many-turn agentic settings — crossagent's own setting.
…ace children (S5)

On a failed delegation (failing check, scope violation, or failing verification
-> delegation_verdict == failed), re-dispatch the task to the next larger peer.
Recorded as option (a): a same-trace child job with parent_job_id set and the
trace_id preserved, resolved through resolve_lineage — exactly what the shipped
analytics.py escalation-rate definition counts, so no analytics change is needed.

Ladder = ordered advisor[:model] rungs; each hop hands the remainder down, and
the check/verify/scope/pass_env posture is propagated so the larger peer is held
to the same write boundary and credential withholding. Bounded twice over: the
rung list shrinks each hop, and MAX_NESTING_DEPTH is enforced (a LineageError is
caught and audited as a skipped escalation, never a crash or a runaway loop).
…d the worker (S5)

Add --verify-with/--verify-model/--escalate-to to 'start', persist them in
command.json, and load them in the worker. The worker runs the fresh
verification pass after the check/scope gates and persists verify_result on the
SAME terminal transition (the only writer of terminal state); after that
transition it re-dispatches up the escalation ladder when the delegation failed.
Refine the result-command verdict line to name the actual failing gate.

E2E worker tests drive real jobs through worker_main and reload from disk: a
structured verify 'fail' blocks green while the delegate exited 0; a pass yields
verified; the verifier argv carries no session flag and the artifact is its
final user-turn arg; a prose verifier stays unverified; a failed delegation
spawns a same-trace child with parent_job_id set at depth+1.
@datj9 datj9 self-assigned this Jul 26, 2026
@datj9
datj9 merged commit 4d040e1 into feat/delegation-scope Jul 26, 2026
6 checks passed
@datj9

datj9 commented Jul 26, 2026

Copy link
Copy Markdown
Owner Author

Three HIGH review findings fixed

Fixes for all three HIGH items from the review on #19, pushed to feat/delegation-verify (PR #20). They reach this branch when #20 merges.

Finding Fix
Escalation fired on a cancelled job — re-dispatching to a larger, costlier advisor, i.e. the opposite of cancelling escalate.py:136 now gates on JobState.SUCCEEDED first, so only a declared gate failure (check/scope/verify) escalates
tempfile.mkstemp() outside its own try broke the documented "never raises" contract — leaving jobs permanently non-terminal with completed work discarded verify.py:284 extracts a @contextmanager _schema_file() with setup in its own try/except; failure degrades to an error outcome instead of propagating. Same OSError hardening applied to escalate.py
Credential scrubbing covered the job path but not the default synchronous path (crossagent --agent … --prompt …), which inherited the full unscrubbed environment with no --pass-env escape hatch cli.py:121 now passes a scrubbed env=, sharing one policy with the job path

TIMED_OUT was a deliberate decision, not a side effect. CANCELLED is unambiguous — explicit user intent, never escalate. TIMED_OUT cut both ways, and the call was do not escalate: a timeout yields no gate verdict, and a larger model is typically slower, so escalating tends to burn budget timing out again. Reasoning is documented at escalate.py:122-133 so a future reader can disagree with the decision rather than rediscover the behaviour.

Verification: 513 passed (was 505), ruff check and format --check green. The regression tests were confirmed genuine by reverting only the three source files to their pre-fix state while keeping the new tests — 6 tests fail against the old source. A test written after a fix that passes either way documents behaviour without proving the bug is gone.

Not addressed here, deliberately, to keep the diff reviewable: the MEDIUM items (obj placeholder naming; jobs.py/cli.py exceeding the 800-line guideline). The run_verification length MEDIUM is incidentally resolved by the context-manager extraction above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants