Validation-First Agent Architecture

A small, self-contained pattern for building agent pipelines where an LLM does part of the work and code needs to trust — or catch — what it produced.

What this is

Two ideas, plus one honestly unsolved problem:

1. A contract pattern for keeping an LLM-authored artifact and its code-side validator from drifting apart, even though only one side of the pair is deterministic.
2. A tiered verification model that assigns cheap, deterministic checks first and reserves expensive, judgment-based review for what only judgment can catch — a shape related to the test pyramid and to generator-critic (reflection / LLM-as-judge) designs, but with the tier boundaries drawn precisely enough to state, for each tier, exactly what the previous one structurally cannot catch.
3. An open problem, named rather than hidden: one tier in this model has no code-enforceable trigger, in any domain. See docs/open-proposals.md.

What this is not

This is not the production system it was extracted from. That system is a private, working pipeline that has run for months against a real workload. This repo keeps the two ideas above and rebuilds them, from scratch, against a small toy domain — structured data extraction from an ambiguous document — chosen specifically because it can produce a case where a mechanical check passes on a wrong answer. Nothing here is a copy-paste of the original code, and nothing in reference-impl/ has run against real traffic.

Repo map

docs/
  contract-pattern.md   — the canonical-spec trick, and why LLM-in-the-loop
                           systems need it more than ordinary software does
  tier-model.md          — the verification tiers and their exact boundaries
  open-proposals.md      — the one tier that isn't solved here, named plainly
  ml-architecture-analogy.md — an honest, precise comparison to real neural
                           and cognitive architectures: what loosely maps,
                           and exactly what's missing to earn the comparison
reference-impl/
  contracts/             — one canonical spec per artifact; writer and
                           validator both generated from it, not from each
                           other
  tiers/                 — Tier 0 (schema), Tier 1 (mechanical completion),
                           Tier 3 (critic review) as separate, testable units
  pipeline.py             — wires the tiers into one runnable flow; the
                           point where Tier 2 would fire is marked and
                           explained, not implemented
  tests/                 — a round-trip suite, including the negative case
                           that proves Tier 1 and Tier 3 catch different
                           things
  example_run/           — one real, captured execution, generated fresh
                           for this repo
evidence/
  README.md              — the actual, narrowly-scoped claim these notes
                           support, and its known limits
  refund-email-eval.md    — the domain that bears on the core claims
  rsvp-csv-eval.md        — a documented risk from a domain outside scope

Status of each claim

Claim	Status
Contract pattern prevents shape drift	Enforced by tests/, runs in CI
Tier 0 / Tier 1 boundary is real (Tier 1 catches what Tier 0 can't)	Enforced by tests/, runs in CI
Tier 1 / Tier 3 boundary is real (Tier 1 passes what only Tier 3 catches)	Enforced by tests/, runs in CI
Tier 3 (critic review) catches the semantic error in this domain	Demonstrated once, captured in example_run/; not asserted on every run, and not claimed to be reliable across all inputs
Tier 2 (pre-execution plan review)	Not implemented. See docs/open-proposals.md
Blueprint helps on a domain with a genuine LLM-authored artifact	Tested twice (n=2, replicated); see evidence/ for the exact, narrow claim and its limits

License

MIT — see LICENSE.