Standalone feature development has ended. This repository preserves the acceptance kernel and its research record. For new systems, use the alternatives listed below. Existing evidence remains subject to its original versions and limitations. Evalopt evaluates supplied evidence against policy; it is not a semantic truth oracle.
Make acceptance policy explicit and replayable.
Your harness runs the checks and supplies observations. Evalopt applies the policy and returns a decision with stable reason codes.
Expected output from the example below, using authored inputs:
ACCEPTED ('policy_satisfied',)
FAILED ('required_gate_failed:tests',)
BLOCKED ('tests_weakened',)
Run the example · Inspect the kernel
Five outcomes: ACCEPTED, BLOCKED, FAILED, UNSUPPORTED and UNVERIFIED.
Boundary: the host runs checks, detects weakened tests and supplies trustworthy evidence. Replay establishes consistency with that input, not the truth of the input.
Engineering: Zero runtime dependencies · State and replay tests 32 authored/generated conformance cases · Release checks
Conformance evidence only; no external capability or comparative result is claimed. Install · Host responsibilities
Apply acceptance policy to your CI observations
Have the host report its test and lint results through
AcceptanceInput. Evaluate them againstGovernancePolicy, save the decision with its policy and input, then deserialize and replay it later. The example shows passing gates, a failed test gate, and a host-reported weakened test suite producing different outcomes.This workflow is exercised with authored inputs. Evalopt neither runs those checks nor discovers weakened tests itself.
New systems should start from one of these destinations. Seven jobs, one destination each, in the order fixed by the migration map. These are job-oriented pointers, not tested drop-in replacements. The four sibling projects are independent of evalopt and of each other. This guide and the sources it inspected supply no automatic or lossless mapping of evalopt's five decision states, reason codes or decision records into any destination. A content hash binds bytes; a trusted signature binds a statement to an identity within its verification system; neither proves a claim true.
| Your job | Go to | What it does, in its own terms |
|---|---|---|
| General policy evaluation | OPA | Evaluates policies over structured data and query input and returns structured decisions. |
| Software provenance and attestation verification | in-toto attestation framework and Witness | A Statement binds a typed predicate to artifact digests; a DSSE envelope carries signatures; Witness verifies signed attestation collections against a signed policy. |
| Native merge enforcement | GitHub rulesets and required checks | Applies required status checks and bypass permissions to targeted branches; check states are set by write-capable producers. |
| Action-policy qualification | Actseal | Tests a model's action policy under a frozen policy against recorded evaluation evidence and replays it offline; the application owns execution. |
| Code-claim warrants | Dorian | Turns explicit code claims into checks sealed beside the code and revokes the warrant when a load-bearing check fails. |
| PR scope | Frontier Scout | Checks a diff against the base commit's scope policy and compiles the same policy into Claude Code permissions and hooks; a research preview. |
| Stochastic regression | ARCI | Freezes a repeated experiment, gates a change on relative non-inferiority and exports a replayable failure; its hashing helpers were adapted from evalopt with attribution, not as a dependency. |
The migration map covers each of the ten stable exports and six cross-cutting topics with its own limit, and the portfolio index describes the four sibling projects. The deprecated compatibility surfaces (graph, CLI, providers, Codex bridge, research loop and adapters) have no supported compatible migration target in this map; see entry 16.
Install into an isolated environment (Python 3.10–3.14), then run the example below:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install evalopt-graph==0.1.1On Windows PowerShell, activate with .venv\Scripts\Activate.ps1.
from evalopt_graph import AcceptanceDecision, AcceptanceInput, GovernancePolicy, evaluate_acceptance
policy = GovernancePolicy(required_gates=("tests", "lint"))
# What the harness observed: both required gates passed, nothing was skipped or deleted.
observed = AcceptanceInput(
observed_at="2026-07-20T12:00:00+00:00",
gate_results=(("tests", "PASS"), ("lint", "PASS")),
)
decision = evaluate_acceptance(policy, observed)
print(decision.status, decision.reasons) # ACCEPTED ('policy_satisfied',)
# Same policy, but the test gate failed.
failed = evaluate_acceptance(
policy,
AcceptanceInput(
observed_at="2026-07-20T12:00:00+00:00",
gate_results=(("tests", "FAIL"), ("lint", "PASS")),
),
)
print(failed.status, failed.reasons) # FAILED ('required_gate_failed:tests',)
# The host reports passing gates and a weakened test suite.
blocked = evaluate_acceptance(
policy,
AcceptanceInput(
observed_at="2026-07-20T12:00:00+00:00",
gate_results=(("tests", "PASS"), ("lint", "PASS")),
tests_weakened=True,
),
)
print(blocked.status, blocked.reasons) # BLOCKED ('tests_weakened',)
# Decisions are plain, content-addressed records: store them, then re-check them later.
restored = AcceptanceDecision.from_dict(decision.to_dict())
assert restored.validate()
assert restored.replay(policy, observed)The evaluator is pure, with no LLM in the loop: the same policy and input produce the same decision,
with stable reason codes and hashes for the policy, input, evidence records, and decision. Its
precedence is fail-closed: malformed or tampered input
FAILED, policy or contradiction block BLOCKED, observed gate failure FAILED, unavailable required
mechanism UNSUPPORTED, incomplete required evidence UNVERIFIED, otherwise ACCEPTED.
The package root exports exactly ten stable symbols in v0.1.x: GovernancePolicy, ClaimRecord,
EvidenceRequest, EvidenceMaterial, EvidenceAttestation, SupportAssessment, EvidenceAuthority,
AcceptanceInput, AcceptanceDecision and evaluate_acceptance.
GovernancePolicydeclares required gates, trust, freshness, evaluator, claim, verifier, and record authorization requirements.EvidenceAuthorityapplies controller-owned adapter policy to material already retrieved by a host.EvidenceAttestationbinds source policy, retrieval metadata, selected content, and hashes.SupportAssessmentseparately records whether selected content supports, contradicts, or is insufficient for a claim.evaluate_acceptanceevaluates a complete immutable observation once and returns a replayableAcceptanceDecision.
See kernel.py for the compact public contract and
test_kernel.py for complete evidence-authority examples.
In words: your host runs the checks, detects weakened tests, collects evidence and builds an AcceptanceInput.
evaluate_acceptance(policy, input) returns an AcceptanceDecision. The host stores policy, input and decision;
replay() later recomputes the decision and compares the record.
| The kernel owns | Your host owns |
|---|---|
| Deterministic policy evaluation | Model calls, prompts, and sampling |
| Terminal decision status and reason codes | Tools, edits, retries, and orchestration |
| Content-bound evidence and support records | Retrieval and protection of controller policy |
| Record authorization and freshness checks | Sandboxes, network and filesystem controls |
| Canonical serialization, hashes, and replay | Persistence, scheduling, graders, and metrics |
This boundary is intentional. Importing evalopt_graph loads the kernel without importing provider
SDKs, the compatibility graph, CLI code, filesystem adapters, benchmark adapters, or optional
frameworks.
Evalopt keeps three questions separate:
- Was this exact material retrieved under an allowed adapter policy? An attestation binds the retrieved bytes, selected span, source class, time, parser identity, and policy.
- Does the selected material support this exact claim? A support assessment records the verifier and its result independently of retrieval integrity.
- May these exact records authorize acceptance? Governance policy allowlists the authority policy, verifier, and evidence records before the decision can use them.
That structure makes evidence tampering and unsupported load-bearing claims visible. It does not prove semantic truth, deployed behavior, source correctness, or model capability.
- IMPLEMENTED / UNIT_PROVEN: pure API, immutable records, distinct terminal states, authorization, serialization, replay, and import isolation.
- INTEGRATION_PROVEN: the compatibility host and optional Harbor verifier mapping terminate through the same kernel decision path.
- CONFORMANCE_PROVEN: deterministic generated and authored cases exercise a bounded evidence boundary mechanism.
- What this does not claim: external benchmark results, comparison against other tools, or independent reproduction. The conformance cases show the mechanism behaves as specified; they say nothing about model capability.
Read the evidence report and external evaluation protocol for the claims, controls, and gaps. The protocol is preserved as a historical document; no new benchmark programme runs. There is no live-model governance campaign or official Docker SWE-bench result in this release.
The kernel was designed to accept host-supplied observations through AcceptanceInput, so that a
host could adapt observations from Codex, Claude Code, OpenHands, LangGraph, CI or its own runtime
into that input and store the returned decision beside the policy and input used to create it. The
host remains responsible for both steps. The examples below are preserved as they were. They
are not an invitation to build new evalopt integrations; new systems should start from the
alternatives above.
An optional Harbor 0.18 conformance task demonstrates verifier-side mapping without importing Harbor into the kernel. It validates integration wiring only; it is not an external benchmark.
The historical graph, standalone CLI, provider clients, Codex bridge, research loop, and filesystem adapters remain as deprecated compatibility surfaces. They receive no feature work, are not stable API or policy authorities, and have no supported compatible migration target in this map (entry 16 of the migration map). Anything that still imports evalopt should depend only on the ten root exports above.
- Migration map: where each evalopt concept goes
- Portfolio index: the four independent sibling projects
- Wind-down decisions
- Architecture and trust boundaries
- Evidence results and limitations
- External evaluation protocol (historical, preserved; no new benchmark programme)
- Harbor integration
- v0.1.1 final deprecation release notes
- v0.1.0 release notes
- Changelog
This repository is in maintenance: new features, integrations and root exports are not accepted.
Bug reports against 0.1.x with a version and a minimal reproduction, corrections to the
documentation and migration map, and private security reports remain open. Start with
the contribution guide and
follow the Code of Conduct.
Please report vulnerabilities privately using GitHub Security Advisories, not a public issue. See the security policy for supported versions and the bounded maintenance statement.
git clone https://github.com/ajaysurya1221/evalopt-graph.git
cd evalopt-graph
uv venv .venv
source .venv/bin/activate
uv pip install -e ".[dev]"
python -m compileall -q src tests scripts
python -m pytest -q
python -m ruff check src tests scripts
python -m ruff format --check src tests scripts
bash scripts/smoke_test.sh
uv buildMaintained by Ajay Surya Senthilrajan, with AI pair-programming recorded in commit trailers. See the tests, design records and release evidence linked here.
Released under the MIT License. Copyright © 2026 Ajay Surya Senthilrajan.