Add eval framework for the Cerbos skills - #4
Open
alexolivier wants to merge 2 commits into
Open
alexolivier wants to merge 2 commits into
alexolivier wants to merge 2 commits into
Conversation
Headless Claude agent runs each skill in an isolated workspace; output is scored with deterministic checks (real cerbos compile) plus a calibrated LLM-as-judge. - 20 narrow binary judge criteria (12 policy, 8 synapse), doc-grounded - 13 deterministic assertions incl. real cerbos compile + test-suite run - Calibration gate: 25 human-labeled fixtures verify judge<->human agreement (100%) before any score is trusted; runs first in CI - CI runs on PRs, selecting only the suite(s) for the changed skill(s); shared-harness changes run both. Manual dispatch + filter for surgical runs - scripts/eval.sh for targeted local runs (--filter-pattern/--filter-failing) Covers compile-clean-but-wrong footguns: conditional DENY fails-open, role-scoped deny defeated across roles, weak happy-path tests, over-broad wildcards, unimported derived roles, role-policy misuse, unwired schemas; Synapse: unpinned extension URLs, wrong Envoy response mode, unverified JWTs. Signed-off-by: Alex Olivier <alex@alexolivier.me>
There was a problem hiding this comment.
Pull request overview
Adds a promptfoo-based evaluation harness for the Cerbos agent skills (cerbos-policy, cerbos-synapse-extension) that runs the real skills in isolated workspaces, scores outputs with deterministic checks (including real cerbos compile for policy), and gates LLM-judge scoring behind a calibration suite.
Changes:
- Introduces promptfoo suites for policy, synapse, and judge calibration, plus new real-world prompt sets and calibration fixtures.
- Adds custom promptfoo providers and a set of deterministic assertions (compile/static checks + targeted invariants) alongside an LLM-as-judge rubric.
- Adds a GitHub Actions workflow to run the evals conditionally on PRs touching skills/eval harness and publish summaries/artifacts.
Reviewed changes
Copilot reviewed 119 out of 120 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Adds a short repo-level note pointing readers to the eval framework and CI workflow. |
| evals/tests/synapse.yaml | Adds synapse prompt cases, including “modify” and supply-chain/JWT/envoy edge cases. |
| evals/tests/policy.yaml | Adds policy prompt cases, including “modify” and common authorization footguns. |
| evals/scripts/summary.js | Adds a script to convert promptfoo JSON output into a GitHub job summary. |
| evals/scripts/eval.sh | Adds a helper script to run suites with optional filtering/“rerun failing” support. |
| evals/README.md | Documents the eval framework, rationale, layout, local usage, CI behavior, and extension guidance. |
| evals/providers/skill-runner.js | Implements a promptfoo provider that runs the real skill via the Claude agent SDK in a temp workspace and returns transcript/files. |
| evals/providers/replay.js | Implements a promptfoo provider that replays calibration fixtures (no agent run) for judge drift detection. |
| evals/promptfoo.trigger.yaml | Adds a promptfoo config intended to validate skill triggering behavior on positive/negative prompts. |
| evals/promptfoo.synapse.yaml | Adds the synapse suite promptfoo config wiring provider + assertions. |
| evals/promptfoo.policy.yaml | Adds the policy suite promptfoo config wiring provider + assertions (incl. compile). |
| evals/promptfoo.calibration.yaml | Adds the calibration gate promptfoo config to verify judge↔human agreement before scoring. |
| evals/package.json | Adds the eval harness Node package definition and npm scripts. |
| evals/criteria/cerbos-synapse-extension.json | Defines narrow binary judge criteria for synapse outputs. |
| evals/criteria/cerbos-policy.json | Defines narrow binary judge criteria for policy outputs. |
| evals/calibration/synapse/good-url-checksum/transcript.txt | Adds calibration fixture transcript for checksum-pinned remote extension URLs. |
| evals/calibration/synapse/good-url-checksum/label.json | Adds calibration fixture label/expectations for checksum-pinned remote extension URLs. |
| evals/calibration/synapse/good-url-checksum/files/config.yaml | Adds calibration fixture files for checksum-pinned remote extension URLs. |
| evals/calibration/synapse/good-jwt-verified/transcript.txt | Adds calibration fixture transcript for verified JWT auxData configuration. |
| evals/calibration/synapse/good-jwt-verified/label.json | Adds calibration fixture label/expectations for verified JWT auxData configuration. |
| evals/calibration/synapse/good-jwt-verified/files/config.yaml | Adds calibration fixture files for verified JWT auxData configuration. |
| evals/calibration/synapse/good-envoy-dynamic-response/transcript.txt | Adds calibration fixture transcript for correct Envoy dynamic response mode. |
| evals/calibration/synapse/good-envoy-dynamic-response/label.json | Adds calibration fixture label/expectations for correct Envoy dynamic response mode. |
| evals/calibration/synapse/good-envoy-dynamic-response/files/extensions/envoy.star | Adds calibration fixture code for Envoy dynamic response mode. |
| evals/calibration/synapse/good-callmapper/transcript.txt | Adds calibration fixture transcript for simplest-kind declarative call mapper. |
| evals/calibration/synapse/good-callmapper/label.json | Adds calibration fixture label/expectations for simplest-kind declarative call mapper. |
| evals/calibration/synapse/good-callmapper/files/config.yaml | Adds calibration fixture files for simplest-kind declarative call mapper. |
| evals/calibration/synapse/bad-wrong-envoy/transcript.txt | Adds calibration fixture transcript for incorrect Envoy callback naming/license behavior. |
| evals/calibration/synapse/bad-wrong-envoy/label.json | Adds calibration fixture label/expectations for incorrect Envoy callback naming/license behavior. |
| evals/calibration/synapse/bad-wrong-envoy/files/extensions/envoy.star | Adds calibration fixture code for incorrect Envoy callback naming. |
| evals/calibration/synapse/bad-url-nochecksum/transcript.txt | Adds calibration fixture transcript for unpinned remote extension URLs. |
| evals/calibration/synapse/bad-url-nochecksum/label.json | Adds calibration fixture label/expectations for unpinned remote extension URLs. |
| evals/calibration/synapse/bad-url-nochecksum/files/config.yaml | Adds calibration fixture files for unpinned remote extension URLs. |
| evals/calibration/synapse/bad-jwt-unverified/transcript.txt | Adds calibration fixture transcript for unverified JWT auxData configuration. |
| evals/calibration/synapse/bad-jwt-unverified/label.json | Adds calibration fixture label/expectations for unverified JWT auxData configuration. |
| evals/calibration/synapse/bad-jwt-unverified/files/config.yaml | Adds calibration fixture files for unverified JWT auxData configuration. |
| evals/calibration/synapse/bad-envoy-static-mapping/transcript.txt | Adds calibration fixture transcript for incorrect Envoy response mode (static mapping). |
| evals/calibration/synapse/bad-envoy-static-mapping/label.json | Adds calibration fixture label/expectations for incorrect Envoy response mode (static mapping). |
| evals/calibration/synapse/bad-envoy-static-mapping/files/extensions/envoy.star | Adds calibration fixture code for incorrect Envoy response mode (static mapping). |
| evals/calibration/synapse/bad-envoy-missing-callback/transcript.txt | Adds calibration fixture transcript for missing map_cerbos_response callback. |
| evals/calibration/synapse/bad-envoy-missing-callback/label.json | Adds calibration fixture label/expectations for missing map_cerbos_response callback. |
| evals/calibration/synapse/bad-envoy-missing-callback/files/extensions/envoy.star | Adds calibration fixture code for missing map_cerbos_response callback. |
| evals/calibration/policy/good-scoped-roles/transcript.txt | Adds calibration fixture transcript for least-privilege wildcard usage patterns. |
| evals/calibration/policy/good-scoped-roles/label.json | Adds calibration fixture label/expectations for least-privilege wildcard usage patterns. |
| evals/calibration/policy/good-scoped-roles/files/resource_policies/report.yaml | Adds calibration fixture policy for least-privilege wildcard usage patterns. |
| evals/calibration/policy/good-schema-wired/transcript.txt | Adds calibration fixture transcript for wired schema validation. |
| evals/calibration/policy/good-schema-wired/label.json | Adds calibration fixture label/expectations for wired schema validation. |
| evals/calibration/policy/good-schema-wired/files/resource_policies/ticket.yaml | Adds calibration fixture policy wiring schemas via schemas: block. |
| evals/calibration/policy/good-schema-wired/files/resource_policies/ticket_test.yaml | Adds calibration fixture tests for schema-wired ticket policy. |
| evals/calibration/policy/good-schema-wired/files/_schemas/resources/ticket.json | Adds calibration fixture resource schema for ticket. |
| evals/calibration/policy/good-schema-wired/files/_schemas/principal.json | Adds calibration fixture principal schema. |
| evals/calibration/policy/good-role-narrow/transcript.txt | Adds calibration fixture transcript for correct role policy narrowing usage. |
| evals/calibration/policy/good-role-narrow/label.json | Adds calibration fixture label/expectations for correct role policy narrowing usage. |
| evals/calibration/policy/good-role-narrow/files/role_policies/billing_agent.yaml | Adds calibration fixture role policy demonstrating narrowing behavior. |
| evals/calibration/policy/good-role-narrow/files/resource_policies/invoice.yaml | Adds calibration fixture resource policy backing the role policy’s allowed actions. |
| evals/calibration/policy/good-negative-tests/transcript.txt | Adds calibration fixture transcript for negative tests and boundary coverage. |
| evals/calibration/policy/good-negative-tests/label.json | Adds calibration fixture label/expectations for negative tests and boundary coverage. |
| evals/calibration/policy/good-negative-tests/files/resource_policies/expense.yaml | Adds calibration fixture expense policy for negative tests. |
| evals/calibration/policy/good-negative-tests/files/resource_policies/expense_test.yaml | Adds calibration fixture tests asserting EFFECT_DENY and boundary cases. |
| evals/calibration/policy/good-multi-role-deny/transcript.txt | Adds calibration fixture transcript for cross-role deny override correctness. |
| evals/calibration/policy/good-multi-role-deny/label.json | Adds calibration fixture label/expectations for cross-role deny override correctness. |
| evals/calibration/policy/good-multi-role-deny/files/resource_policies/artifact.yaml | Adds calibration fixture policy using roles:["*"] deny with role-membership condition. |
| evals/calibration/policy/good-multi-role-deny/files/resource_policies/artifact_test.yaml | Adds calibration fixture tests proving deny holds for dual-role principals. |
| evals/calibration/policy/good-expense/transcript.txt | Adds calibration fixture transcript for a complete “good” expense solution. |
| evals/calibration/policy/good-expense/label.json | Adds calibration fixture label/expectations for the “good” expense solution. |
| evals/calibration/policy/good-expense/files/resource_policies/expense.yaml | Adds calibration fixture expense policy for the “good” solution. |
| evals/calibration/policy/good-expense/files/resource_policies/expense_test.yaml | Adds calibration fixture expense tests for the “good” solution. |
| evals/calibration/policy/good-derived-import/transcript.txt | Adds calibration fixture transcript for correct derived roles wiring/importing. |
| evals/calibration/policy/good-derived-import/label.json | Adds calibration fixture label/expectations for correct derived roles wiring/importing. |
| evals/calibration/policy/good-derived-import/files/resource_policies/document.yaml | Adds calibration fixture policy importing derived roles and using derivedRoles field. |
| evals/calibration/policy/good-derived-import/files/derived_roles/common_roles.yaml | Adds calibration fixture derived roles policy used by the document resource policy. |
| evals/calibration/policy/good-conditional-deny-guard/transcript.txt | Adds calibration fixture transcript for guarding denies against fail-open on missing attrs. |
| evals/calibration/policy/good-conditional-deny-guard/label.json | Adds calibration fixture label/expectations for guarding denies against fail-open. |
| evals/calibration/policy/good-conditional-deny-guard/files/resource_policies/record.yaml | Adds calibration fixture policy implementing guarded deny with has() pattern. |
| evals/calibration/policy/good-conditional-deny-guard/files/resource_policies/record_test.yaml | Adds calibration fixture tests covering missing-attribute denial behavior. |
| evals/calibration/policy/bad-wildcard-allow/transcript.txt | Adds calibration fixture transcript for an over-broad wildcard allow anti-pattern. |
| evals/calibration/policy/bad-wildcard-allow/label.json | Adds calibration fixture label/expectations for an over-broad wildcard allow anti-pattern. |
| evals/calibration/policy/bad-wildcard-allow/files/resource_policies/report.yaml | Adds calibration fixture policy demonstrating the wildcard allow failure mode. |
| evals/calibration/policy/bad-schema-orphan/transcript.txt | Adds calibration fixture transcript for orphaned schema files. |
| evals/calibration/policy/bad-schema-orphan/label.json | Adds calibration fixture label/expectations for orphaned schema files. |
| evals/calibration/policy/bad-schema-orphan/files/resource_policies/ticket.yaml | Adds calibration fixture policy missing schemas: wiring. |
| evals/calibration/policy/bad-schema-orphan/files/_schemas/resources/ticket.json | Adds calibration fixture schema file that is not referenced (orphan). |
| evals/calibration/policy/bad-schema-orphan/files/_schemas/principal.json | Adds calibration fixture schema file that is not referenced (orphan). |
| evals/calibration/policy/bad-role-grant/transcript.txt | Adds calibration fixture transcript for the “role policy used to grant” failure mode. |
| evals/calibration/policy/bad-role-grant/label.json | Adds calibration fixture label/expectations for the “role policy used to grant” failure mode. |
| evals/calibration/policy/bad-role-grant/files/role_policies/billing_agent.yaml | Adds calibration fixture role policy demonstrating the grant-misuse failure. |
| evals/calibration/policy/bad-multi-role-deny/transcript.txt | Adds calibration fixture transcript for deny overridden across roles failure mode. |
| evals/calibration/policy/bad-multi-role-deny/label.json | Adds calibration fixture label/expectations for deny overridden across roles failure mode. |
| evals/calibration/policy/bad-multi-role-deny/files/resource_policies/artifact.yaml | Adds calibration fixture policy demonstrating deny overridden across roles. |
| evals/calibration/policy/bad-happy-path-tests/transcript.txt | Adds calibration fixture transcript for happy-path-only tests failure mode. |
| evals/calibration/policy/bad-happy-path-tests/label.json | Adds calibration fixture label/expectations for happy-path-only tests failure mode. |
| evals/calibration/policy/bad-happy-path-tests/files/resource_policies/expense.yaml | Adds calibration fixture policy paired with happy-path-only tests. |
| evals/calibration/policy/bad-happy-path-tests/files/resource_policies/expense_test.yaml | Adds calibration fixture tests that fail to assert denials/boundaries. |
| evals/calibration/policy/bad-derived-orphan/transcript.txt | Adds calibration fixture transcript for derived roles not imported/orphaned policy failure mode. |
| evals/calibration/policy/bad-derived-orphan/label.json | Adds calibration fixture label/expectations for derived roles not imported/orphaned policy. |
| evals/calibration/policy/bad-derived-orphan/files/resource_policies/document.yaml | Adds calibration fixture policy missing importDerivedRoles for referenced derived role. |
| evals/calibration/policy/bad-derived-orphan/files/derived_roles/common_roles.yaml | Adds calibration fixture derived roles policy that becomes orphaned. |
| evals/calibration/policy/bad-deny-hole/transcript.txt | Adds calibration fixture transcript for deny-hole/coverage/audit/compile failure modes. |
| evals/calibration/policy/bad-deny-hole/label.json | Adds calibration fixture label/expectations for deny-hole/coverage/audit/compile failure modes. |
| evals/calibration/policy/bad-deny-hole/files/resource_policies/expense.yaml | Adds calibration fixture policy demonstrating a deny-hole case. |
| evals/calibration/policy/bad-conditional-deny-fails-open/transcript.txt | Adds calibration fixture transcript for unguarded deny conditions failing open. |
| evals/calibration/policy/bad-conditional-deny-fails-open/label.json | Adds calibration fixture label/expectations for unguarded deny conditions failing open. |
| evals/calibration/policy/bad-conditional-deny-fails-open/files/resource_policies/record.yaml | Adds calibration fixture policy demonstrating fail-open deny behavior. |
| evals/assertions/synapse-static.js | Adds static checks for common synapse extension mistakes (base64, entrypoints, wiring). |
| evals/assertions/synapse-jwt-keyset-verification.js | Adds deterministic checking for JWT keyset verification when auxData.jwt.token is used. |
| evals/assertions/synapse-extension-url-checksum.js | Adds deterministic checking for checksum pinning of remote extension URLs. |
| evals/assertions/synapse-envoy-response-mode.js | Adds deterministic checking for required map_cerbos_response when using cerbos_check_request mode. |
| evals/assertions/skill-fired.js | Adds an assertion validating whether the expected skill was invoked. |
| evals/assertions/policy-test-negative-coverage.js | Adds deterministic checking that policy tests assert at least one EFFECT_DENY. |
| evals/assertions/policy-static.js | Adds structural checks for policy output headers, rule names, and presence of tests. |
| evals/assertions/policy-schema-enforcement-wired.js | Adds deterministic checking for schema files being referenced via schemas: blocks. |
| evals/assertions/policy-role-policy-cannot-grant.js | Adds deterministic checking to catch role-policy-only “granting” attempts. |
| evals/assertions/policy-least-privilege-wildcards.js | Adds deterministic checking for unconditional wildcard-role EFFECT_ALLOW rules. |
| evals/assertions/policy-derived-roles-imported.js | Adds deterministic checking for derived roles wiring and orphan derived role policies. |
| evals/assertions/policy-conditional-deny-fails-open.js | Adds deterministic checking for fail-open deny conditions when missing attributes are signaled. |
| evals/assertions/judge.js | Adds LLM-as-judge evaluation + calibration agreement mode. |
| evals/assertions/cerbos-compile.js | Adds deterministic compile-and-run-tests check via Dockerized Cerbos compiler. |
| evals/.gitignore | Ignores local eval artifacts (JSON/JUnit outputs, node_modules, etc.). |
| .github/workflows/skill-evals.yml | Adds CI workflow to run calibration + selected suites, summarize, and upload artifacts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Add eval:trigger npm script and include it in the aggregate eval run - Run the trigger suite in CI whenever any suite is selected - Include trigger.json in the job summary; skip missing result files silently - Fail the workflow on trigger-suite regressions - Fix EVAL_JUDGE_MODEL default in judge.js docstring Signed-off-by: Alex Olivier <alex@alexolivier.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A promptfoo-based eval framework for the Cerbos agent skills (
cerbos-policy,cerbos-synapse-extension). A headless Claude agent runs each skill in an isolated workspace; the output is scored by deterministic checks (realcerbos compile) plus a calibrated LLM-as-judge.How it works
evals/criteria/*.json) — 20 narrow, binary failure-mode checks (12 policy, 8 synapse), each grounded in the Cerbos docs.evals/assertions/*.js) — realcerbos compile+ bundled test run, plus structural/static checks.evals/calibration/, 25 human-labeled good/bad fixtures) — replays labeled outputs through the same judge and fails fast if the judge no longer reproduces the human labels, so scores are never trusted blind. Runs first in CI.evals/providers/skill-runner.js) — drives the real skill via@anthropic-ai/claude-agent-sdk, captures the transcript + generated files.Coverage — the compile-clean-but-wrong footguns
Policy: conditional
DENYthat fails-open on a missing attribute · role-scoped deny defeated by an allow on another role · weak happy-path-only tests · over-broad wildcards · unimported derived roles · role policies misused to grant · unwired schemas — plus deny-holes, audit comments, requirement coverage, no fabricated attributes, actually-ran-compile.Synapse: unpinned remote extension URLs (checksum) · wrong Envoy response mode / missing
map_cerbos_response· unverified JWT (auxData.jwtwithoutkeySetID) — plus ask-for-licence-first, simplest kind/runtime, references loaded, correct entry-point names.CI (
.github/workflows/skill-evals.yml)Runs on PRs touching a skill or the harness, and selects only the suite(s) for the changed skill(s) — a change under
cerbos-policy/**or its eval files runs the policy suite; synapse likewise; a shared-harness change (judge/providers/calibration/deps) runs both. Calibration hard-gates. Manualworkflow_dispatchtakesrun(all/policy/synapse) and afilterregex. Locally,scripts/eval.sh <suite> [pattern](or--failing <prior.json>) targets a subset for surgical iteration.Validation
Known findings — intentionally left RED (real signal, not framework bugs)
cerbos-policyunder-triggers on "modify an existing policy" prompts. When a policy already exists in the workspace, the agent tends to edit the YAML directly without invoking the skill, soskill-firedfails on those modify cases. Left red as a genuine discoverability finding for the skill (the skill'sdescriptionshould cover modify/fix/strengthen-existing tasks).tests-when-askedshouldn't penalize unrequested tests;simplest-kindover-flagged an explicitly-WASM case.Setup
Requires the repo secret
ANTHROPIC_API_KEY. Optional repo varsEVAL_ACTOR_MODEL(defaultclaude-sonnet-5) andEVAL_JUDGE_MODEL(defaultclaude-haiku-4-5).