Evergreen: Efficient Claim Verification for Semantic Aggregates

This repository contains the code and experiments for Evergreen: Efficient Claim Verification for Semantic Aggregates.

Repository Layout

src/evergreen/     # The Evergreen system (compiler, optimizer, provenance)
tests/             # Unit and integration tests
docs/              # API reference (built into site/llms-full.txt)
experiments/       # Benchmark, baselines, evaluation harness, and results
  eval/semantic_aggregate/    # Aggregation queries (AI_AGG) that produce the aggregates
  eval/claim_evaluator/       # The 32 benchmark claims as verification queries
  results/semantic_aggregate/ # Raw aggregate outputs and decomposed claims
  results/claim_evaluator/    # Per-claim results (cost, latency, verdict, provenance)
  figures/                    # Plots and tables

Prerequisites

• Python 3.12+
• uv
• DuckDB
• Snowflake account for Cortex AI language and embedding model access
• Deno (only required for RLM experiments)

Setup

Install dependencies:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync

To run experiments with RLM, install Deno and then restart your shell (or re-source your shell profile) so the updated PATH takes effect:

curl -fsSL https://deno.land/install.sh | sh

Configure Snowflake connection by creating ~/.snowflake/connections.toml:

[evergreen]
account = "<account identifier>"
user = "<login name>"
password = "<programmatic access token>"
role = "<role>"
database = "<database>"
schema = "<schema>"
warehouse = "<warehouse>"

Set cache directory:

export EVERGREEN_CACHE_DIR_ROOT=~/.cache/evergreen/

Set log directory (holds experiment logs and dataframe checkpoints):

export EVERGREEN_EXPERIMENT_DIR_ROOT=experiments/

Build the documentation, which generates site/llms-full.txt (the API reference used by the claim compiler):

uv run mkdocs build

Run tests to ensure correct setup:

uv run pytest

Data Preparation

The datasets are sourced from the Yelp Open Dataset and the Customer Support on Twitter Dataset.

Download both datasets and build the evaluation datasets:

./experiments/scripts/prepare_data.sh

Add embeddings (requires a configured Snowflake connection):

./experiments/scripts/add_embeddings.sh

Reproducing Results

Run the full evaluation:

uv run python -m experiments.scripts.run_claim_evaluators \
    --impls base_rm rag_agent rlm \
    --lms claude-opus-4-6 claude-sonnet-4-6 claude-haiku-4-5

uv run python -m experiments.scripts.run_claim_evaluators \
    --impls evg_ref

uv run mkdocs build && uv run python -m experiments.scripts.run_claim_evaluators --compile

uv run python -m experiments.scripts.run_claim_evaluators \
    --impls evg_opt evg_opt_ref_query \
    --lms claude-opus-4-6 claude-sonnet-4-6 claude-haiku-4-5 qwen3-vl-235b-a22b qwen3-next-80b-a3b

uv run python -m experiments.scripts.run_claim_evaluators \
    --impls evg_unopt evg_unopt_ref_query \
    --lms claude-opus-4-6 claude-sonnet-4-6 claude-haiku-4-5

uv run python -m experiments.scripts.run_claim_evaluators \
    --impls evg_abl_no_es evg_abl_no_rs evg_abl_no_ecs evg_abl_no_of evg_abl_no_sf evg_abl_no_pc \
    --lms claude-haiku-4-5

uv run python -m experiments.scripts.run_claim_evaluators --eval_sim_filter

Consider moving any existing results in experiments/results/ to a separate directory to avoid overwriting or double counting results.

Generate figures:

uv run python -m experiments.scripts.plot_results

Figures are saved to experiments/figures/.