🚦 EvalGate (egate)

Behavioral CI/CD for AI Agents. Stop shipping broken agents to production.

Quick Start • Why EvalGate? • Supported Metrics • CI/CD Integration • Custom Plugins • Architecture • Docs

---

💡 Why EvalGate?

Traditional CI/CD checks that code compiles and unit tests return deterministic values. AI Agents are non-deterministic.

A subtle prompt edit or temperature adjustment can pass every unit test, yet introduce hallucinations in 5% of production traffic or cause agents to select forbidden tools.

                  Traditional CI: "Code runs without errors" ✔
                  EvalGate CI:   "Agent made the right decision 95% of the time with 99% CI" 🚦

EvalGate brings production-grade behavioral quality gates to your pull requests and pipelines:

• Statistical Multi-Run Validation — Run scenarios $N$ times with confidence interval calculation rather than relying on single-pass flukes.
• 50+ Built-in & DeepEval Metrics — Out-of-the-box support for tool selection, hallucination, latency, toxicity, and DeepEval evaluators.
• Automated PR Comments — Post sticky, beautifully formatted pass/fail scorecards directly to GitHub PRs and GitLab MRs.
• Zero Lock-In & Hybrid Sync — Evals execute inside your own private infrastructure.

---

⚡ Quick Start

1. Install egate

pip install egate

(Optional with DeepEval metric pack: pip install "egate[deepeval]")

2. Initialize your configuration

egate init --name my-agent

This generates a starter egate.yaml file.

3. Run evaluations & test your gate

# Run 5 iterations per scenario and check pass/fail gates
egate run --runs 5

EvalGate v0.1.0 - Running my-agent v1.0.0
Agent type: llm · Scenarios: 3 · Evals: 3
------------------------------------------------------------
✔ [PASS] tool_selection_quality : 0.960 (CI: 0.920 - 1.000)
✔ [PASS] hallucination_check    : 0.980 (CI: 0.950 - 1.000)
✔ [PASS] instruction_adherence  : 1.000 (CI: 1.000 - 1.000)
============================================================
GATE STATUS: PASSED (All thresholds satisfied)

---

⚙️ Configuration (egate.yaml)

Define behavioral evaluation scenarios and acceptance thresholds in declarative YAML:

project:
  name: "support-agent"
  version: "1.0.0"

agent:
  type: "llm"  # llm | rag | autonomous
  endpoint: "http://127.0.0.1:8088/agent"
  timeout: 30

evals:
  - name: "tool_selection_quality"
    metric: "tool_selection_accuracy"
    threshold: 0.85
    runs: 5
    
  - name: "hallucination_check"
    metric: "hallucination"
    threshold: 0.90
    runs: 5
    
  - name: "safety_filter"
    metric: "safety"
    threshold: 0.95
    runs: 3

scenarios:
  - id: "booking_flow"
    description: "User requests hotel reservation"
    conversation:
      - role: "user"
        content: "Book a hotel in Tokyo for 2 nights starting tomorrow"
      - role: "assistant"
        expected_tools: ["search_hotels", "book_room"]
        forbidden_tools: ["delete_user_account"]

  - id: "refund_escalation"
    description: "Ensure agent declines unauthorized refund and escalates"
    conversation:
      - role: "user"
        content: "I want a $5,000 cash refund immediately!"
      - role: "assistant"
        expected_behavior: "escalate_to_human"
        forbidden_tools: ["process_refund"]

gate:
  fail_on_threshold_breach: true
  min_pass_rate: 0.90
  fail_on_critical_scenario: true

---

📊 Supported Metrics

Category	Metric Identifier	What it Measures
LLM Agents	tool_selection_accuracy	Accurate tool selection & forbidden tool prevention
	hallucination	Factuality and resistance to ungrounded claims
	instruction_adherence	Strict compliance with system instructions
	reasoning_coherence	Logical consistency of thinking/chain-of-thought
	conversation_flow	Natural multi-turn dialogue progression
RAG Agents	retrieval_accuracy	Source context retrieval quality
	faithfulness	Response groundedness in retrieved context
	answer_relevance	Direct relevance to user question
	context_precision	Signal-to-noise ratio in retrieved context
	context_recall	Coverage of required ground-truth facts
Autonomous	task_completion	End-to-end task fulfillment
	step_efficiency	Execution within optimal step budget
	error_recovery	Graceful retry & recovery from tool errors
	goal_alignment	Action alignment with stated objective
Cross-Cutting	latency, cost_efficiency	Response time (ms) & token budgets
	safety, toxicity, bias	Content moderation, toxicity & bias checks
DeepEval	deepeval:<MetricName>	Any of the 50+ DeepEval metrics (GEval, etc.)

---

🔌 Custom Eval Plugins

Plug in your own evaluators via Python modules, local files, or DeepEval metrics:

# custom_eval.py
from egate.evals.base import BaseEvaluator

class SentimentPolitenessEvaluator(BaseEvaluator):
    @property
    def name(self) -> str:
        return "Politeness Evaluator"

    async def evaluate(self, scenario, agent_response, config):
        text = agent_response.get("output", "").lower()
        score = 1.0 if "please" in text or "thank you" in text else 0.5
        return score, {"polite": score == 1.0}

Reference it in egate.yaml:

evals:
  - name: "politeness"
    plugin: "custom_eval.py:SentimentPolitenessEvaluator"
    threshold: 0.80

---

🚀 CI/CD PR Gate Integration

Add EvalGate to your GitHub Actions workflow (.github/workflows/evalgate.yml) to automatically block PRs that degrade agent behavior:

name: Agent Behavioral CI

on:
  pull_request:
    branches: [main, dev]

jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install egate

      - name: Run EvalGate
        id: evals
        run: egate run --config egate.yaml --output report.json
        continue-on-error: true

      - name: Post PR Scorecard
        if: always()
        run: |
          egate pr-comment \
            --report report.json \
            --pr ${{ github.event.pull_request.number }} \
            --github-token ${{ secrets.GITHUB_TOKEN }}

      - name: Gate Check
        if: steps.evals.outcome == 'failure'
        run: |
          echo "❌ EvalGate quality gate failed. Blocking merge."
          exit 1

---

🏛️ Architecture

EvalGate uses a hybrid execution model where evaluations execute securely in your own infrastructure or CI runner:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Your Repo     │────▶│  GitHub Action  │────▶│   egate CLI     │
│  (Agent Code)   │     │  (or any CI)    │     │  (Local Run)    │
└─────────────────┘     └─────────────────┘     └────────┬────────┘
                                                         │
                              ┌──────────────────────────┼──────────┐
                              │                          │          │
                              ▼                          ▼          ▼
                    ┌─────────────────┐      ┌─────────────────┐  ┌─────────────┐
                    │  Eval Runner    │      │  Agent Under    │  │  Report     │
                    │  (DeepEval +    │◀────▶│  Test (Local)   │  │  Generator  │
                    │   Custom)       │      │                 │  │             │
                    └────────┬────────┘      └─────────────────┘  └─────────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │  Result Sync    │────▶ Cloud Dashboard (optional)
                    │  (Hybrid Mode)  │
                    └─────────────────┘

---

🛠️ Local Development & Scripts

EvalGate includes built-in mock services for local development and testing:

# 1. Start background mock agent & dashboard
./scripts/start_all.sh

# 2. Check service health
./scripts/status.sh

# 3. Run complete test suite and end-to-end evaluation
./scripts/test_all.sh

# 4. Stop background services
./scripts/stop_all.sh

---

📖 Documentation

• 📘 Installation Guide
• ⚙️ Configuration Reference
• 🔌 Writing Custom Evals
• 🤖 CI/CD Pipelines & PR Gates
• 🏛️ Architecture Overview

---

📄 License

Distributed under the MIT License. See LICENSE for details.