AgentOps — AI Agent Observability, Debugging & Evaluation

A mini-LangSmith: record, visualize, evaluate, and compare AI agent executions. Records agent runs (planner → tool → retrieval → LLM), renders them as an interactive execution graph, scores answers for faithfulness / hallucination, and tracks cost + latency.

Stack

• Backend FastAPI · SQLAlchemy 2 (async) · Pydantic v2 · PostgreSQL
• Frontend Next.js 14 (App Router) · TypeScript · Tailwind · React Flow · Recharts
• SDK agentops_sdk — thin Python tracer
• Infra Docker Compose

Quick start

cp .env.example .env
docker compose up --build          # db :5432, backend :8000, frontend :3000

# in another shell, once backend is healthy:
pip install httpx                  # for the demo/seed scripts
python seed.py                     # seeds prompts + tests, runs demo agent, evaluates

Open http://localhost:3000 (API docs at http://localhost:8000/docs).

What you get

• Dashboard — total runs, success rate, cost today, avg latency, cost-by-model, latency-by-stage.
• Trace Explorer — searchable/filterable run list.
• Run Detail — React Flow execution graph, per-node JSON inspector, cost breakdown, evaluation scorecard.
• Evaluations — failure analysis + per-run faithfulness / relevance / hallucination.
• Prompt Lab — version A/B: success-rate, cost, latency, hallucination deltas.
• Settings — API key + SDK snippet.

Instrument your own agent

from agentops_sdk import AgentOps
ao = AgentOps(base_url="http://localhost:8000", api_key="dev-key-change-me")

with ao.run("support-agent", user_id="u1", user_query="refund policy?") as run:
    with run.step("retrieval", "vector_search", query="refund policy") as s:
        s.detail["documents"] = [{"text": "Refund period is 30 days.", "source": "policy.md",
                                  "score": 0.9, "rank": 1, "used": True}]
    with run.step("llm", "answer", model="claude-haiku-4-5", provider="anthropic") as s:
        s.detail.update(response="The refund period is 30 days.",
                        input_tokens=800, output_tokens=32, cost=0.0015)
    run.finish("The refund period is 30 days.")

Steps buffer client-side and flush in one POST /trace when the run exits. Exceptions inside a run/step block are captured as status=failed — tracing never crashes your app.

Evaluation engine

Offline heuristics by default (token-overlap cosine) — zero cost, no API key. To use an LLM judge, set in .env:

AGENTOPS_EVAL_LLM_PROVIDER=gemini
AGENTOPS_EVAL_LLM_API_KEY=<google-ai-studio-key>
AGENTOPS_EVAL_LLM_MODEL=gemini-2.5-flash

Both paths return {faithfulness, retrieval_relevance, hallucination_score, unsupported_claims}.

API (FastAPI, see /docs)

Method	Path	Purpose
POST	/runs/create	open a run (auth: X-API-Key)
POST	/trace	submit steps
POST	/runs/end	close a run, roll up totals
GET	/runs	list runs (filters: status, agent_name, q)
GET	/runs/{id}	full trace + evaluation
POST	/evaluate/{id}	score a run
GET	/analytics/cost · /analytics/latency	analytics
POST	/prompts · GET /prompts/compare	prompt versions + A/B
POST	/tests/cases · /tests/run	regression suite
WS	/ws/traces	live trace stream

Database (12 tables)

users · agents · agent_runs · trace_steps · tool_calls · retrieval_events · llm_calls · evaluations · prompt_versions · test_cases · test_results · cost_metrics. Schema is created on backend startup (create_all). Swap in Alembic when it churns.

Layout

backend/app/   models, schemas, routers/{ingest,query,evaluate,analytics,prompts,tests}, eval/engine, ws
sdk/           agentops_sdk tracer
demo_agent.py  realistic traces (success / hallucinated / tool-failure)
seed.py        prompts + test cases + demo + evaluate
docker-compose.yml

Security model & known simplifications (deliberate)

• Writes require the API key (X-API-Key): ingest (/runs/*, /trace), /evaluate (prevents unauth triggering paid LLM-judge calls / economic DoS), and /prompts + /tests writes.
• Read endpoints are open (GET /runs, /runs/{id}, /analytics/*, /prompts) so the keyless browser dashboard can load them. This exposes trace contents to anyone who can reach the API and is not multi-tenant-safe — a shared browser key is not a real secret. For production, front reads with per-user session auth and scope every query by user_id.
• Heuristic eval is bag-of-words cosine, not embeddings — swap eval/engine._cosine.
• Analytics aggregate on-read; precompute if run volume grows.