Track AI agent traffic on agent-facing endpoints (server-side PostHog)

[jannikmaierhoefer]

Why

AI crawlers and fetchers (GPTBot, ClaudeBot, PerplexityBot, claude-code, ChatGPT-User, ...) don't execute JavaScript, so posthog-js never sees them — neither on HTML pages nor on the agent-facing markdown endpoints, which are static files with no analytics attached. This traffic is currently invisible. For reference, Mintlify reports ~half of docs traffic is now AI agents; PostHog classified only ~207 pageviews as ai_crawler in the last 30 days on langfuse.com.

What

• proxy.ts (Next 16's middleware): fires a server-side agent_content_fetch PostHog event for

  • agent-facing content endpoints, any caller: .md markdown mirrors, llms*.txt, and Accept: text/markdown content negotiation;
  • regular HTML page routes, AI agents only: gated by a conservative allowlist of documented AI user agents (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, claude-code, Google-Extended, Amazonbot, Bytespider, CCBot, meta-external*, MistralAI-User, ...). Human traffic never fires an event — posthog-js covers that. Static assets and /api are excluded from the matcher.

  Events include $raw_user_agent and $ip, so PostHog's built-in $virt_* traffic classification (traffic type, bot name, bot operator) applies at query time. Fire-and-forget via event.waitUntil; content delivery is never blocked. Kill switch: DISABLE_AGENT_FETCH_TRACKING=true.

• MCP server segmentation: docs_mcp:execute_tool events now carry the caller's user-agent + IP (via AsyncLocalStorage populated in the route handler) instead of a fixed distinct id, and a new docs_mcp:initialize event captures MCP clientInfo (name/version) from the handshake — so we can tell Claude Code vs Cursor vs ChatGPT apart.

• The MCP server's own outbound fetches to langfuse.com now send User-Agent: langfuse-docs-mcp-server so they're distinguishable from external agents in the new events.

Events feed the new AI & agent traffic dashboard (the 'Agent content fetches by agent' tile populates once this deploys).

Verified

Ran the dev server with PostHog pointed at a local mock endpoint:

• GET /docs.md (GPTBot UA) → agent_content_fetch with content_kind: markdown_file; same for a human UA (agent endpoints track every caller).
• GET /docs/observability/overview with Accept: text/markdown → content_kind: markdown_negotiation.
• GET /faq (HTML) with GPTBot UA and GET / with ChatGPT-User UA → content_kind: page.
• GET /faq with a regular Chrome UA → no event.
• .md and content-negotiated responses still serve correctly through the proxy (200 + markdown body).
• Static export (Cloudflare mirror) is unaffected: Next only warns and disables middleware for output: export.

Notes for reviewers

• The proxy now runs on all page routes (assets/API excluded) — sub-ms regex check per request; the PostHog fetch only happens on matches. Vercel bills middleware invocations, roughly a few USD/month at current traffic.
• The HTML-route gate requires the UA allowlist in our code (unlike the markdown endpoints, where PostHog classifies at query time). It's deliberately conservative and worth an occasional refresh against openai.com/gptbot, docs.perplexity.ai/guides/bots, and github.com/ai-robots-txt.
• Event volume is unknown until deployed — if AI crawl volume turns out very high, we can add sampling or use the kill switch.
• proxy.ts is Next 16's convention (middleware.ts is deprecated there).

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
langfuse-docs	Ready	Preview, Comment	Jul 24, 2026 1:46pm

[github-actions]

@claude review

[claude]

LGTM — approving with the one inline nit noted; it's a labeling edge case in analytics only and doesn't affect content delivery.

Extended reasoning...

Overview

This PR adds server-side PostHog tracking for AI agent/crawler traffic across three files: a new proxy.ts (Next 16 middleware) that fires an agent_content_fetch event for .md, llms*.txt, and Accept: text/markdown requests; and updates to app/api/mcp/route.ts / lib/mcp-handler.ts that thread per-request user-agent/IP context through AsyncLocalStorage for MCP tool-usage and a new initialize handshake event, plus a self-identifying User-Agent header on the MCP server's own outbound fetches.

Security risks

None of significance. No auth, permissions, or user input trust boundaries are touched. The proxy event payload only forwards headers already visible to any observer (user-agent, IP, referrer, path) to PostHog, and analytics delivery is fire-and-forget (event.waitUntil / waitUntil) wrapped in try/catch so a PostHog outage cannot affect content delivery. The MCP initialize handler parses request bodies defensively (clones the request, catches JSON-parse failures) so malformed bodies from SSE reconnects are safely ignored.

Level of scrutiny

This warrants light-to-moderate scrutiny: it is a pure analytics/observability addition with no effect on served content, gated behind an explicit DISABLE_AGENT_FETCH_TRACKING kill switch and a posthogKey presence check, and the author documented manual verification of the exact request shapes affected (.md fetch, Accept: text/markdown negotiation, and MCP tool calls). It is not the kind of change (auth, billing, data write paths) that needs a deep line-by-line audit.

Other factors

One nit was found and is posted inline: the proxy.ts header-based matcher fires for any path with Accept: text/markdown, while the actual markdown-negotiation rewrite it is meant to track (next.config.mjs:265) excludes /api, /_next, md-src, .well-known, and .md/.txt/.json paths — so a subset of requests could get mislabeled content_kind: markdown_negotiation in the new dashboard even though they never received a markdown response. This is metric-noise only, does not affect any user-facing behavior, and does not block approval. No CODEOWNERS-protected paths are touched, and the diff is self-contained to the three listed files.

[claude]

🟡 The proxy.ts middleware matcher's header-based rule fires agent_content_fetch for any path carrying an Accept: text/markdown header, but the actual markdown-negotiation rewrite in next.config.mjs explicitly excludes /api, /_next, md-src, .well-known, and .md/.txt/.json paths. Requests to those excluded paths with that header get mislabeled content_kind: markdown_negotiation even though the real response is JSON/JS/API output, skewing the exact metric this PR is meant to measure. Fix: mirror the same negative-lookahead exclusions in the matcher's has-header source pattern.

Extended reasoning...

The bug: proxy.ts's config.matcher includes a header-based entry:

{
  source: "/:path*",
  has: [{ type: "header", key: "accept", value: "(.*text/markdown.*)" }],
}

This matches every path when the request carries an Accept header containing text/markdown — there is no exclusion list. Inside proxy(), contentKind(pathname) falls through to "markdown_negotiation" for anything that isn't /llms*.txt or *.md.

Why that's wrong: the actual content-negotiation rewrite this event is meant to track lives in next.config.mjs (beforeFiles, lines ~263-268):

{
  source:
    "/:path((?!api|_next|md-src|\\.well-known)(?!.*\\.md$)(?!.*\\.txt$)(?!.*\\.json$).*)",
  has: [{ type: "header", key: "accept", value: ".*text/markdown.*" }],
  destination: "/md-src/:path.md",
}

This rewrite is only applied when the path is not api, _next, md-src, .well-known, and does not end in .md, .txt, or .json. The middleware matcher was clearly intended to track exactly this rewrite (per the PR description and the comment above config.matcher: "Only agent-facing content endpoints are matched"), but the regex exclusions were dropped when the matcher's has-header arm was written.

Concrete trigger: any client that sends Accept: text/markdown on a request to an excluded path — e.g. GET /api/mcp, a .json asset, or a path under /_next/ — will:

1. Match the middleware's header-based matcher entry (no exclusions there).
2. Fire event.waitUntil(fetch(posthogHost + "/i/v0/e/", ...)) with content_kind: contentKind(pathname).
3. Since the pathname isn't /llms*.txt or *.md, contentKind() falls through to "markdown_negotiation".
4. Meanwhile, in next.config.mjs, the same path fails the negative-lookahead and is not rewritten to /md-src/... — the real response is the actual JSON/JS/API payload, not markdown.

The result is an agent_content_fetch event mislabeled content_kind: markdown_negotiation for a request that never actually got a markdown response. Since this PR's whole purpose is measuring markdown-negotiation traffic accurately (per the dashboard referenced in the description), this directly pollutes that metric.

Why nothing else catches it: the middleware and the rewrite are two independent regex definitions that happen to need the same exclusion set, but nothing enforces they stay in sync — there's no shared constant. The PR's own "Verified" section only tested the happy path (/docs/observability/overview with the header), which does match both the middleware and the rewrite, so this divergence wasn't caught in manual testing.

Fix: extract the exclusion pattern (api, _next, md-src, .well-known, .md/.txt/.json) into a single source of truth (e.g. a shared regex string) used by both the next.config.mjs rewrite source and the proxy.ts matcher's header-arm source, so the tracked scope can never drift from the rewritten scope again.

Severity: this is analytics-only — event.waitUntil never blocks or alters content delivery, so no user-facing behavior changes. The realistic trigger (an agent that sends Accept: text/markdown on non-page requests like /api or .json assets) is a narrower case than a full crawl, so the practical impact is limited data noise on the new dashboard rather than a functional defect. Recommending as a nit to fix before the metric is relied upon for reporting.