fix(tracing): continue inbound W3C trace context at the ACP boundary

[NiteshDhanpal]

Targets next, and builds on the merged #484 (business↔obs correlation) and #485 (Temporal interceptor) — this is a single logical change, no longer stacked.

Problem

After #484 (business↔obs correlation + Option A) and #485 (Temporal interceptor), an async turn's obs trace still detached from the ingress: the workflow + every activity started fresh traces, unlinked from the event/send API call.

Root cause (proven on infra-staging with per-hop traceparent probes): the ingress traceparent arrives at the agent's ACP server in the HTTP header, but FastACP never extracts it — the active OTel context stays empty. So start_workflow/signal (and the asyncio.create_task background dispatch) fire with no active span, the Temporal interceptor injects nothing, and the workflow/activities root fresh traces.

Fix

Extract + attach the inbound W3C context in the ASGI RequestIDMiddleware, which wraps the whole request (so the background task inherits it via create_task's context copy). Now the interceptor propagates the ingress trace across the Temporal boundary and the workflow/activity inherit it → one connected trace. Fail-open (obs never breaks a request).

Parsing is hardened: it uses the explicit TraceContextTextMapPropagator (immune to OTEL_PROPAGATORS=datadog silently disabling W3C), combines repeated tracestate header lines (W3C/RFC7230 MUST) instead of collapsing to the last, and excludes inbound baggage so attacker-controlled keys aren't pulled into downstream context. traceparent is single-valued, so trace linkage is unchanged.

Verified (infra-staging)

Trace 12c6290e…: 347 spans across all 4 services in one tree — gateway POST /agents/{id}/rpc → agent ACP → RunWorkflow → HandleSignal:receive_event → RunActivity → egp/identity/DB. Per-hop probes confirmed traceparent went from <none> → the ingress trace at every hop (ACP + worker). (Requires the control-plane header-forward fix, scaleapi/scale-agentex#396, to deliver the header.)

Tests

tests/test_trace_context_extraction.py — extract-makes-inbound-active, fail-open on missing header, safe detach, repeated tracestate combined, and inbound baggage not extracted. ruff clean.

🤖 Generated with Claude Code

Greptile Summary

The PR continues inbound W3C trace context at the ACP request boundary so asynchronous Temporal dispatch inherits the ingress trace.

• Adds fail-open extraction and lifecycle-safe detachment of inbound traceparent and tracestate.
• Preserves repeated tracestate header values while excluding inbound baggage.
• Adds focused tests for extraction, repeated headers, baggage exclusion, and safe cleanup.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/agentex/lib/sdk/fastacp/base/base_acp_server.py	Adds request-scoped W3C context extraction, repeated-tracestate preservation, fail-open handling, and guaranteed context detachment.
tests/test_trace_context_extraction.py	Adds focused regression coverage for active trace continuation, repeated tracestate fields, baggage exclusion, and safe detachment.

Sequence Diagram

sequenceDiagram
    participant Gateway
    participant ACP as ACP Middleware
    participant Handler
    participant Temporal
    Gateway->>ACP: HTTP request with traceparent/tracestate
    ACP->>ACP: Extract and attach W3C context
    ACP->>Handler: Run request under ingress context
    Handler->>Temporal: Start workflow or signal
    Note over Handler,Temporal: Async dispatch inherits attached context
    Temporal-->>Handler: Dispatch accepted
    Handler-->>ACP: Response
    ACP->>ACP: Detach context safely
    ACP-->>Gateway: HTTP response

Loading

_{Reviews (4): Last reviewed commit: "fix(tracing): parse ACP ingress headers ..." | Re-trigger Greptile}