Skip to content

perf(ingest): gzip and sample the gateway's own telemetry - #1214

Merged
Makisuo merged 2 commits into
fix/ingest-wal-shutdown-recoveryfrom
perf/ingest-self-telemetry-sampling
Oct 2, 2026
Merged

Makisuo merged 2 commits into
fix/ingest-wal-shutdown-recoveryfrom
perf/ingest-self-telemetry-sampling

docs(otel-spec): point gateway head sampling at INGEST_SELF_TRACE_SAM…

de19115
Select commit
Loading
Failed to load commit list.
Maple Review Bot / Maple / review succeeded Oct 2, 2026 in 3m 23s

Confidence 4/5 · No issues found

🟢 Confidence 4/5 · likely safe to merge
The sampler's weighting and the parent-based decision path are unit-tested and consistent with SAMPLE_RATE_EXPR; the only lasting risk is the prd volume cut, which is config-driven.
quality 100/100 · no findings · tests covered · risk medium

The gateway's four OTLP exporters now gzip, and its own traces are head-sampled by a new SelfTraceSampler driven by INGEST_SELF_TRACE_SAMPLE_RATIO (prd 0.05), which stamps SampleRate so warehouse throughput stays weighted. Safe to merge.

  • SelfTraceSampler wraps ParentBased(TraceIdRatioBased) and stamps SampleRate = 1/ratio on sampled spans
  • INGEST_SELF_TRACE_SAMPLE_RATIO goes through parse_sample_ratio, refused outside (0, 1]
  • All four OTLP exporters set OtlpCompression::Gzip; gzip-http added to Cargo features
  • prd sets 0.05 via resolveIngestSelfTraceSampleRatio; log and metric export stay complete

Fixed since the last review

  • ✅ F1 · with_sampler overrides the documented OTEL_TRACES_SAMPLER knob
What was checked
  • Stamped weight is 1/ratio (20 at 0.05), so SAMPLE_RATE_EXPR's >= 1 branch wins over TraceState (packages/domain/src/tinybird/datasources.ts:150)
  • Gateway extracts no incoming trace context (apps/ingest/src/main.rs:1990), so every root is local and ParentBased never follows a foreign sampled flag
  • Gzip targets the unchanged stock OTLP/HTTP receiver on the collector (packages/infra/otel-collector/collector-config.yaml:20)

de19115 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.