perf(ingest): gzip and sample the gateway's own telemetry

[Makisuo]

Stacked on #1213.

Problem

The gateway traces 100% of its own requests and exports them as uncompressed OTLP/HTTP protobuf. In prd that's ~4.6M spans/hour (~7 spans per request), about 13x every other Maple service combined. It costs cross-AZ transfer to the in-VPC collector, collector CPU, the collector's uploads, and warehouse ingestion and storage for the internal org.

Change

• Gzip on all four OTLP exporters (spans, logs, two metric exporters) via the gzip-http feature.
• New INGEST_SELF_TRACE_SAMPLE_RATIO (default 1.0; anything outside (0, 1] refuses to boot). SelfTraceSampler wraps ParentBased(TraceIdRatioBased). When the ratio is below 1 it stamps SampleRate = 1/ratio on every sampled span, so the warehouse's SampleRate weighting (docs/sampling-throughput.md) keeps throughput and request counts right.
• prd sets 0.05 through resolveIngestSelfTraceSampleRatio(stage); previews stay at 1.0. Logs and metrics are not sampled.

Alert check

At 5%, prd keeps ~39k gateway spans per 10 min, so "Ingest gateway silent" (count < 500) and "Ingest pipeline stopped" (count < 1000) keep a wide margin. Both use count(); switching them to sum(SampleRate) would make them sampling-proof.

Tests

• self_trace_sample_ratio_must_be_in_the_unit_interval
• sampled_self_spans_carry_their_sample_rate: sampled root, dropped root, child of a sampled parent, and ratio 1.0 adds nothing.
• stage.test.ts case for the per-stage ratio.

Notes

The gateway doesn't extract incoming trace context, so every root is local. If it ever starts honouring a caller's traceparent, ParentBased would follow the caller's sampled flag and the stamped SampleRate would be wrong.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 8fb6f1ef-6750-4ee2-a301-6542fdedd03e

📥 Commits

Reviewing files that changed from the base of the PR and between ea5f8ef and de19115.

⛔ Files ignored due to path filters (1)

• apps/ingest/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (6)

• apps/ingest/Cargo.toml
• apps/ingest/alchemy.run.ts
• apps/ingest/src/main.rs
• docs/otel-spec/resource-and-config.md
• packages/infra/src/aws/stage.test.ts
• packages/infra/src/aws/stage.ts

 _______________________________________________
< Optimization hinders evolution. - Alan Perlis >
 -----------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[maple-review-bot]

Maple review

🟢 Confidence 4/5 · likely safe to merge
Sampling 95% of the gateway's own traces changes what the internal-org dashboards and alerts see; the weighting path (SampleRate >= 1) is correct, but nobody measured the alert thresholds at 5%.
_{quality 98/100 · 1 note · tests covered · risk medium}

Gzips all four OTLP exporters and head-samples the gateway's own traces through a new INGEST_SELF_TRACE_SAMPLE_RATIO (0.05 in prd), stamping SampleRate = 1/ratio so the warehouse's weighting keeps request counts true. The sampling math and the (0, 1] validation are sound.

• SelfTraceSampler wraps ParentBased(TraceIdRatioBased) and stamps SampleRate on sampled spans
• INGEST_SELF_TRACE_SAMPLE_RATIO parsed in (0, 1], default 1.0, refuses to boot otherwise
• resolveIngestSelfTraceSampleRatio returns 0.05 for prd only
• All four OTLP HTTP exporters now send gzip

Findings

🔵 Note · F1 · with_sampler overrides the documented OTEL_TRACES_SAMPLER knob

correctness · apps/ingest/src/main.rs:1968

Building the provider with an explicit sampler replaces the sampler the SDK derives from OTEL_TRACES_SAMPLER/OTEL_TRACES_SAMPLER_ARG, which docs/otel-spec/resource-and-config.md still names as how apps/ingest sets head sampling (the comment this diff deletes at line 1954 said the same). A deployment that sets either variable now gets 100% of the ratio's behaviour and no error, in every stage — not just prd. Either fall back to the env-configured sampler when INGEST_SELF_TRACE_SAMPLE_RATIO is unset, or drop those variables from the ingest row of that doc.

🤖 Prompt to fix this finding with an AI agent

Findings from an automated review of commit 334defd79626053c6c06e3e88c6a4cba7bf17a8a. Verify each one against the current code before changing anything, fix only those that still apply, and keep each fix to the lines it names.

---

F1 · Note · correctness · apps/ingest/src/main.rs:1968
`with_sampler` overrides the documented `OTEL_TRACES_SAMPLER` knob
Building the provider with an explicit sampler replaces the sampler the SDK derives from `OTEL_TRACES_SAMPLER`/`OTEL_TRACES_SAMPLER_ARG`, which `docs/otel-spec/resource-and-config.md` still names as how `apps/ingest` sets head sampling (the comment this diff deletes at line 1954 said the same). A deployment that sets either variable now gets 100% of the ratio's behaviour and no error, in every stage — not just prd. Either fall back to the env-configured sampler when `INGEST_SELF_TRACE_SAMPLE_RATIO` is unset, or drop those variables from the ingest row of that doc.

What was checked

• SampleRate weight of 1/ratio is >= 1, so SAMPLE_RATE_EXPR picks it up (docs/sampling-throughput.md:24)
• Ratio 1.0 yields no attribute, matching today's parentbased_always_on default
• The gateway never extracts traceparent (no match in apps/ingest), so every root really is local and ParentBased cannot follow a caller's flag

_{334defd · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}

[maple-review-bot]

Maple review

🟢 Confidence 4/5 · likely safe to merge
The sampler's weighting and the parent-based decision path are unit-tested and consistent with SAMPLE_RATE_EXPR; the only lasting risk is the prd volume cut, which is config-driven.
_{quality 100/100 · no findings · tests covered · risk medium}

The gateway's four OTLP exporters now gzip, and its own traces are head-sampled by a new SelfTraceSampler driven by INGEST_SELF_TRACE_SAMPLE_RATIO (prd 0.05), which stamps SampleRate so warehouse throughput stays weighted. Safe to merge.

• SelfTraceSampler wraps ParentBased(TraceIdRatioBased) and stamps SampleRate = 1/ratio on sampled spans
• INGEST_SELF_TRACE_SAMPLE_RATIO goes through parse_sample_ratio, refused outside (0, 1]
• All four OTLP exporters set OtlpCompression::Gzip; gzip-http added to Cargo features
• prd sets 0.05 via resolveIngestSelfTraceSampleRatio; log and metric export stay complete

Fixed since the last review

• ✅ F1 · with_sampler overrides the documented OTEL_TRACES_SAMPLER knob

What was checked

• Stamped weight is 1/ratio (20 at 0.05), so SAMPLE_RATE_EXPR's >= 1 branch wins over TraceState (packages/domain/src/tinybird/datasources.ts:150)
• Gateway extracts no incoming trace context (apps/ingest/src/main.rs:1990), so every root is local and ParentBased never follows a foreign sampled flag
• Gzip targets the unchanged stock OTLP/HTTP receiver on the collector (packages/infra/otel-collector/collector-config.yaml:20)

_{de19115 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}