Repository navigation
test: shared fixture corpus for tests and benches - #1472
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fa7a94d02a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
lym953
left a comment
lym953
left a comment
There was a problem hiding this comment.
Could you explain more context? It's hard for me to understand what this PR does without reading your other PRs.
duncanista
changed the title
- dogstatsd: advance the metric name every 100 lines so n lines give min(n, 5000) contexts - traces: set BASE_NS to 2026-10-27T12:00:00Z and encode meta/metrics in key order - events: pad the SQS body to the exact size and put the LWA deadline 60 s from now - telemetry: emit extension logs in the logger::Formatter JSON shape
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e499bbbed
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- write_corpus: 600 events for the JSON shapes so each POST fits the 1 MiB maxBytes - traces: TRACE_HEADERS matches the reference payload's 100 traces
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e895b770b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Keep the #1467 dependency pins without ddsketch-agent, regenerate Cargo.lock and LICENSE-3rdparty.csv, and list libddwaf-sys in the cargo-machete ignore list: it is a dependency only to enable its aws-lc-rs feature.
…/cargo-machete/feature
…-fixtures/feature
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 36e52f855c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🟡 Changes recommended
Zero-sized packet requests omit all metrics, and directory iteration errors are silently discarded, allowing incomplete fixture corpora without diagnostics.
2 open issues
What changed in this PR
Adds a shared, generated fixture corpus for the planned benchmarks, fault-injection harness, and Lambda simulator. Extension runtime behavior is unchanged.
Changes:
- Adds generators for telemetry, logs, traces, DogStatsD packets, and trigger events.
- Adds corpus self-tests, on-demand file export, and usage documentation.
- Exposes the existing telemetry stitcher through a documentation-hidden public module for tests and benchmarks.
| File | Description |
|---|---|
bottlecap/tests/fixtures/traces.rs |
Generates traces and deterministic v0.4 MessagePack payloads. |
bottlecap/tests/fixtures/telemetry.rs |
Generates telemetry batches and split log records. |
bottlecap/tests/fixtures/README.md |
Documents generators and corpus export. |
bottlecap/tests/fixtures/mod.rs |
Organizes generators and writes the corpus with a manifest. |
bottlecap/tests/fixtures/logs.rs |
Supplies text, JSON, and out-of-memory log fixtures. |
bottlecap/tests/fixtures/events.rs |
Loads trigger fixtures and generates large SQS and DynamoDB events. |
bottlecap/tests/fixtures/dogstatsd.rs |
Generates metric lines and UDP datagrams. |
bottlecap/tests/fixtures_corpus.rs |
Checks fixture shapes, encoding, and stitching; supports export. |
bottlecap/src/extension/telemetry/stitch.rs |
Makes stitcher types and entry point public. |
bottlecap/src/extension/telemetry/mod.rs |
Exposes the stitcher module while hiding it from documentation. |
.gitignore |
Excludes generated corpus files. |
Review effort: Balanced
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3cfb84e43c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 021e3900d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- traces: identify the Node.js tracer, to match the nodejs22.x runtime of the telemetry, log and DogStatsD fixtures. - telemetry: invocation_batch ends with a warm platform.report without initDurationMs. report() stays the cold-start report. - telemetry: function and extension log event times are nondecreasing and stay before platform.runtimeDone. - write_corpus removes the files the previous MANIFEST.txt lists before it writes, so renamed or removed fixtures do not stay on disk.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac7b8591f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🟡 Changes recommended
Cleanup suppresses I/O failures that can leave stale fixtures untracked, while generated INSERT images omit the declared primary key and therefore do not represent valid stream records.
2 open issues
2 resolved since last review
Review effort: Balanced
…y, strict corpus cleanup - Add `telemetry::XRAY_TRACE_HEADER` and use it in `platform_start` and `lwa_next_headers`, so both describe the same invocation trace. - Drop `runtime` and `_dd.runtime_family` from `enrich_tags()`: Lambda does not give `AWS_EXECUTION_ENV` to extensions. Add `init_type:on-demand`. - Put the `Id` key in each DynamoDB `NewImage` and make it unique per record. - `write_corpus` ignores only `NotFound` when it cleans up the previous corpus.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab9ba948ac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Add `fixtures/invocation.rs` with the function, request ID, X-Ray root,
times, tracer and tags of the one simulated invocation, and build every
fixture from it. `telemetry::{FUNCTION_ARN, REQUEST_ID, XRAY_TRACE_HEADER}`
and `traces::BASE_NS` are re-exports.
- trace roots use the function name as resource, span the handler inside
the invocation window and contain their children; `_dd.p.tid`, service,
env and version match the invocation
- json logs use the shared X-Ray root, a cold start and per-line times;
text log and split-record times fall inside the invocation
- invocation_batch ends with the cold-start report, as REQUEST_ID is the
cold start that platform_batch describes
- enrich tags, DIST_8_TAGS, unique_tag_line, SQS and DynamoDB ARNs and
regions come from the shared values
- write_corpus refuses a manifest entry that is not a relative path under
the corpus dir, before it removes anything
- fixtures_share_one_invocation fails when the fixtures disagree
- docs state what the fixtures guarantee
…at pass today (#1473) **TL;DR:** Tests that run the real `bottlecap` binary against a fake Lambda API and an intake that fails on purpose. This PR adds the harness and the 10 checks that pass today. **Why:** Our worst bugs (hangs past the invocation deadline, SIGKILL at shutdown, lost data on intake errors) show up only at the process level, so unit tests cannot catch them. Each later fix adds its own test here. ## Overview Jira: [SVLS-9978](https://datadoghq.atlassian.net/browse/SVLS-9978) Stacked on #1472 (`jordan.gonzalez/test-fixtures/feature`). Contract tests that run the real `bottlecap` binary against a fake Lambda control plane and a Datadog intake that fails on demand. Each test asserts what Lambda or the customer sees: `/next` before the deadline, exit on SHUTDOWN, no panic. See `bottlecap/tests/HARNESS.md`. - `tests/common/`: fake Extensions and Telemetry API, fault intake (blackhole, SYN blackhole, slow, 5xx, 429, close mid body, size limits), process wrapper with RSS and CPU, port guard. - 4 test binaries with the 10 tests that pass on `main` today: invoke then shutdown exits cleanly; a 503 storm does not hold `/next`; an oversized trace is dropped and later traces flow; an intake that closes mid body does not hold `/next`; 4 config fields with a wrong type cost only that field; ports 8124 and 8126 held by another process. - `bottlecap/.config/nextest.toml`: the 4 binaries run one test at a time (`fixed-ports` group), because the binary binds fixed ports. - `AGENTS.md`: lists the build, test, lint and format commands that CI runs, and points to `bottlecap/tests/HARNESS.md` and the fixtures README, so agents find the harness. No production code. One Cargo change: `protobuf` 3.7 as a dev-dependency, already in `Cargo.lock`, so the tests can decode the metrics payloads. The tests for the open bugs come in a later PR, each ignored with the Jira ticket that tracks its fix. Size: 20 core lines, 2184 lines in total. ## Testing - applies on the previous PR of the stack: pass - `cargo fmt --all -- --check`: pass - `cargo clippy --workspace --all-targets --features default -- -D warnings`: pass - `cargo clippy --workspace --all-targets --features default,test-mode -- -D warnings`: pass - `cargo nextest run --workspace`: pass - plain `cargo test --test <binary>` for the 4 harness binaries: pass - the 4 test files hold exactly the 10 passing tests, no `#[ignore]`, no backlog reference: pass - Linux arm64 in Docker (rust 1.93.1), on the full harness at the end of the stack: the same 10 tests pass. [SVLS-9978]: https://datadoghq.atlassian.net/browse/SVLS-9978?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
TL;DR: One shared set of generated test inputs: Telemetry API batches, Lambda log lines, traces, DogStatsD packets and trigger events.
Why: The fault-injection harness (#1473) and the benchmarks (#1474) need the same realistic inputs. Today each test builds its own by hand. The extension does not change.
Overview
Jira: SVLS-9977
Stacked on #1471 (
jordan.gonzalez/cargo-machete/feature). This PR does not need the CI PRs below it. It sits on this stack so that the bench PR can build on both.Why this PR
We are adding test tools for the extension:
bottlecapbinary against a fake Lambda API and an intake that fails on demand (test: process-level fault-injection harness and the contract tests that pass today #1473)All of them need the same realistic inputs: Telemetry API batches, Lambda log lines, trace payloads, DogStatsD packets and trigger events. Today each test builds its own inputs by hand. This PR adds one small set of generators for those inputs, so every tool tests and measures the same shapes. Large inputs, such as a 220 KB SQS batch, are generated at test time instead of being checked into git.
On its own, this PR only adds test code. The extension does not change. The only non-test change is 9 lines that make the existing
stitchmodule#[doc(hidden)] pub, so integration tests and benches can call it.What to check here: that the generated shapes match what Lambda sends (for example the split-record format that the stitcher reassembles since #1415), and the self-test in
tests/fixtures_corpus.rs.What is in it
bottlecap/tests/fixtures/generates the inputs that the benches, the fault-injectionharness and the Lambda simulator share, so no large blob is in git:
telemetry: Telemetry API batches (platform events, function logs in 3 formats, splitrecords for the stitcher,
runtimeDonewith or without metrics,logsDropped, unknownevents);
logs: Lambda text and JSON lines, one OOM line per runtime;traces: synthetic traces and the 2000-span v0.4 msgpack payload, at a fixed base time;dogstatsd: datagrams for 50 metric names x 100 endpoints;events: the checked-in trigger payloads plus a 220 KB SQS and a 100-record DynamoDB event.tests/fixtures_corpus.rschecks the generators (for example, a split record stitches back).It needs
extension::telemetry::stitchfrom the integration tests, so the module andFragmentBufferbecome#[doc(hidden)] pub. No behavior change.Size: 13 core lines, 1058 lines in total.
Testing
cargo fmt --all -- --check: passcargo clippy --workspace --all-targets --features default -- -D warnings: passcargo clippy --workspace --all-targets --features default,test-mode -- -D warnings: passcargo nextest run --workspace: pass