Repository navigation
bench: criterion suite for the extension hot paths - #1474
duncanista wants to merge 30 commits into
Conversation
…act tests that pass today
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 341f23b39d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
duncanista
changed the title
- bench-docker-arm64.sh: pass the cargo arguments to the container shell as positional parameters, so criterion filters are not re-parsed. - bench-compare.sh: make a relative CARGO_TARGET_DIR absolute before cd, so cargo, critcmp and the baseline notes use the same directory. - bench-compare.sh: record a dirty marker and a hash of the uncommitted diff in the baseline notes. - benches/README.md: drop the old branch name.
- dogstatsd: advance the metric name every 100 lines so n lines give min(n, 5000) contexts - traces: set BASE_NS to 2026-10-27T12:00:00Z and encode meta/metrics in key order - events: pad the SQS body to the exact size and put the LWA deadline 60 s from now - telemetry: emit extension logs in the logger::Formatter JSON shape
- fault_injection_test: measure the /next budget from when the INVOKE was queued - harness: shutdown_cleanly asserts the exit came within the SHUTDOWN deadline
- write_corpus: 600 events for the JSON shapes so each POST fits the 1 MiB maxBytes - traces: TRACE_HEADERS matches the reference payload's 100 traces
- oversized trace: wait for /next against the 10 s INVOKE deadline, and assert the rejected trace is never sent again after the rejection is lifted - lifecycle: read the intake before SHUTDOWN so the final flush cannot hide a missing end-of-invocation flush
Keep the #1467 dependency pins without ddsketch-agent, regenerate Cargo.lock and LICENSE-3rdparty.csv, and list libddwaf-sys in the cargo-machete ignore list: it is a dependency only to enable its aws-lc-rs feature.
…modes in blackhole
…/cargo-machete/feature
…-fixtures/feature
…tdown deadline in shutdown()
…TDOWN, deliver telemetry only after an accepted subscribe
…t-harness/feature
…h-suite/feature Adapt the benches to what the merge brings in from main and the fixtures: - init: the fixtures build the enrichment tags with enrich_tags(), not ENRICH_TAGS. - traces: the libdatadog bump (#1467) renamed obfuscate_span to obfuscate_pb_span, HttpConfig.remove_paths_with_digits to remove_path_digits, and new_without_connection_pooling to new_periodic. tag_replace_rules is now built by startup::build_tag_replace_rules, made doc(hidden) pub so the bench builds it as production does.
**TL;DR:** One shared set of generated test inputs: Telemetry API batches, Lambda log lines, traces, DogStatsD packets and trigger events. **Why:** The fault-injection harness (#1473) and the benchmarks (#1474) need the same realistic inputs. Today each test builds its own by hand. The extension does not change. ## Overview Jira: [SVLS-9977](https://datadoghq.atlassian.net/browse/SVLS-9977) Stacked on #1471 (`jordan.gonzalez/cargo-machete/feature`). This PR does not need the CI PRs below it. It sits on this stack so that the bench PR can build on both. ### Why this PR We are adding test tools for the extension: - criterion benchmarks for the hot paths (#1474) - a fault-injection harness that runs the real `bottlecap` binary against a fake Lambda API and an intake that fails on demand (#1473) - later, a local Lambda simulator All of them need the same realistic inputs: Telemetry API batches, Lambda log lines, trace payloads, DogStatsD packets and trigger events. Today each test builds its own inputs by hand. This PR adds one small set of generators for those inputs, so every tool tests and measures the same shapes. Large inputs, such as a 220 KB SQS batch, are generated at test time instead of being checked into git. On its own, this PR only adds test code. The extension does not change. The only non-test change is 9 lines that make the existing `stitch` module `#[doc(hidden)] pub`, so integration tests and benches can call it. What to check here: that the generated shapes match what Lambda sends (for example the split-record format that the stitcher reassembles since #1415), and the self-test in `tests/fixtures_corpus.rs`. ### What is in it `bottlecap/tests/fixtures/` generates the inputs that the benches, the fault-injection harness and the Lambda simulator share, so no large blob is in git: - `telemetry`: Telemetry API batches (platform events, function logs in 3 formats, split records for the stitcher, `runtimeDone` with or without metrics, `logsDropped`, unknown events); - `logs`: Lambda text and JSON lines, one OOM line per runtime; - `traces`: synthetic traces and the 2000-span v0.4 msgpack payload, at a fixed base time; - `dogstatsd`: datagrams for 50 metric names x 100 endpoints; - `events`: the checked-in trigger payloads plus a 220 KB SQS and a 100-record DynamoDB event. `tests/fixtures_corpus.rs` checks the generators (for example, a split record stitches back). It needs `extension::telemetry::stitch` from the integration tests, so the module and `FragmentBuffer` become `#[doc(hidden)] pub`. No behavior change. Size: 13 core lines, 1058 lines in total. ## Testing - applies on the previous PR of the stack: pass - `cargo fmt --all -- --check`: pass - `cargo clippy --workspace --all-targets --features default -- -D warnings`: pass - `cargo clippy --workspace --all-targets --features default,test-mode -- -D warnings`: pass - `cargo nextest run --workspace`: pass [SVLS-9977]: https://datadoghq.atlassian.net/browse/SVLS-9977?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8c95bd0b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🟡 Changes recommended
The trace benchmark filters out every chunk, while the initialization benchmark drops clients before RSS sampling and includes teardown in its timing, making the performance and memory comparisons misleading.
4 open issues
What changed in this PR
Adds Criterion benchmarks for the Lambda extension’s hot paths, using shared fixtures from the preceding test work to support before-and-after performance comparisons. Production changes expose existing helpers without changing behavior.
Changes:
- Adds six benchmark targets covering logs, traces, invocation parsing, enhanced metrics, configuration, and initialization.
- Adds baseline comparison tooling and a Linux arm64 Docker runner.
- Documents measurement practices, initial results, and benchmark coverage.
| File | Description |
|---|---|
scripts/bench-docker-arm64.sh |
Runs benchmarks in an isolated Linux arm64 container. |
scripts/bench-compare.sh |
Saves, compares, exports, and lists baselines. |
bottlecap/src/startup.rs |
Exposes tag-replacement rule construction. |
bottlecap/src/proc/mod.rs |
Exposes path-based readers for fixture benchmarks. |
bottlecap/src/logs/lambda/processor.rs |
Exposes the OOM detection helper. |
bottlecap/Cargo.toml |
Adds Criterion and registers six benchmark targets. |
bottlecap/Cargo.lock |
Locks Criterion and its dependencies. |
bottlecap/benches/traces.rs |
Benchmarks trace decoding, processing, stats, and flushing. |
bottlecap/benches/README.md |
Documents execution, comparisons, coverage, and baseline results. |
bottlecap/benches/logs.rs |
Benchmarks log processing, stitching, batching, and compression. |
bottlecap/benches/invocation.rs |
Benchmarks trigger identification and payload handling. |
bottlecap/benches/init.rs |
Measures initialization components and client construction. |
bottlecap/benches/enhanced_metrics.rs |
Benchmarks resource readers and metric tag construction. |
bottlecap/benches/config.rs |
Benchmarks configuration loading and rule compilation. |
bottlecap/benches/common/mod.rs |
Shares allocator, runtime, configuration, and benchmark setup. |
bottlecap/benches/baselines/mac-main.txt |
Records initial baseline metadata. |
.gitignore |
Excludes baseline exports and arm64 build output. |
Review effort: Balanced
- bench-docker-arm64.sh writes the baseline-<name>.txt note (machine, container, load, profile, commit) that bench-compare.sh compare prints; fix its BENCH_FAST profile label. - bench-compare.sh records every CARGO_PROFILE_BENCH_* override in the note. - init: read RSS from /proc/self/status on Linux, so images without ps work. Keep what each step builds alive, and drop the warm-build clients outside the measurement. - traces: the regex require filter matches the fixture service; assert that every span is kept. - The shared setup and the config bench take the function identity from the fixtures and do not set the variables Lambda gives only to the runtime (AWS_EXECUTION_ENV, LAMBDA_TASK_ROOT and others). The config benches drop their results outside the measurement. - README: per-element time is the batch time divided by the count, the FIPS command needs --no-default-features, and the numbers recorded before these fixes are marked pre-fix.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc8831c964
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| (cd "$crate" && cargo bench ${benches[@]+"${benches[@]}"} -- --save-baseline "$name" ${extra[@]+"${extra[@]}"}) | ||
| mkdir -p "$CARGO_TARGET_DIR/criterion" | ||
| printf '%s\nsaved: %s\ngit: %s\n' "$note" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ | ||
| "$(git_note)" \ |
There was a problem hiding this comment.
When tracked files change while a benchmark is running—particularly likely for the documented production-profile suite, which can take several minutes—this evaluates git_note only after Cargo has finished, so the note describes the post-edit tree rather than the code compiled into the benchmark binary. Capture the revision and dirty-diff hash before invoking cargo bench, then write that saved value into the metadata.
Useful? React with 👍 / 👎.
| echo "$note" | ||
| # `${arr[@]+"${arr[@]}"}` expands to nothing when the array is empty; a bare "${arr[@]}" | ||
| # is an unbound variable under `set -u` on bash 3.2 (/bin/bash on macOS). | ||
| (cd "$crate" && cargo bench ${benches[@]+"${benches[@]}"} -- --save-baseline "$name" ${extra[@]+"${extra[@]}"}) |
There was a problem hiding this comment.
When an existing baseline name is saved again with only selected bench targets or a Criterion filter, the executed IDs are overwritten but results for skipped IDs remain from the prior run, while the single baseline-<name>.txt note is replaced with the new revision. Criterion stores and copies baselines independently under each benchmark ID, so it does not clear those unexecuted IDs (Criterion analysis source); a later critcmp can therefore present one baseline containing measurements from multiple revisions under one provenance note. Reject reused names, clear every prior ID for that name, or preserve provenance per benchmark before allowing partial saves; the Docker wrapper has the same reuse problem.
Useful? React with 👍 / 👎.
|
|
||
| /// Sets the AWS variables the tags provider and the config read, for the fixture function. No | ||
| /// `AWS_EXECUTION_ENV`: Lambda gives it to the runtime process only, not to extensions. | ||
| pub fn setup_env() { |
There was a problem hiding this comment.
When the caller already has AWS_EXECUTION_ENV set, setup_env leaves it intact even though this setup explicitly models the extension environment where that variable is absent. tags::lambda::get_tags_map reads the inherited value and adds a runtime tag, so the logs and traces benchmarks revert to the 17-tag workload that the README says was corrected to 16, making results dependent on the launch environment. Explicitly remove AWS_EXECUTION_ENV before constructing providers.
Useful? React with 👍 / 👎.
|
|
||
| # Named volumes keep the registry and git checkouts between runs. | ||
| docker run --rm \ | ||
| --platform linux/arm64 \ |
There was a problem hiding this comment.
On an x86 host with Docker Desktop or binfmt/QEMU configured, this --platform linux/arm64 invocation succeeds by emulating ARM and still saves the result as a Linux-arm64 baseline. Those measurements are dominated by instruction translation rather than Graviton behavior—especially the suite's compression and other compute-heavy cases; Docker explicitly warns that these workloads take a large performance hit under QEMU (Docker multi-platform documentation). Check the host architecture and refuse to save a performance baseline on non-arm64 hosts, unless an explicit functional-smoke override is supplied.
Useful? React with 👍 / 👎.
TL;DR: Criterion benchmarks for the hot paths: log processing, trace processing, invocation parsing, enhanced metrics, config load and init.
Why: The extension runs on about 0.07 vCPU at 128 MB, so per-line and per-span costs show up in the customer's billed duration. We need before and after numbers for each performance fix.
Overview
Jira: SVLS-9977
Stacked on #1473 (
jordan.gonzalez/fault-harness/feature). It needs the fixtures and the dependency removal; the harness PR is only stacking order.The extension runs at about 0.07 vCPU at 128 MB, so per-line, per-span and per-invocation
costs show up in billed duration. This PR adds 6 criterion benches (
bottlecap/benches/):logs,traces,invocation,enhanced_metrics,config,init. Benches namedalt_*measure an alternative from the review, as the target for a fix.
bottlecap/benches/README.md: what each bench measures, how to run it, noise control, thefirst baseline, and a table from each review item to its bench and number.
scripts/bench-compare.sh: save and compare criterion baselines with critcmp.scripts/bench-docker-arm64.sh: the suite in a 2-CPU Linux arm64 container.Cargo.toml:criterion0.5 as a dev-dependency (no plotters, no rayon), the 6[[bench]]targets, and
bench = falseon the lib and binaries socargo benchruns only criterion.#[doc(hidden)] pubso the benches can reach them(
is_oom_error, the/procreaders*_from_path,startup::build_tag_replace_rules).No behavior change.
.gitignore: criterion exports and the Linux arm64 target dir.Size: 270 core lines, 3032 lines in total.
Testing
cargo fmt --all -- --check: passcargo metadata --locked: passcargo clippy --workspace --all-targets --features default -- -D warnings: passcargo clippy --workspace --all-targets --features default,test-mode -- -D warnings: passcargo nextest run --workspace: passcargo bench --no-run(LTO off, 16 codegen units): passcargo deny --locked check: passcargo machete: passdd-rust-license-tool check: passcargo clippy --workspace --all-targets --no-default-features --features fips -- -D warnings: pass. It cannot run on macOS (libddwaf-sys load error).