Repository navigation
test: process-level fault-injection harness and the contract tests that pass today - #1473
Conversation
…act tests that pass today
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0e4939146
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
duncanista
changed the title
- dogstatsd: advance the metric name every 100 lines so n lines give min(n, 5000) contexts - traces: set BASE_NS to 2026-10-27T12:00:00Z and encode meta/metrics in key order - events: pad the SQS body to the exact size and put the LWA deadline 60 s from now - telemetry: emit extension logs in the logger::Formatter JSON shape
- fault_injection_test: measure the /next budget from when the INVOKE was queued - harness: shutdown_cleanly asserts the exit came within the SHUTDOWN deadline
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51f541581a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- write_corpus: 600 events for the JSON shapes so each POST fits the 1 MiB maxBytes - traces: TRACE_HEADERS matches the reference payload's 100 traces
- oversized trace: wait for /next against the 10 s INVOKE deadline, and assert the rejected trace is never sent again after the rejection is lifted - lifecycle: read the intake before SHUTDOWN so the final flush cannot hide a missing end-of-invocation flush
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 399b90b8e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Keep the #1467 dependency pins without ddsketch-agent, regenerate Cargo.lock and LICENSE-3rdparty.csv, and list libddwaf-sys in the cargo-machete ignore list: it is a dependency only to enable its aws-lc-rs feature.
…modes in blackhole
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e32723992
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…/cargo-machete/feature
…-fixtures/feature
…tdown deadline in shutdown()
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e24e897cc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be8ca84877
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved test-harness defects can mask deadline violations and failed uploads, while fault scenarios may pass without exercising log flushing under injected failures.
5 open issues
What changed in this PR
Adds a process-level fault-injection harness for the real bottlecap binary, building on the shared fixture corpus. This establishes regression coverage for lifecycle and intake failures without changing production code or Cargo dependencies.
Changes:
- Adds fake Lambda APIs, a configurable fault intake, and process and port-management helpers.
- Adds 10 contract tests covering lifecycle completion, intake failures, malformed configuration, and port conflicts.
- Serializes the test binaries in nextest and documents harness usage.
| File | Description |
|---|---|
bottlecap/tests/port_conflict_test.rs |
Tests lifecycle and trace listener port conflicts. |
bottlecap/tests/lifecycle_contract_test.rs |
Checks invocation flushing and clean shutdown. |
bottlecap/tests/HARNESS.md |
Documents harness APIs, conventions, and execution. |
bottlecap/tests/fault_injection_test.rs |
Tests 503 responses, oversized traces, and truncated responses. |
bottlecap/tests/config_robustness_test.rs |
Checks isolation of four malformed configuration values. |
bottlecap/tests/common/telemetry_events.rs |
Adapts shared telemetry fixtures for the harness. |
bottlecap/tests/common/port_guard.rs |
Holds and probes fixed listener ports. |
bottlecap/tests/common/harness.rs |
Provides scenario orchestration and payload helpers. |
bottlecap/tests/common/fault_intake.rs |
Implements configurable intake faults and request recording. |
bottlecap/tests/common/fake_extensions_api.rs |
Emulates Lambda registration, polling, and telemetry delivery. |
bottlecap/tests/common/bottlecap_process.rs |
Manages child processes, captured logs, and resource measurements. |
bottlecap/.config/nextest.toml |
Serializes fixed-port tests and sets their timeout. |
Review effort: Balanced
…xt against the deadline The fake Extensions API now answers 403 to /next and subscribe without the identifier issued at register, withholds telemetry events whose type the subscription did not ask for, and records each /next arrival so wait_for_next_calls compares call n with an absolute deadline.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65ea94a287
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…odies, check logs under faults - The fake answers 403 to a register without `Lambda-Extension-Name: datadog-agent`. - `decoded_body()` and `body_text()` return an error instead of partial output. - The fault intake does not record or answer an upload that breaks off. - The 503 and truncation tests send `platform.initStart` on the cold start and check that a logs request hit the fault before it clears.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ce513efff6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Subscribe answers 400 unless the body follows the Telemetry API reference: a known `schemaVersion`, non-empty `types` of `platform`, `function` or `extension`, an `HTTP` destination with a `sandbox` URI and port, and `buffering` values in range. - Register answers 403 to an invalid body or event type, as the runtime interface emulator does, and accepts a body without `events`. - Register returns `accountId` only on `Lambda-Extension-Accept-Feature: accountId`.
Like Lambda's runtime (the agent is in the ready state while its /next waits), the fake rejects a second /next with 403 and does not count it. A poll the client drops stays pending, as in Lambda.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0026b430b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… in the fake - Deliver telemetry to the port and path of the subscribed URI. - Reject an https:// telemetry destination with 400. - In Managed Instance mode (set by BottlecapBuilder::managed_instance()), answer 403 to a register with INVOKE and 400 to a subscribe with a schema version other than 2025-01-29. - Issue a UUID extension identifier and send register and /next bodies as application/json. - Note that Lambda sends the SHUTDOWN reason in lower case.
…the tests in this PR
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 120286fa37
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab7593b207
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fbedd391af
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05057f456d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…content - push_invoke and push_shutdown return the event's deadline, taken before the push. Scenario::invoke returns it and takes the deadline its poll must meet. The startup deadline counts from before the spawn. - FaultIntake records when each request arrived, and wait_for takes a deadline, so a late request counts as missed. - IntakeRequest::trace_ids and metric_names decode the trace and metrics payloads. Each data assertion now matches the log text, the invocations metric or the trace ID, not only a path, status or key. - The config tests send platform.initStart, as Lambda does: without it the cold-start invocations metric is held until SHUTDOWN.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0fc774046
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…in Lambda's order - Scenario::shutdown fails only when a poll saw the process running after the SHUTDOWN deadline (BottlecapProcess::seen_running), so a slow test thread cannot fail an exit that came in time. The port-conflict log line reads the recorded /next and poll instants too. - deliver_start sends platform.initStart on a cold start, then platform.start; deliver_runtime_done sends only platform.runtimeDone. Every test sends start, then its function logs, then runtimeDone. - The fake answers 403 to a telemetry subscribe after the first /next. - FaultIntake records a Slow request before its delay, so wait_for sees it from its arrival. - Every positive intake assertion in fault_injection_test requires TEST_API_KEY, the faulted log requests and both trace requests too. - HARNESS.md: one rule per gap, and the API summary.
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: done
and removed
mergequeue-status: queued
labels
TL;DR: Tests that run the real
bottlecapbinary against a fake Lambda API and an intake that fails on purpose. This PR adds the harness and the 10 checks that pass today.Why: Our worst bugs (hangs past the invocation deadline, SIGKILL at shutdown, lost data on intake errors) show up only at the process level, so unit tests cannot catch them. Each later fix adds its own test here.
Overview
Jira: SVLS-9978
Stacked on #1472 (
jordan.gonzalez/test-fixtures/feature).Contract tests that run the real
bottlecapbinary against a fake Lambda control plane and aDatadog intake that fails on demand. Each test asserts what Lambda or the customer sees:
/nextbefore the deadline, exit on SHUTDOWN, no panic. Seebottlecap/tests/HARNESS.md.tests/common/: fake Extensions and Telemetry API, fault intake (blackhole, SYN blackhole,slow, 5xx, 429, close mid body, size limits), process wrapper with RSS and CPU, port guard.
maintoday: invoke then shutdown exitscleanly; a 503 storm does not hold
/next; an oversized trace is dropped and later tracesflow; an intake that closes mid body does not hold
/next; 4 config fields with a wrong typecost only that field; ports 8124 and 8126 held by another process.
bottlecap/.config/nextest.toml: the 4 binaries run one test at a time (fixed-portsgroup), because the binary binds fixed ports.
AGENTS.md: lists the build, test, lint and format commands that CI runs, and points tobottlecap/tests/HARNESS.mdand the fixtures README, so agents find the harness.No production code. One Cargo change:
protobuf3.7 as a dev-dependency, already inCargo.lock, so the tests can decode the metrics payloads. The tests for the open bugs come in a later PR, eachignored with the Jira ticket that tracks its fix.
Size: 20 core lines, 2184 lines in total.
Testing
cargo fmt --all -- --check: passcargo clippy --workspace --all-targets --features default -- -D warnings: passcargo clippy --workspace --all-targets --features default,test-mode -- -D warnings: passcargo nextest run --workspace: passcargo test --test <binary>for the 4 harness binaries: pass#[ignore], no backlog reference: pass