[lucaspimentel]

Problem

bottlecap (*) compile jobs fail intermittently in the Docker build at yum install. The failure is always early (median 57 s into the job) and always recovers on the job's built-in retry, but each occurrence forces a full recompile of that flavor.

Cause

The BuildKit cache mount for /var/cache/yum in images/Dockerfile.bottlecap.compile has no sharing mode, so every concurrently-scheduled bottlecap (*) flavor that uses this Dockerfile shares one cache mount. All of them mutate it at once, so a build can read a repodata file or a cached .rpm that another build is rewriting or pruning.

This PR

Adds sharing=private to that one mount, so each concurrent build gets its own yum cache. Same approach and same syntax as #1369, which did this for the cargo target mount in this file.

Not changed: the alpine Dockerfile (it uses apk, not yum) and the cargo git/registry mounts, which #1369 deliberately left shared.

Tradeoff: private mounts mean no yum cache reuse across concurrent builds, so each concurrent build re-downloads its packages. Ephemeral runners start with empty caches anyway; unmeasured.

Testing

Test 1: the failure is current, and its signature matches a shared-cache race

Steps

1. List every failed bottlecap (*) compile job created between 2026-10-14 and 2026-11-05 in DataDog/datadog-lambda-extension, all refs (GET /projects/:id/jobs?scope[]=failed, paginated). 137 of them failed with script_failure.
2. Download those 137 job traces and classify each by the BuildKit step that failed and by the yum error text.

Result

71 of the 137 (52%) failed in the yum install step with a symptom of the cache being mutated underneath the build:

Trace literal	n
Metadata file does not match checksum, then [Errno 256] No more mirrors to try on repodata/primary.sqlite.gz	58
[Errno 16] error setting timestamp on file on /var/cache/yum/.../*.rpm	29
Error: Cannot open file /var/cache/yum/.../*.rpm: [Errno 2] No such file or directory during Installing :	12
[Errno 14] HTTPS Error 416 - Requested Range Not Satisfiable on a .rpm URL	23

The subsets overlap; each of the 71 matches at least one.

This is not a stale sample. The failures recur at a steady rate through 2026-11-04: 6 on 10-28, 2 on 10-29, 4 on 10-30, 5 on 11-02, 6 on 11-03, 6 on 11-04, then none on the partial day of 11-05.

They also cluster by minute, which is what a shared mount predicts: on 2026-10-19, 8 failed within 09:18-09:20 across different pipelines.

Cost: of the 67 (pipeline, job) pairs involved, 66 passed on a retry (62 on the first, 4 on the second). Failed attempts run 33 to 98 s, median 57 s, so the cost is the retry and the recompile it triggers, not a blocked pipeline.

Test 2: the Dockerfile still parses

Steps

1. docker build --check -f images/Dockerfile.bottlecap.compile images/

Result

Check complete, no warnings found.

Limits

• That the shared cache causes this is an inference from the error signatures and the burst pattern; the race was not reproduced on demand.
• The failures span all refs (only 3 of the 71 were on main). That is expected: the same Dockerfile builds on every ref, and the builds it competes with come from the whole project, not from one branch.
• The effect of the fix is unmeasured until it ships. To validate after merge: watch whether script_failure compile attempts with these literals disappear.

[datadog-datadog-prod-us1]

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

DDCI - DataDog/datadog-lambda-extension | e2e-test-status (amd64) — 🔧 Needs a code fix, caused by this PR

View more details · View in DDCI

DataDog/datadog-lambda-extension | e2e-test-status (amd64) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: cb3bc0b | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-07T05:10:00.864113Z	cb3bc0b	New commits
🔒 Security Review	✅ Completed 2026-11-07T05:11:18.295522Z	cb3bc0b	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

🟢 Approval recommended

The change isolating the yum cache for concurrent builds has no active review issues.

No open issues 🎉

What changed in this PR

This PR isolates the yum cache used by concurrent Bottlecap compile builds to address intermittent dependency-install failures attributed to shared-cache races.

Changes:

• Adds sharing=private to the /var/cache/yum BuildKit cache mount, following the approach used for the cargo target cache in #1369.
• Documents the isolation rationale; Alpine builds and cargo caches remain unchanged.

File	Description
images/​Dockerfile.bottlecap.compile	Makes the yum cache private to concurrent builds and adds explanatory comments.

_{Review effort: Balanced}