fix(bazel): key execlog reader on client pid; probe with startup flags by jeffpignataro · Pull Request #1428 · aspect-build/aspect-cli

jeffpignataro · 2026-09-09T18:43:38Z

Problem

A customer's aspect test jobs started failing after a successful build:

INFO: Build completed successfully, 2 total actions
...
error: execlog stream thread error: io error: process exited before the file was created

A pre-command hook ran aspect ci bazelrc, writing --output_base=/mnt/ephemeral/output/dirt/dirt into ~/.bazelrc.
Build::spawn probed bazel info server_pid release with no startup flags. Bazel resolved that from the rc files, so the probe landed on the task's own output base but with rc-derived startup options, and returned that server's pid.
The real bazel test ran with the runner's explicit startup flags. Bazel saw different startup options on the running server and restarted it — Starting local Bazel server (9.2.0) three seconds after bazel health check passed.
The execlog reader had been started before spawn with the probed server pid as its liveness signal. That pid was gone, so StreamingFile::open returned BrokenPipe immediately.
wait() joined the streams after a green build, hit the stream error and bail!ed, never reporting bazel's exit status.

Until that hook change the rc files pointed the probe at a different, long-lived output base (Rosetta's __main__), so the wrong-but-alive pid masked the bug. It only bites with an exec-log sink registered (the artifacts feature's upload_exec_log).

Fix

Probe the server the invocation will use. Build::spawn now calls server_info_with_startup_flags(&startup_flags), as query already did. The version probes behind version-gated flags (resolve_flags_for_running_bazel, resolve_rc_version) take the startup flags too, so they no longer read rc files and never start a stray server.
Bound the execlog reader by the client pid. The --execution_log_compact_file path is chosen before spawn; the reader thread starts after cmd.spawn() with child.id(), mirroring the BES reader. Bazel's CompactSpawnLogContext opens the file in its constructor and closes it in afterCommand, both before the client exits, so the client's exit is the end-of-stream signal. galvanize::StreamingFile keys on is_pid_alive(pid) instead of "does pid hold the file open" — the server holds it, and the server may be replaced mid-invocation.
Keep bazel's exit code visible. wait() collects stream errors instead of bailing on the first. If bazel failed, the stream errors are printed as warnings and bazel's status is returned. If bazel succeeded and a stream failed, the call still fails (a truncated artifact must not pass silently), but the message now carries the exit code.

Not changed here: WorkspaceEventStream::spawn_with_pipe and the BES Pipe's IfOpenForPid policy still receive the pre-spawn server pid. With the probe fixed that pid is the right server, so they are correct unless bazel restarts for some other reason; making them client-pid-bounded is a follow-up.

Test plan

bazel test //crates/galvanize:galvanize_test — two new tests: the reader gives up when the bounding pid is already dead before the file appears, and it keeps reading past EOF while the pid lives (with no process holding the file open) and ends only when the pid exits.
bazel build //crates/axl-runtime
bazel test //crates/axl-runtime:test
Manual: on a Workflows runner with aspect ci bazelrc in a pre-command hook and upload_exec_log on, aspect test completes and the .execlog.zstd sink is written.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HTa7Rk2dND4P5AXWZ7VBcV

…on the client pid `Build::spawn` asked `bazel info server_pid` with no startup flags, so the probe read whatever rc files were on disk and could land on a different output base than the invocation — or the same one with different options, in which case the real invocation restarted the server and the captured pid was dead before the build started. The execlog reader used that server pid as its liveness signal and gave up at once with "process exited before the file was created", and `wait()` surfaced that stream error in place of bazel's own exit status, so a green build failed the task. - Probe with the invocation's startup flags, here and in the version probes behind version-gated flags (`query` already did this). - Start the execlog reader after `cmd.spawn()` and bound it by the client pid, as the BES reader already does; bazel closes the log before the client exits, so the client's exit is the end-of-stream signal. - In `wait()`, when bazel itself failed, report stream errors as warnings and return bazel's exit code; only a stream failure on a green build fails the call, and the message now carries the exit code. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HTa7Rk2dND4P5AXWZ7VBcV

coderabbitai · 2026-09-09T18:43:49Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 516f9b70-8049-4da5-85dd-b6954b7b743d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

aspect-workflows · 2026-09-09T18:54:39Z

Aspect Workflows Tasks

📅 Mon Sep 14 17:41:28 UTC 2026

Task Results

✅ axl-smoke-gha-bootstrap [build] · complete (1 built) · 1m 9s · Aspect · GitHub Actions · Check
✅ axl-tests-gha-bootstrap [build] · complete (1 built) · 1m 7s · Aspect · GitHub Actions · Check
✅ build-gha-debug [build] · complete (178 built) · 4m 54s · Aspect · GitHub Actions · Check
✅ build-gha [build] · complete (178 built) · 5m 4s · Aspect · GitHub Actions · Check
✅ build-gha-ephemeral [build] · complete (10 built) · 1m 10s · GitHub Actions · Check
✅ buildifier-gha-debug [buildifier] · complete (clean) · 1m 48s · GitHub Actions · Check
✅ buildifier-gha [buildifier] · complete (clean) · 1m · GitHub Actions · Check
⚠️ delivery-gha-debug [delivery] · complete (1 delivered · 2 warn · 4 skipped) · 53.8s · Aspect · GitHub Actions · Check
✅ delivery-no-remote-exec [delivery] · complete (no deliveries) · 6.7s · Aspect · GitHub Actions
⚠️ delivery-gha [delivery] · complete (1 delivered · 2 warn · 4 skipped) · 53.8s · Aspect · GitHub Actions · Check
❌ delivery-uncacheable [delivery] · failed in deliver · failed (1 delivery fail) · 36.5s · Aspect · GitHub Actions
⚠️ delivery-uncacheable-warn [delivery] · complete (1 warn) · 14.5s · Aspect · GitHub Actions
✅ format-gha-debug [format] · complete (clean) · 1m 16s · GitHub Actions · Check
✅ format-format-repeat-task [format] · complete (clean) · 1m 35s · GitHub Actions · Check
✅ format-format-repeat-task-2 [format] · complete (clean) · 14.1s · GitHub Actions · Check
✅ format-format-repeat-task-3 [format] · complete (clean) · 15.5s · GitHub Actions · Check
✅ format-format-repeat-task-4 [format] · complete (clean) · 13.4s · GitHub Actions · Check
✅ format-gha [format] · complete (clean) · 2m 4s · GitHub Actions · Check
✅ gazelle-gha-debug [gazelle] · complete (clean) · 49.3s · GitHub Actions · Check
✅ gazelle-from-source-gha-debug [gazelle] · complete (clean) · 2m 16s · GitHub Actions · Check
✅ gazelle-from-source-gha [gazelle] · complete (clean) · 1m 53s · GitHub Actions · Check
✅ gazelle-gha [gazelle] · complete (clean) · 41.3s · GitHub Actions · Check
✅ init-shell [build] · complete (10 built) · 43.6s · GitHub Actions · Check
✅ lint-gha-debug [lint] · complete (clean) · 53.1s · GitHub Actions · Check
✅ lint-gha [lint] · complete (clean) · 35.5s · GitHub Actions · Check
✅ test-gha-debug [test] · complete (32/32 passed · 32 cached) · 4m 53s · Aspect · GitHub Actions · Check
✅ test-gha-bazel-flag-spellings [build] · complete (1 built) · 13.8s · Aspect · GitHub Actions · Check
✅ test-gha-ide-target-pattern-file [build] · complete (1 built) · 14.3s · Aspect · GitHub Actions · Check
✅ test-gha-coverage [test] · complete (1/1 passed · 1 cached) · 25.8s · Aspect · GitHub Actions · Check
✅ test-gha-target-pattern-file [test] · complete (1/1 passed · 1 cached) · 25.3s · Aspect · GitHub Actions · Check
✅ test-gha [test] · complete (32/32 passed · 32 cached) · 4m 54s · Aspect · GitHub Actions · Check
✅ test-gha-ephemeral [test] · complete (1/1 passed) · 1m 20s · GitHub Actions · Check

Reproduce

❌ delivery (delivery-uncacheable · delivery-gha-debug · delivery-gha · delivery-uncacheable-warn)

# --mode=always --track-state=false for off-runner with no state backend.
aspect delivery \
  --commit-sha=39b7db9129e5beba9c4ca222e7dbfba08ef0fa22 \
  --mode=always \
  --track-state=false \
  --dry-run=true

_{Install aspect: aspect.build/docs/cli/install}

_{⏱ Last updated Mon Sep 14 18:36:34 UTC 2026 · 📊 GitHub API quota 110/15,000 (1% used, resets in 56m)}
_{🚀 Powered by Aspect CLI (v0.0.0-dev) | Aspect Build · X · LinkedIn · YouTube}

chatgpt-codex-connector · 2026-09-09T21:27:43Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-09T21:33:36.915365Z	`804cd79`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

jbedard · 2026-09-14T16:53:29Z

    pub fn spawn_with_file(
-        pid: u32,
-        out_path: Option<PathBuf>,
+        path: PathBuf,


path is a pretty horrible name here

yeah good call - changed to execlog_path for clarity.

jbedard · 2026-09-14T16:54:47Z

        rt: AsyncRuntime,
    ) -> Result<Build, std::io::Error> {
-        let (pid, version) = super::info::server_info()?;
+        // Probe with the invocation's own startup flags so the answer describes


This seems like a pretty over detailed bloated comment

Yeah, definitely. Tightened up the verbiage.

thesayyn · 2026-09-14T18:54:01Z

to me this sounds like we should just forward the correct startup flags that should fix the issue here?

thesayyn · 2026-09-14T18:56:04Z

Also relying on the client_pid is not an answer here. if you have bazel/tools wrapper, bazelisk etc the client_pid is never the real bazel client, even when it is, its not the client that writes this file, its the server that does it.

jeffpignataro · 2026-09-14T18:58:42Z

Closing based on #1428 (comment) - I'll try to look for a simpler solution here.

jeffpignataro marked this pull request as ready for review September 9, 2026 21:27

jeffpignataro requested a review from gregmagolan September 9, 2026 21:27

jeffpignataro requested review from jbedard and thesayyn September 14, 2026 14:20

jbedard reviewed Sep 14, 2026

View reviewed changes

refactor(bazel): clarify execlog naming and probe comment

39b7db9

jeffpignataro closed this Sep 14, 2026

Navigation Menu

fix(bazel): key execlog reader on client pid; probe with startup flags - #1428

fix(bazel): key execlog reader on client pid; probe with startup flags#1428
jeffpignataro wants to merge 2 commits into
mainfrom
jeff/fix/execlog-child-pid-probe-flags

jeffpignataro commented Sep 9, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 9, 2026 •

edited

Loading

Review skipped

Uh oh!

aspect-workflows Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

jbedard Sep 14, 2026

Uh oh!

jeffpignataro Sep 14, 2026

Uh oh!

jbedard Sep 14, 2026

Uh oh!

jeffpignataro Sep 14, 2026

Uh oh!

thesayyn commented Sep 14, 2026

Uh oh!

thesayyn commented Sep 14, 2026

Uh oh!

jeffpignataro commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jeffpignataro commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Test plan

Uh oh!

coderabbitai Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

aspect-workflows Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Aspect Workflows Tasks

Task Results

Reproduce

❌ delivery (delivery-uncacheable · delivery-gha-debug · delivery-gha · delivery-uncacheable-warn)

Uh oh!

chatgpt-codex-connector Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

jbedard Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

jeffpignataro Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

jbedard Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

jeffpignataro Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

thesayyn commented Sep 14, 2026

Uh oh!

thesayyn commented Sep 14, 2026

Uh oh!

jeffpignataro commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jeffpignataro commented Sep 9, 2026 •

edited

Loading

coderabbitai Bot commented Sep 9, 2026 •

edited

Loading

aspect-workflows Bot commented Sep 9, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 9, 2026 •

edited

Loading