feat(axl): read a compact execution log, and diff two of them

[gregmagolan]

Two builds that should have been identical were not. Which action moved, and why?

That question is answerable from the compact execution log and almost nobody answers it, because the log is a zstd frame of protobufs that reference each other by id, and reconstructing it is a project. execlog-diff (Go) does it. This brings the same analysis into the CLI as an AXL task, and adds the one runtime primitive it needed.

The runtime bit

bazel.execution_log.read(path = ...) decodes a finished --execution_log_compact_file into the same ExecLogEntry iterator the live build path already yields. The decoder existed; what was missing was pointing it at a file nobody is still writing.

entries = bazel.execution_log.read(path = "before.binpb.zst")
for entry in entries:
    if type(entry.type) == "spawn":
        ...
if entries.error() != None:
    ctx.std.process.exit(1, "before.binpb.zst: " + entries.error())

Decoding runs on a background thread with blocking sends, so a log larger than memory can be walked as long as the loop body does not keep every entry. Unlike the live path, nothing is dropped under back-pressure: the build already finished, so there is nothing to slow down, and a dropped entry would make a diff wrong rather than slow.

aspect execlog diff

$ aspect execlog diff /tmp/before.binpb.zst /tmp/after.binpb.zst
2 action(s) changed and would not have been a cache hit:

Genrule  //gen:banner
  output: bazel-out/darwin_arm64-fastbuild/bin/gen/banner.txt
  inputs: 1 changed, 0 added, 0 removed
      changed  gen/greeting.txt

Genrule  //gen:report
  output: bazel-out/darwin_arm64-fastbuild/bin/gen/report.txt
  inputs: 1 changed, 0 added, 0 removed
      changed  bazel-out/darwin_arm64-fastbuild/bin/gen/banner.txt

A source file moved, so the action reading it moved, so the action reading its output moved. //gen:stable is absent from the report because it did not change. That chain is the thing you actually want when a cache hit rate drops and nobody knows why.

Five dimensions of the action key are compared — args, env, inputs, tools, platform — and every one that moved is reported rather than the first, because an action with two problems otherwise sends you round the loop twice. env is the one that finds non-hermeticity:

Genrule  //gen:stable
  env: 1 changed, 0 added, 0 removed
      changed  DEMO_TOKEN

Exits 0 by default, because a changed action is a finding rather than an error. --fail-on-change makes it fail a CI step that asserts two builds are identical.

aspect execlog list is the --list equivalent: mnemonic, label, runner, cache hit, with --output=json adding each action's computed key.

Memory is the design

A 20 MB compact log expands to gigabytes once every input set is flattened, and a diff needs two of them. The Go tool solves this by streaming and diffing concurrently. AXL is single-threaded, so it solves it by never holding a flattened log:

1. Index each log once, keeping three id-keyed maps and one Fingerprint per spawn. A fingerprint is five hashes and a label, so it scales with the number of actions, not the number of files.
2. Detail only the spawns whose fingerprints differ, by walking the logs again. On a healthy build that is a handful out of tens of thousands, and it is skipped entirely when nothing differs.

Two passes over a compressed file are far cheaper than one flattened log in memory. --summary-only stops after the first pass when even that is too much.

What I got wrong, and what zstd will not tell us

Truncation is usually, not always, detected. A log cut off part-way through sets error() and ends the stream early, so a caller that checks does not compare a prefix and call it clean. The detection comes from zstd reporting incomplete frame when it runs out of input mid-block. When the cut lands exactly on a block boundary, the decoder sees a clean end and the entries before it are a structurally valid stream.

Measured against a real 134 KB log: cuts at 40%, 50%, 60%, 75%, 90% and 99% were all reported; cuts at 10% and 25% were not. Closing that gap needs zstd's end-of-frame marker, which zstd::Decoder does not expose — it needs zstd_safe::DCtx directly. Documented on spawn_from_path. Worth doing if a partial log is ever mistaken for a real answer in practice.

The Rust tests found a real hazard. In fibre's SPMC broadcast, every Receiver clone is an independent subscriber whose tail the sender must not lap. My first test helper held the stream (and therefore its internal receiver clone) while consuming, which deadlocked the producer on any log longer than the 1000-entry channel bound. Production already drops it — the Starlark read() does drop(stream) right after taking the receiver — but the test did not, and hung. Both now do the same thing, with the reason written down.

Other limits worth knowing

• Directory entries are matched by path, not contents. A directory output whose contents changed but whose path did not will not be flagged. The Go tool resolves those; this does not yet.
• The spawn key is the primary output path, since Bazel gives no stable action id. Unique within a build and identical across two builds of the same target, which is the property a diff key needs. A spawn with no usable output (a failed action that produced nothing) falls back to label!mnemonic, which can collide with its siblings.
• Comparing a cold build against a warm one is not meaningful — the warm log only contains what re-ran. The only in first / only in second counts make that visible, but the tool cannot stop you.

Tests

6 Rust (cargo test -p axl-runtime --lib engine::bazel::stream::execlog): a complete log yields every entry with no failure; a non-zstd file, a too-short file and a missing file each fail on the calling thread with a message naming the problem; a corrupt frame sets the failure slot; a truncated log yields a prefix.

14 AXL (tests axl, 1005 total, exit 0): input-set flattening flat, transitive, cyclic, with a dangling id, and with Bazel's id-0 "no inputs"; the maps differ separating changed from added from removed; the list differ reporting the index so an inserted flag reads as a shift; and the hash not colliding across element boundaries, which is what stops a diff missing a real change.

Verified by hand end to end against a clone of aspect-starters/shell with three genrules: one whose source changed, one downstream of it, and one that did not change. All three classify correctly, and the env and args dimensions were confirmed separately with --action_env and a cmd edit.

Changes are visible to end-users: yes

• Searched for relevant documentation and updated as needed: yes (builtins/aspect/README.md)
• Breaking change (forces users to change their own code or config): no — two new commands, one new runtime function
• Suggested release notes appear below: yes

Suggested release notes:

• New aspect execlog diff <before> <after>: compare two compact execution logs and explain every action that would not have been a cache hit, naming the arguments, environment variables or input files behind each one.
• New aspect execlog list <log>: list the actions in a compact execution log, with --output=json.
• New bazel.execution_log.read(path = ...): read a compact execution log from disk in AXL.

Test plan

• New test cases added: 6 Rust, 14 AXL.
• Manual testing; please provide instructions so we can reproduce:

  cargo build -p aspect-cli
  cargo test -p axl-runtime --lib engine::bazel::stream::execlog
  ./target/debug/aspect-cli tests axl
  
  # In any Bazel repo, with .aspect/version.axl pointed at the built binary:
  bazel clean && bazel build //... --execution_log_compact_file=/tmp/before.binpb.zst
  # change a source file
  bazel clean && bazel build //... --execution_log_compact_file=/tmp/after.binpb.zst
  aspect execlog diff /tmp/before.binpb.zst /tmp/after.binpb.zst

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: af18cbc1-f769-4378-bf6d-977090bd3bc1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[aspect-workflows]

Aspect Workflows Tasks

📅 Sun Sep 20 18:27:55 UTC 2026

Task Results

• ✅ axl-smoke-gha-bootstrap [build] · complete (1 built) · 41.4s · Aspect · GitHub Actions · Check
• ✅ axl-tests-gha-bootstrap [build] · complete (1 built) · 19.3s · Aspect · GitHub Actions · Check
• ✅ build-gha-debug [build] · complete (179 built) · 7m 43s · Aspect · GitHub Actions · Check
• ✅ build-gha [build] · complete (179 built) · 7m 54s · Aspect · GitHub Actions · Check
• ✅ build-gha-ephemeral [build] · complete (10 built) · 31.6s · Aspect · GitHub Actions · Check
• ✅ buildifier-gha-debug [buildifier] · complete (clean) · 56.6s · Aspect · GitHub Actions · Check
• ✅ buildifier-gha [buildifier] · complete (clean) · 30.8s · Aspect · GitHub Actions · Check
• ⚠️ delivery-gha-debug [delivery] · complete (1 delivered · 2 warn · 4 skipped) · 25.2s · Aspect · GitHub Actions · Check
• ✅ delivery-no-remote-exec [delivery] · complete (no deliveries) · 4.6s · Aspect · GitHub Actions
• ⚠️ delivery-gha [delivery] · complete (1 delivered · 2 warn · 4 skipped) · 28.2s · Aspect · GitHub Actions · Check
• ❌ delivery-uncacheable [delivery] · failed in deliver · failed (1 delivery fail) · 10.3s · Aspect · GitHub Actions
• ✅ delivery-uncacheable-warn [delivery] · complete (no deliveries) · 5.8s · Aspect · GitHub Actions
• 🔄 format-gha-debug [format] · Building formatter... (//tools/format) · 9m 13s · Aspect · GitHub Actions · Check
• 🔄 format-format-repeat-task [format] · Building formatter... (//tools/format) · 9m 11s · Aspect · GitHub Actions · Check
• 🔄 format-gha [format] · Building formatter... (//tools/format) · 9m 13s · Aspect · GitHub Actions · Check
• ✅ gazelle-gha-debug [gazelle] · complete (clean) · 36.3s · Aspect · GitHub Actions · Check
• ✅ gazelle-from-source-gha-debug [gazelle] · complete (clean) · 45.1s · Aspect · GitHub Actions · Check
• ✅ gazelle-from-source-gha [gazelle] · complete (clean) · 1m 13s · Aspect · GitHub Actions · Check
• ✅ gazelle-gha [gazelle] · complete (clean) · 33s · Aspect · GitHub Actions · Check
• 🔄 init-shell [build] · running... (9 targets built) · 27m 20s · Aspect · GitHub Actions · Check
• ✅ lint-gha-debug [lint] · complete (clean) · 1m 20s · Aspect · GitHub Actions · Check
• ✅ lint-gha [lint] · complete (clean) · 1m 22s · Aspect · GitHub Actions · Check
• ✅ test-gha-debug [test] · complete (33/33 passed · 26 cached) · 8m 8s · Aspect · GitHub Actions · Check
• ✅ test-gha-bazel-flag-spellings [build] · complete (1 built) · 9.1s · Aspect · GitHub Actions · Check
• ✅ test-gha-ide-target-pattern-file [build] · complete (1 built) · 15.9s · Aspect · GitHub Actions · Check
• ✅ test-gha-coverage [test] · complete (1/1 passed · 1 cached) · 18.8s · Aspect · GitHub Actions · Check
• ✅ test-gha-target-pattern-file [test] · complete (1/1 passed · 1 cached) · 14.8s · Aspect · GitHub Actions · Check
• ✅ test-gha [test] · complete (33/33 passed · 31 cached) · 8m 38s · Aspect · GitHub Actions · Check
• ✅ test-gha-ephemeral [test] · complete (1/1 passed) · 39.6s · Aspect · GitHub Actions · Check

Reproduce

❌ delivery (delivery-uncacheable · delivery-gha-debug · delivery-gha)

# --mode=always --track-state=false for off-runner with no state backend.
aspect delivery \
  --commit-sha=7acadacfa51a39e42f56a5c1029be28606ae0ef0 \
  --mode=always \
  --track-state=false \
  --dry-run=true

_{Install aspect: aspect.build/docs/cli/install}

---

_{⏱ Last updated Sun Sep 20 18:55:57 UTC 2026 · 📊 GitHub API quota 0/7,700 (0% used, resets in 59m)
🚀 Powered by Aspect CLI (v0.0.0-dev)  |  Aspect Build · X · LinkedIn · YouTube}