delivery: recover when the local action cache outlives the remote cache

[ewhauser]

Problem

Selective aspect delivery can leave unchanged targets pending when the normal build reuses a persistent local Bazel action-cache entry whose corresponding remote-cache entry has expired or been evicted.

The phase-one build succeeds, but the checksum phase runs on the separate -checksum output base with --experimental_remote_require_cached. If an upstream action is no longer present remotely, Bazel never reaches the target's DeliveryHash action, so the gRPC log contains no output SHA for that target.

This turns remote-cache retention into a correctness requirement for selective delivery even though the artifact is still valid in the runner's local action cache.

Observed behavior

Environment:

• Aspect CLI v2026.28.4
• Bazel 9.2.0
• persistent Aspect Workflows runner output base
• writable BuildBarn remote cache

An initial run built and delivered a set of image targets successfully. A later run used the same CLI version and checksum architecture, but phase one reused the warm local action cache. Phase two could resolve most delivery digests, while targets whose upstream actions were absent from the remote cache remained pending.

Representative upstream failures were LayerTar and LayerInputFilesCAS actions reporting:

Action must be cached due to --experimental_remote_require_cached but it is not

The final summary reported the affected targets as pending rather than skipped, despite no source changes to those targets.

0 delivered, <most targets> skipped, <affected targets> pending, 0 failed

Rolling the release build back onto the shared output base did not help because the checksum phase correctly remains on its dedicated -checksum server.

Expected behavior

Selective delivery should recover from an ordinary remote-cache miss. Rebuilding an artifact needed to calculate its digest is acceptable, but an unchanged target must still be compared with delivery state and skipped. Recovery must not dispatch or bazel run every deliverable.

Proposed fix

Add one bounded repair attempt inside _get_output_shas after parsing the first checksum probe:

1. Collect targets absent from output_shas.
2. Run bazel build for only those unresolved labels, using the same unstamped flags/configuration as phase one plus:
   • --nouse_action_cache to prevent the warm local action cache from short-circuiting remote lookup/upload
   • --remote_upload_local_results
   • --noremote_cache_async so repaired results are visible before retrying
3. Re-run the cache-only checksum probe for only those labels.
4. Merge recovered output SHAs into the original result.
5. Preserve the current warning/error behavior for labels still unresolved after the single retry.

This is a build-only repair. Phase-three state comparison and dispatch remain unchanged, so containers are pushed only when the recovered digest differs from recorded delivery state.

cache_diff.axl already uses --nouse_action_cache to force fresh remote lookups instead of trusting the persistent local action cache:

https://github.com/aspect-build/aspect-cli/blob/v2026.28.4/crates/aspect-cli/src/builtins/aspect/cache_diff.axl#L104-L183

The relevant Delivery implementation is here:

https://github.com/aspect-build/aspect-cli/blob/v2026.28.4/crates/aspect-cli/src/builtins/aspect/delivery.axl#L383-L710

Suggested regression test

1. Run selective Delivery once to populate delivery state, the local output base, and the remote cache.
2. Preserve the local output base but clear or replace only the remote action cache.
3. Run selective Delivery again.
4. Assert that the repair build occurs for unresolved labels.
5. Assert that the target is reported SKIP, not pending or delivered, and that its entrypoint is not dispatched a second time.