Fix: align full-rank chip run activation

[high-cloud]

Summary

Stabilize multi-rank chip runs under a sustained depth-two asynchronous workload.

• Prepare every member of a chip-run group before releasing native execution, so a slow host-side bind on one rank cannot turn into a late runner_run start.
• Keep one completion owner resident for each chip-run lane instead of relying on a later Python call to advance predecessor completion.
• Add a dedicated mailbox generation notification with futex wakeup for blocking completion waits and shutdown. The generic mailbox store remains syscall-free.
• Keep diagnostic and direct runs on their calling thread so thread-affine capture state is preserved and a direct depth-two successor can stage before its predecessor is finalized; scheduler-leased Serving runs retain the resident completion owner.
• Gate staged admission by the physical pipeline frame as well as the logical lane, so consecutive groups inside one L3 run cannot publish two tasks to the same mailbox frame while different-slot Serving steps still overlap.
• Cover full-rank activation, lane completion ownership, blocking notification, and shutdown behavior with C++ and Python tests.

This is the Simpler runtime part of hw-native-sys/pypto-serving#179. PyPTO group metadata/codegen counterpart: hw-native-sys/pypto#2490.

Verification

• C++ non-hardware suite: 117/117 passed.
• Python worker suite: 923 passed, 3 skipped.
• a2a3sim host-build-graph dep-gen smoke: 2/2 passed, with deps.json produced.
• One-device a2a3 onboard host-build-graph concurrent-prepare suite: 4/4 passed (task_20260823_100011_126283323169, exit 0).
• Three-device a2a3 onboard consecutive-group reservation regression: 1/1 passed (task_20260823_102832_2163987332, exit 0).
• Pre-commit on the diagnostic-ownership change: passed, including clang-format, clang-tidy, and cpplint.
• Four-device EP4/TP4, 5 warmups + 1000 measured rounds, async depth 2: completed successfully.
  • post-frame-guard task: task_20260823_103208_224612532435, exit 0
  • runner-start skew p50/p95/p99/max: 0.047/0.056/0.072/0.470 ms
  • runner-end skew p50/p95/p99/max: 0.020/0.041/0.112/4.657 ms
  • step-end skew p50/p95/p99/max: 0.140/1.048/1.650/4.658 ms
  • max pre-bind/bind/post-bind/gap/validate: 0.073/0.423/2.153/0.054/3.723 ms; no >5 ms samples
• DeepSeek V4 Serving MTP-1, GBS32, DP8/EP8, 32 concurrent requests x 256 output tokens: 8192/8192 tokens, 426.909 tok/s, task exit 0.
  • runner-start skew p95/max: 0.096/1.089 ms, with no >5 ms late start
  • runner-to-validate gap max: 0.052 ms; validate max: 0.227 ms

The Serving run still observed seven random-rank runner-end skews above 5 ms (maximum 7.420 ms). Device-phase markers were disabled in the acceptance run, so this PR does not claim that every residual completion/fence tail is eliminated.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: db8a5c5d-61ab-4f9e-923e-efea517ef988

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}