Conversation
_probe allocates its destination with torch.zeros, which enqueues the fill on the current stream, then enqueues the verification copy on a fresh probe stream. Nothing joins the two. When the current stream has a backlog the copy completes first on the independent probe stream and the fill lands on top of it, so the probe reads back zeros and load_batch_memcpy raises "cudaMemcpyBatchAsync probe copied wrong bytes" on a GPU that supports the API. OffloadMoeCache catches that and falls back to full-layer copies, so --moe-prefill-hit-d2d silently does nothing whenever the current stream is busy as the probe runs -- in practice during prefill warmup, which is exactly when the flag is first exercised. A cold process hides the bug: the first torch.zeros pays a cudaMalloc and the first torch.cuda.Stream() populates the per-device stream pool, and each of those synchronizes the device, draining the backlog before the copy is enqueued. That is why the probe passes when run standalone and fails inside a warmed-up server. Join the probe stream to the current stream before the copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 18, 2026
gdevenyi
force-pushed
the
5080/11-msgpack-frames-466
branch
from
September 19, 2026 21:31
412afc9 to
fff12a0
Compare
gdevenyi
force-pushed
the
5080/12-memcpy-probe-order-414
branch
from
September 19, 2026 21:31
65cb410 to
b63d524
Compare
…nst the current stream
gdevenyi
force-pushed
the
5080/11-msgpack-frames-466
branch
from
September 23, 2026 23:36
fff12a0 to
7c58f01
Compare
gdevenyi
force-pushed
the
5080/12-memcpy-probe-order-414
branch
from
September 23, 2026 23:36
b63d524 to
0a62baa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #49. Merges upstream FlashML-org#414 (head
b6e7995) into5080deploy, rebuilt on upstream maincac247a(v0.1.3).Why it is in scope for this deployment: Generic kernel fix (the expert-cache batch memcpy probe); this deployment uses the offload expert cache it probes for.
Merge: Clean merge.
Testing (RTX 5080 16 GB, production flags: FTW-fp8, nvfp4 KV, 262,144 tokens, 1,280 expert slots):
pytest tests/ -m "not slow"at69a5efa(the tip before the middleware commit8adde91, which touches onlyserver/api_server.pyand adds one test;tests/serverrerun at8adde91: 871 passed): 2412 passed, 206 skipped; the 4 failures are environment-only (3xtest_muse_glimmer_vision.pyfail identically on pristine upstreamcac247awith transformers 5.15.1, andtest_async_pull_coalescedis anasync deftest that needspytest-asyncio, which the project does not declare).Rebased 2026-09-19 onto upstream main
cc1f5c2(4 commits pastcac247a: FlashML-org#471 greedy sampling in mixed batches, FlashML-org#518 WeightLoadError, FlashML-org#521 tvm-ffi jit arch, FlashML-org#524 install index). Same replay as before viagit rerere; the replayed stack differs from the previous tip8adde91by exactly thecac247a..cc1f5c2file set. This PR's head is nowb63d524.🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt