transport ci : nic amd gpu emulation - #3178
Draft
multiphaseCFD wants to merge 9 commits into
Draft
multiphaseCFD wants to merge 9 commits into
multiphaseCFD wants to merge 9 commits into
Conversation
The GPU half of gpu_verbs is hipMalloc plus hipMemGetHandleForAddressRange(DmaBufFd), which hipcc maps onto cuMemGetHandleForAddressRange. Whether a given NVIDIA GPU exports device memory that way is a property of the device and driver, needs no RDMA device to establish, and currently gates any plan for testing the GPU RDMA path. The probe attempts the export directly rather than reading CU_DEVICE_ATTRIBUTE_DMA_BUF_SUPPORTED, which is new in CUDA 13.0 and so absent from the cu129 stack this job installs. Runs only on workflow_dispatch; probe_only=true skips the build job so the GPU host is held for seconds rather than the full build.
The A10G answered CUDA_ERROR_INVALID_VALUE for the page-locked host allocation because the probe handed cuMemGetHandleForAddressRange the device pointer from cuMemHostGetDevicePointer. The documented exportable range for cuMemHostAlloc is the host pointer, so the error said nothing about whether the device supports the export.
Catch2 exits 4 when a binary ran no tests, which is exactly what an all-skipped suite looks like from outside: on a host with no RDMA device the verbs suites skip every case. Under set -e that aborted the loop, so the suites after it never ran and the job reported a test failure -- on a device-less machine runner_tests_transport_memcpy was silently lost that way. --allow-running-no-tests makes the skip exit 0 instead. The same glob also matched the non-executable *_include.cmake and *_tests.cmake files catch_discover_tests writes beside the binaries. Two of the three loops had no -x guard and would have tried to execute them. All three loops were copies of each other, so they now share one define.
The GPU coprocessor registers device memory with the NIC as a dma-buf -- hipMalloc, then hipMemGetHandleForAddressRange in GpuRuntime.hip, then the MemoryRegion dma-buf constructor -- and nothing exercised that constructor off a GPU host. udmabuf produces an equivalent fd from host memory, so the registration branch is reachable with neither a GPU nor a HIP toolchain. Skips on three counts: no rxe0, no /dev/udmabuf, or a provider without reg_user_mr_dmabuf. Soft-RoCE is the last of those on 6.8 (dma-buf support for rxe was only submitted upstream in March 2026), so the case skips today and turns into a real assertion once it runs against a provider that implements it.
Test_TransportGpu.cpp is entirely STATIC_REQUIRE: the Payload frame layout, the 64 B ring slot, the 16 B handoff slot, K_RING_SLOTS. None of it needs a GPU at runtime, and neither GpuRuntime.hpp nor GpuCoprocessorSession.hpp includes a HIP header -- hipStream_t is kept as an opaque void* precisely so they stay includable. Gating the whole target on TRANSPORT_HAS_HIP therefore confined these ABI checks to GPU hosts, where a struct-reordering regression could have merged from any transport PR. Only the library it links needs the gate. transport_tests_all gated the dependency the same way, which is why the target could configure while the binary never built.
gpu_verbs_selftest exercises the coprocessor session end to end -- RC bring-up, MR and key exchange, the ring and handoff protocol, launcher dispatch, collect, teardown -- but needed a GPU for two things only: the allocations and the persistent kernel. TRANSPORT_GPU_STUB=ON supplies both from the host, so the session is testable on any machine with an RDMA device, and the selftest passes against a cpu_verbs controller over rxe0. GpuRuntimeHost.cpp implements the same GpuRuntime.hpp declarations from 64 B aligned host pages; one address space, so each mapping's device view is the host pointer. GpuLaunchersHost.cpp mirrors fused<DECODE> step for step in a host thread, with acquire/release fences where the kernel uses __threadfence_system and the stop flag sampled on the same interval. Both decoders reuse the constexpr tables in SteaneLut.hpp rather than duplicating them. alloc_memory has to register host pages, since they have no dma-buf export. That is under #ifdef, not a runtime fallback: on real hardware the export always yields an fd, and quietly registering host memory when it did not would turn a failed GPUDirect setup into a slow wrong success. The #else branch is unchanged, and HIP still wins wherever it is present.
The GPU backends are HIP sources that only ever get built by nvcc against fetched hip-dev headers (lib/transport/CMakeLists.txt), so nothing checks them against the compiler they are written for and an AMD-only error would surface first on real AMD hardware. Device-only compilation needs no GPU and no driver, so this runs on the GH-hosted x86_64 runners, gated on the existing transport path filter. Both sources are clean for gfx950 and gfx90a today. Two things that cost time to find, recorded in the job and the script: * The ROCm repo has to be pinned above Ubuntu universe. Otherwise apt satisfies the HIP dependencies with universe's libamdhip64-dev 5.7.1, whose /usr/include/hip headers shadow ROCm's and predate hipMemRangeHandleTypeDmaBufFd -- the build then fails on an enum that does exist in 7.1.1, at hip_runtime_api.h:2006. * rocm-device-libs is a separate package from rocm-llvm and carries the amdgcn bitcode; without it amdclang++ refuses device compilation. compile_check.sh runs amdclang++ directly when it is on PATH and otherwise re-executes itself inside a container that has one, so CI and a laptop run the same check. `make check-hip-amd` is the local entry point. ROCm publishes x86_64 packages only, hence --platform linux/amd64 in the container path.
rocm-ernic is AMD's libvfio-user RDMA NIC emulator. It builds and self-tests on arm64 (3 of 5 ctest cases pass; the two data-path cases need a VM by upstream's own account), and its guest driver implements reg_user_mr_dmabuf -- the call Soft-RoCE answers EOPNOTSUPP to, and the one the GPU coprocessor needs. That makes it the only dma-buf capable provider reachable without hardware, which is the reason to keep this around. The README records what a real device costs from here: nested virt, a QEMU with the vfio-user client (Debian sid packages 11.1.1 with it), the guest driver and custom rdma-core, and a GPU passed through with vfio-pci for the GPU path. Two upstream arm64 gaps are noted with it: INSTALL.md omits libcmocka-dev, and rocm-ernic's find_library fallback hints name x86_64-linux-gnu only. The rocjitsu patch is separate: aarch64 has no dup2 syscall, so its KMD interposer does not compile there. Worth sending upstream.
multiphaseCFD
changed the base branch from
main
to
shuli/add-transport-runtime-ci
September 2, 2026 18:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before submitting
Please complete the following checklist when submitting a PR:
All new functions and code must be clearly commented and documented.
Ensure that code is properly formatted by running
make format.The latest version of black and
clang-format-20are used in CI/CD to check formatting.All new features must include a unit test.
Integration and frontend tests should be added to
frontend/test,Quantum dialect and MLIR tests should be added to
mlir/test, andRuntime tests should be added to
runtime/tests.When all the above are checked, delete everything above the dashed
line and fill in the pull request template.
Context:
Description of the Change:
Benefits:
Possible Drawbacks:
Related GitHub Issues: