Keep accelerator data reachable when the devices cannot all read each other - #802
Merged
Merged
Conversation
parsec_cuda_all_devices_attached() cleared peer_access_mask and only set the self-access bit while walking the peers, a loop it never reaches for a device excluded from device_cuda_nvlink_mask. Such a device was left believing it could not even read its own memory. The mask is how the rest of the runtime tells a copy that is already on the device it needs from one that has to be moved, so a device that answers no for itself is reported as unable to reach a copy it is holding. Set the self bit before the mask can be abandoned, and let the peer loop skip the device it is asked about. Signed-off-by: George Bosilca <gbosilca@nvidia.com>
pushout was only ever forced for a remote successor, on the grounds that the communication engine cannot send from accelerator memory. A local successor was assumed to be able to read the producer wherever it ran, which holds only as long as every accelerator of the process can read every other one. On a machine whose devices do not all reach each other, either because the hardware cannot or because device_cuda_nvlink_mask excludes some of them, a successor scheduled on a device that cannot read the producer has no source for its input. Where a local successor will be scheduled is not known when the producer is submitted, so any device has to be treated as a possible destination. Derive once, from the peer access masks of the attached devices, whether they all reach each other, and expose the answer as parsec_device_peer_mesh_incomplete. A successor walk is then worth its cost when either a remote successor cannot be served from accelerator memory or a local one may be unable to read the producer, and each successor is charged against the condition that concerns it. The walk no longer depends on DISTRIBUTED, only the read of parsec_mpi_allow_gpu_memory_communications does, as that symbol lives in the MPI communication engine. Level Zero has no device to device transfer, so its devices only ever reach their own memory and report a mesh that is incomplete as soon as there is more than one of them. Signed-off-by: George Bosilca <gbosilca@nvidia.com>
An incoming remote dependency is placed in the memory of the device the sender advertised as preferred, a choice made before the placement of the successors that will read it is known. Nothing mirrors such a value back to the host afterwards, as the host detour is only arranged for the successors of an accelerator task, and an incoming dependency is not one. A successor scheduled on a device that cannot read the one that received the data therefore has no reachable source for its input. Where the devices all reach each other this is invisible, since the successor simply reads its peer. Where they do not, the successor falls back on a host copy that was never filled, and the computation proceeds on whatever that buffer held. Take the host detour for incoming data whenever the devices do not all reach each other. This is the receive-side counterpart of the host mirroring that already covers the successors of an accelerator task. Signed-off-by: George Bosilca <gbosilca@nvidia.com>
Falling back on the host copy in parsec_device_data_stage_in() was conditioned on having found a peer that holds the version, so the check that the host copy is usable at all was only reached when such a peer existed. When no peer was even a candidate the host copy was read unconditionally, including when it carries another version or was never filled, and the kernel ran on the wrong bytes with nothing reported. Evaluate the host copy on its own terms. When a peer does hold the version and is merely busy, waiting is still the answer and the task is deferred. When nothing reachable holds it, neither waiting nor reading is defensible: say so and report the device, the task, the flow, the version that was required and the state of the host copy. A host copy that is present but marked invalid is one of the shapes this takes, and it is not benign: a mirror allocated to receive a stage out is given a version derived from the accelerator copy, which for a copy still at its first version is that same version, so an empty buffer can advertise exactly the version a consumer is looking for. Signed-off-by: George Bosilca <gbosilca@nvidia.com>
bosilca
force-pushed
the
topic/gpu-peer-mesh-pushout
branch
from
September 25, 2026 20:05
df47bd7 to
e9bfbff
Compare
A mirror allocated to receive a stage out was given the version of the accelerator copy less one. The buffer is empty at that point, so the number described nothing; it existed only to get past an assertion in the epilog that demanded the host copy be strictly behind the copy it was about to adopt. The value it invents is not always even distinct. A copy still at its first version leaves no room below it, so the subtraction is clamped and the mirror ends up announcing the very version the accelerator copy carries, which is also the version a consumer of a first generation value is looking for. The assertion the number was invented for is then violated as well. Nothing is lost by saying nothing. An empty copy is already marked invalid, and invalid is what every reader consults before a version means anything, so the version on such a copy is never the deciding factor. Leave it at the value the constructor gives, and let the epilog expect an ordering only from the copies that do carry one. Signed-off-by: George Bosilca <gbosilca@nvidia.com>
bosilca
force-pushed
the
topic/gpu-peer-mesh-pushout
branch
from
September 25, 2026 20:13
92532d7 to
35048c4
Compare
Contributor
|
15 minutes between open and merge? Mhhh. Anyway, I think this is excessive as it forces every modified data back to the host, even if it is not consumed by another device (of which there may be many if the device mapping is done well by the application). Alternative: the peer device already requests a copy via the d2d task it schedules on the source device. If we receive one such task for a copy, we can do the pushout then if the two devices cannot access each other. Otherwise the data stays on the device. |
Contributor
Author
|
This was responsible for 99% of the CUDA failing tests. Depending on what GPUs the CI got sometimes the checks will work and sometime not, all boiling down to the availability of nvlink. If a process has multiple GPU, and these GPUs cannot access each other memory, no d2d tasks are created and it hangs. Moreover, there was a different issue, if at least one of the successors did not have a GPU bound chore, then the data must land on CPU. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A local successor was assumed to be able to read its producer wherever that producer ran. That holds only while every accelerator of a process can read every other one. When it does not hold, a successor can be scheduled on a device that has no reachable source for its input, falls back on a host copy that was never filled, and the computation proceeds on whatever that buffer held.
dplasma_dpotrf_2gpu_cuda_mpireproduces this as ananresidual.Two paths leave a value on a single unreachable device.
pushoutwas forced only for remote successors, on the grounds that the communication engine cannot send from accelerator memory; local successors were never considered. And an incoming remote dependency is received straight into the memory of the device the sender advertised, a choice made before the placement of its consumers is known, with nothing mirroring it back to the host afterwards.Both now take the host detour when the devices do not all reach each other, a property derived once from the peer access masks at attach time. Where the mesh is complete nothing changes, so machines with full peer access pay nothing.
The stage-in fallback is also made honest: it used to read the host copy without checking it whenever no peer was a candidate. It now defers when a peer holds the version and is merely busy, and reports a fatal error when nothing reachable holds it, rather than computing on the wrong bytes silently. Since that check makes the coherency state the deciding signal, a host mirror allocated for a stage out no longer invents a version it cannot honour.