[GSD-13346] Multi-device visibility (ZE_AFFINITY_MASK=0,1) costs ~1 GiB host RAM per GiB of device memory allocated (driver-side, invisible to meminfo buckets)

[SergiioB]

Summary

On a system with two Arc Pro B70 (xe driver, kernel 7.0, Level Zero runtime libze_intel_gpu.so.1.15.39122 / intel-opencl-icd 26.27.39122.11, as shipped inside the vllm/vllm-openai-xpu image), any process in which Level Zero can see both GPUs (ZE_AFFINITY_MASK=0,1) consumes approximately 1 GiB of host RAM per 1 GiB of device memory it allocates, on either GPU. With a single visible device (ZE_AFFINITY_MASK=0 or 1) the same allocations cost almost no host RAM. The host memory is not attributable to any standard accounting bucket and is released only when the process's DRM contexts are torn down.

This makes multi-GPU serving on modest-RAM hosts fail: loading a 29 GiB FP8 model across two 32 GB B70s through two worker processes (each seeing both GPUs, as vLLM configures them) exhausts 30 GiB of host RAM, while the identical load with per-process single-device visibility succeeds with ~2 GiB of host overhead.

Reproducer (~60 s, no files read, no host->device copies)

Run in any torch-xpu container (--device /dev/dri -v /dev/dri:/dev/dri:ro), then watch host /proc/meminfo MemAvailable while the process holds the tensors:

docker run --rm --device /dev/dri -v /dev/dri:/dev/dri:ro \
  -e ZE_AFFINITY_MASK=0,1 \
  --entrypoint python3 <torch-xpu image> - <<'EOF'
import torch, time
ts = []
for i in range(10):                       # 10 GiB of device memory on xpu:0
    t = torch.empty(1 << 30, dtype=torch.uint8, device="xpu:0")
    t.zero_()                             # commit pages on device; no host->device copy
    ts.append(t)
print("HOLDING")                          # inspect host /proc/meminfo NOW
time.sleep(60)
EOF

Measured matrix (host MemAvailable delta while tensors held)

Process visibility	Device memory held	Host RAM cost	Ratio
ZE_AFFINITY_MASK=0 (1 GPU visible)	5.6 GiB (GPU0)	0.77 GiB	~0.14 (process overhead)
ZE_AFFINITY_MASK=0,1	10.8 GiB (GPU0)	10.59 GiB	0.98
ZE_AFFINITY_MASK=0,1 (+ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE)	11.0 GiB (GPU0)	10.97 GiB	1.00
2 processes, masks 0 and 1	10.5 GiB (both GPUs)	0.82 GiB	~0.08

All allocations landed on GPU 0 in the affected runs (GPU 1 nearly untouched), so the cost tracks device-memory allocations, not per-device context creation.

Forensics (while 10.9 GiB VRAM held under mask 0,1)

• Worker RSS (VmHWM): <= 2.6 GiB — the host memory is not process anonymous memory.
• AnonPages 121 MiB, Cached 80 MiB, Slab 0.83 GiB, Shmem 4 MiB, VmallocUsed 144 MiB, HugePages_Total 0, Unevictable/Mlocked/VmPin ~0 — no visible bucket accounts for the ~10 GiB; /proc/buddyinfo confirms the pages are genuinely allocated.
• Fully recovered after the container exits (DRM context teardown).
• Reproduces with zero file I/O and with allocations never copied from host buffers (t.zero_() only), so this is not page cache, not checkpoint staging, not pinned transfer buffers.

Questions

1. What kernel/driver structure scales ~1:1 with zeDriverAllocDeviceMemory when the process sees more than one device?
2. Is there a supported way to disable it (environment variable or allocation flag)?
3. If intended, could it be documented? Currently it presents like a host-memory leak in every multi-GPU XPU process.

Environment: 2 x Intel Arc Pro B70 (8086:e223), xe KMD, kernel 7.0.0, Level Zero 1.15.39122 / intel-opencl-icd 26.27.39122.11 (pinned inside the vllm-openai-xpu image; newest compute-runtime release not yet re-tested — happy to re-verify on request), docker with --device /dev/dri plus a read-only /dev/dri bind mount, AMD-IOMMU enabled, resizable BAR active (32 GB aperture per card).

[weddle]

Reproduced on a dual-B65 host: 8 GiB allocations caused negligible host-memory loss with one visible GPU, but approximately 1:1 host-memory loss when both GPUs were visible; memory recovered after process exit.

[kgibala]

Hello @SergiioB,

Thank you for your contribution.

Could you please follow the Issue Submission Guide and create an issue based on it? You can edit this one.

Thank you.

[SergiioB]

This duplicates #980, which already follows the Linux issue template and isolates the same trigger: one context spanning two devices causes approximately 1:1 host-memory backing, while per-process single-device visibility avoids it.

I added this issue's B70 visibility matrix and the independent B65 reproduction to #980 so the extra evidence is retained. Closing this report as duplicate rather than maintaining a third parallel investigation. The version-bisected defer-backing regression remains tracked separately in #953.