vmm: sanitize GPUs through vfio-ioctls hot reset - #1065
Open
kvinwang wants to merge 4 commits into
Open
Conversation
The sanitize-on-attach path issued the Secondary Bus Reset by writing Bridge Control in the upstream bridge sysfs config space and re-probed devices through /sys/bus/pci/drivers_probe. Both files are writable by root only, so the feature could not be enabled in production where dstack-vmm runs as an unprivileged user with no sudo. Switch to the VFIO_DEVICE_PCI_HOT_RESET ioctl, which makes the kernel perform the same bus reset. The ioctl is authorized by device ownership rather than privilege: the caller presents fds for every VFIO group affected by the reset, and the /dev/vfio group nodes are the same ones QEMU opens to attach the GPU, so the VMM user already has access. A single group fd suffices because every sanitized GPU sits alone behind a dedicated PCIe bridge and alone in its IOMMU group. The bridge topology check is kept as defense, and the kernel-reported set of affected devices must all belong to the GPU own group or the launch is aborted. Devices stay bound to vfio-pci across the reset, so the drivers_probe re-probe logic is no longer needed and is removed. Not yet validated on GPU hardware; see plans/2026-08-14-vfio-gpu-hot-reset.md for the pending experiment.
Expose the sanitize path as "dstack-vmm sanitize-gpu <slot>..." so operators can reset GPUs by hand and the pending hardware experiment can exercise exactly the code path used at VM launch, running as the unprivileged VMM user. The subcommand needs no server configuration, only /dev/vfio access, and is handled before config loading like the other special modes.
kvinwang
force-pushed
the
fix/gpu-sbr-vfio-ioctls
branch
from
August 17, 2026 13:46
ff467dc to
3d9c161
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VFIO_DEVICE_PCI_HOT_RESETrust-vmm/vfioabstractions instead of maintaining handwritten VFIO ioctl numbers, ABI structs, and variable-length buffer parsing in dstackvfio-pciacross the resetdstack-vmm sanitize-gpu <slot>...for standalone validation and operationsDependency strategy
rust-vmm/vfioalready provides the VFIO bindings and container/group/device lifecycle abstractions, but its public API does not yet expose PCI hot reset operations. This PR temporarily pins the following fork commit:The fork adds
pci_hot_reset_infoandpci_hot_resetAPIs. After validating the complete flow on GPU hardware, we plan to submit those APIs upstream and replace the Git dependency with a released crates.io version.Motivation
The existing sanitize path writes Bridge Control through sysfs and re-probes devices through
/sys/bus/pci/drivers_probe. Both operations require root privileges. The VMM normally runs as a dedicated unprivileged user that already has access to the VFIO group nodes required for GPU passthrough.The VFIO hot-reset ioctl asks the kernel to perform the bus reset and authorizes it through ownership of every affected VFIO group. This is the same mechanism used by VMM implementations such as QEMU.
Safety
Tests
cargo check -p dstack-vmmcargo test -p dstack-vmm— 119 passedcargo clippy -p dstack-vmm --all-targets -- -D warningsThe new reset path has not yet been validated on target GPU hardware. The standalone
sanitize-gpucommand is included to exercise the exact launch-time path during that validation.Relationship to #1058
This is an alternative implementation of #1058 that addresses the review feedback to use
vfio-ioctls. PR #1058 has not been modified.