feat(validator): commit pending MIG-mode change via targeted GPU reset - #2603
Open
lexfrei wants to merge 1 commit into
Open
feat(validator): commit pending MIG-mode change via targeted GPU reset#2603lexfrei wants to merge 1 commit into
lexfrei wants to merge 1 commit into
Conversation
lexfrei
marked this pull request as ready for review
July 4, 2026 21:33
lexfrei
requested review from
cdesiniotis,
karthikvetrivel,
rahulait,
rajathagasthya,
shivamerla and
tariq1890
as code owners
July 4, 2026 21:33
Contributor
|
Just linking back to the ongoing discussion at #2600 (comment) (thanks for the additional context there!). Let's revisit this PR once we decide on a direction for #2600. |
On the vGPU (sandbox) workload path, a MIG-mode change is not applied until the GPU is reset, and neither the MIG mode nor the SR-IOV VFs survive a node reboot. The MIG manager can set MIG mode to Enabled, but committing that change requires a GPU reset it does not perform (its only reset mechanism reboots the whole node), so after a reboot MIG-backed vGPU devices cannot be created until the state is re-established out-of-band by host systemd units. Add an opt-in step to vGPU Manager validation that commits a pending-but- uncommitted MIG-mode enable via a targeted 'nvidia-smi --gpu-reset', right before VFs are waited for. The reset is destructive, so it is tightly gated: it runs only when explicitly enabled (commit-mig-mode-with-gpu-reset, off by default), only on the vGPU path, and only on a GPU whose MIG-mode enable is requested but not yet applied, that has no SR-IOV VFs enabled (no vGPU VM attached), and that has no running compute processes. In the steady state it is a no-op. It is best-effort: reset failures are logged and validation continues, preserving current behavior for setups that commit MIG mode and create VFs out-of-band. This complements the SR-IOV VF re-enable seam so MIG-backed vGPU can recover after a reboot without a hand-rolled host unit. VF re-enablement must run after this reset (a GPU reset requires SR-IOV to be disabled first). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
lexfrei
force-pushed
the
feat/validator-commit-mig-mode-gpu-reset
branch
from
August 10, 2026 10:37
7a40a92 to
95e02db
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
On Ampere (A100, A30) enabling MIG mode needs a GPU reset, and the driver tries that reset itself. It doesn't always work: the reset is refused while the GPU is in use, and under passthrough the hypervisor can disallow it. The GPU then sits with
mig.mode.pending=Enabledandmig.mode.currentnotEnabled, so MIG-backed vGPU devices can't be created until something resets it later. Right now that means a hand-rolled host-side step.This adds an opt-in step to vGPU Manager validation that commits such a pending enable with a targeted
nvidia-smi --gpu-reset. It runs before the wait for SR-IOV VFs, since a GPU reset needs SR-IOV disabled first.What this is not: it does not recreate MIG state lost to a reboot, and it isn't needed for that. MIG mode survives a reboot on Ampere via an InfoROM status bit, and from Hopper on, enabling it no longer needs a reset. MIG instances don't survive on any generation, but recreating those is
mig-parted's job.Destructive GPU reset, why it is safe here
nvidia-smi --gpu-resettears down anything running on the GPU. The step is off by default and it resets a GPU only when all of these hold:--commit-mig-mode-with-gpu-reset(envCOMMIT_MIG_MODE_WITH_GPU_RESET), defaultfalse. Unset, behavior is identical to today: no reset,validate()unchanged.commitMIGModeis reached only fromVGPUManager.validate(), which early-returns for non-vm-vgpuworkloads.mig.mode.pending == Enabledandmig.mode.current != Enabled. Something else already requested the change; this only commits it, never decides it.nvidia-smi --gpu-resetrefuses to reset a GPU that is in use anyway, which is the real safety net.In the steady state, MIG already committed or VFs present, it does nothing. Best-effort: reset failures are logged and validation continues, so setups that commit MIG mode out-of-band keep working as before.
On NVSwitch / Fabric Manager systems a single-GPU reset can interact with fabric partition state. This change only commits MIG mode and doesn't manage fabric partitions, which is one of the open design questions for the RFC.
Checklist
make lint): the changed package is clean,GOOS=linux golangci-lint run ./cmd/nvidia-validator/reports 0 issues. Repo-widemake lintwas not run locally.make validate-generated-assets): this change adds no API/CRD changes, so no generated assets change; the full target was not run.make validate-modules)Testing
gofmt,go build,go vet, andgolangci-lint runon the changed package, cross-compiledGOOS=linux(the validator is a Linux-only binary): clean, 0 issues. No asset orgo.mod/go.sumchanges.go test ./cmd/nvidia-validator/):TestNormalizePCIAddress(nvidia-smi 8-digit vs go-nvlib 4-digit PCI domains normalize to the same key),TestParseMIGModes(CSV parsing,[N/A], malformed rows, the pending-vs-current guard),TestMIGModeNeedsCommit(the enable-only commit predicate across every current/pending combination, including case-insensitivity), andTestShouldResetForMIGCommit(the reset-decision guard matrix over uncommitted-enable, VFs-present and workload-running).nvidia-smi --gpu-resetpath relies on CI and cluster testing. This is why the PR is opened as a draft/RFC.Refs #2600.