feat(validator): re-enable SR-IOV vGPU VFs before waiting for them

[lexfrei]

Description

On the vGPU (sandbox) workload path, SR-IOV Virtual Functions (VFs) are runtime state that does not survive a node reboot. The vGPU Manager validator already waits for VFs to appear (waitForVFs) before the vGPU device-manager applies its config, but nothing re-creates them. After a node reboot the VFs are gone, so validation blocks until they are re-established out-of-band (e.g. host systemd units running sriov-manage -e) — the operator does not recover this on its own.

This adds the smallest useful part of the reboot-recovery seam discussed in #2600: the operator re-enables the SR-IOV VFs itself. Right before the existing wait, VGPUManager.validate() runs sriov-manage -e ALL inside the resolved driver root (the driver container root, or the host root when the vGPU Manager driver is pre-installed on the host).

The call is:

• gated to the vm-vgpu workload path — the function already early-returns for other workloads;
• idempotent — skipped when every SR-IOV-capable GPU already has its full VF count, which is the normal steady state, so it is a no-op except right after a reboot when no VMs are running yet;
• best-effort — on failure it logs and falls through to waitForVFs, preserving the current behavior for setups that create VFs out-of-band.

The PF-counting loop is extracted into a shared countVFs helper so the new idempotency guard and waitForVFs agree on the accounting.

Scope is intentionally limited to VF re-enablement. GPU reset, MIG-mode commit, and nvidia-persistenced coordination (the rest of #2600) are out of scope here.

Refs #2600.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

• go vet and golangci-lint run on the changed package, cross-compiled GOOS=linux (the validator is a Linux-only binary): clean, 0 issues. No asset or go.mod/go.sum changes.
• New unit test TestCountVFs covers the shared VF-accounting helper across the boundary cases (no SR-IOV GPUs, VFs missing after reboot, fully enabled, partially enabled across multiple PFs, and VFs not miscounted as PFs); it passes in a Linux container.
• Not yet exercised on GPU hardware; the runtime sriov-manage/chroot path relies on CI and cluster testing.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rajathagasthya]

/ok-to-test 7d6d61f

[rajathagasthya]

Thanks @lexfrei! I will defer to @cdesiniotis to assess this PR since he has more knowledge on SR-IOV parts.

[karthikvetrivel]

Hey @lexfrei! Sorry for the delay on the response here. For some context, the validator infers the GPU's vGPU mode from side effects: totalVFs > 0 and numVFs > 0.

enableVFs gates on the PCIe capability bit (IsPF() && TotalVFs > 0) rather than the driver's vGPU mode, so a Turing T4, which advertises sriov_totalvfs=16 but runs PF-based mdev and never creates VFs, reads as 0/16 "missing" and gets sriov-manage -e ALL. The sriov-manage still unbinds the PF from nvidia, binds pci-pf-stub, and rebinds on a PF that may have live mdev vGPUs.

We can't reuse isDriverUsingSRIOV because it requires NumVFs > 0. It could be worth switching to using nvmlDeviceGetHostVgpuMode() to get a signal from the driver directly.

[karthikvetrivel]

Another high level comment:

Every other check in this pod just reads state and reports, while this one detaches GPU from its driver and reattaches it. This could be out of scope of a "validator." Given the vGPU Manager container already calls sriov-manage at startup,
is the validator the right home for this, or should VF creation live with whichever component owns the driver lifecycle?

@cdesiniotis interested to hear your thoughts on this thread.

[cdesiniotis]

Another high level comment:

Every other check in this pod just reads state and reports, while this one detaches GPU from its driver and reattaches it. This could be out of scope of a "validator." Given the vGPU Manager container already calls sriov-manage at startup, is the validator the right home for this, or should VF creation live with whichever component owns the driver lifecycle?

@cdesiniotis interested to hear your thoughts on this thread.

Agreed. SRIOV creation is out-of-scope for the validator.

If the vGPU Manager is deployed by the operator, then the VFs will be created by the vGPU Manager pod on a reboot. If the vGPU Manager is installed directly on the host (not managed by the operator), then some host-side component (i.e. a systemd service, a udev rule) should be responsible for ensuring the VFs are recreated on a reboot.

Closing this PR as out-of-scope.