Skip to content

Normalize measured GPU VRAM in SSH and Kubernetes - #4406

Closed
peterschmidt85 wants to merge 1 commit into
masterfrom
pr_normalize_gpu_memory
Closed

peterschmidt85 wants to merge 1 commit into
masterfrom
pr_normalize_gpu_memory

Conversation

@peterschmidt85

Copy link
Copy Markdown
Contributor

Kubernetes can infer the wrong VRAM from a GPU model name that has multiple capacities, while SSH fleet discovery rounds unmatched measurements. This can misrepresent available VRAM and affect memory-based scheduling.

  • Read NVIDIA's nvidia.com/gpu.product and nvidia.com/gpu.memory labels and normalize measured VRAM through gpuhunt. Match both labels when scheduling so cards with different capacities stay distinct.
  • Use the same normalization for SSH fleets, preserving measured MiB when no capacity matches. Both older string and newer numeric shim memory formats remain accepted.
  • Warn with node or instance details when a model or capacity cannot be normalized. Kubernetes also warns when the measured-memory label is missing or invalid.

Slurm code and configuration syntax stay unchanged. Existing bare P100 mappings continue resolving to 16GB; A100 mappings continue specifying A100:40GB or A100:80GB.

Depends on dstackai/gpuhunt#279. Release that change and bump the gpuhunt dependency before merging this draft.

@peterschmidt85

Copy link
Copy Markdown
Contributor Author

Created by mistake. Cloud-offer VRAM normalization is handled in gpuhunt; these dstack changes are not required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant