Skip to content

Normalize GPU VRAM against registered capacities - #279

Merged
peterschmidt85 merged 3 commits into
mainfrom
pr_normalize_gpu_memory
Oct 29, 2026
Merged

peterschmidt85 merged 3 commits into
mainfrom
pr_normalize_gpu_memory

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Offers can report VRAM that differs from the GPU SKU's known capacity. This makes expected-VRAM filters unreliable and splits equivalent offers into separate groups when grouping by GPU.

  • Normalize full-GPU, per-device VRAM against KNOWN_ACCELERATORS, accepting a correction only when exactly one capacity matches the existing ±7% window.
  • Round reported VRAM to whole GiB when the model is unknown or no unique capacity matches, with warnings identifying the offer, original measurement, and returned value. Partial allocations keep their assigned VRAM.
  • Correct DigitalOcean's combined-VRAM handling and the MI325X registry capacity, and add common RTX models found in real offers.

Examples, in GiB per GPU:

GPU Backend Before After
B300 AWS 268.59 270
B300 OCI 262.5 270
B300 RunPod 288 270
B300 Verda 268 270
L40S AWS 44.70 48
MI325X, 8 GPUs DigitalOcean 2,048 256

Compatibility:

  • SSH GPU discovery uses the existing helper; shim formats and configuration stay unchanged.
  • Kubernetes continues using the registry for GPU memory; MI325X detection now reports 256GB instead of 288GB. No dstack code or configuration changes are required.
  • Slurm syntax and P100/A100 mappings are unchanged. In gpu_partitions, gpu: MI325X:288GB will fail to resolve; change it to gpu: MI325X:256GB. Bare gpu: MI325X needs no edit and now resolves to 256GB.

@peterschmidt85
peterschmidt85 requested a review from jvstme October 26, 2026 11:18
@peterschmidt85
peterschmidt85 marked this pull request as ready for review October 26, 2026 11:18
Comment thread src/gpuhunt/__init__.py Outdated
KNOWN_TPUS as KNOWN_TPUS,
correct_gpu_memory_gib as correct_gpu_memory_gib,
find_accelerators as find_accelerators,
get_gpu_memory_gib as get_gpu_memory_gib,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) Is this necessary? I think we usually only export gpuhunt utility functions if we need to call them in dstack

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) You've switched from correct_gpu_memory_gib to get_gpu_memory_gib throughout the codebase. The only behavioral difference is that get_gpu_memory_gib doesn't round the memory of unknown GPUs to a whole number. Wouldn't it make sense to round it? I don't think any GPU models have a nominal memory that isn't a whole number, so we probably don't want to display fractional values in dstack.

If you decide to round everywhere, I don't see a reason to keep both correct_gpu_memory_gib and get_gpu_memory_gib. I'd keep only correct_gpu_memory_gib.

@peterschmidt85
peterschmidt85 marked this pull request as draft October 29, 2026 08:52
@peterschmidt85
peterschmidt85 marked this pull request as ready for review October 29, 2026 08:53
@peterschmidt85
peterschmidt85 merged commit 02bce82 into main Oct 29, 2026
6 checks passed
@peterschmidt85
peterschmidt85 deleted the pr_normalize_gpu_memory branch October 29, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants