Support Blackwell GPUs and MIG slices for structure inference - #52
Merged
Conversation
Size the unified-memory fraction from the MIG slice rather than its parent
card. structure_inference_xla_mem_fraction: auto computes host RAM / GPU
VRAM from `nvidia-smi --query-gpu=memory.total`, which on a MIG slice
reports the whole physical card - 32623 MiB for a 16 GB 1g.16gb instance.
That roughly halves the fraction and so switches off host spill exactly on
the small slices that need it most: with 32 GB of host RAM the fraction came
out 0.981 (no spill past the ~16 GB slice) instead of 1.953. Read the
slice's own profile from `nvidia-smi -L` when CUDA_VISIBLE_DEVICES is a MIG
UUID, falling back to --query-gpu on whole cards.
Measured on EMBL gpu-el8 with 32000 MiB host RAM:
GPU gpu_mem_mb fraction jax sees
RTX PRO 4500 MIG 1g.16gb 16384 1.953 ~16032 MiB
(was) 32623 0.981
RTX PRO 6000 Blackwell 96GB 97887 0.327 ~97252 MiB
RTX 3090 24576 1.302 ~24126 MiB
Whole cards are unchanged, so this only affects MIG nodes.
Also document Blackwell as verified rather than expected. AlphaPulldown
2.5.0 containers run on RTX PRO 6000 (gpu50-53) and on the RTX PRO 4500
16 GB MIG slices (gpu60-68) for both AF3 and AF2, with confidence scores
matching A100/H100/3090; only pre-2.5.0 images fail there, at the first
kernel compilation with "ptxas too old" / UNIMPLEMENTED. Add both node
groups to the gpu_tiers example (a 16 GB tier for monomers and small
complexes, a 96 GB tier for large ones) and note that a plain gpu:1 gres
already lands on a MIG slice, so no gres change is needed to use them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the new Blackwell hardware actually usable for structure inference: one real bug fix for MIG
slices, plus documentation/routing for both Blackwell node types.
Companion to KosinskiLab/AlphaPulldown#629 (container side).
The bug: unified memory is silently disabled on MIG slices
structure_inference_xla_mem_fraction: autocomputeshost RAM / GPU VRAMfromnvidia-smi --query-gpu=memory.total. On a MIG slice that query reports the parent card, notthe slice — 32623 MiB for a 16 GB
1g.16gbinstance. The fraction therefore comes out roughlyhalved, which switches off host spill on exactly the small slices that need it most: borderline
complexes OOM instead of spilling to host RAM.
Fixed by reading the slice's own profile from
nvidia-smi -LwhenCUDA_VISIBLE_DEVICESis a MIGUUID, falling back to
--query-gpuon whole cards.Measured on EMBL
gpu-el8with 32000 MiB host RAM:gpu_mem_mb1g.16gbWhole cards are byte-identical before/after — this only affects MIG nodes.
Docs / routing
RTX PRO 6000 (
gpu50-53) and on the RTX PRO 4500 16 GB MIG slices (gpu60-68) for both AF3 andAF2, with confidence scores matching A100/H100/3090. Only pre-2.5.0 images fail there, at the
first kernel compilation with
ptxas too old/UNIMPLEMENTED— so excluding those nodes is anold-image workaround, not a hardware statement.
structure_inference_gpu_tiersexample: a 16 GB tier (monomers andsmall complexes) and a 96 GB tier (large ones).
gpu:1gres already lands on a MIG slice (SLURM setsCUDA_VISIBLE_DEVICES=MIG-<uuid>), so noslurm_greschange is needed to use them.Testing
above; non-MIG output unchanged.
shell:block verified to expand under Snakemake's formatter (brace escaping) and toproduce valid bash (
bash -n).🤖 Generated with Claude Code