Skip to content

Support Blackwell GPUs and MIG slices for structure inference - #52

Merged
DimaMolod merged 1 commit into
mainfrom
feature/blackwell-mig-support
Aug 11, 2026
Merged

Support Blackwell GPUs and MIG slices for structure inference#52
DimaMolod merged 1 commit into
mainfrom
feature/blackwell-mig-support

Conversation

@DimaMolod

Copy link
Copy Markdown
Collaborator

Summary

Makes the new Blackwell hardware actually usable for structure inference: one real bug fix for MIG
slices, plus documentation/routing for both Blackwell node types.

Companion to KosinskiLab/AlphaPulldown#629 (container side).

The bug: unified memory is silently disabled on MIG slices

structure_inference_xla_mem_fraction: auto computes host RAM / GPU VRAM from
nvidia-smi --query-gpu=memory.total. On a MIG slice that query reports the parent card, not
the slice — 32623 MiB for a 16 GB 1g.16gb instance. The fraction therefore comes out roughly
halved, which switches off host spill on exactly the small slices that need it most: borderline
complexes OOM instead of spilling to host RAM.

Fixed by reading the slice's own profile from nvidia-smi -L when CUDA_VISIBLE_DEVICES is a MIG
UUID, falling back to --query-gpu on whole cards.

Measured on EMBL gpu-el8 with 32000 MiB host RAM:

GPU gpu_mem_mb fraction JAX actually sees
RTX PRO 4500 MIG 1g.16gb 16384 (was 32623) 1.953 (was 0.981) ~16032 MiB
RTX PRO 6000 Blackwell 96 GB 97887 0.327 ~97252 MiB
RTX 3090 24576 1.302 ~24126 MiB

Whole cards are byte-identical before/after — this only affects MIG nodes.

Docs / routing

  • Blackwell documented as verified rather than expected: AlphaPulldown 2.5.0 containers run on
    RTX PRO 6000 (gpu50-53) and on the RTX PRO 4500 16 GB MIG slices (gpu60-68) for both AF3 and
    AF2, with confidence scores matching A100/H100/3090. Only pre-2.5.0 images fail there, at the
    first kernel compilation with ptxas too old / UNIMPLEMENTED — so excluding those nodes is an
    old-image workaround, not a hardware statement.
  • Added both node groups to the structure_inference_gpu_tiers example: a 16 GB tier (monomers and
    small complexes) and a 96 GB tier (large ones).
  • Noted that a plain gpu:1 gres already lands on a MIG slice (SLURM sets
    CUDA_VISIBLE_DEVICES=MIG-<uuid>), so no slurm_gres change is needed to use them.

Testing

  • Fraction logic run on real hardware — MIG slice, RTX PRO 6000, and RTX 3090 — producing the table
    above; non-MIG output unchanged.
  • The edited shell: block verified to expand under Snakemake's formatter (brace escaping) and to
    produce valid bash (bash -n).

🤖 Generated with Claude Code

Size the unified-memory fraction from the MIG slice rather than its parent
card. structure_inference_xla_mem_fraction: auto computes host RAM / GPU
VRAM from `nvidia-smi --query-gpu=memory.total`, which on a MIG slice
reports the whole physical card - 32623 MiB for a 16 GB 1g.16gb instance.
That roughly halves the fraction and so switches off host spill exactly on
the small slices that need it most: with 32 GB of host RAM the fraction came
out 0.981 (no spill past the ~16 GB slice) instead of 1.953. Read the
slice's own profile from `nvidia-smi -L` when CUDA_VISIBLE_DEVICES is a MIG
UUID, falling back to --query-gpu on whole cards.

Measured on EMBL gpu-el8 with 32000 MiB host RAM:

  GPU                              gpu_mem_mb   fraction    jax sees
  RTX PRO 4500 MIG 1g.16gb         16384        1.953       ~16032 MiB
    (was)                          32623        0.981
  RTX PRO 6000 Blackwell 96GB      97887        0.327       ~97252 MiB
  RTX 3090                         24576        1.302       ~24126 MiB

Whole cards are unchanged, so this only affects MIG nodes.

Also document Blackwell as verified rather than expected. AlphaPulldown
2.5.0 containers run on RTX PRO 6000 (gpu50-53) and on the RTX PRO 4500
16 GB MIG slices (gpu60-68) for both AF3 and AF2, with confidence scores
matching A100/H100/3090; only pre-2.5.0 images fail there, at the first
kernel compilation with "ptxas too old" / UNIMPLEMENTED. Add both node
groups to the gpu_tiers example (a 16 GB tier for monomers and small
complexes, a 96 GB tier for large ones) and note that a plain gpu:1 gres
already lands on a MIG slice, so no gres change is needed to use them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@DimaMolod
DimaMolod merged commit 1c16324 into main Aug 11, 2026
2 checks passed
@DimaMolod
DimaMolod deleted the feature/blackwell-mig-support branch August 11, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant