Skip to content

Fix ROCm misidentified as CUDA/FP8-capable via device.type check - #160

Open
boxwrench (boxwrench) wants to merge 1 commit into
Lightricks:mainfrom
boxwrench:rocm-accelerator-fixes
Open

Fix ROCm misidentified as CUDA/FP8-capable via device.type check#160
boxwrench (boxwrench) wants to merge 1 commit into
Lightricks:mainfrom
boxwrench:rocm-accelerator-fixes

Conversation

@boxwrench

Copy link
Copy Markdown

Summary

ROCm PyTorch reports torch.cuda.is_available() and device.type == "cuda" identically to real NVIDIA CUDA builds — the only reliable discriminator is torch.version.hip (set on ROCm, None on CUDA). Two places in the backend assumed device.type == "cuda" meant NVIDIA/FP8-capable hardware:

  • services_utils.device_supports_fp8() would attempt FP8 quantization on ROCm, which has no fp8_cast kernel support there.
  • runtime_policy.decide_local_generation_mode()'s >=31 GB -> full_models_loading floor assumes the FP8-halved (~23 GB) transformer footprint. Without FP8, the full ~42-46 GB bf16 transformer tries to stay resident instead and OOMs at that floor — reproduced live: a HIP OOM crash on an AMD Radeon AI PRO R9700 (31.86 GB reported VRAM), fixed, and re-verified with a successful generation afterward.

Adds runtime_config/accelerator.py with a torch.version.hip-based accelerator_backend() helper ("rocm" | "cuda" | "mps" | "cpu"), and an fp8_capable parameter (default True, so existing CUDA/MPS behavior is unchanged) that forces streaming_models_loading when FP8 isn't available.

Found while bringing up a community ROCm port (boxwrench/LTX-Desktop-ROCm); this PR is scoped to just the two backend-agnostic correctness fixes, none of the ROCm-specific packaging/dependency-pinning work.

Test plan

  • Full backend test suite: 927 passed, 1 skipped (unchanged from base)
  • pyright strict on the four touched files: 0 errors, 0 warnings
  • Reproduced the runtime_policy OOM live on an R9700, applied the fix, re-ran the same generation successfully
  • Maintainer sanity check that fp8_capable defaulting to True correctly preserves all existing CUDA/MPS call sites (no other caller currently passes it)

🤖 Generated with Claude Code

ROCm PyTorch reports torch.cuda.is_available() and device.type == "cuda"
identically to real NVIDIA CUDA builds — the only reliable discriminator
is torch.version.hip (set on ROCm, None on CUDA). Two places assumed
device.type == "cuda" meant NVIDIA/FP8-capable hardware:

- services_utils.device_supports_fp8() would have attempted FP8
  quantization on ROCm, which has no fp8_cast kernel support there.
- runtime_policy.decide_local_generation_mode()'s >=31 GB ->
  full_models_loading floor assumes the FP8-halved (~23 GB) transformer
  footprint; without FP8 the full ~42-46 GB bf16 transformer would try
  to stay resident instead and OOM at that floor.

Add runtime_config/accelerator.py with a torch.version.hip-based
accelerator_backend() helper ("rocm" | "cuda" | "mps" | "cpu"), and an
fp8_capable parameter (default True, so CUDA/MPS behavior is unchanged)
that forces streaming_models_loading when FP8 isn't available.

Found and fixed while bringing up a community ROCm port; the OOM was
reproduced live on an AMD Radeon AI PRO R9700. Full test suite and
pyright pass unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant