Fix ROCm misidentified as CUDA/FP8-capable via device.type check - #160
Open
boxwrench (boxwrench) wants to merge 1 commit into
Open
Fix ROCm misidentified as CUDA/FP8-capable via device.type check#160boxwrench (boxwrench) wants to merge 1 commit into
boxwrench (boxwrench) wants to merge 1 commit into
Conversation
ROCm PyTorch reports torch.cuda.is_available() and device.type == "cuda"
identically to real NVIDIA CUDA builds — the only reliable discriminator
is torch.version.hip (set on ROCm, None on CUDA). Two places assumed
device.type == "cuda" meant NVIDIA/FP8-capable hardware:
- services_utils.device_supports_fp8() would have attempted FP8
quantization on ROCm, which has no fp8_cast kernel support there.
- runtime_policy.decide_local_generation_mode()'s >=31 GB ->
full_models_loading floor assumes the FP8-halved (~23 GB) transformer
footprint; without FP8 the full ~42-46 GB bf16 transformer would try
to stay resident instead and OOM at that floor.
Add runtime_config/accelerator.py with a torch.version.hip-based
accelerator_backend() helper ("rocm" | "cuda" | "mps" | "cpu"), and an
fp8_capable parameter (default True, so CUDA/MPS behavior is unchanged)
that forces streaming_models_loading when FP8 isn't available.
Found and fixed while bringing up a community ROCm port; the OOM was
reproduced live on an AMD Radeon AI PRO R9700. Full test suite and
pyright pass unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ROCm PyTorch reports
torch.cuda.is_available()anddevice.type == "cuda"identically to real NVIDIA CUDA builds — the only reliable discriminator istorch.version.hip(set on ROCm,Noneon CUDA). Two places in the backend assumeddevice.type == "cuda"meant NVIDIA/FP8-capable hardware:services_utils.device_supports_fp8()would attempt FP8 quantization on ROCm, which has nofp8_castkernel support there.runtime_policy.decide_local_generation_mode()'s>=31 GB -> full_models_loadingfloor assumes the FP8-halved (~23 GB) transformer footprint. Without FP8, the full ~42-46 GB bf16 transformer tries to stay resident instead and OOMs at that floor — reproduced live: a HIP OOM crash on an AMD Radeon AI PRO R9700 (31.86 GB reported VRAM), fixed, and re-verified with a successful generation afterward.Adds
runtime_config/accelerator.pywith atorch.version.hip-basedaccelerator_backend()helper ("rocm" | "cuda" | "mps" | "cpu"), and anfp8_capableparameter (defaultTrue, so existing CUDA/MPS behavior is unchanged) that forcesstreaming_models_loadingwhen FP8 isn't available.Found while bringing up a community ROCm port (
boxwrench/LTX-Desktop-ROCm); this PR is scoped to just the two backend-agnostic correctness fixes, none of the ROCm-specific packaging/dependency-pinning work.Test plan
927 passed, 1 skipped(unchanged from base)pyrightstrict on the four touched files:0 errors, 0 warningsruntime_policyOOM live on an R9700, applied the fix, re-ran the same generation successfullyfp8_capabledefaulting toTruecorrectly preserves all existing CUDA/MPS call sites (no other caller currently passes it)🤖 Generated with Claude Code