PF passthrough: PCI config space returns 0xFF after VFIO reset on MI300X/MI355X

[johnahull]

Summary

When an AMD Instinct GPU PF (Physical Function) is bound to vfio-pci and a VFIO reset is triggered (FLR or bus reset), the PCI config space becomes permanently corrupted: all reads return 0xFF. The device cannot be recovered via PCI remove+rescan; only a full host reboot restores it.

This prevents PF passthrough to VMs on MI300X and MI355X. GIM SR-IOV VF passthrough is unaffected.

Environment

• GPUs: AMD Instinct MI355X (device ID 0x75a3), AMD Instinct MI300X (device ID 0x740f)
• Host OS: Fedora 44, kernel 7.0.12-201.fc44.x86_64
• IOMMU: Enabled (AMD-Vi)
• GIM version: 9.1.0.K (also reproduced on earlier versions)
• QEMU: 10.0
• Hypervisor: KVM via KubeVirt / libvirt

Steps to Reproduce

1. Boot with GPUs on the amdgpu driver (no GIM loaded)
2. Bind a PF to vfio-pci:

   echo "0000:0c:00.0" > /sys/bus/pci/drivers/amdgpu/unbind
   echo "vfio-pci" > /sys/bus/pci/devices/0000:0c:00.0/driver_override
   echo "0000:0c:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

3. Start a QEMU/KubeVirt VM with the PF as a VFIO passthrough device
4. Stop the VM (triggers VFIO device reset / FLR)
5. Unbind from vfio-pci, attempt to rebind to amdgpu

Expected Behavior

PF rebinds to amdgpu and is usable for compute or another VM session.

Actual Behavior

After the VM shuts down and VFIO triggers a device reset:

• lspci -s 0000:0c:00.0 -xxx shows all config space bytes as 0xFF
• Vendor/device ID reads as 0xFFFF:0xFFFF
• Rebinding to amdgpu fails (driver can't identify the device)
• echo 1 > /sys/bus/pci/devices/0000:0c:00.0/remove + echo 1 > /sys/bus/pci/rescan does not recover the device
• Only a full host reboot restores the PCI config space

Workaround

Disabling the PCI bus reset method before binding to vfio-pci prevents the corruption:

echo "" > /sys/bus/pci/devices/0000:0c:00.0/reset_method

With this workaround, the full PF passthrough lifecycle works correctly:

• PF binds to vfio-pci
• VM boots with GPU passthrough
• VM shutdown completes without config space corruption
• PF rebinds to amdgpu successfully

This workaround prevents QEMU from triggering a PCI bus reset on the device. However, it means the GPU state is not fully cleaned between VM sessions.

Relationship to VF FLR Issue

The GIM 9.1.0.K release notes mention a related issue:

"In configurations with 8 VFs per GPU, VF FLR may intermittently fail with 'SMU FW not responding' or 'SMU Timeout' errors."

This PF issue may share the same root cause: the SMU firmware not handling FLR correctly on CDNA3/CDNA4 ASICs. The PF case is 100% reproducible (not intermittent like the VF case).

Context

This issue was discovered while implementing on-demand VFIO passthrough support for the AMD GPU DRA driver (ROCm/k8s-gpu-dra-driver PR #50). The DRA driver dynamically binds GPU PFs from amdgpu to vfio-pci during Kubernetes resource preparation and unbinds on release. The bind/unbind cycle works correctly with the reset_method workaround but fails without it due to this config space corruption.

[johnahull]

Additional Workaround: Kernel Module

In addition to the per-device sysfs workaround, a kernel module can disable bus reset for all affected devices at load time by setting PCI_DEV_FLAGS_NO_BUS_RESET:

#include <linux/module.h>
#include <linux/pci.h>

#define AMD_VENDOR_ID 0x1002

static void disable_reset(unsigned int device_id) {
    struct pci_dev *dev = NULL;
    while ((dev = pci_get_device(AMD_VENDOR_ID, device_id, dev)) != NULL) {
        dev->dev_flags |= PCI_DEV_FLAGS_NO_BUS_RESET;
        pr_info("no_bus_reset_amd: disabled bus reset for %s (0x%04x)\n",
                pci_name(dev), device_id);
    }
}

static void enable_reset(unsigned int device_id) {
    struct pci_dev *dev = NULL;
    while ((dev = pci_get_device(AMD_VENDOR_ID, device_id, dev)) != NULL) {
        dev->dev_flags &= ~PCI_DEV_FLAGS_NO_BUS_RESET;
        pr_info("no_bus_reset_amd: re-enabled bus reset for %s\n", pci_name(dev));
    }
}

static int __init no_bus_reset_init(void) {
    disable_reset(0x75a3);  /* MI355X PF */
    disable_reset(0x75b3);  /* MI355X VF */
    disable_reset(0x740f);  /* MI300X */
    return 0;
}

static void __exit no_bus_reset_exit(void) {
    enable_reset(0x75a3);
    enable_reset(0x75b3);
    enable_reset(0x740f);
}

module_init(no_bus_reset_init);
module_exit(no_bus_reset_exit);
MODULE_LICENSE("GPL");

This is equivalent to echo "" > /sys/bus/pci/devices/<bdf>/reset_method for each device, but applies automatically at module load to all matching PCI device IDs.

A proper fix would be a PCI quirk in the kernel's drivers/pci/quirks.c that sets PCI_DEV_FLAGS_NO_BUS_RESET for these device IDs, similar to existing quirks for other devices with broken FLR (e.g., quirk_no_bus_reset entries for various Intel, Mellanox, and AMD devices).