Backport DirectNIC patches to nvidia_unstable-11.0 - #36
Open
tdavenvidia wants to merge 6 commits into
Open
Conversation
Nvidia’s next generation GB200 platform has Blackwell GPU and CX8 directly
connected through PCIe Gen6 x16 link. Direct P2P PCIe traffic between GPU
and NIC is possible however it requires ATS at its core and Grace CPU does
not support PCIe ATS. GPA=HPA solution removes the need for GPA to HPA
address translation by configuring PCIe BARs in the VM with HPA. It also
enables ACPI PCI DSM by setting ‘preserve_config’ to true to avoid VM from
reconfiguring the PCI BARs during boot.
Here is the example of PCIe topology that shows GPU and CX8 behind the PCIe Switch:
$ lspci -vt
-[0000:00]---00.0-[01-07]----00.0-[02-07]--+-00.0-[03]--+-00.0 Mellanox Technologies CX8 Family [ConnectX-8]
| \-00.1 Mellanox Technologies CX8 Family [ConnectX-8]
\-03.0-[04-07]----00.0-[05-07]--+-08.0-[06]--
\-0c.0-[07]--
-[0002:00]---00.0-[01-07]----00.0-[02-07]--+-00.0-[03]--+-00.0 Mellanox Technologies CX8 Family [ConnectX-8]
| \-00.1 Mellanox Technologies CX8 Family [ConnectX-8]
\-01.0-[04-07]----00.0-[05-07]--+-08.0-[06]--
\-0c.0-[07]--
-[0005:00]---00.0-[01-0a]----00.0-[02-0a]--+-01.0-[03]--
+-02.0-[04]--
+-03.0-[05]--
+-04.0-[06-07]----00.0-[07]----00.0 ASPEED Technology, Inc. ASPEED Graphics Family
+-05.0-[08]----00.0 Renesas Technology Corp. uPD720201 USB 3.0 Host Controller
+-06.0-[09]----00.0 Intel Corporation I210 Gigabit Network Connection
\-07.0-[0a]--
-[0006:00]---00.0-[01-09]----00.0-[02-09]--+-00.0-[03]--+-00.0 Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller
| +-00.1 Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller
| \-00.2 Mellanox Technologies MT43244 BlueField-3 SoC Management Interface
\-02.0-[04-09]----00.0-[05-09]--+-00.0-[06]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
+-04.0-[07]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
+-08.0-[08]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
\-0c.0-[09]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
-[0008:00]---00.0-[01-06]----00.0-[02-06]--+-00.0-[03]----00.0 Mellanox Technologies Device 2100
\-03.0-[04-06]----00.0-[05-06]----00.0-[06]----00.0 NVIDIA Corporation Device 2941
-[0009:00]---00.0-[01-06]----00.0-[02-06]--+-00.0-[03]----00.0 Mellanox Technologies Device 2100
\-01.0-[04-06]----00.0-[05-06]----00.0-[06]----00.0 NVIDIA Corporation Device 2941
-[0010:00]---00.0-[01-07]----00.0-[02-07]--+-00.0-[03]--+-00.0 Mellanox Technologies CX8 Family [ConnectX-8]
| \-00.1 Mellanox Technologies CX8 Family [ConnectX-8]
\-03.0-[04-07]----00.0-[05-07]--+-08.0-[06]--
\-0c.0-[07]--
-[0012:00]---00.0-[01-07]----00.0-[02-07]--+-00.0-[03]--+-00.0 Mellanox Technologies CX8 Family [ConnectX-8]
| \-00.1 Mellanox Technologies CX8 Family [ConnectX-8]
\-01.0-[04-07]----00.0-[05-07]--+-08.0-[06]--
\-0c.0-[07]--
-[0015:00]---00.0-[01]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
-[0016:00]---00.0-[01-09]----00.0-[02-09]--+-00.0-[03]--+-00.0 Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller
| +-00.1 Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller
| \-00.2 Mellanox Technologies MT43244 BlueField-3 SoC Management Interface
\-02.0-[04-09]----00.0-[05-09]--+-00.0-[06]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
+-04.0-[07]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
+-08.0-[08]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
\-0c.0-[09]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
-[0018:00]---00.0-[01-06]----00.0-[02-06]--+-00.0-[03]----00.0 Mellanox Technologies Device 2100
\-03.0-[04-06]----00.0-[05-06]----00.0-[06]----00.0 NVIDIA Corporation Device 2941
-[0019:00]---00.0-[01-06]----00.0-[02-06]--+-00.0-[03]----00.0 Mellanox Technologies Device 2100
\-01.0-[04-06]----00.0-[05-06]----00.0-[06]----00.0 NVIDIA Corporation Device 2941
GPA=HPA is expected to work with PCIe topology in the VM that resembles to
baremetal. In other words, for P2P PCIe traffic (using GPA=HPA) over Gen6,
CX8 NIC(the DMA-PF) and GPU assigned to VM should be under the same PCIe switch.
Note: PCIe Switch needs special non-conventional ACS configuration such that
minimal P2P routes needed for GPU Direct RDMA should be allowed.
Signed-off-by: Tushar Dave <tdave@nvidia.com>
(backported from commit 31a853d https://github.com/NVIDIA/QEMU/commits/nvidia_stable-10.1)
[tdave: context adjustment in hw/arm/virt-acpi-build.c build_dsdt() due to
VFIO_PCI_BASE renamed to VFIO_PCI_DEVICE and build_dsdt signature change;
context adjustment in include/hw/arm/virt.h due to new fields added in 11.0]
Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
Signed-off-by: Tushar Dave <tdave@nvidia.com>
Signed-off-by: Matthew R. Ochs <mochs@nvidia.com>
Acked-by: Matthew R. Ochs <mochs@nvidia.com>
Acked-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Signed-off-by: Matthew R. Ochs <mochs@nvidia.com>
Grace Blackwell GPU PCIe BAR1 is real BAR exposed to VM that can be
used for GPUdirect RDMA [1].
This patch assigns HPA to BAR1 in the VM for the reason mentioned in
the commit 54db2e4a632 ("hw/arm: GB200 DirectNIC GPA=HPA").
This patch also assigns appropriate GPA to GPU BAR2 (exposed to VM with
the same size as BAR 1 that emulates C2C cache coherent address space)
to avoid region conflict in PCI bus resource assignment.
[1]: https://lore.kernel.org/lkml/20241006102722.3991-1-ankita@nvidia.com/
Signed-off-by: Tushar Dave <tdave@nvidia.com>
(cherry picked from commit c332310 https://github.com/NVIDIA/QEMU/commits/nvidia_stable-10.1)
Signed-off-by: Tushar Dave <tdave@nvidia.com>
Signed-off-by: Matthew R. Ochs <mochs@nvidia.com>
Acked-by: Matthew R. Ochs <mochs@nvidia.com>
Acked-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Signed-off-by: Matthew R. Ochs <mochs@nvidia.com>
Simialr to GB200, GB300 also requires workaround to make GPU BAR 1 GPA=HPA. Signed-off-by: Tushar Dave <tdave@nvidia.com> Acked-by: Matthew R. Ochs <mochs@nvidia.com> Acked-by: Mitchell Augustin <mitchell.augustin@canonical.com> Signed-off-by: Matthew R. Ochs <mochs@nvidia.com>
…e ports When PASID capable device is added behind the PCIe downstream port, for example Nvidia GPU, the PCIe downstream ports must expose ACS capability otherwise PASID won't get enabled. In addition, the other usecase is GPUDirect RDMA using Data Direct that must require special ACS controls at the PCIe downstream ports for P2P communication. Signed-off-by: Tushar Dave <tdave@nvidia.com> (cherry picked from commit 69ff099 https://github.com/NVIDIA/QEMU/commits/nvidia_stable-10.1) Signed-off-by: Tushar Dave <tdave@nvidia.com> Acked-by: Matthew R. Ochs <mochs@nvidia.com> Acked-by: Mitchell Augustin <mitchell.augustin@canonical.com> Signed-off-by: Matthew R. Ochs <mochs@nvidia.com>
To support P2P on Guest we must expose to the guest OS the actual PCIe topology and configuration as set by the HYP. Otherwise, the behavior is considered as un-defined. It might fail by SW or HW. Extend both root port and downstream port to get acs caps that should match the HYP and use them in the guest. Signed-off-by: Tushar Dave <tdave@nvidia.com> (backported from commit 78accd6 https://github.com/NVIDIA/QEMU/commits/nvidia_stable-10.1) [tdave: context adjustment in hw/pci-bridge/pcie_root_port.c rp_realize() and rp_props[] due to PCIESlot::disable_acs removed in 11.0; context adjustment in include/hw/pci/pcie.h due to pcie_insert_capability() added in 11.0] Signed-off-by: Yishai Hadas <yishaih@nvidia.com> Signed-off-by: Tushar Dave <tdave@nvidia.com> Acked-by: Matthew R. Ochs <mochs@nvidia.com> Acked-by: Mitchell Augustin <mitchell.augustin@canonical.com> Signed-off-by: Matthew R. Ochs <mochs@nvidia.com>
GPUDirect RDMA uisng data-direct requires a specific ACS configuration on PCIe Root Ports and Downstream Ports. While ACS can be configured via QEMU's 'acs-caps' property, the guest kernel may overwrite ACS during standard programming. This change blocks all guest writes to the PCIe ACS Control register and preserves QEMU-provided ACS settings across device resets on PCIe Root Ports and Downstream Ports. Signed-off-by: Tushar Dave <tdave@nvidia.com> (cherry picked from commit b1bcf26 https://github.com/NVIDIA/QEMU/commits/nvidia_stable-10.1) Signed-off-by: Tushar Dave <tdave@nvidia.com> Acked-by: Matthew R. Ochs <mochs@nvidia.com> Acked-by: Mitchell Augustin <mitchell.augustin@canonical.com> Signed-off-by: Matthew R. Ochs <mochs@nvidia.com>
BaseOS Kernel ReviewSummaryARM virt can fail to link without VFIO PCI, while ACPI ROM relocation can loop forever if MMIO32 has no fitting gap. Unversioned ACS addition also breaks incoming migration for xio3130 downstream ports. Findings: Critical: 0, High: 3, Medium: 7, Low: 12 Latest watcher review: open review Head: This comment is maintained by BaseOS Reviewer. It is updated when the GitHub watcher publishes a newer review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains GPA=HPA and ACS changes required for DirectNIC GB300 w/CX8.