diff --git a/README.md b/README.md index 6003210..778af9e 100644 --- a/README.md +++ b/README.md @@ -22,9 +22,64 @@ options nvidia NVreg_RegistryDwords="RMDisableFeatureDisablement=1;EnableResizab `EnableResizableBar=1` resizes BAR1 from 256 MB to the full VRAM — BAR1 is the P2P data aperture. -Together, those two flags are sufficient. Full reverse‑engineering write-up: +Together, those two flags are sufficient on the tested RTX 5090 / GB202 system. +They are not sufficient on the tested RTX 5060 Ti / GB206 system. Full +reverse‑engineering write-up: [`docs/RMDISABLEFEATUREDISABLEMENT.md`](docs/RMDISABLEFEATUREDISABLEMENT.md). +## RTX 5060 Ti / GB206 + +Driver 610.43.03 contains explicit GB206 BAR1 P2P support, but physical RM/GSP +reports its default PCIe read/write capability as `GPU_NOT_SUPPORTED`. Forcing +only the capability bits can leave the driver on the legacy GM200 mailbox path, +which fails with an invalid or unaligned remote write-mailbox address. + +The experimental GB206 configuration must also select BAR1 explicitly: +The following parameters were tested together on NVIDIA driver 610.43.03: + +```conf +options nvidia NVreg_RegistryDwords="RMDisableFeatureDisablement=1;EnableResizableBar=1;ForceP2P=0x11;RMPcieP2PType=1;RMForceP2PType=1;RMForceStaticBar1=1" +``` + +- `ForceP2P=0x11` overrides the cached GB206 GSP read/write denial. +- `RMPcieP2PType=1` selects `P2P_CONNECTIVITY_PCIE_BAR1`, causing the existing + GH100/Blackwell dispatcher to use `kbusCreateP2PMappingForBar1P2P_GH100` + instead of `kbusCreateP2PMappingForMailbox_GM200`. +- `RMForceP2PType=1` pins connection selection to PCIe P2P. +- `RMForceStaticBar1=1` forces the internal static BAR1 mapping required by + BAR1 is the GPU PCIe Base Address Register 1 aperture: a host-visible virtual + address window into GPU VRAM. P2P BAR1 mapping lets one GPU access the other + GPU’s framebuffer directly over PCIe. Without this selection, the driver can + fall back to a mailbox path, where small control messages are exchanged + through a remote write mailbox instead of mapping the peer framebuffer. + GB206 BAR1 P2P. This can exhaust BAR1; keep the rollback configuration. + +The repository keeps the default RTX 5090 profile in `modprobe.d/nvidia-p2p.conf` +and the RTX 5060 profile in `modprobe.d/nvidia-p2p-rtx5060.conf`. The enable +script installs only the selected profile as `/etc/modprobe.d/nvidia-p2p.conf`. +The tested RTX 5060 pair must be headless: disconnect display cables and do not +attach either card to a monitor. If either card is driving a display, the two +GPUs are not mapped as peers and the CUDA tests will not validate P2P transfers. +Note also that this is very experimental and has been tested only on the 610.43.03 +Nvidia open source driver. + +Install, reboot, and verify with: + +```bash +sudo ./scripts/enable-p2p.sh --rtx5060 +sudo reboot +./scripts/verify-p2p.sh +``` + +Build the functional tests from NVIDIA’s [cuda-samples](https://github.com/NVIDIA/cuda-samples.git) project. +The verification uses `simpleIPC` and `simpleP2P`: `simpleIPC` checks CUDA inter-process communication +and `simpleP2P` performs peer-access/data validation. Set `CUDA_SAMPLES_BUILD` if the samples are built elsewhere. + +See [`RTX5060.md`](RTX5060.md) for the investigation, success criteria, and +rollback procedure. An `OK` result from `nvidia-smi topo -p2p` advertises a +capability; a successful `simpleP2P` data-validation run is still required to +prove that peer mapping and transfers work. + ## Quick start ```bash @@ -44,6 +99,7 @@ sudo ./scripts/disable-p2p.sh - NVIDIA proprietary or open-kernel driver, 590.x or later (tested on 590.48.01) - Consumer Blackwell GPU (tested on RTX 5090 / GB202) +- For RTX 5060 Ti / GB206: cards must be headless and disconnected from displays; otherwise peer mappings are not established - Re-Size BAR enabled in motherboard UEFI / BIOS - On AMD systems, IOMMU in passthrough mode: add `amd_iommu=on iommu=pt` to the GRUB kernel command line - Multiple GPUs on the same host for P2P to be meaningful @@ -51,7 +107,7 @@ sudo ./scripts/disable-p2p.sh ## Repository layout ``` -modprobe.d/ NVIDIA modprobe drop-in (the two registry keys) +modprobe.d/ Default RTX 5090 and separate RTX 5060 modprobe profiles scripts/ enable / disable / verify helpers, cupy peer-bandwidth probe docs/ GSP firmware reverse-engineering notes ``` @@ -62,18 +118,24 @@ Ghidra + Capstone RISC-V reverse engineering of the GSP firmware (`gsp_elf_9.bin See [`docs/RMDISABLEFEATUREDISABLEMENT.md`](docs/RMDISABLEFEATUREDISABLEMENT.md), [`docs/IS_GEFORCE.md`](docs/IS_GEFORCE.md), and [`docs/REGISTRY_KEYS.md`](docs/REGISTRY_KEYS.md). -## Related keys (not required, but relevant) +## Related registry keys | Key | Effect | |---|---| | `RMForceP2PType=1` | Force PCIe P2P path (skip NVLink/C2C detection) | -| `RMPcieP2PType=1` | Force BAR1 mailbox P2P mode | -| `ForceP2P=0x11` | Force-set the P2P read/write enable bits | +| `RMPcieP2PType=1` | Select the static-BAR1 PCIe P2P path | +| `ForceP2P=0x11` | Force P2P capability recognition (read/write bits); does not select the mapping path | +| `RMForceStaticBar1=1` | Avoid mailbox fallback and force static BAR1 peer mapping (address-space exhaustion risk) | | `NvLinkPeerThroughL2=1` | Route peer traffic through local L2 (undocumented, untested) | | `RmEnableLocalizedMemory=1` | NUMA-aware peer memory (undocumented, untested) | | `RmCeExposeAllLce=1` | Expose all copy engines to user space | -The first three are redundant once `RMDisableFeatureDisablement=1` + `EnableResizableBar=1` are set; they just skip detection paths that already succeed. The last three are speculative and documented in [`docs/REGISTRY_KEYS.md`](docs/REGISTRY_KEYS.md) for reference. +The first three force-path keys are redundant on the tested RTX 5090 / GB202 +configuration. They and `RMForceStaticBar1=1` are required by the experimental +RTX 5060 Ti / GB206 path described above: `ForceP2P` overrides the cached +capability denial, `RMPcieP2PType` prevents mailbox fallback, and static BAR1 +provides the peer aperture. The final three keys are speculative and documented in [`docs/REGISTRY_KEYS.md`](docs/REGISTRY_KEYS.md) +for reference. ## License diff --git a/docs/RTX5060.md b/docs/RTX5060.md new file mode 100644 index 0000000..b5dfa10 --- /dev/null +++ b/docs/RTX5060.md @@ -0,0 +1,136 @@ +# RTX 5060 Ti / GB206 P2P investigation + +## Conclusion + +NVIDIA driver 610.43.03 contains an explicit BAR1 P2P implementation for +GB206, the GPU used by the RTX 5060 Ti. However, the two-key configuration +documented for the RTX 5090 does not enable P2P on the tested GB206 system. +The GSP/physical RM reports the GPU-specific read and write capabilities as +`GPU_NOT_SUPPORTED`, which produces `GNS` in `nvidia-smi topo -p2p` and causes +`cuDeviceCanAccessPeer` to return false. + +The force keys are therefore not redundant for this configuration. The next +See the parameter descriptions below for the distinction between capability +recognition (`ForceP2P`) and selecting the BAR1 mapping path. +test should explicitly override the cached GSP capability result and select the +BAR1 path: + +```text +RMDisableFeatureDisablement=1 +EnableResizableBar=1 +ForceP2P=0x11 +RMPcieP2PType=1 +RMForceP2PType=1 +RMForceStaticBar1=1 +``` + +### Parameter roles + +- `RMDisableFeatureDisablement=1` bypasses the GeForce feature gate in GSP + firmware and permits the otherwise-disabled professional P2P initialization + path. +- `EnableResizableBar=1` expands BAR1 from the small default aperture to the + framebuffer-sized PCIe aperture needed for peer VRAM access. +- `ForceP2P=0x11` forces the P2P capability recognition result: the low read + and write capability bits are reported as enabled instead of the GB206 + `GPU_NOT_SUPPORTED` result. This makes the peer relationship recognizable; + it does not by itself choose how the peer memory is mapped. +- `RMPcieP2PType=1` selects the PCIe BAR1 P2P connectivity type rather than the + legacy PCIe mailbox type. +- `RMForceP2PType=1` pins the connection decision to PCIe P2P and prevents other + interconnect detection or fallback from changing the selected path. +- `RMForceStaticBar1=1` forces the static BAR1 peer mapping and avoids the + mailbox mapping path. BAR1 is the PCIe Base Address Register 1 aperture: a + host-visible address window through which a GPU can access framebuffer VRAM. + With static BAR1 mapping, one GPU can access the other GPU’s VRAM directly + over PCIe instead of sending remote write-mailbox control messages. The + mapping consumes BAR1 address space and may exhaust it, so rollback remains + important. + +The six-key combination was tested on NVIDIA driver 610.43.03 with two RTX +5060 Ti / GB206 cards. It remains experimental and is not a guarantee of +data integrity on other driver versions or GPU combinations. + +This remains experimental. The source shows that the forced path is reachable, +but it does not prove that a real peer mapping will complete without a GSP crash +or data corruption. + +## Tested system + +- Two NVIDIA GeForce RTX 5060 Ti cards, PCI device ID `10de:2d04` +- GB206 implementation +- NVIDIA open-kernel driver 610.43.03 +- Linux 6.8.0-136-generic +- AMD Ryzen 7 3800XT +- GPUs connected through the same PCIe host bridge (`PHB`) +- Kernel command line includes `amd_iommu=on iommu=pt` +- Re-Size BAR enabled in the BIOS as well as IOMMU + +With only the original two keys loaded: + +```text +RegistryDwords: "RMDisableFeatureDisablement=1;EnableResizableBar=1" +BAR1: 16 GiB on each GPU +nvidia-smi topo -p2p r/w: GNS in both directions +cuDeviceCanAccessPeer: 0 in both directions +``` + +This rules out an undersized BAR1 aperture. `GNS` also identifies the immediate +failure as a GPU capability decision rather than a topology decision. + +## Evidence in driver 610.43.03 + +The published RM source contains the complete GB206 path: + +1. PCI ID `0x2D04` is identified as `NVIDIA GeForce RTX 5060 Ti`. +2. GB206 sets `PDB_PROP_KBUS_SUPPORT_BAR1_P2P_BY_DEFAULT` to true. +3. GB206 dispatches BAR1 DMA information, mapping creation/removal, mapping + detection, and capability checks to the real GH100/Blackwell implementation, + not to unsupported stubs. +4. `kbusIsPcieBar1P2PMappingSupported_GH100` accepts non-loopback peers when + static BAR1 is enabled and no conflicting mailbox mapping exists. + +The rejection happens earlier in the capability pipeline: + +1. `_gpuInitPcieP2PCapability` calls physical RM/GSP through + `NV2080_CTRL_CMD_INTERNAL_GET_PCIE_P2P_CAPS`. +2. It caches the returned per-GPU read and write status. +3. `_kp2pCapsGetStatusOverPcie` converts that cached result into + `NV0000_P2P_CAPS_STATUS_GPU_NOT_SUPPORTED`/`GNS`. +4. `ForceP2P=0x11` is checked before the cached GSP result and forces both + statuses to `OK`. +5. `RMPcieP2PType=1` is required to select BAR1 instead of the default mailbox + PCIe path. `RMForceP2PType=1` pins selection to PCIe P2P. +6. `RMForceStaticBar1=1` forces the internal static mapping required by the + GB206 BAR1 path. It may exhaust BAR1, so rollback access is mandatory. + +## Controlled test + +Use the accompanying script from this repository: + +```bash +# Install the six-key test configuration and rebuild initramfs. +sudo ./scripts/enable-p2p.sh --rtx5060 +sudo reboot + +# After reboot, run as the normal user. +./scripts/verify-p2p.sh + +# Restore the previous configuration if the test fails. +sudo ./scripts/disable-p2p.sh +sudo reboot +``` + +To reboot automatically after install or rollback, append `--reboot`. + +Success requires all of the following: + +- the six keys appear in `/proc/driver/nvidia/params`; +- BAR1 remains the full framebuffer size on both GPUs; +- `nvidia-smi topo -p2p r` and `w` report `OK` off-diagonal; +- CUDA `simpleP2P` completes its data-validation test; +- `p2pBandwidthLatencyTest` detects peer access and completes; +- no NVRM Xid or GSP recovery appears in the kernel log. + +Atomics may remain unsupported and are not part of the success criteria. +Note for Vllm usage: DISABLE_CUSTOM_ALL_REDUCE="1" diff --git a/modprobe.d/nvidia-p2p-rtx5060.conf b/modprobe.d/nvidia-p2p-rtx5060.conf new file mode 100644 index 0000000..90520d7 --- /dev/null +++ b/modprobe.d/nvidia-p2p-rtx5060.conf @@ -0,0 +1,19 @@ +# rtx-p2p RTX 5060 Ti / GB206: enable PCIe peer-to-peer on consumer GeForce (Blackwell) +# +# RMDisableFeatureDisablement=1 -- bypasses the GSP-side GeForce SKU gate +# and nine firmware ebreak crash guards +# (undocumented, found in firmware only) +# EnableResizableBar=1 -- resizes BAR1 from 256 MB to full VRAM, +# required as the P2P data aperture +# +# See docs/RMDISABLEFEATUREDISABLEMENT.md for the full RE write-up. + +# RTX 5060 / GB206 parameter roles (tested with NVIDIA driver 610.43.03): +# ForceP2P=0x11 force recognition of P2P read/write capability. +# RMPcieP2PType=1 select PCIe BAR1 P2P instead of mailbox connectivity. +# RMForceP2PType=1 force the PCIe P2P connection type. +# RMForceStaticBar1=1 force static BAR1 mapping and avoid mailbox fallback. +# BAR1 is the PCIe Base Address Register 1 VRAM aperture used for direct peer +# framebuffer access; static mapping consumes BAR1 address space. + +options nvidia NVreg_RegistryDwords="RMDisableFeatureDisablement=1;EnableResizableBar=1;ForceP2P=0x11;RMPcieP2PType=1;RMForceP2PType=1;RMForceStaticBar1=1" diff --git a/scripts/enable-p2p.sh b/scripts/enable-p2p.sh index de01461..66a7586 100755 --- a/scripts/enable-p2p.sh +++ b/scripts/enable-p2p.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# enable-p2p.sh — install modprobe drop-in and reload the NVIDIA driver. +# enable-p2p.sh — install a P2P modprobe profile and reload the NVIDIA driver. # # WARNING: this kills every process holding /dev/nvidia*. Do not run while # workloads you care about are using the GPU. @@ -11,17 +11,32 @@ if [[ $EUID -ne 0 ]]; then exit 1 fi +PROFILE="rtx5090" +if [[ ${1:-} == "--rtx5060" ]]; then + PROFILE="rtx5060" +elif [[ ${1:-} == "--help" || ${1:-} == "-h" ]]; then + echo "Usage: sudo ./scripts/enable-p2p.sh [--rtx5060]" + exit 0 +elif (( $# > 0 )); then + echo "error: unknown option: $1" >&2 + exit 2 +fi + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" - -SRC="${REPO_ROOT}/modprobe.d/nvidia-p2p.conf" DST="/etc/modprobe.d/nvidia-p2p.conf" - +if [[ ${PROFILE} == "rtx5060" ]]; then + SRC="${REPO_ROOT}/modprobe.d/nvidia-p2p-rtx5060.conf" +else + SRC="${REPO_ROOT}/modprobe.d/nvidia-p2p.conf" +fi if [[ ! -f "${SRC}" ]]; then echo "error: ${SRC} not found" >&2 exit 1 fi + + echo "[1/4] installing ${DST}" install -m 0644 "${SRC}" "${DST}" cat "${DST}"