Skip to content

Add specific flags for RTX 5060 and document the changes #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
laurentdavid wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add specific flags for RTX 5060 and document the changes #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 68 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,64 @@ options nvidia NVreg_RegistryDwords="RMDisableFeatureDisablement=1;EnableResizab

`EnableResizableBar=1` resizes BAR1 from 256 MB to the full VRAM — BAR1 is the P2P data aperture.

Together, those two flags are sufficient. Full reverse‑engineering write-up:
Together, those two flags are sufficient on the tested RTX 5090 / GB202 system.
They are not sufficient on the tested RTX 5060 Ti / GB206 system. Full
reverse‑engineering write-up:
[`docs/RMDISABLEFEATUREDISABLEMENT.md`](docs/RMDISABLEFEATUREDISABLEMENT.md).

## RTX 5060 Ti / GB206

Driver 610.43.03 contains explicit GB206 BAR1 P2P support, but physical RM/GSP
reports its default PCIe read/write capability as `GPU_NOT_SUPPORTED`. Forcing
only the capability bits can leave the driver on the legacy GM200 mailbox path,
which fails with an invalid or unaligned remote write-mailbox address.

The experimental GB206 configuration must also select BAR1 explicitly:
The following parameters were tested together on NVIDIA driver 610.43.03:

```conf
options nvidia NVreg_RegistryDwords="RMDisableFeatureDisablement=1;EnableResizableBar=1;ForceP2P=0x11;RMPcieP2PType=1;RMForceP2PType=1;RMForceStaticBar1=1"
```

- `ForceP2P=0x11` overrides the cached GB206 GSP read/write denial.
- `RMPcieP2PType=1` selects `P2P_CONNECTIVITY_PCIE_BAR1`, causing the existing
GH100/Blackwell dispatcher to use `kbusCreateP2PMappingForBar1P2P_GH100`
instead of `kbusCreateP2PMappingForMailbox_GM200`.
- `RMForceP2PType=1` pins connection selection to PCIe P2P.
- `RMForceStaticBar1=1` forces the internal static BAR1 mapping required by
BAR1 is the GPU PCIe Base Address Register 1 aperture: a host-visible virtual
address window into GPU VRAM. P2P BAR1 mapping lets one GPU access the other
GPU’s framebuffer directly over PCIe. Without this selection, the driver can
fall back to a mailbox path, where small control messages are exchanged
through a remote write mailbox instead of mapping the peer framebuffer.
GB206 BAR1 P2P. This can exhaust BAR1; keep the rollback configuration.

The repository keeps the default RTX 5090 profile in `modprobe.d/nvidia-p2p.conf`
and the RTX 5060 profile in `modprobe.d/nvidia-p2p-rtx5060.conf`. The enable
script installs only the selected profile as `/etc/modprobe.d/nvidia-p2p.conf`.
The tested RTX 5060 pair must be headless: disconnect display cables and do not
attach either card to a monitor. If either card is driving a display, the two
GPUs are not mapped as peers and the CUDA tests will not validate P2P transfers.
Note also that this is very experimental and has been tested only on the 610.43.03
Nvidia open source driver.

Install, reboot, and verify with:

```bash
sudo ./scripts/enable-p2p.sh --rtx5060
sudo reboot
./scripts/verify-p2p.sh
```

Build the functional tests from NVIDIA’s [cuda-samples](https://github.com/NVIDIA/cuda-samples.git) project.
The verification uses `simpleIPC` and `simpleP2P`: `simpleIPC` checks CUDA inter-process communication
and `simpleP2P` performs peer-access/data validation. Set `CUDA_SAMPLES_BUILD` if the samples are built elsewhere.

See [`RTX5060.md`](RTX5060.md) for the investigation, success criteria, and
rollback procedure. An `OK` result from `nvidia-smi topo -p2p` advertises a
capability; a successful `simpleP2P` data-validation run is still required to
prove that peer mapping and transfers work.

## Quick start

```bash
Expand All @@ -44,14 +99,15 @@ sudo ./scripts/disable-p2p.sh

- NVIDIA proprietary or open-kernel driver, 590.x or later (tested on 590.48.01)
- Consumer Blackwell GPU (tested on RTX 5090 / GB202)
- For RTX 5060 Ti / GB206: cards must be headless and disconnected from displays; otherwise peer mappings are not established
- Re-Size BAR enabled in motherboard UEFI / BIOS
- On AMD systems, IOMMU in passthrough mode: add `amd_iommu=on iommu=pt` to the GRUB kernel command line
- Multiple GPUs on the same host for P2P to be meaningful

## Repository layout

```
modprobe.d/ NVIDIA modprobe drop-in (the two registry keys)
modprobe.d/ Default RTX 5090 and separate RTX 5060 modprobe profiles
scripts/ enable / disable / verify helpers, cupy peer-bandwidth probe
docs/ GSP firmware reverse-engineering notes
```
Expand All @@ -62,18 +118,24 @@ Ghidra + Capstone RISC-V reverse engineering of the GSP firmware (`gsp_elf_9.bin

See [`docs/RMDISABLEFEATUREDISABLEMENT.md`](docs/RMDISABLEFEATUREDISABLEMENT.md), [`docs/IS_GEFORCE.md`](docs/IS_GEFORCE.md), and [`docs/REGISTRY_KEYS.md`](docs/REGISTRY_KEYS.md).

## Related keys (not required, but relevant)
## Related registry keys

| Key | Effect |
|---|---|
| `RMForceP2PType=1` | Force PCIe P2P path (skip NVLink/C2C detection) |
| `RMPcieP2PType=1` | Force BAR1 mailbox P2P mode |
| `ForceP2P=0x11` | Force-set the P2P read/write enable bits |
| `RMPcieP2PType=1` | Select the static-BAR1 PCIe P2P path |
| `ForceP2P=0x11` | Force P2P capability recognition (read/write bits); does not select the mapping path |
| `RMForceStaticBar1=1` | Avoid mailbox fallback and force static BAR1 peer mapping (address-space exhaustion risk) |
| `NvLinkPeerThroughL2=1` | Route peer traffic through local L2 (undocumented, untested) |
| `RmEnableLocalizedMemory=1` | NUMA-aware peer memory (undocumented, untested) |
| `RmCeExposeAllLce=1` | Expose all copy engines to user space |

The first three are redundant once `RMDisableFeatureDisablement=1` + `EnableResizableBar=1` are set; they just skip detection paths that already succeed. The last three are speculative and documented in [`docs/REGISTRY_KEYS.md`](docs/REGISTRY_KEYS.md) for reference.
The first three force-path keys are redundant on the tested RTX 5090 / GB202
configuration. They and `RMForceStaticBar1=1` are required by the experimental
RTX 5060 Ti / GB206 path described above: `ForceP2P` overrides the cached
capability denial, `RMPcieP2PType` prevents mailbox fallback, and static BAR1
provides the peer aperture. The final three keys are speculative and documented in [`docs/REGISTRY_KEYS.md`](docs/REGISTRY_KEYS.md)
for reference.

## License

Expand Down
136 changes: 136 additions & 0 deletions docs/RTX5060.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# RTX 5060 Ti / GB206 P2P investigation

## Conclusion

NVIDIA driver 610.43.03 contains an explicit BAR1 P2P implementation for
GB206, the GPU used by the RTX 5060 Ti. However, the two-key configuration
documented for the RTX 5090 does not enable P2P on the tested GB206 system.
The GSP/physical RM reports the GPU-specific read and write capabilities as
`GPU_NOT_SUPPORTED`, which produces `GNS` in `nvidia-smi topo -p2p` and causes
`cuDeviceCanAccessPeer` to return false.

The force keys are therefore not redundant for this configuration. The next
See the parameter descriptions below for the distinction between capability
recognition (`ForceP2P`) and selecting the BAR1 mapping path.
test should explicitly override the cached GSP capability result and select the
BAR1 path:

```text
RMDisableFeatureDisablement=1
EnableResizableBar=1
ForceP2P=0x11
RMPcieP2PType=1
RMForceP2PType=1
RMForceStaticBar1=1
```

### Parameter roles

- `RMDisableFeatureDisablement=1` bypasses the GeForce feature gate in GSP
firmware and permits the otherwise-disabled professional P2P initialization
path.
- `EnableResizableBar=1` expands BAR1 from the small default aperture to the
framebuffer-sized PCIe aperture needed for peer VRAM access.
- `ForceP2P=0x11` forces the P2P capability recognition result: the low read
and write capability bits are reported as enabled instead of the GB206
`GPU_NOT_SUPPORTED` result. This makes the peer relationship recognizable;
it does not by itself choose how the peer memory is mapped.
- `RMPcieP2PType=1` selects the PCIe BAR1 P2P connectivity type rather than the
legacy PCIe mailbox type.
- `RMForceP2PType=1` pins the connection decision to PCIe P2P and prevents other
interconnect detection or fallback from changing the selected path.
- `RMForceStaticBar1=1` forces the static BAR1 peer mapping and avoids the
mailbox mapping path. BAR1 is the PCIe Base Address Register 1 aperture: a
host-visible address window through which a GPU can access framebuffer VRAM.
With static BAR1 mapping, one GPU can access the other GPU’s VRAM directly
over PCIe instead of sending remote write-mailbox control messages. The
mapping consumes BAR1 address space and may exhaust it, so rollback remains
important.

The six-key combination was tested on NVIDIA driver 610.43.03 with two RTX
5060 Ti / GB206 cards. It remains experimental and is not a guarantee of
data integrity on other driver versions or GPU combinations.

This remains experimental. The source shows that the forced path is reachable,
but it does not prove that a real peer mapping will complete without a GSP crash
or data corruption.

## Tested system

- Two NVIDIA GeForce RTX 5060 Ti cards, PCI device ID `10de:2d04`
- GB206 implementation
- NVIDIA open-kernel driver 610.43.03
- Linux 6.8.0-136-generic
- AMD Ryzen 7 3800XT
- GPUs connected through the same PCIe host bridge (`PHB`)
- Kernel command line includes `amd_iommu=on iommu=pt`
- Re-Size BAR enabled in the BIOS as well as IOMMU

With only the original two keys loaded:

```text
RegistryDwords: "RMDisableFeatureDisablement=1;EnableResizableBar=1"
BAR1: 16 GiB on each GPU
nvidia-smi topo -p2p r/w: GNS in both directions
cuDeviceCanAccessPeer: 0 in both directions
```

This rules out an undersized BAR1 aperture. `GNS` also identifies the immediate
failure as a GPU capability decision rather than a topology decision.

## Evidence in driver 610.43.03

The published RM source contains the complete GB206 path:

1. PCI ID `0x2D04` is identified as `NVIDIA GeForce RTX 5060 Ti`.
2. GB206 sets `PDB_PROP_KBUS_SUPPORT_BAR1_P2P_BY_DEFAULT` to true.
3. GB206 dispatches BAR1 DMA information, mapping creation/removal, mapping
detection, and capability checks to the real GH100/Blackwell implementation,
not to unsupported stubs.
4. `kbusIsPcieBar1P2PMappingSupported_GH100` accepts non-loopback peers when
static BAR1 is enabled and no conflicting mailbox mapping exists.

The rejection happens earlier in the capability pipeline:

1. `_gpuInitPcieP2PCapability` calls physical RM/GSP through
`NV2080_CTRL_CMD_INTERNAL_GET_PCIE_P2P_CAPS`.
2. It caches the returned per-GPU read and write status.
3. `_kp2pCapsGetStatusOverPcie` converts that cached result into
`NV0000_P2P_CAPS_STATUS_GPU_NOT_SUPPORTED`/`GNS`.
4. `ForceP2P=0x11` is checked before the cached GSP result and forces both
statuses to `OK`.
5. `RMPcieP2PType=1` is required to select BAR1 instead of the default mailbox
PCIe path. `RMForceP2PType=1` pins selection to PCIe P2P.
6. `RMForceStaticBar1=1` forces the internal static mapping required by the
GB206 BAR1 path. It may exhaust BAR1, so rollback access is mandatory.

## Controlled test

Use the accompanying script from this repository:

```bash
# Install the six-key test configuration and rebuild initramfs.
sudo ./scripts/enable-p2p.sh --rtx5060
sudo reboot

# After reboot, run as the normal user.
./scripts/verify-p2p.sh

# Restore the previous configuration if the test fails.
sudo ./scripts/disable-p2p.sh
sudo reboot
```

To reboot automatically after install or rollback, append `--reboot`.

Success requires all of the following:

- the six keys appear in `/proc/driver/nvidia/params`;
- BAR1 remains the full framebuffer size on both GPUs;
- `nvidia-smi topo -p2p r` and `w` report `OK` off-diagonal;
- CUDA `simpleP2P` completes its data-validation test;
- `p2pBandwidthLatencyTest` detects peer access and completes;
- no NVRM Xid or GSP recovery appears in the kernel log.

Atomics may remain unsupported and are not part of the success criteria.
Note for Vllm usage: DISABLE_CUSTOM_ALL_REDUCE="1"
19 changes: 19 additions & 0 deletions modprobe.d/nvidia-p2p-rtx5060.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# rtx-p2p RTX 5060 Ti / GB206: enable PCIe peer-to-peer on consumer GeForce (Blackwell)
#
# RMDisableFeatureDisablement=1 -- bypasses the GSP-side GeForce SKU gate
# and nine firmware ebreak crash guards
# (undocumented, found in firmware only)
# EnableResizableBar=1 -- resizes BAR1 from 256 MB to full VRAM,
# required as the P2P data aperture
#
# See docs/RMDISABLEFEATUREDISABLEMENT.md for the full RE write-up.

# RTX 5060 / GB206 parameter roles (tested with NVIDIA driver 610.43.03):
# ForceP2P=0x11 force recognition of P2P read/write capability.
# RMPcieP2PType=1 select PCIe BAR1 P2P instead of mailbox connectivity.
# RMForceP2PType=1 force the PCIe P2P connection type.
# RMForceStaticBar1=1 force static BAR1 mapping and avoid mailbox fallback.
# BAR1 is the PCIe Base Address Register 1 VRAM aperture used for direct peer
# framebuffer access; static mapping consumes BAR1 address space.

options nvidia NVreg_RegistryDwords="RMDisableFeatureDisablement=1;EnableResizableBar=1;ForceP2P=0x11;RMPcieP2PType=1;RMForceP2PType=1;RMForceStaticBar1=1"
23 changes: 19 additions & 4 deletions scripts/enable-p2p.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# enable-p2p.sh — install modprobe drop-in and reload the NVIDIA driver.
# enable-p2p.sh — install a P2P modprobe profile and reload the NVIDIA driver.
#
# WARNING: this kills every process holding /dev/nvidia*. Do not run while
# workloads you care about are using the GPU.
Expand All @@ -11,17 +11,32 @@ if [[ $EUID -ne 0 ]]; then
exit 1
fi

PROFILE="rtx5090"
if [[ ${1:-} == "--rtx5060" ]]; then
PROFILE="rtx5060"
elif [[ ${1:-} == "--help" || ${1:-} == "-h" ]]; then
echo "Usage: sudo ./scripts/enable-p2p.sh [--rtx5060]"
exit 0
elif (( $# > 0 )); then
echo "error: unknown option: $1" >&2
exit 2
fi

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"

SRC="${REPO_ROOT}/modprobe.d/nvidia-p2p.conf"
DST="/etc/modprobe.d/nvidia-p2p.conf"

if [[ ${PROFILE} == "rtx5060" ]]; then
SRC="${REPO_ROOT}/modprobe.d/nvidia-p2p-rtx5060.conf"
else
SRC="${REPO_ROOT}/modprobe.d/nvidia-p2p.conf"
fi
if [[ ! -f "${SRC}" ]]; then
echo "error: ${SRC} not found" >&2
exit 1
fi



echo "[1/4] installing ${DST}"
install -m 0644 "${SRC}" "${DST}"
cat "${DST}"
Expand Down