Backport: system/memory: Backport direct access support for RAM device regions

[nvmochs]

Summary

Backport the three-patch upstream QEMU series that makes RAM device regions directly accessible while preserving exact-width access semantics:

QEMU memory: Make RAM device regions directly accessible (https://lore.kernel.org/all/20260728031731.286666-1-gshan@redhat.com/)

This resolves a virtio-blk failure observed when guest I/O buffers are allocated from passed-through GPU coherent memory.

Problem

When NVIDIA GPU coherent memory is onlined in a guest, the corresponding VFIO BAR memory is represented by QEMU as a RAM device region. Current versions of QEMU treat RAM device regions as indirectly accessible.

When virtio-blk processes an I/O request whose buffer resides in GPU coherent memory, address_space_map() therefore allocates a bounce buffer instead of returning a direct mapping.

The bounce-buffer budget defaults to only 4096 bytes per PCI device. Once that budget is exhausted, address_space_map() returns NULL, and virtqueue_map_desc() reports:

virtio: bogus descriptor or out of resources

The virtio device is then marked broken and enters VIRTIO_CONFIG_S_NEEDS_RESET. Outstanding block completions may be lost, leaving guest processes blocked in uninterruptible I/O and potentially locking up the VM.

The issue was initially observed during DCGM diagnostics and highly parallel CUDA sample builds. It was later reduced to a deterministic direct-I/O reproducer.

Reproducer

On a VM with a passed-through GPU and a memory-bearing GPU NUMA node:

numactl --membind=<gpu-numa-node> dd if=/dev/zero of=/root/test bs=1M count=512 oflag=direct

The unpatched QEMU baseline reports virtio: bogus descriptor or out of resources immediately and marks the virtio-blk device broken.

Increasing the bounce-buffer limit only mitigates the issue. It does not address the underlying unnecessary bounce operation.

Backported series

1. system/memory: Use memmove() for directly accessible regions

   Uses memmove() in direct-access paths where source and destination ranges may overlap.

2. system/memory: Use qemu_ram_move() for directly accessible regions

   Introduces qemu_ram_move(). Naturally aligned 1-, 2-, 4-, and 8-byte accesses use exact-width atomic operations; other accesses fall back to memmove(). This preserves the access semantics required when RAM-device backing may represent PCI BAR memory.

3. system/memory: Make ram device region directly accessible

   Removes the indirect RAM-device MemoryRegionOps implementation and initializes RAM device regions using the normal RAM-pointer path. memory_access_is_direct() can consequently return true for these regions, allowing address_space_map() to return the underlying RAM pointer instead of allocating a bounce buffer.

Verification

Validation was performed on the QEMU baseline and fixed builds using the same VM configuration.

Baseline

• The direct-I/O loop failed immediately with:

  virtio: bogus descriptor or out of resources

Fixed build

• Confirmed the I/O buffer was allocated on the memory-bearing GPU NUMA node using numastat and /proc//numa_maps.

• Confirmed x-max-bounce-buffer-size remained at its default 4096-byte value; no bounce-buffer increase was used.

• Completed 8 GiB direct writes and reads using GPU-node-bound buffers.

• Completed repeated 32-cycle bidirectional direct-I/O tests across independent QEMU boots.

• Switched back to the baseline and reproduced the failure immediately, then restored the fixed build and passed another 32-cycle test.

• Completed an 8 GiB multijob fio CRC verification workload with:

  • err=0
  • No short operations
  • No dropped operations

• Completed a five-minute, eight-job mixed random read/write fio workload at queue depth 64 with:

  • err=0
  • No short operations
  • No dropped operations

• Confirmed guest dmesg contained no virtio errors, I/O errors, hung tasks, soft lockups, or GPU Xids.

• Confirmed through the QEMU monitor:

  • broken: false
  • disabled: false
  • No VIRTIO_CONFIG_S_NEEDS_RESET
  • All four virtio queues had inuse: 0
  • used_idx, last_avail_idx, shadow_avail_idx, and signalled_used matched after I/O quiesced

The baseline/fixed/baseline comparison demonstrates that the failure follows the QEMU code version, while the fixed build remains healthy under sustained multiqueue direct I/O from GPU coherent memory.

[nirmoy]

BaseOS Kernel Review

Summary

Only commit-message clarity issues were found: awkward grammar in two messages and several spelling errors in another. No runtime behavior issues were identified.

Findings: Critical: 0, High: 0, Medium: 0, Low: 3

Latest watcher review: open review

Head: b37bf99d9458

This comment is maintained by nv-pr-bot. It is updated when the GitHub watcher publishes a newer review.

[shamiali2008]

Backport LGTM:

Acked-by: Shameer Kolothum skolothumtho@nvidia.com

[NathanChenNVIDIA]

LGTM
Acked-by: Nathan Chen <nathanc@nvidia.com>

[mxc42]

LGTM, and ran a successful test build.

Acked-by: Morgan Hunter <morgan.hunter@canonical.com>

[MitchellAugustin]

All of the changes here look correct to me, and everything looks consistent with the latest version of the patch series that was approved and queued into the upstream tree here.

Please let me know when you're ready for a build of this version in the staging PPA @nvmochs .

Acked-by: Mitchell Augustin <mitchell.augustin@canonical.com>

[nvmochs]

Thanks all for the reviews!

Merged, closing PR.

d3b9383469e7 (HEAD -> nvidia_stable-11.0) 1:11.0.0+nvidia6-1
ac146ef2f62f NVIDIA: SAUCE: system/memory: Make ram device region directly accessible
d51cd18fe1c0 NVIDIA: SAUCE: system/memory: Use qemu_ram_move() for directly accessible regions
3a880a826ebf NVIDIA: SAUCE: system/memory: Use memmove() for directly accessible regions