Skip to content

nvidia: reduce NVML polling overhead - #2104

Open
DevL0rd wants to merge 1 commit into
flightlessmango:masterfrom
DevL0rd:nvml-reduce-polling-overhead
Open

nvidia: reduce NVML polling overhead#2104
DevL0rd wants to merge 1 commit into
flightlessmango:masterfrom
DevL0rd:nvml-reduce-polling-overhead

Conversation

@DevL0rd

@DevL0rd DevL0rd commented Aug 4, 2026

Copy link
Copy Markdown

I was chasing bad frame pacing in a game on my machine. Frametimes were constantly spiky, but neither the CPU nor the GPU were being used much, which didn't add up.

I ended up tracing it into the kernel with bpftrace, and found the game's render and submission threads were blocking on NVIDIA's global RM lock. So I traced who was holding that lock, and it turned out to be MangoHud's own NVML thread. It was holding it for roughly 10 seconds out of every 15.

Looking at the code, the NVML thread samples every metric every 25ms (40 times a second), but the HUD only publishes an averaged value every 500ms. So 19 of every 20 calls get averaged away. That makes sense for GPU load since it's genuinely spiky, but temperature, power, clocks, VRAM and fan speed don't change meaningfully in 25ms.

On top of that, nvmlDeviceGetGraphicsRunningProcesses was being called on every sample regardless of config, even though the only thing that consumes it is proc_vram, which is off by default. That call takes 4.5ms on my machine.

Per-call timings I measured (median, RTX 4090 laptop, driver 610.57.04):

nvmlDeviceGetGraphicsRunningProcesses   4491.4us   (2 calls per sample)
nvmlDeviceGetFanSpeed                    137.1us
nvmlDeviceGetMemoryInfo                   11.6us
nvmlDeviceGetUtilizationRates              2.5us
nvmlDeviceGetClockInfo                     1.6us
nvmlDeviceGetTemperature                   0.9us
nvmlDeviceGetPowerUsage                    0.8us

nvmlDeviceGetFanSpeed_v2                   0.5us
nvmlDeviceGetMemoryInfo_v2                 4.0us

This matters more than just CPU time. On the proprietary driver these calls serialise on one global lock that the render path also uses, so the polling stalls other processes, not just MangoHud.

What this PR changes:

  • Everything except GPU load is sampled at the HUD update rate instead of every 25ms, with values cached in between, so the published averages are unchanged.
  • Running-process info is only queried when proc_vram is actually enabled.
  • Uses nvmlDeviceGetFanSpeed_v2 and nvmlDeviceGetMemoryInfo_v2 when the driver exports them, falling back to v1 otherwise.
  • Checks the fan speed return value before storing it. v1 returns NotSupported on my card and the old code stored the value anyway.

Before and after on my system, measuring how long MangoHud's NVML thread holds the NVIDIA global lock over 15 seconds:

before:  38664 acquisitions, 10234.6ms held
after:      30 acquisitions,      0.6ms held

And the effect on the game, counting how often the vkd3d submission thread had to block on that lock over 20 seconds:

before:  132 stalls, 34.59ms average
after:   none

This gives insanely smooth frame pacing, and almost doubled my 1% lows as well, and finally after a long investigation over the last 2 months, this fixes the issues.

Please consider merging this as this doesn't in any way change mangohuds functionality, but it greatly increases game performance and frame pacing. <3

Let me know if there is anything to patch up, but I think this is pretty clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant