nvidia: reduce NVML polling overhead - #2104
Open
DevL0rd wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I was chasing bad frame pacing in a game on my machine. Frametimes were constantly spiky, but neither the CPU nor the GPU were being used much, which didn't add up.
I ended up tracing it into the kernel with bpftrace, and found the game's render and submission threads were blocking on NVIDIA's global RM lock. So I traced who was holding that lock, and it turned out to be MangoHud's own NVML thread. It was holding it for roughly 10 seconds out of every 15.
Looking at the code, the NVML thread samples every metric every 25ms (40 times a second), but the HUD only publishes an averaged value every 500ms. So 19 of every 20 calls get averaged away. That makes sense for GPU load since it's genuinely spiky, but temperature, power, clocks, VRAM and fan speed don't change meaningfully in 25ms.
On top of that, nvmlDeviceGetGraphicsRunningProcesses was being called on every sample regardless of config, even though the only thing that consumes it is proc_vram, which is off by default. That call takes 4.5ms on my machine.
Per-call timings I measured (median, RTX 4090 laptop, driver 610.57.04):
This matters more than just CPU time. On the proprietary driver these calls serialise on one global lock that the render path also uses, so the polling stalls other processes, not just MangoHud.
What this PR changes:
Before and after on my system, measuring how long MangoHud's NVML thread holds the NVIDIA global lock over 15 seconds:
And the effect on the game, counting how often the vkd3d submission thread had to block on that lock over 20 seconds:
This gives insanely smooth frame pacing, and almost doubled my 1% lows as well, and finally after a long investigation over the last 2 months, this fixes the issues.
Please consider merging this as this doesn't in any way change mangohuds functionality, but it greatly increases game performance and frame pacing. <3
Let me know if there is anything to patch up, but I think this is pretty clean.