fix: stop the per-frame GPU readback that cost the wall 40x - #275
Merged
Conversation
The videowall measured 1.4 fps on the split-flap board -- about 700ms a frame -- on an RTX PRO 4000 Blackwell with gpu_compositing enabled, where the same board runs at 118.9 fps in a harness. The instrumentation added in #271 is what made that visible. The cause is the Art-Net frame observer (#59), and it was wrong in two ways that compounded. **It was registered unconditionally at startup**, on the reasoning that "the observer is a no-op while disabled". That is true of the observer and false of the readback: gl-base reads the frame BEFORE calling anyone, so the wall paid the full cost with `artnetEnabled: false` in its settings the entire time. Every stall fed a callback that did nothing. **The readback was never rate-limited.** The doc said observers were "expected to rate-limit itself" -- impossible, because by the time an observer is called the pixels have already been read. The Art-Net client sends at 1Hz while the readback ran at frame rate, so 59 of every 60 were discarded. Each notify is TILES_X*TILES_Y = 32 synchronous gl.readPixels. On a discrete GPU each one is a pipeline stall plus a PCIe transfer; dual view runs two boards as two runtimes, so 64 stalls per frame. 700ms / 64 is about 11ms each, which is what a synchronous readback costs on that hardware. **Why this was missed for so long, and why I refuted it once already.** I measured this exact hypothesis yesterday and got 4% -- on Apple Silicon, where unified memory makes readPixels close to a memcpy. The figure was correct for that machine and worthless as a prediction for a discrete GPU. Measuring on the development machine was the mistake, not the reasoning. Two changes, both small: - the observer is registered only while Art-Net is enabled, re-evaluated on every settings save so toggling without a restart still works -- the property the unconditional registration was protecting - the readback is rate-limited in gl-base to 1Hz, so the feature is usable on a discrete GPU even when it IS enabled 9 new tests, 629 -> 638. Reverting the conditional registration fails 5; removing the rate limit fails 1. Not yet confirmed on the wall: the fix is expected to take the board from 1.4 fps back towards its harness figure, and #271's report is how that gets verified rather than assumed. The 1.4 fps baseline was deliberately captured before this change for exactly that comparison.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The wall measured 1.4 fps on the split-flap board — about 700ms a frame — on an RTX PRO
4000 Blackwell with
gpu_compositing enabled, where the same board runs at 118.9 fps in aharness. #271's instrumentation is what made that visible:
The cause, wrong in two compounding ways
Registered unconditionally, on the reasoning that "the observer is a no-op while
disabled". True of the observer, false of the readback — gl-base reads the frame before
calling anyone. So the wall paid full price with
artnetEnabled: falsethe whole time, andevery stall fed a callback that did nothing.
Never rate-limited. The doc said observers were "expected to rate-limit itself" — which
they cannot: by the time one is called, the pixels have already been read. Art-Net sends at
1Hz while the readback ran at frame rate, so 59 of every 60 were thrown away.
Each notify is
TILES_X * TILES_Y= 32 synchronousgl.readPixels. On a discrete GPUeach is a pipeline stall plus a PCIe transfer, and dual view runs two boards as two runtimes —
64 stalls per frame. 700ms ÷ 64 ≈ 11ms each, which is about right for that hardware.
Why I refuted this hypothesis yesterday
I tested exactly this and measured 4%. On Apple Silicon, where unified memory makes
readPixelsclose to a memcpy. The number was correct for this Mac and worthless as aprediction for a discrete GPU on PCIe.
Measuring on the development machine was the mistake — not the reasoning. Worth recording,
because the same trap is waiting for the next GPU-cost question.
The fix
save so toggling without a restart still works — the property the unconditional registration
was protecting
discrete GPU even when it is enabled
Tests
9 new, 629 → 638. Both halves confirmed by breaking them:
Two of these read gl-base's source, so they use
projectRootfrom the DOM helper —import.meta.urlis not a file URL under jsdom, which is a trap the helper exists todocument and which caught me here first.
Not yet confirmed
That this actually restores the wall's frame rate. The expectation is 1.4 fps → back toward
the harness figure, and #271's report is how that gets checked rather than assumed. The 1.4
fps baseline was captured deliberately before this change for exactly this comparison —
which is why the instrumentation shipped separately in the first place.
Filed #274 for the case this does not cover: Art-Net enabled on the wall has still never
been run, and the 1Hz limit being affordable there is arithmetic rather than a measurement.