Fix Dash loopback deadlock during spectrum rendering - #228
Merged
mwang87 merged 1 commit intoAug 31, 2026
Conversation
Contributor
Author
|
CI note: both required workflows are currently failing on
I intentionally did not mix CI modernization into this production deadlock fix. The two new regression tests pass under pytest in an ephemeral container using the exact image currently deployed on |
Contributor
Author
|
End-to-end compatibility verification on
The observable behavior and existing cache-backed render path are preserved; only the redundant synchronous server-side loopback request is removed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #227.
What changed
requests.get("http://localhost:5000/svg/...")cache-warm request from single-spectrum Dash rendering./svg/and/svg/mirror/. The browser still requests those endpoints normally, so rendering remains cached without doing the work twice.Why
The Dash callback already occupies a Gunicorn thread. Its synchronous request back into the same finite Gunicorn thread pool can deadlock under concurrency: all threads serve outer callbacks, and every outer callback waits for an inner SVG request that cannot acquire a thread.
This matches production symptoms on
ucr-web1:Loading...because its assets and callbacks stall.GET /requests are recorded by nginx as HTTP 499 exactly at the monitor's 5-second timeout./json/and directly upstream.Local reproduction before the fix
A one-worker/two-thread Gunicorn app was used where
/outersynchronously requested its own/innerendpoint. Sending two concurrent requests occupied both threads and prevented either inner request from running:Verification
The new tests were run inside an ephemeral container using the exact image currently deployed on
ucr-web1:The tests replace
requests.getwith an exception-raising mock, invoke both rendering paths, verify that no HTTP request occurs, and assert that the resulting single-spectrum and mirror SVG URLs are unchanged.