Skip to content

Fix Dash loopback deadlock during spectrum rendering - #228

Merged
mwang87 merged 1 commit into
mwang87:devfrom
Michael-C-Strobel:fix/dash-loopback-deadlock
Aug 31, 2026
Merged

Fix Dash loopback deadlock during spectrum rendering#228
mwang87 merged 1 commit into
mwang87:devfrom
Michael-C-Strobel:fix/dash-loopback-deadlock

Conversation

@Michael-C-Strobel

Copy link
Copy Markdown
Contributor

Closes #227.

What changed

  • Remove the synchronous requests.get("http://localhost:5000/svg/...") cache-warm request from single-spectrum Dash rendering.
  • Remove the equivalent loopback request from mirror rendering.
  • Add regression tests that fail if either rendering path performs HTTP I/O while constructing its response.
  • Preserve the generated SVG URLs and the existing joblib/Celery caches behind /svg/ and /svg/mirror/. The browser still requests those endpoints normally, so rendering remains cached without doing the work twice.

Why

The Dash callback already occupies a Gunicorn thread. Its synchronous request back into the same finite Gunicorn thread pool can deadlock under concurrency: all threads serve outer callbacks, and every outer callback waits for an inner SVG request that cannot acquire a thread.

This matches production symptoms on ucr-web1:

  • Dash remains on Loading... because its assets and callbacks stall.
  • Uptime Kuma's GET / requests are recorded by nginx as HTTP 499 exactly at the monitor's 5-second timeout.
  • Docker health checks intermittently exceed 10 seconds.
  • Gunicorn workers hit their 90-second timeout and autoheal eventually restarts the container.
  • Host CPU, memory, and disk are not saturated, and the tested GNPS2 spectrum resolves quickly through /json/ and directly upstream.

Local reproduction before the fix

A one-worker/two-thread Gunicorn app was used where /outer synchronously requested its own /inner endpoint. Sending two concurrent requests occupied both threads and prevented either inner request from running:

$ seq 2 | xargs -P 2 -I{} curl --max-time 3 -o /dev/null \
    -w 'request {}: HTTP=%{http_code} total=%{time_total}s\n' \
    http://127.0.0.1:5099/outer
curl: (28) Operation timed out after 3001 milliseconds with 0 bytes received
request 1: HTTP=000 total=3.001550s
curl: (28) Operation timed out after 3001 milliseconds with 0 bytes received
request 2: HTTP=000 total=3.001776s

Verification

The new tests were run inside an ephemeral container using the exact image currently deployed on ucr-web1:

  • Python 3.11.15
  • Dash 1.20.0
  • dash-bootstrap-components 0.9.2
  • Flask 2.1.3
  • Werkzeug 2.0.0
  • Gunicorn 26.1.0
  • Celery 5.3.6
  • joblib 1.5.3
  • Matplotlib 3.6.3
  • NumPy 1.26.4
  • Requests 2.34.2
$ python -m pytest -q test/test_dashinterface.py
..                                                                       [100%]
2 passed

The tests replace requests.get with an exception-raising mock, invoke both rendering paths, verify that no HTTP request occurs, and assert that the resulting single-spectrum and mirror SVG URLs are unchanged.

@Michael-C-Strobel

Copy link
Copy Markdown
Contributor Author

CI note: both required workflows are currently failing on dev independently of this PR.

  • The latest dev push before this PR failed both Lint and unittest; the preceding dev push failed both as well.
  • This PR's lint job crashes inside psf/black@stable before checking repository files because the action uses str | None syntax while the workflow runs the action under Python 3.8.
  • This PR's unittest workflow declares Python 3.8 but its unpinned Conda environment resolves to Python 3.14.7. Collection then fails because imghdr was removed and the workflow did not install Dash or Flask-Limiter. The same missing dependencies also prevent the new test module from importing the application in that broken environment.

I intentionally did not mix CI modernization into this production deadlock fix. The two new regression tests pass under pytest in an ephemeral container using the exact image currently deployed on ucr-web1 (Python 3.11.15, Dash 1.20.0, Flask 2.1.3, Werkzeug 2.0.0, Gunicorn 26.1.0, and Celery 5.3.6):

..                                                                       [100%]
2 passed

@Michael-C-Strobel

Copy link
Copy Markdown
Contributor Author

End-to-end compatibility verification on ucr-web1 using the exact deployed image/dependency set:

  • Ran this branch as an isolated web + API + Redis + Celery stack (production containers/files were not changed).
  • For both the reported GNPS2 task USI and CCMSLIB00005436077, /json/ responses were byte-for-byte identical to production.
  • /proxi/v0.1/spectra responses were byte-for-byte identical to production.
  • /png/ output was byte-for-byte identical to production (task: 32,624 bytes; library: 27,284 bytes).
  • Loaded both USIs through a real Firefox browser against /dashinterface/; all Dash callback POSTs returned 200 and the browser fetched the final annotated /svg/ successfully.

The observable behavior and existing cache-backed render path are preserved; only the redundant synchronous server-side loopback request is removed.

@mwang87
mwang87 merged commit 84eb45c into mwang87:dev Aug 31, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dash rendering can deadlock Gunicorn through synchronous loopback requests

2 participants