Feature/decouple websocket - #1427
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1427 +/- ##
==========================================
- Coverage 74.10% 74.06% -0.04%
==========================================
Files 214 215 +1
Lines 14140 14178 +38
==========================================
+ Hits 10478 10501 +23
- Misses 3662 3677 +15 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
|
||
| def generate_and_run(self, initiator): | ||
| self.validate_standard_analysis() | ||
|
|
There was a problem hiding this comment.
Should add another
self.num_events_total = 0
self.num_events_complete = 0
self.save()
|
|
||
|
|
||
| def send_task_status_message(items: dict): | ||
| if settings.DISABLE_WORKER_WS: |
There was a problem hiding this comment.
Claude:
DISABLE_WORKER_WS=true silently disables k8s autoscaling. send_task_status_message feeds both the UI and the worker-controller autoscaler, which subscribes to ws/v2/queue-status/ (kubernetes/worker-controller/src/oasis_websocket.py L34). The flag is safe in compose but in Helm it would stop scaling with only a DEBUG log. Needs a loud warning at startup, or the flag scoped to the analysis-status broadcast only.
Needs: OasisLMF/OasisLMF#2105
Fixes to run platform without redis websocket
New: HTTP progress endpoint
src/server/oasisapi/queues/views.py(new) —AnalysisStatusView, an unauthenticatedAPIViewat/analysis-status/(outside the versionedv1/v2API, alongsidehealthcheck//server_info/). Appliesevents_completeas an atomicF()SQL increment soconcurrent worker processes can't clobber each other's updates.
src/server/oasisapi/queues/serializers.py—AnalysisStatusSerializer(analysis_pk,optional
events_total/events_complete).src/server/oasisapi/base_urls.py— wires/analysis-status/in at the root, alongsidehealthcheck//server_info/(moved out of the versionedv2router it started in).src/server/oasisapi/queues/consumers.py—send_task_status_messagenow short-circuitswhen
settings.DISABLE_WORKER_WSis set, skipping the websocket broadcast entirely.Hardening: unreachable channel layer no longer crashes callers
send_task_status_message'sDISABLE_WORKER_WScheck only helps when someone remembers to setit — e.g. the debug compose file drops the
channel-layer(redis) service but that flag needsto be set separately, and nothing enforces the two changes travel together. Without it, an
unreachable channel layer raised straight out of
group_send(...), uncaught by any of thisfunction's callers — some of which (
AnalysisTaskStatusQuerySet.create_statuses()inanalyses/models.py, called when a run's chunk sub-task rows are created) sit on the actual runlifecycle, not just UI notification. That could fail or error out a real analysis purely because
of a broken websocket broadcast.
src/server/oasisapi/queues/consumers.py—send_task_status_messagenow wraps thegroup_send(...)call intry/except redis.exceptions.RedisError, logging a warning andreturning instead of propagating. This complements
DISABLE_WORKER_WS(an explicit "don't eventry" opt-out) by also covering redis being down, misconfigured, or unreachable for any other
reason — no reliance on remembering to set the flag correctly.
Bug fixs: progress reset race across chunks
events_totalis reported once per chunk, not once per analysis (each chunk is its ownworker process running
generate-losses-chunk). The original code resetnum_events_completeto
0every time anevents_totalping arrived, so a later-starting chunk would wipe outprogress already reported by chunks further along in the same run.
Fixed by moving the reset to the one place that unambiguously means "a run (or re-run) is
starting":
src/server/oasisapi/analyses/models.py—Analysis.run()now resetsnum_events_total = 0andnum_events_complete = 0at the same point it setsstatus = RUN_QUEUED.src/server/oasisapi/queues/views.py/consumers.py—events_totalpings nowonly update
num_events_total; they no longer touchnum_events_complete.