Skip to content

Feature/decouple websocket - #1427

Open
sambles wants to merge 6 commits into
mainfrom
feature/decouple-websocket
Open

Feature/decouple websocket#1427
sambles wants to merge 6 commits into
mainfrom
feature/decouple-websocket

Conversation

@sambles

@sambles sambles commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Needs: OasisLMF/OasisLMF#2105

Fixes to run platform without redis websocket

New: HTTP progress endpoint

  • src/server/oasisapi/queues/views.py (new) — AnalysisStatusView, an unauthenticated
    APIView at /analysis-status/ (outside the versioned v1/v2 API, alongside
    healthcheck//server_info/). Applies events_complete as an atomic F() SQL increment so
    concurrent worker processes can't clobber each other's updates.
  • src/server/oasisapi/queues/serializers.pyAnalysisStatusSerializer (analysis_pk,
    optional events_total/events_complete).
  • src/server/oasisapi/base_urls.py — wires /analysis-status/ in at the root, alongside
    healthcheck//server_info/ (moved out of the versioned v2 router it started in).
  • src/server/oasisapi/queues/consumers.pysend_task_status_message now short-circuits
    when settings.DISABLE_WORKER_WS is set, skipping the websocket broadcast entirely.

Hardening: unreachable channel layer no longer crashes callers

send_task_status_message's DISABLE_WORKER_WS check only helps when someone remembers to set
it — e.g. the debug compose file drops the channel-layer (redis) service but that flag needs
to be set separately, and nothing enforces the two changes travel together. Without it, an
unreachable channel layer raised straight out of group_send(...), uncaught by any of this
function's callers — some of which (AnalysisTaskStatusQuerySet.create_statuses() in
analyses/models.py, called when a run's chunk sub-task rows are created) sit on the actual run
lifecycle, not just UI notification. That could fail or error out a real analysis purely because
of a broken websocket broadcast.

  • src/server/oasisapi/queues/consumers.pysend_task_status_message now wraps the
    group_send(...) call in try/except redis.exceptions.RedisError, logging a warning and
    returning instead of propagating. This complements DISABLE_WORKER_WS (an explicit "don't even
    try" opt-out) by also covering redis being down, misconfigured, or unreachable for any other
    reason — no reliance on remembering to set the flag correctly.

Bug fixs: progress reset race across chunks

events_total is reported once per chunk, not once per analysis (each chunk is its own
worker process running generate-losses-chunk). The original code reset num_events_complete
to 0 every time an events_total ping arrived, so a later-starting chunk would wipe out
progress already reported by chunks further along in the same run.

Fixed by moving the reset to the one place that unambiguously means "a run (or re-run) is
starting":

  • src/server/oasisapi/analyses/models.pyAnalysis.run() now resets
    num_events_total = 0 and num_events_complete = 0 at the same point it sets
    status = RUN_QUEUED.
  • src/server/oasisapi/queues/views.py / consumers.pyevents_total pings now
    only update num_events_total; they no longer touch num_events_complete.

@sambles
sambles requested a review from Ha-Ree August 14, 2026 12:32
@sambles sambles added the feature A main feature, captured on the backlog label Aug 14, 2026
@sambles sambles self-assigned this Aug 14, 2026
@sambles sambles moved this to Waiting for Review in Oasis Dev Team Tasks Aug 14, 2026
@sambles sambles added Enhancement Small improvement or refinement. and removed feature A main feature, captured on the backlog labels Aug 14, 2026
@codecov

codecov Bot commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.00000% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.06%. Comparing base (e46412b) to head (8db6abe).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/server/oasisapi/queues/views.py 47.82% 12 Missing ⚠️
src/server/oasisapi/queues/consumers.py 55.55% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1427      +/-   ##
==========================================
- Coverage   74.10%   74.06%   -0.04%     
==========================================
  Files         214      215       +1     
  Lines       14140    14178      +38     
==========================================
+ Hits        10478    10501      +23     
- Misses       3662     3677      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.


def generate_and_run(self, initiator):
self.validate_standard_analysis()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should add another

self.num_events_total = 0
self.num_events_complete = 0
self.save()



def send_task_status_message(items: dict):
if settings.DISABLE_WORKER_WS:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:
DISABLE_WORKER_WS=true silently disables k8s autoscaling. send_task_status_message feeds both the UI and the worker-controller autoscaler, which subscribes to ws/v2/queue-status/ (kubernetes/worker-controller/src/oasis_websocket.py L34). The flag is safe in compose but in Helm it would stop scaling with only a DEBUG log. Needs a loud warning at startup, or the flag scoped to the analysis-status broadcast only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Enhancement Small improvement or refinement.

Projects

Status: Waiting for Review

Development

Successfully merging this pull request may close these issues.

Task controller depends on WebSocket component ~ decouple or make optional

3 participants