Skip to content

feat(progress): Redis live bar, SQLite for stage start/end - #147

Open
nexus69420 wants to merge 4 commits into
OpenAgriNet:mainfrom
nexus69420:fix/live-progress-counters
Open

feat(progress): Redis live bar, SQLite for stage start/end#147
nexus69420 wants to merge 4 commits into
OpenAgriNet:mainfrom
nexus69420:fix/live-progress-counters

Conversation

@nexus69420

@nexus69420 nexus69420 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Live done/ otal come from a Redis cache (pipeline:progress:{workflow_id}) that the worker updates on each activity heartbeat.
  • SQLite stays canonical for stage start/end and finished page/chunk rows. The bar is hidden when SQLite leaves OCR/translation/chunking/ingest. No live ticker is written to document_jobs.config_json.
  • Redis is an ephemeral UI ticker, not content SoT. Compose adds a
    edis service (not published). API + worker get REDIS_URL. Tests use the in-memory fallback when REDIS_URL is unset.
  • Document Ops polls every 1s while a live stage is running.
  • OCR default segment size is 5 pages (OCR_SEGMENT_PAGES) so the bar ticks during Chandra, not once at the end of a 20-page block.
  • Ingest heartbeats include workflow_id so Redis actually updates during ingest.
  • Translation remaining-work ticks use the event denominator (pages still to translate), not len(pages) or the language-detection total.

Test plan

  • pytest tests/test_runtime_progress.py tests/test_activities.py::TestTranslationRetryProgressPersistence (43 passed)
  • Worker heartbeat path moves OCR / translation / chunking / ingest bars, then hides them on review/completed
  • Translation callback keeps event pages_total (remaining work), does not inflate to the full page set
  • LIVE_PROGRESS_UI_ENABLED=true with Redis up on H100: bars move from cache, disappear when SQLite hits review
  • Redis down: pipeline still runs; bar may stay at the SQLite start snapshot until Redis recovers
  • Flag off: progress is null

…king

Persist OCR and translation job progress like chunking, keep a stale 0/N heartbeat from hiding SQLite, and poll Document Ops every 1s while a stage is running.
Worker heartbeats write pipeline:progress:{workflow_id}. GET /runtime reads that cache for done/total and still uses SQLite for stage start/end and finished pages/chunks. Drop the job_config ticker writes.
@nexus69420 nexus69420 changed the title fix(ui): make live progress counters move during OCR/translation/chunking feat(progress): Redis live bar, SQLite for stage start/end Sep 2, 2026
@nexus69420
nexus69420 requested a review from KDwevedi September 2, 2026 14:10
…remaining work

Ingest heartbeats omitted workflow_id so Redis never moved; translation inherited detection 100%. Stage bars now update through the worker heartbeat path.
The translation callback was republishing len(pages), which undid remaining-work ticks. Drop unused runtime kwargs and document Redis as an ephemeral cache, not SoT.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant