Bound PRISM tip refresh and watchdog recovery

[kiwidream]

Summary

• bound every blocking phase in the qbit tip-refresh path and expose the active phase, tip/generation, phase age, helper PID, last success times, and newest pending generation
• prevent qbit-prism-build-audit-bundle from blocking on its stdin pipe, then terminate, kill, reap, and clean up timed-out or superseded helpers
• build collection bundles outside coordinator/client locks, bound lock admission and fanout waits, and preserve latest-generation-wins publication fencing
• replace watchdog hard exit with a controlled fatal shutdown that closes writer admission, performs a bounded exact-session lease release, and exits non-zero without weakening lease ownership
• add bounded qbit RPC and PostgreSQL operations plus phase, timeout, supersession, and watchdog lease-release metrics

Root cause

build_audit_bundle() streamed JSON into the helper's stdin pipe before entering process.wait(timeout=...). If the helper stopped reading, the pipe filled and process.stdin.write() blocked qbit_blockpoll before the existing subprocess timeout began. Collection refresh also held a client job lock around expensive bundle preparation.

Configuration defaults

• whole tip refresh: PRISM_TIP_REFRESH_TIMEOUT_SECONDS=90
• bundle helper: PRISM_BUNDLE_BUILD_TIMEOUT_SECONDS=60
• helper terminate/kill grace: PRISM_AUDIT_BUNDLE_TERMINATE_GRACE_SECONDS=2
• qbit RPC: PRISM_QBIT_RPC_TIMEOUT_SECONDS=10
• PostgreSQL operation: PRISM_POSTGRES_OPERATION_TIMEOUT_SECONDS=30
• refresh lock: PRISM_TIP_REFRESH_LOCK_TIMEOUT_SECONDS=5
• client job lock: PRISM_CLIENT_JOB_LOCK_TIMEOUT_SECONDS=2
• watchdog lease release: PRISM_WATCHDOG_LEASE_RELEASE_TIMEOUT_SECONDS=5

Validation

• python3 -m unittest -q tests.test_prism_coordinator_vardiff — 235 passed
• python3 -m unittest -q tests.test_prism_tip_refresh_validation tests.test_prism_coordinator_shutdown tests.test_prism_coordinator_job_cache tests.test_prism_compose_profile — 140 passed
• python3 -m unittest -q tests.test_prism_share_ledger — 107 passed
• Docker Python 3.12 compile check for the changed PRISM modules/tests — passed
• Docker Compose configuration validation for permissionless, real-miner-smoke, auxpow, and prism profiles — passed

Production risk

Timed-out refreshes fail closed and retry the newest generation, so persistent helper, database, RPC, or lock failures can delay new jobs but cannot publish obsolete work. If the watchdog's exact-session lease release fails or times out, the process still exits and restart remains fenced by the existing lease TTL; this intentionally retains the current restart delay rather than risking overlapping writers.

No deployment or live-host changes are included.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Note

Medium Risk
Changes core coordinator tip refresh, job publication fencing, and watchdog fatal shutdown around ledger lease release; failures delay jobs but are designed fail-closed with retries rather than publishing stale work.

Overview
Adds configurable timeouts across the qbit tip-refresh path (whole refresh, RPC, Postgres, lock waits, audit-bundle helper terminate/kill grace, watchdog lease release) and startup validation so those bounds stay below the liveness watchdog.

Tip refresh now runs under a refresh deadline: lock acquisition uses timed waits, phases are tracked (qbit_blockpoll in health + Prometheus histograms/timeouts), and TipRefreshPhaseTimeout schedules immediate retry instead of hanging. Collection mode builds per-worker bundles outside client locks and fans out via the prepared-job path with per-submission chain checks.

Audit bundle helper no longer streams JSON on a blocking stdin pipe; payload is written to a temp file first, then the subprocess is polled with deadline-aware communicate, terminate → kill → reap on timeout/supersession, with helper PID exposed in blockpoll state.

Watchdog no longer os._exit(1) immediately: it calls controlled shutdown() with bounded lease release, records outcomes, then exits non-zero. Lease release can time out on a background thread with a new timeout outcome.

JsonRpc default timeout is env-driven; production requires positive PRISM_STRATUM_SEND_TIMEOUT_SECONDS.

^{Reviewed by Cursor Bugbot for commit 44d4a4f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44d4a4ffed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Guard watchdog timeout checks behind enablement

When PRISM_WATCHDOG_ENABLED=0, the startup path does not launch the watchdog thread, but this new validation still rejects any bounded operation timeout that is greater than or equal to PRISM_WATCHDOG_TIMEOUT_SECONDS. That breaks valid debugging/external-supervisor configurations that intentionally disable the watchdog and raise PRISM_TIP_REFRESH_TIMEOUT_SECONDS or PRISM_BUNDLE_BUILD_TIMEOUT_SECONDS above the default 120s threshold; the coordinator exits before starting even though there is no watchdog deadline to violate. Gate this comparison on self.watchdog_enabled (or skip it when the watchdog is disabled).

Useful? React with 👍 / 👎.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 44d4a4f. Configure here.}

[cursor]

Job fanout ignores refresh deadline

Medium Severity

Operations within the _fanout_prepared_tip_refresh loop and the audit bundle helper's input serialization can exceed PRISM_TIP_REFRESH_TIMEOUT_SECONDS without checking the refresh deadline. This prevents TipRefreshPhaseTimeout from being raised, causing the blockpoll heartbeat to go stale and triggering the liveness watchdog instead of a bounded refresh retry.

Additional Locations (1)

• lab/prism/prism_coordinator.py#L10871-L10886

 

^{Reviewed by Cursor Bugbot for commit 44d4a4f. Configure here.}