Extract PRISM observability and audit HTTP

[kiwidream]

Position in the stack

PR 7 of 9. Base: prism-vardiff-finalization. Depends on PR 78. Next: PR 80.

Review this PR against its configured base. See the stack guide for the complete order.

Why this slice exists

Health, metrics, and audit/public HTTP were assembled inside the coordinator and could perform expensive work near request time. This slice makes observability a cached background concern and gives HTTP routing/listener lifecycle a dedicated owner while preserving the public contracts.

What changes

• Move base health snapshots, progress overlay, delivery health policy, and complete cached metrics assembly into observability.
• Move audit/public routing, response assembly, bounded request execution, and listener lifecycle into audit_http.
• Serve /healthz and /metrics from complete cached state without backend, ledger, artifact, or RPC work on the owned request path.
• Preserve route schemas, status codes, cache headers, metric names/labels, startup ordering, and shutdown behavior.
• Add PRISM_METRICS_REFRESH_SECONDS with a five-second default.
• Ensure every audit HTTP serve-loop exit closes and retires its listener before restart, including unexpected post-readiness exits.

Why the diff is sizeable

The roughly 2,670 additions and 630 deletions include two new owners, about 730 lines of direct observability/HTTP tests, and coordinator wiring. This is more than a textual move because request-time assembly is converted into explicitly refreshed immutable snapshots with staleness diagnostics.

Review cached-state completeness and staleness semantics first, then route/response compatibility, listener start/stop/restart, background-service registration, and coordinator ports.

Behavior and risk

This is a medium-risk operational slice because probes and Prometheus scraping depend on it. Public schemas and defaults remain compatible. The externally managed compatibility handler may still render metrics synchronously before its cache is initialized because it owns no refresher thread.

Validation

• 88 focused observability, HTTP, metrics, lifecycle, and shutdown tests passed at this boundary.
• The focused audit HTTP suites pass 20 tests.
• The unexpected-exit restart regression passed 100 consecutive runs.
• Public dashboard and audit API contracts pass in full discovery.
• Docker Python compile and Docker Ruff passed.

Operator impact

No schema migration is required. PRISM_METRICS_REFRESH_SECONDS is optional and defaults to five seconds.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Note

Medium Risk
Operational behavior of health checks and Prometheus scraping changed to cached snapshots with new staleness semantics; incorrect caching or lifecycle bugs could cause false unhealthy states or probe timeouts, though contracts and defaults are preserved.

Overview
Moves PRISM health, metrics, and audit/public HTTP out of the coordinator into observability and audit_http, so probes and scrapes read background-cached snapshots instead of doing ledger or full renderer work on the request path.

/healthz and /metrics now use cached payloads with staleness rules (503 when missing or stale); metrics get a new background refresher and PRISM_METRICS_REFRESH_SECONDS (default 5s). Health keeps caching base state while re-applying progress overlay on every response. Mining delivery health policy lives in ObservabilityService; the coordinator wires thin ports and compatibility shims for existing fields.

Audit HTTP is owned by AuditHttpFacade (routing, public API cache, bounded listener lifecycle, startup cancel, shutdown during non-writer drain). make_audit_handler becomes a thin wrapper over the facade; production starts health and metrics refreshers before the HTTP listener.

Extensive tests cover observability semantics, HTTP lifecycle, cached metrics behavior, compose/config loading, and shutdown ordering.

^{Reviewed by Cursor Bugbot for commit 584a1f1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4bb1f0b3cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cursor]

Stop clobbers restarted lifecycle

Medium Severity

stop always writes _lifecycle to stopped or stop_timeout after joining, even when _server/_thread no longer refer to the instance it shut down. After the serve thread retires those fields under stopping, a concurrent start can install a new listener; the finishing stop then marks the facade stopped while that listener is still running, so state() lies and later control decisions see the wrong lifecycle.

 

^{Reviewed by Cursor Bugbot for commit bccba5a. Configure here.}

[cursor]

Unused delivery timestamp field

Low Severity

MiningDeliveryInputs.last_initial_job_delivery_monotonic is populated from the coordinator adapter but never read when building the mining delivery snapshot. The field is dead weight on every health and metrics collection path.

Additional Locations (1)

• lab/prism/prism_coordinator.py#L1287-L1290

 

^{Reviewed by Cursor Bugbot for commit bccba5a. Configure here.}

[cursor]

Stale metrics trigger live fallback

Medium Severity

_CoordinatorAuditHttp.cached_metrics_payload falls back to a live metrics_payload() render on any 503 when allow_uncached_compatibility is enabled. Cached metrics also return 503 when a snapshot is stale, so the compatibility handler can bypass the staleness contract, hide a wedged collector, and redo expensive scrape-time work.

 

^{Reviewed by Cursor Bugbot for commit bccba5a. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 4 total unresolved issues (including 3 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 584a1f1. Configure here.}

[cursor]

Delivery reset races snapshot timers

Medium Severity

reset_delivery_failure only takes the observability _lock, not _mining_delivery_lock, so it can run between input capture and timer advancement. A refresh that sampled poor coverage can rewrite the gap timer after delivery already cleared it, undoing the restored grace window and leaving cached health degraded after recovery.

Additional Locations (1)

• lab/prism/prism_coordinator.py#L5282-L5300

 

^{Reviewed by Cursor Bugbot for commit 584a1f1. Configure here.}