Stabilize PRISM payout snapshot publication under share churn

[kiwidream]

Summary

• freeze an immutable payout epoch for each ready-tip publication attempt so accepted-share and settlement advances cannot invalidate an in-flight build
• keep jobs, share-window boundaries, coinbase inputs, payout policy, CTV settlement inputs, durable block-candidate intents, and final block reconstruction bound to the same snapshot ID and digest
• coalesce any number of same-tip payout advances behind an active build into one follow-up refresh while preserving immediate real-tip supersession
• retain submit-capable snapshot state behind a hard bound and evict only after issued jobs and queued candidates no longer reference it
• add bounded snapshot lifecycle, coalescing, supersession, age, retention, and eviction metrics

Concurrency invariants

• The tip refresh single-flight lane and snapshot owner allow at most one active build for a chain tip and selected payout snapshot.
• Accepted-share commits and snapshot selection share one short linearization lock. A commit is wholly before the selected boundary or advances the next epoch.
• Payout advances never cancel a snapshot-bound build. They update one latest epoch and schedule exactly one same-tip follow-up after successful publication.
• A newly observed chain tip still cancels old-tip construction and fanout immediately. Existing stale-tip validation and block-submit authority checks remain authoritative.
• Serialization, policy and CTV construction, template work, signing, and fanout run without coordinator, ledger, database, or client locks held.
• Publication validates the complete immutable build key and retained snapshot before authority changes. No partial or cross-snapshot bundle can publish.
• Build and snapshot ownership is released on success, failure, cancellation, and shutdown through finally paths.

Observability

Adds bounded-cardinality metrics for selected, published, and latest snapshot IDs; coalesced advances; scheduled and completed follow-ups; chain-tip versus payout supersession; oldest unpublished age; retention count; and eviction reason. Snapshot selection, publication, follow-up, and eviction also emit lifecycle logs.

Tests

• 222 focused unit and concurrency tests covering immutable refresh, payout mutation, initial delivery, publication boundaries, representative selection, payout formulas, durable candidate replay, and block submission
• Dockerized Python 3.12 compile of CI source roots
• Dockerized ShellCheck for tracked shell scripts

Rollout risks

• Same-tip payout activity can now produce one additional clean refresh after the current snapshot publishes; monitor follow-up scheduled/completed deltas and current-tip coverage.
• Snapshot retention adds bounded memory proportional to still-submit-capable issued epochs. The hard cap fails closed instead of evicting valid submission state; monitor retention count and eviction reasons.
• Old payout generations may cross the delivery gate only when backed by an exact retained snapshot digest. Chain-tip and template validation are unchanged; monitor chain-tip supersession and candidate audit-mismatch counters.

Rollback

Revert commit 82392ed and restart the PRISM coordinator. This patch has no database migration and does not change payout formulas or stale-grace policy. Durable candidate intents keep the existing schema version and add optional fields, so rollback readers ignore the extra snapshot metadata. A restart invalidates connected issued jobs in the normal way and restores the prior payout-generation supersession behavior.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Note

High Risk
Large concurrent changes to tip refresh, payout publication, job delivery, and block submission in the mining coordinator; incorrect snapshot binding could mis-attribute payouts or reject valid blocks.

Overview
Introduces immutable payout publication snapshots so ready-tip job builds pin a single ledger epoch (shares, balances, payout policy, CTV settlement) instead of being torn down when accepted shares or payout state advance mid-build.

Each snapshot gets an ID and canonical digest; that identity flows through JobBuildKey, issued jobs, tip-refresh validation tokens, block-candidate intents, and block submission checks. Tip refresh and startup prewarm select and optionally own a snapshot before building; share commits advance latest_available under a short lock but do not cancel an active snapshot-bound build. Multiple same-tip advances coalesce into one scheduled follow-up refresh after publication, while a new chain tip still supersedes immediately.

Retention is bounded (MAX_PRISM_RETAINED_PAYOUT_SNAPSHOTS) with eviction only when no submit-capable job still references a snapshot. The payout delivery gate gains allow_stale for snapshot-pinned work; payout-generation supersession skips builds that already carry payout_snapshot_id > 0. New Prometheus metrics cover snapshot IDs, coalescing, follow-ups, supersession reasons, retention, and evictions. Tests add PayoutSnapshotEpochTests and adjust initial-delivery / tip-refresh expectations for pinned bundles.

^{Reviewed by Cursor Bugbot for commit 82392ed. Bugbot is set up for automated code reviews on this repo. Configure here.}

[blacksmith-sh]

Found 14 test failures on Blacksmith runners:

Failures

Test	View Logs
test_accepted_direct_block_refreshes_clean_job_after_submit_response (test_prism_coordi nator_vardiff.PrismCoordinatorVardiffTests.test_accepted_direct_block_refreshes_clean_j ob_after_submit_response)/ test_accepted_direct_block_refreshes_clean_job_after_submit_response (test_prism_coordi nator_vardiff.PrismCoordinatorVardiffTests.test_accepted_direct_block_refreshes_clean_j ob_after_submit_response)	View Logs
test_direct_block_preparation_does_not_hold_delivery_gate (test_prism_coordinator_vardi ff.PrismCoordinatorVardiffTests.test_direct_block_preparation_does_not_hold_delivery_ga te)/ test_direct_block_preparation_does_not_hold_delivery_gate (test_prism_coordinator_vardi ff.PrismCoordinatorVardiffTests.test_direct_block_preparation_does_not_hold_delivery_ga te)	View Logs
test_event_only_stop_cannot_strand_late_admitted_share (test_prism_coordinator_shutdown .PrismCoordinatorShutdownTests.test_event_only_stop_cannot_strand_late_admitted_share)/ test_event_only_stop_cannot_strand_late_admitted_share (test_prism_coordinator_shutdown .PrismCoordinatorShutdownTests.test_event_only_stop_cannot_strand_late_admitted_share)	View Logs
test_idle_cached_collection_bundle_refreshes_readiness (test_prism_coordinator_vardiff. PrismCoordinatorVardiffTests.test_idle_cached_collection_bundle_refreshes_readiness)/ test_idle_cached_collection_bundle_refreshes_readiness (test_prism_coordinator_vardiff. PrismCoordinatorVardiffTests.test_idle_cached_collection_bundle_refreshes_readiness)	View Logs
test_idle_cached_ready_bundle_rebinds_same_tip_observation (test_prism_coordinator_vard iff.PrismCoordinatorVardiffTests.test_idle_cached_ready_bundle_rebinds_same_tip_observa tion)/ test_idle_cached_ready_bundle_rebinds_same_tip_observation (test_prism_coordinator_vard iff.PrismCoordinatorVardiffTests.test_idle_cached_ready_bundle_rebinds_same_tip_observa tion)	View Logs
test_idle_retarget_delivers_fresh_bundle_when_cache_is_disabled (test_prism_coordinator _vardiff.PrismCoordinatorVardiffTests.test_idle_retarget_delivers_fresh_bundle_when_cac he_is_disabled)/ test_idle_retarget_delivers_fresh_bundle_when_cache_is_disabled (test_prism_coordinator _vardiff.PrismCoordinatorVardiffTests.test_idle_retarget_delivers_fresh_bundle_when_cac he_is_disabled)	View Logs
test_idle_retarget_queue_is_globally_bounded (test_prism_coordinator_vardiff.PrismCoord inatorVardiffTests.test_idle_retarget_queue_is_globally_bounded)/ test_idle_retarget_queue_is_globally_bounded (test_prism_coordinator_vardiff.PrismCoord inatorVardiffTests.test_idle_retarget_queue_is_globally_bounded)	View Logs
test_idle_sweep_cache_miss_builds_only_on_bounded_worker (test_prism_coordinator_vardif f.PrismCoordinatorVardiffTests.test_idle_sweep_cache_miss_builds_only_on_bounded_worker )/ test_idle_sweep_cache_miss_builds_only_on_bounded_worker (test_prism_coordinator_vardif f.PrismCoordinatorVardiffTests.test_idle_sweep_cache_miss_builds_only_on_bounded_worker )	View Logs
test_idle_vardiff_failure_restores_pending_and_idle_window (test_prism_coordinator_vard iff.PrismCoordinatorVardiffTests.test_idle_vardiff_failure_restores_pending_and_idle_wi ndow)/ test_idle_vardiff_failure_restores_pending_and_idle_window (test_prism_coordinator_vard iff.PrismCoordinatorVardiffTests.test_idle_vardiff_failure_restores_pending_and_idle_wi ndow)	View Logs
test_idle_vardiff_shutdown_after_delivery_keeps_committed_window (test_prism_coordinato r_vardiff.PrismCoordinatorVardiffTests.test_idle_vardiff_shutdown_after_delivery_keeps_ committed_window)/ test_idle_vardiff_shutdown_after_delivery_keeps_committed_window (test_prism_coordinato r_vardiff.PrismCoordinatorVardiffTests.test_idle_vardiff_shutdown_after_delivery_keeps_ committed_window)	View Logs
test_idle_vardiff_success_sends_paired_job_and_resets_window (test_prism_coordinator_va rdiff.PrismCoordinatorVardiffTests.test_idle_vardiff_success_sends_paired_job_and_reset s_window)/ test_idle_vardiff_success_sends_paired_job_and_resets_window (test_prism_coordinator_va rdiff.PrismCoordinatorVardiffTests.test_idle_vardiff_success_sends_paired_job_and_reset s_window)	View Logs
test_post_accept_notification_does_not_run_failing_template_build (test_prism_coordinat or_vardiff.PrismCoordinatorVardiffTests.test_post_accept_notification_does_not_run_fail ing_template_build)/ test_post_accept_notification_does_not_run_failing_template_build (test_prism_coordinat or_vardiff.PrismCoordinatorVardiffTests.test_post_accept_notification_does_not_run_fail ing_template_build)	View Logs
test_post_confirm_publication_loss_completes_candidate_and_fences (test_prism_coordinat or_vardiff.PrismCoordinatorVardiffTests.test_post_confirm_publication_loss_completes_ca ndidate_and_fences)/ test_post_confirm_publication_loss_completes_candidate_and_fences (test_prism_coordinat or_vardiff.PrismCoordinatorVardiffTests.test_post_confirm_publication_loss_completes_ca ndidate_and_fences)	View Logs
test_repeated_idle_sweeps_do_not_enqueue_duplicate_connection_work (test_prism_coordina tor_vardiff.PrismCoordinatorVardiffTests.test_repeated_idle_sweeps_do_not_enqueue_dupli cate_connection_work)/ test_repeated_idle_sweeps_do_not_enqueue_duplicate_connection_work (test_prism_coordina tor_vardiff.PrismCoordinatorVardiffTests.test_repeated_idle_sweeps_do_not_enqueue_dupli cate_connection_work)	View Logs

^{Need help on this PR? Tag /codesmith with what you need.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 82392ed. Configure here.}

[cursor]

Idle poll publishes payout snapshots

Medium Severity

In ready mode, every tip poll claims an immutable payout snapshot and later calls _mark_payout_snapshot_published even when no prepared ready bundle was built or faned out (use_prepared_fanout is false). That marks an epoch published and can schedule a coalesced follow-up without the audit-bundle validation path used elsewhere, so retained_published may point at a snapshot that never completed a successful job build.

Additional Locations (1)

• lab/prism/prism_coordinator.py#L3776-L3835

 

^{Reviewed by Cursor Bugbot for commit 82392ed. Configure here.}