Revert the anchor-scoped payout-artifact reuse line (#103, #102) - #104
Merged
Conversation
…s production share rates (#102)" This reverts commit fcc2c3e. Anchor-scoped reuse livelocked mainnet payouts on 2026-07-29 (born-expired artifact livelock); together with the #103 revert this restores the payout-artifact machinery to its #101 (e2fb33d) state: exact-count fence semantics, no staleness/rearm/anchor-age knobs, no admission-deadline machinery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6aca4fa5f2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
18363
to
+18365
| if floor_ms is None: | ||
| return issued_at_ms - 1 | ||
| return min(issued_at_ms - 1, floor_ms - 1) | ||
| return issued_at_ms | ||
| return min(issued_at_ms, floor_ms - 1) |
There was a problem hiding this comment.
When a share is stamped after this clamp but within the same millisecond, returning the inclusive issued_at_ms lets that share fall inside the declared audit window even though it was absent from the snapshot because its commit was still pending. A later qbit_audit_share_window replay then includes the share and cannot reproduce the published bundle; the accepted-count fences do not close this race because the commit can land after their final read. Preserve the one-millisecond subtraction even when no pending floor is currently visible.
Useful? React with 👍 / 👎.
djh58
added a commit
that referenced
this pull request
Aug 3, 2026
…witch (#107) * Revert "Revert the anchor-scoped payout-artifact reuse line (#103, #102) (#104)" This reverts commit 1c474c5. * Make payout-artifact reuse event-driven below the audit ceiling The 2026-07-29 rollback and the 2026-07-30 brownout were the same arithmetic error at different anchors: a wall-clock validity budget shorter than the production interval between consecutive shared builds, on an object whose rebuild pays a multi-second reward-window walk. Round 1 measured the budget from the pre-walk anchor, so every artifact arrived born expired; round 2 moved it to install time, so the budget expired between consecutive builds and every build fell back to the synchronous walk while the re-arm worker walked the same window in parallel. Wall-clock time is no longer a validity input below the audit ceiling: - The reuse probe serves any armed artifact whose payout generation, difficulty, and balances fences hold and whose declared anchor is inside PRISM_PAYOUT_ARTIFACT_MAX_ANCHOR_AGE_SECONDS. Past the new PRISM_PAYOUT_ARTIFACT_REANCHOR_SECONDS floor it schedules the debounced background re-anchor and KEEPS SERVING; post-anchor shares settle in the next window by construction, so a re-anchor is freshness maintenance, not correctness. - PRISM_PAYOUT_ARTIFACT_REUSE_STALENESS_SECONDS is removed. Cached no-artifact ready bundles gate on the re-anchor floor instead (their declared anchor is already ceiling-gated, and the bundle-cache TTL bounds them tighter in production). - PRISM_PAYOUT_ARTIFACT_REUSE (default on) is a master kill-switch: off restores the pre-reuse deployment exactly -- probe refuses, no background walks -- so a production regression is an env flip instead of an emergency rollback (both prior incidents required redeploys). Test changes mirror the semantics: install-age rejection tests become serve-past-install-age brownout regressions, rearm/debounce and landed-preview suppression drive aging through the anchor, in-flight retention ages through the ceiling, and the kill-switch has dedicated coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add reuse probe outcomes, semantic work-currency gauge, slow-wave log Three observability gaps the incident review had to work around: - The reuse probe was invisible: nothing distinguished served reuse from ceiling rejections. served_reuse / probe_rejected_ceiling join the payout-artifact event family (recorded outside the cache lock, matching the family's lock ordering). - current_tip_job_coverage demands template-generation equality and template object identity, so under generation churn it reads 0.0 essentially continuously even while every miner holds semantically current work. The new semantic_current_work_ratio gauge compares template fingerprint and payout generation only. The strict gauge is untouched -- it drives mining-health gating and stays fail-closed. - Slow waves logged nothing attributable: latency had to be reconstructed by pairing tip-observation lines with "refreshed N client job(s)" lines, which cannot see superseded passes. Waves over 1s now log outcome, refreshed count, pass count, and elapsed time in one line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Flush worker-thread build phases into the exported counters The job-build executor and payout-artifact preparation workers never pass through the per-client delivery entry points that clear and flush the thread-local phase dict, so every wave-level phase they accrued -- template, ledger, assembly, bundle, payout_artifact -- exported as exactly 0.0 forever despite continuous template fetches and multi-second window walks. The bundle-build cost had to be discovered from the separate tip_refresh_bundle_phase histograms and ad-hoc profiling because this family was blind. _flush_job_build_phases folds phase seconds only (build counts and the duration histogram stay per-delivery) and clears the dict so a long-lived worker cannot double-report an earlier request's accruals. Wired at the two worker boundaries: _execute_job_build_request and each _payout_artifact_preparation_loop iteration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Harden the reuse re-land per adversarial review Review verdicts: the revert purity, round-2 diagnosis, serve-window correctness, test archaeology, and zero-regression claims survived; the kill-switch equivalence claim was refuted as written, and the review surfaced one genuinely dangerous miss plus four smaller ones. This commit addresses all of them: - Startup validation (the dangerous miss): a ceiling at or below what the background re-anchor can beat re-creates the 2026-07-30 walk-per-build shape with a single env var. __init__ now refuses MAX_ANCHOR_AGE <= 2x REANCHOR via validate_payout_artifact_age_bounds, with direct test coverage. - Kill-switch now gates ARMING too: _install_payout_ledger_artifact, the payout-publication pointer swap, and sync-build window seeding all no-op while disabled. Previously artifacts kept arming with the switch off, churning install events and re-keying the idle-bundle fast path to a generation the probe refuses -- a permanent idle cache miss in the disabled state. Comments and the PR description no longer claim the disabled state matches the pre-reuse deployment "exactly": it restores pre-reuse delivery economics while the synchronous path keeps the re-landed anchor-selection semantics. - prior_balances_sha256 is memoized on the artifact at construction and re-derived at the accepted-preview balance patch; the reuse probe and both reused-build fences stop paying O(accounts) JSON canonicalization per serving decision. The restamp-race test now drives the un-memoized fallback path it was written to protect. - The idle issuance path enforces the anchor ceiling like every other NEW serving decision (it was the one route that could issue a window past the ceiling), and the probe docstring states the in-flight-retention exception instead of overselling the ceiling. - probe_past_floor joins the event family so the canary can distinguish serving-past-floor from never-crossing-the-floor; rearm_scheduled alone under-counts through debounce/suppression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #103 (
cd32c13) and #102 (fcc2c3e) from 1.x.x, in reverse order, as two plaingit revertcommits. The result is tree-identical to the #101 squash commit (e2fb33d) — the state mainnet has been pinned to since the second rollback.Why: two production incidents in two days
2026-07-29 — born-expired artifact livelock (#102). At production reward windows (100k+ shares) the window walk takes multiple seconds, and #102 measured reuse freshness from the wall-clock snapshot anchor — stamped before the walk — so every artifact arrived already past the 10s staleness bound and was discarded as born expired. Installs unconditionally reset the re-arm backoff, so the single preparation worker re-walked the reward window continuously while holding the prepare lock every synchronous ready build needs, driving tip refreshes from 1–6s to a 27s mean and parking initial jobs into the 90s client timeout; the deploy was emergency-rolled-back after ~15 minutes.
2026-07-30 — generation-churn brownout (#103 re-land). The re-land moved reuse freshness to install time and re-gated cached bundles on artifact-generation currency, but under production template-generation churn those validity paths kept superseding and rebuilding otherwise-valid artifacts, and job delivery browned out. It was emergency-rolled-back the same day, and mainnet has run pinned to #101 (
e2fb33d) since.Decision
The anchor-scoped payout-artifact reuse line is abandoned on 1.x.x rather than re-landed a third time. This PR removes all #102/#103 behavior:
PRISM_PAYOUT_ARTIFACT_REUSE_STALENESS_SECONDS,PRISM_PAYOUT_ARTIFACT_MAX_ANCHOR_AGE_SECONDS, orPRISM_ROUTINE_ADMISSION_DEADLINE_SECONDSknobspayout_artifact_events_totalmetrics, no admission-deadline machinery (JobBuildAdmissionDeadlineExceeded)What stays
PR #99's serve-daemon/spool plumbing is not part of this revert: it predates #102, is dormant without publishable artifacts (the pre-#102 state), and ran harmlessly in prod under #101 for two days. Note the two remaining
PRISM_PAYOUT_ARTIFACT_REARM_*names inlab/prism/prism_coordinator.py(rebuild-pacing floor and backoff cap) are #99's, introduced in5a0b1fcand present ine2fb33d— they are part of the pinned mainnet state, not #102/#103 residue.Validation
git diff --exit-code e2fb33d HEAD— empty; tree-identical to the PRISM: converge tip refreshes on latest epochs #101 pinpython3 -m unittest discover -s tests -p 'test_*.py'— 1347 tests, OK (3 skipped)cargo test -p qbit-prism— all greenREUSE_STALENESS/MAX_ANCHOR_AGE/admission-deadline/payout_artifact_events_total/ anchor-age / install-age references remain inlab/ortests/🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Note
High Risk
This is a large revert of payout-artifact reuse and job-build admission paths in the prism coordinator—production-critical coordination logic that already caused two emergency rollbacks when the replaced line shipped.
Overview
Reverts the anchor-scoped payout-ledger-artifact reuse line (#102/#103) and restores the pre-incident coordinator behavior pinned on mainnet after the 2026-07-29/30 rollbacks.
Payout ledger artifact validity and install ordering again use accepted-share counts (with bracketed capture per template generation via
_job_build_anchor_counts), not snapshot-anchor age or install-time staleness. Background and synchronous builds abort when a pending commit clamps the anchor, when before/after share counts disagree mid-read, or when the live count no longer matches the anchor-scoped count—rather than publishing anchor-exact windows under continuous share traffic.Removed: env knobs
PRISM_PAYOUT_ARTIFACT_REUSE_STALENESS_SECONDS,PRISM_PAYOUT_ARTIFACT_MAX_ANCHOR_AGE_SECONDS, andPRISM_ROUTINE_ADMISSION_DEADLINE_SECONDS; born-expired / anchor-ceiling install paths;payout_artifact_events_totalmetrics and JSON lifecycle logging;JobBuildAdmissionDeadlineExceededand routine admission deadline parking; cached-bundle freshness gates tied to build age or declared anchor age.Adjusted:
_job_snapshot_anchor_msno longer forces anchors strictly below the clamp instant when no pending floor is held; in-flight reuse re-validation calls_usable_payout_ledger_artifactagain instead of only generation/balances fences.Reviewed by Cursor Bugbot for commit 6aca4fa. Bugbot is set up for automated code reviews on this repo. Configure here.