Survive production build latency in payout-artifact reuse (2026-07-29 re-land) - #103
Merged
Merged
Conversation
… re-land) The 2026-07-29 deploy of #102 rolled back after ~15 minutes: at 100k+ share reward windows the window walk takes multiple seconds, and reuse freshness measured from the wall-clock snapshot anchor -- which is stamped BEFORE the walk -- declared every artifact born expired on arrival. Installs unconditionally reset the re-arm backoff, so the single preparation worker re-walked the reward window continuously while holding the prepare lock every synchronous ready build needs; generation churn superseded in-flight builds and the cached-bundle anchor gate declared rebuilt bundles dead once their template generation outlived the bound. Tip refreshes went 1-6s -> 27s mean, initial jobs parked silently to the 90s client timeout, and the artifact lifecycle logged nothing. Fixes, per the incident brief: - Reuse freshness is measured from install (prepared_monotonic), never from the anchor: PRISM_PAYOUT_ARTIFACT_REUSE_STALENESS_SECONDS (default 10s). The wall-clock anchor bound becomes a separate, deliberately loose audit ceiling (PRISM_PAYOUT_ARTIFACT_MAX_ANCHOR_AGE_SECONDS, now 300s) on how far a served window may trail the live ledger. - Born-expired installs (anchor past the ceiling after a slow walk) are discarded and pace the re-arm backoff like failed preparations -- _install_payout_ledger_artifact returns arming success, and only success resets the backoff -- with a warning log and metric. - Generation bumps no longer scrap valid work: the in-build re-validation checks only the payout generation and the published balances fence (the selected window stays audit-reproducible at its declared anchor), and cached ready bundles gate on artifact-generation currency or built_monotonic freshness instead of the wall-clock anchor, which is frozen per template generation and predates the walk. - The routine/initial admission park behind publication priority is bounded by PRISM_ROUTINE_ADMISSION_DEADLINE_SECONDS (default 15s): expiry raises JobBuildAdmissionDeadlineExceeded (a TemplateRefreshBlocked, so callers retry with normal pacing), logs, and counts under initial_job_prepared_work_total. - Artifact lifecycle observability: structured single-line JSON logs for build/install/refresh/discard/born-expired/re-arm with durations, ages, and window sizes, plus a new qbit_prism_payout_artifact_events_total counter family. Existing metric names unchanged. Tests close the gap that let this through -- nothing had modeled walk duration against the bound: a slow-walk build (clock advanced past the freshness budget mid-walk) must still arm, serve reuse, and reset backoff; a glacial walk past the ceiling must pace backoff without arming; a generation bump mid-build must not scrap the build; rebuilt bundles must serve under template generations older than the budget; the admission deadline must fail fast and visibly. Full unittest suite: failure set byte-identical to base fcc2c3e in this environment (20 pre-existing sandbox failures, zero regressions). cargo test -p qbit-prism: 182 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e6710b791e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
PR #103 round-1 review (Cursor + Codex P1): prepared_monotonic was stamped at build completion, but the freshness budget must run from the moment the window becomes reusable. - Equal-anchor re-proves now credit freshness: a pending-commit floor pins the snapshot anchor, so a fence-failure rebuild re-proves the same window at the same anchor; without the credit the prep worker re-walked the reward window (resetting backoff each round as an already-current success) until the audit ceiling. - The atomic payout publication restamps the candidate artifact at its pointer-swap install: the delivery-gate drain between candidate build and publication can outlive the reuse budget, and a freshly published generation must never arm an already-stale artifact. - The ordinary install path restamps too, covering the sync-seeded artifact whose construction happens mid-bundle-build, before the audit builder runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
@codex review |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 41b1dbe. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41b1dbe8f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Round-3 review fixes: Cursor: an equal-window freshness restamp (already_current/refreshed) replaces the armed artifact object while a reuse probe hashes balances outside the cache lock, so the probe failed closed on object identity and fell through to a synchronous reward-window walk without scheduling a re-arm -- routine under a pinned pending-commit floor, where the restamp is the intentional recovery path. The probe now re-reads the armed slot and admits the restamped copy as the same armed window (same generation, window sha, and balances object; freshness and anchor only move forward under a restamp) while any real re-key still fails closed. Codex P1: the atomic payout publication armed the candidate ledger artifact with no anchor-age admission, so a candidate whose declared anchor crossed the audit ceiling during construction plus the delivery-gate drain armed born-expired: every reuse probe rejected it on anchor age while accepted-preview durability suppressed the re-arm, forcing repeated synchronous walks. Publication now applies the same born-expired rule as _install_payout_ledger_artifact -- discard, count born_expired, log with during_publication -- and the existing post-publication probe schedules recovery. Both fixes carry regression tests that fail against the previous commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9360079217
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex round-3 P2: the atomic payout publication arms the candidate artifact through its own pointer swap, bypassing _install_payout_ledger_artifact, so successful publication installs never incremented the installed lifecycle counter or emitted the install log -- the critical publication path was invisible to the observability this change introduced. Thread the successful arm out of the locked section and record the installed event plus lifecycle log (tagged during_publication) after the coordinator locks release, the same shape as the born-expired publication path. The regression test fails against the previous commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
RobertClarke
added a commit
that referenced
this pull request
Jul 30, 2026
* Revert "Survive production build latency in payout-artifact reuse (2026-07-29 re-land) (#103)" This reverts commit cd32c13. The re-land brownouted mainnet on 2026-07-30 (generation-churn brownout); the payout-artifact reuse line is abandoned and 1.x.x returns to the #101 (e2fb33d) behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revert "Make payout-ledger-artifact reuse anchor-scoped so it survives production share rates (#102)" This reverts commit fcc2c3e. Anchor-scoped reuse livelocked mainnet payouts on 2026-07-29 (born-expired artifact livelock); together with the #103 revert this restores the payout-artifact machinery to its #101 (e2fb33d) state: exact-count fence semantics, no staleness/rearm/anchor-age knobs, no admission-deadline machinery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
djh58
added a commit
that referenced
this pull request
Aug 3, 2026
…witch (#107) * Revert "Revert the anchor-scoped payout-artifact reuse line (#103, #102) (#104)" This reverts commit 1c474c5. * Make payout-artifact reuse event-driven below the audit ceiling The 2026-07-29 rollback and the 2026-07-30 brownout were the same arithmetic error at different anchors: a wall-clock validity budget shorter than the production interval between consecutive shared builds, on an object whose rebuild pays a multi-second reward-window walk. Round 1 measured the budget from the pre-walk anchor, so every artifact arrived born expired; round 2 moved it to install time, so the budget expired between consecutive builds and every build fell back to the synchronous walk while the re-arm worker walked the same window in parallel. Wall-clock time is no longer a validity input below the audit ceiling: - The reuse probe serves any armed artifact whose payout generation, difficulty, and balances fences hold and whose declared anchor is inside PRISM_PAYOUT_ARTIFACT_MAX_ANCHOR_AGE_SECONDS. Past the new PRISM_PAYOUT_ARTIFACT_REANCHOR_SECONDS floor it schedules the debounced background re-anchor and KEEPS SERVING; post-anchor shares settle in the next window by construction, so a re-anchor is freshness maintenance, not correctness. - PRISM_PAYOUT_ARTIFACT_REUSE_STALENESS_SECONDS is removed. Cached no-artifact ready bundles gate on the re-anchor floor instead (their declared anchor is already ceiling-gated, and the bundle-cache TTL bounds them tighter in production). - PRISM_PAYOUT_ARTIFACT_REUSE (default on) is a master kill-switch: off restores the pre-reuse deployment exactly -- probe refuses, no background walks -- so a production regression is an env flip instead of an emergency rollback (both prior incidents required redeploys). Test changes mirror the semantics: install-age rejection tests become serve-past-install-age brownout regressions, rearm/debounce and landed-preview suppression drive aging through the anchor, in-flight retention ages through the ceiling, and the kill-switch has dedicated coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add reuse probe outcomes, semantic work-currency gauge, slow-wave log Three observability gaps the incident review had to work around: - The reuse probe was invisible: nothing distinguished served reuse from ceiling rejections. served_reuse / probe_rejected_ceiling join the payout-artifact event family (recorded outside the cache lock, matching the family's lock ordering). - current_tip_job_coverage demands template-generation equality and template object identity, so under generation churn it reads 0.0 essentially continuously even while every miner holds semantically current work. The new semantic_current_work_ratio gauge compares template fingerprint and payout generation only. The strict gauge is untouched -- it drives mining-health gating and stays fail-closed. - Slow waves logged nothing attributable: latency had to be reconstructed by pairing tip-observation lines with "refreshed N client job(s)" lines, which cannot see superseded passes. Waves over 1s now log outcome, refreshed count, pass count, and elapsed time in one line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Flush worker-thread build phases into the exported counters The job-build executor and payout-artifact preparation workers never pass through the per-client delivery entry points that clear and flush the thread-local phase dict, so every wave-level phase they accrued -- template, ledger, assembly, bundle, payout_artifact -- exported as exactly 0.0 forever despite continuous template fetches and multi-second window walks. The bundle-build cost had to be discovered from the separate tip_refresh_bundle_phase histograms and ad-hoc profiling because this family was blind. _flush_job_build_phases folds phase seconds only (build counts and the duration histogram stay per-delivery) and clears the dict so a long-lived worker cannot double-report an earlier request's accruals. Wired at the two worker boundaries: _execute_job_build_request and each _payout_artifact_preparation_loop iteration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Harden the reuse re-land per adversarial review Review verdicts: the revert purity, round-2 diagnosis, serve-window correctness, test archaeology, and zero-regression claims survived; the kill-switch equivalence claim was refuted as written, and the review surfaced one genuinely dangerous miss plus four smaller ones. This commit addresses all of them: - Startup validation (the dangerous miss): a ceiling at or below what the background re-anchor can beat re-creates the 2026-07-30 walk-per-build shape with a single env var. __init__ now refuses MAX_ANCHOR_AGE <= 2x REANCHOR via validate_payout_artifact_age_bounds, with direct test coverage. - Kill-switch now gates ARMING too: _install_payout_ledger_artifact, the payout-publication pointer swap, and sync-build window seeding all no-op while disabled. Previously artifacts kept arming with the switch off, churning install events and re-keying the idle-bundle fast path to a generation the probe refuses -- a permanent idle cache miss in the disabled state. Comments and the PR description no longer claim the disabled state matches the pre-reuse deployment "exactly": it restores pre-reuse delivery economics while the synchronous path keeps the re-landed anchor-selection semantics. - prior_balances_sha256 is memoized on the artifact at construction and re-derived at the accepted-preview balance patch; the reuse probe and both reused-build fences stop paying O(accounts) JSON canonicalization per serving decision. The restamp-race test now drives the un-memoized fallback path it was written to protect. - The idle issuance path enforces the anchor ceiling like every other NEW serving decision (it was the one route that could issue a window past the ceiling), and the probe docstring states the in-flight-retention exception instead of overselling the ceiling. - probe_past_floor joins the event family so the canary can distinguish serving-past-floor from never-crossing-the-floor; rearm_scheduled alone under-counts through debounce/suppression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident
The 2026-07-29 mainnet deploy of #102 (
fcc2c3e) rolled back after ~15 minutes of impaired initial delivery (no lost blocks/shares, zero tracebacks). Root causes, verified against the shipped code:snapshot_anchor_msis stamped before the reward-window walk; at 100k+ share windows the walk takes multiple seconds, so freshness measured from the wall-clock anchor rejected artifacts on arrival. The fatal detail: a born-expired build still "succeeded", and install unconditionally reset the re-arm backoff to 1 — the single prep worker re-walked the reward window continuously, holding_payout_state_prepare_lock, which every synchronous ready build also needs. Tip refreshes went 1–6s → 27s mean; build time exceeded tip cadence → supersede/retry livelock. (Pre-Make payout-ledger-artifact reuse anchor-scoped so it survives production share rates #102, the removed pending-commit abort fired before the walk and backoff grew to 80s — the "broken" fence was accidentally protecting the critical path.)Fixes (per the incident brief's requirements)
prepared_monotonic) via newPRISM_PAYOUT_ARTIFACT_REUSE_STALENESS_SECONDS(default 10s) — never from the anchor.PRISM_PAYOUT_ARTIFACT_MAX_ANCHOR_AGE_SECONDSbecomes the separate, deliberately loose audit ceiling (default now 300s) on how far a served window's declared anchor may trail the live ledger; shares stamped after an anchor deterministically belong to the next window, so nothing is lost._install_payout_ledger_artifactreturns arming success and only success resets the re-arm backoff; an artifact whose anchor already crossed the ceiling at install is discarded with a warning log and aborn_expiredmetric, and paces the backoff exactly like an aborted build.built_monotonicfreshness inside the audit ceiling, never on the tight wall-clock knob.PRISM_ROUTINE_ADMISSION_DEADLINE_SECONDS(default 15s) withJobBuildAdmissionDeadlineExceeded(aTemplateRefreshBlocked, so callers retry with normal pacing), a rate-limited log line, and anadmission_deadlineresult inqbit_prism_initial_job_prepared_work_total.payout_artifact_built/build_aborted/installed/refreshed/discarded/born_expired/rearm_scheduled) with durations, anchor ages, and window sizes, plus a newqbit_prism_payout_artifact_events_totalcounter family. All existing metric names unchanged.Interim note from the brief:
PRISM_PAYOUT_ARTIFACT_MAX_ANCHOR_AGE_SECONDS=300alone would have made the old design self-consistent; this PR lands that ceiling and the structural fixes so the freshness knob no longer interacts with build latency at all.Validation
fcc2c3ein the sandbox (20 pre-existing environment failures — splice/spool syscalls, postgres, deploy-pin, signal formatting; zero regressions).cargo test -p qbit-prism: 182 passed.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Note
High Risk
Changes core payout-ledger artifact reuse, job-bundle admission, and publication arming in
prism_coordinator—the same path that caused a mainnet rollback; mistakes could stall initial job delivery or hold payout preparation locks again.Overview
Re-lands structural fixes for the 2026-07-29 payout-artifact rollback: reuse freshness no longer ties to wall-clock anchor age during slow reward-window walks.
Freshness split: New
PRISM_PAYOUT_ARTIFACT_REUSE_STALENESS_SECONDS(default 10s) gates reuse fromprepared_monotonicat install;PRISM_PAYOUT_ARTIFACT_MAX_ANCHOR_AGE_SECONDSis loosened to a 300s audit ceiling on declared anchor lag only. Born-expired installs are discarded, log/metricborn_expired, and pace re-arm backoff like aborted builds instead of resetting it.Work preservation: In-flight bundle builds re-validate only payout generation and published-balances SHA—not armed-slot generation bumps. Cached ready bundles use artifact-generation currency or
built_monotonicinside the staleness budget, not tight anchor-age on the template’s frozenissued_at_ms.Admission & ops: Routine/initial jobs parked behind publication priority fail at
PRISM_ROUTINE_ADMISSION_DEADLINE_SECONDS(15s) viaJobBuildAdmissionDeadlineExceeded, with counters/logs. Payout-artifact lifecycle adds JSON logs andqbit_prism_payout_artifact_events_total.Tests model walk duration vs bounds and cover slow walks, born-expired backoff, publication-path installs, and admission deadlines.
Reviewed by Cursor Bugbot for commit dacc1db. Bugbot is set up for automated code reviews on this repo. Configure here.