From f4f6773c12c444eaca23735d80791d8a38f43d91 Mon Sep 17 00:00:00 2001 From: Alex Date: Mon, 20 Jul 2026 10:01:41 -0400 Subject: [PATCH] Document PRISM refactor review stack --- crates/qbit-prism/README.md | 6 + docs/README.md | 3 + docs/prism-coordinator-refactor/README.md | 41 ++++++ .../a1-audit-artifacts.md | 78 +++++++++++ docs/prism-coordinator-refactor/invariants.md | 112 ++++++++++++++++ docs/prism-coordinator-refactor/roadmap.md | 49 +++++++ .../prism-coordinator-refactor/stacked-prs.md | 33 +++++ docs/prism-coordinator-refactor/structure.md | 69 ++++++++++ docs/prism-coordinator-refactor/validation.md | 122 ++++++++++++++++++ docs/prism-ledger-ops.md | 47 ++++++- lab/prism/block_finalization.py | 5 +- 11 files changed, 560 insertions(+), 5 deletions(-) create mode 100644 docs/prism-coordinator-refactor/README.md create mode 100644 docs/prism-coordinator-refactor/a1-audit-artifacts.md create mode 100644 docs/prism-coordinator-refactor/invariants.md create mode 100644 docs/prism-coordinator-refactor/roadmap.md create mode 100644 docs/prism-coordinator-refactor/stacked-prs.md create mode 100644 docs/prism-coordinator-refactor/structure.md create mode 100644 docs/prism-coordinator-refactor/validation.md diff --git a/crates/qbit-prism/README.md b/crates/qbit-prism/README.md index f7929e3..d615ff6 100644 --- a/crates/qbit-prism/README.md +++ b/crates/qbit-prism/README.md @@ -19,9 +19,15 @@ defines: - `qbit_share_ledger`: append-only canonical share log ordered by `share_seq` - `qbit_ledger_writer_lease`: one-row coordination table for a single logical writer/failover epoch +- `qbit_pool_blocks.audit_publication_sequence`: durable confirmation and + reactivation order for current audit evidence, independent of block height - `qbit_prism_window(...)`: deterministic newest-backward window query with partial oldest-share weighting +The schema file is cumulative and idempotent. Existing deployments must rerun +it before upgrading when automatic schema initialization is disabled; see the +[ledger operations contract](../../docs/prism-ledger-ops.md#audit-publication-ordering-migration). + Stratum frontends should enqueue share submissions outside this table. Only the active ledger writer inserts rows into `qbit_share_ledger`; that is what keeps all miners in the same reward universe. diff --git a/docs/README.md b/docs/README.md index 1bb0d28..9ae82f9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -33,6 +33,9 @@ This is useful for reviewers and operators who need implementation detail: writer-lease behavior, compaction contract, and readiness probes. - [prism-capacity-readiness.md](prism-capacity-readiness.md): optional Stratum-to-Postgres qualification artifact and standalone validator contract. +- [prism-coordinator-refactor/README.md](prism-coordinator-refactor/README.md): + completed coordinator decomposition, ownership map, validation evidence, and + stacked-PR publication plan. ## Public-Site Guidance diff --git a/docs/prism-coordinator-refactor/README.md b/docs/prism-coordinator-refactor/README.md new file mode 100644 index 0000000..f6532a2 --- /dev/null +++ b/docs/prism-coordinator-refactor/README.md @@ -0,0 +1,41 @@ +# PRISM Coordinator Refactor + +Status: **complete** and organized as a nine-PR review stack. No required +roadmap item remains. + +The completed tree is integrated with `origin/1.x.x` at `b002caa`. It preserves +the base branch's public hashrate, refresh/livelock, initial/reconnect delivery, +queue-reclamation, and latest-tip priority fixes in the extracted owners. It +also ports the latest retry pacing (`#71`) and delivery-health grace (`#82`) +behavior, plus share hot-path lock isolation (`#83`), into the template, +refresh, candidate, submission, vardiff, delivery, ledger, observability, and +metrics owners. The former exact-hash, literal-authorization, +mandatory-reviewer, and per-slice full-suite workflow is retired. + +## Result + +`lab/prism/prism_coordinator.py` is now the construction, startup, signal, +stable-facade, and top-level shutdown root. Domain state machines, background +loops, queues, locks, cached observability, persistence, HTTP, and mining work +live in dedicated owners. See [Structure](structure.md) for the boundary and +the documented size exception. + +B3 is intentionally omitted: the available evidence does not justify a second +finalization lane or its additional durable handoff. See the +[decision record](b3-decision.md). + +The final runnable validation matrix passes. Docker-dependent PostgreSQL, +container lint/build, and both live Stratum targets are `UNAVAILABLE` in the +current environment because the OrbStack daemon is stopped; `qbitd` is also +absent. These are missing-environment evidence, not passes or product +failures. Exact results are in [Validation](validation.md). + +## Reference documents + +- [Invariants](invariants.md): release behavior that must remain true. +- [Roadmap](roadmap.md): completed slices and decisions. +- [Structure](structure.md): final ownership map and structural audit. +- [Validation](validation.md): risk-based cadence and final evidence. +- [Stacked PRs](stacked-prs.md): publication order and reconstruction rules. +- [A1 audit artifacts](a1-audit-artifacts.md): durable storage contract. +- [B3 decision](b3-decision.md): finalization concurrency evidence. diff --git a/docs/prism-coordinator-refactor/a1-audit-artifacts.md b/docs/prism-coordinator-refactor/a1-audit-artifacts.md new file mode 100644 index 0000000..5323f9e --- /dev/null +++ b/docs/prism-coordinator-refactor/a1-audit-artifacts.md @@ -0,0 +1,78 @@ +# A1 Audit Artifact Contract + +A1 is complete. `lab/prism/audit_artifacts.py` owns audit +filesystem authority; `bundle_compiler.py` owns compiler/subprocess work; the +ledger owns database authorization and accepted-block transactions; the +coordinator only composes them and retains narrow compatibility calls. + +## Directory and file authority + +- Pin audit root and evidence-parent directory descriptors with no-follow + semantics. Perform owned reads, creates, replacement, fsync, scans, and + removals relative to those descriptors. +- A path replacement revokes the store until the original inode is restored or + `reconfigure()` adopts a new authority pair atomically. +- Parse only exact owned names. Malformed lookalikes, symlinks, nonregular + entries, traversal, and out-of-root URIs never grant read, pin, repair, or + deletion authority. +- Track candidate inode identity. Cleanup removes only the exact file created + by the current attempt, including builder-create-then-raise and swapped-path + failures. + +## Compiler and verifier boundary + +J1 receives a duplicated A1 directory descriptor and reserved candidate name, +creates the canonical output relative to it, and transfers the still-open exact +inode to A1. Compatibility builders use canonicalization fallback and are not +reported as exact compiler output. + +Verification uses an unlinked read-only descriptor snapshot, a bounded process +group, timeout, and independent stdout/stderr limits. Trust source, writer key, +literal digest/bytes, and normalized report form the verification identity. A +retry never reuses a prior attempt's success. + +## Publication and replay + +- PostgreSQL assigns a unique durable `audit_publication_sequence` at the + serialized confirmation boundary. Exact confirmation replay and later + inactive/reactivation transitions reuse it; reactivation does not create an + unpublished ordinal. +- Sequence, not height, hash ordering, process order, or mtime, chooses current + evidence. Hash, height, coinbase, digest, and verification identity remain + integrity fields. +- Allocation/publication lock order is payout balance mutation then the A1 + publication guard. The guard uses one pinned internal lock inode with an + in-process reentrant lock plus cross-process `flock`. +- Reload disk evidence while guarded before replay, repair, publish, or prune. + An already-valid exact pair may replay behind a later floor; damaged state + repairs only at the fresh durable-row floor. +- Legacy evidence grants no ordering or pin authority after restart until it is + re-proved against exact confirmed ledger state and adopted at its durable + sequence. + +Mutable publication installs the envelope first and evidence second, fsyncing +file and required parent boundaries. Only then does in-memory current evidence +advance. A failure preserves the previous valid pair. Exact replay is +byte/inode stable except for documented global observational counters. + +## Bodies, segments, and retention + +Inline and external audit bodies expose the same canonical bytes, digest, and +response metadata in memory and PostgreSQL modes. Share-slot merges are +serialized, lossless for disjoint updates, idempotent for identical overlap, +and preserving on conflict. Audit bodies and share segments are durable and not +reference-blind garbage-collected. + +Retention runs after successful publication, is best effort, and revalidates +both directory authorities and current/reserved identities at each removal. +Retention 0/1, tied mtimes, concurrent stores/processes, and prune failure may +not delete or regress current evidence. The internal lock file is never exposed +as an artifact. + +## Completion evidence + +A1 was reconciled with the integrated owners and validated through its direct +artifact/API/ledger/candidate/metrics suites, PostgreSQL parity/migration/process +helpers, Rust audit CLI tests, Docker compile/lint, and diff/temporary-artifact +hygiene. Exact hashes, literal authorization, and same-reviewer verdicts were +intentionally not part of completion. diff --git a/docs/prism-coordinator-refactor/invariants.md b/docs/prism-coordinator-refactor/invariants.md new file mode 100644 index 0000000..7ab6c4c --- /dev/null +++ b/docs/prism-coordinator-refactor/invariants.md @@ -0,0 +1,112 @@ +# Refactor Invariants + +These are release constraints, not process gates. If a proposed boundary +conflicts with one, change the boundary. + +## Architecture + +- `prism_coordinator.py` is the construction, signal, startup, and top-level + shutdown root. Leaf services must not import it. +- Each service owns its mutable state, lock, bounded queue/executor, metrics, + and worker lifecycle. Do not replace the coordinator with a broad context + object. +- Cross-service work that can be superseded carries immutable identity: + connection generation, tip/template generation, payout generation, worker, + difficulty, or durable candidate identity as applicable. +- No socket, RPC, PostgreSQL, subprocess, filesystem, or blocking queue work + runs while a shared coordinator/session lock is held. +- When both are required, a client's vardiff lock is acquired before the + coordinator control lock. Normal share submission takes one immutable + coordinator control snapshot; deduplication and share accounting use their + dedicated owner locks rather than reacquiring the coordinator lock. +- Compatibility delegates may remain only while an in-repository caller needs + them. The implementation body and mutable state remain in one owner. + +## Mining work and delivery + +- Detected tip, published tip, template artifacts, and payout state remain + distinct. Older work cannot overwrite newer observation authority. +- Publication-critical latest-tip work preempts or defers routine/initial work + without allowing concurrent heavy builders. +- `mining.set_difficulty` and its matching `mining.notify` commit as one client + update after final connection, authorization, tip, template, payout, worker, + and difficulty validation. +- Slow or failed sends do not hold global locks or block unrelated clients. +- Clean-job retirement, same-tip retention, stale grace, and initial/reconnect + queue bounds retain their current behavior. +- Vardiff changes become active only with a job stamped at that difficulty; + every failure or supersession restores speculative state. +- Failed tip refreshes use bounded, jittered owner-held pacing tied to the + observed tip. A successful refresh or a newer tip clears the holdoff, and a + same-tip poll may reuse only a still-fresh coherent template snapshot. + +## Shares, candidates, and payout + +- A normal share is acknowledged only after its durable ledger transaction. + Accepted counters and vardiff accounting advance at the same boundary. +- A share and required block-candidate intent commit atomically. The durable + outbox is authoritative; the in-memory queue is only a bounded wakeup. +- Duplicate identity uses the immutable job worker and header. Retry or + reauthorization cannot bypass it. +- Candidate replay is idempotent, keeps parents ahead of dependent children, + distinguishes retryable from terminal outcomes, and cannot double-account a + block or regress published evidence. +- Candidate attempt time is durable before processing. Intentional retry waits + wake in bounded heartbeat slices, expose pending/unattempted age and backoff + state, and keep blocked processing phases eligible for watchdog detection. +- A candidate whose network outcome is known but whose durable outbox + finalization failed resumes finalization with bounded pacing. It does not + resubmit the block, recount acceptance, rebuild audit evidence, or re-adopt + the released share-writer floor. +- Payout generations are monotonic and a stale prepared source cannot publish. + Accepted-block preview, final coinbase, confirmed balances, and delivered + jobs must agree. +- Post-accept and blockwait paths only notify the single refresh owner; they do + not enter the heavy refresh lane themselves. + +## Audit artifacts + +- Owned paths are descriptor-relative, no-follow, strictly parsed, and confined + to pinned directory identities. +- Canonical bytes are verified before durable publication. Writes and mutable + pair replacement are atomic and fsync their required parent boundaries. +- Ledger-assigned audit publication sequence, not height or process order, + decides current evidence across replay, restart, same-height replacement, and + reorg. +- Retention is post-publication best effort and cannot remove a current, + reserved, malformed, symlinked, or unowned entry. +- Audit bodies and share segments remain digest-checked and reconstructable in + memory and PostgreSQL modes. + +## Health, HTTP, and metrics + +- Health deadlines use monotonic time. Pending age starts at the first + unresolved change and does not slide on churn. +- First-job starvation and current-tip coverage loss have separate grace + clocks. Previously delivered clients do not masquerade as first-job + starvation, and restored coverage resets the coverage-loss clock. +- Publication and successful socket delivery are separate proofs. Cached base + health never masks a newer progress failure. +- Publication divergence is cleared only by a coherent completed publication, + including a no-op poll that proves existing work is current. +- Public routes, status codes, schemas, cache headers, metric names/labels, and + environment defaults remain compatible unless a roadmap item explicitly + authorizes a bounded behavior change. +- `/healthz` and `/metrics` perform no backend, ledger, artifact, or RPC work + after cached observability is complete. +- Metrics copy share counters under the dedicated accounting lock and expose + coordinator-lock contention plus durable candidate pending/backoff state + through narrow snapshots owned by their source services. +- The coordinator-owned HTTP listener reads complete cached metrics only. The + externally managed compatibility handler may render metrics synchronously + while its cache is still uninitialized because it owns no refresher thread. +- An audit HTTP serve-loop exit closes and retires its owned listener before a + replacement listener can start, including unexpected post-readiness exits. + +## Shutdown and liveness + +Shutdown remains: close admission and signal stop; cancel refresh work; drain +admitted writers; release or deliberately withhold the exact writer lease; +then drain non-writer threads, sockets, and executors. Publication-progress +watchdog protection remains active even when ordinary heartbeat checking is +disabled. diff --git a/docs/prism-coordinator-refactor/roadmap.md b/docs/prism-coordinator-refactor/roadmap.md new file mode 100644 index 0000000..3edcf83 --- /dev/null +++ b/docs/prism-coordinator-refactor/roadmap.md @@ -0,0 +1,49 @@ +# Completed Roadmap + +There is no active work order. Completed commits are implementation history, +not gates that must be reconstructed. + +| Area | Result | +| --- | --- | +| foundations, configuration, RPC, shutdown, executors | extracted | +| progress, background services, CTV | extracted | +| payout, templates, bundles, refresh scheduler | extracted | +| sessions, delivery, share writer/recovery | extracted | +| `origin/1.x.x` through `b002caa` | complete, including retry pacing (`#71`), delivery-health grace (`#82`), and share hot-path lock isolation (`#83`) | +| A1 audit artifact ownership | complete | +| B1 candidate submission | complete | +| S4 share classification | complete | +| D1 bounded duplicate index | complete | +| V1 vardiff and idle retarget | complete | +| B2 accepted-block finalization phases | complete | +| B3 second finalization lane | `OMIT` ([decision](b3-decision.md)) | +| O2 cached health | complete | +| O3 cached metrics | complete | +| H1 audit/public HTTP | complete | +| X1 reorg, metrics, watchdog, and seam cleanup | complete | + +X1 removed unused re-exports and drifting state mirrors, moved the remaining +reorg, metrics, and watchdog domain bodies to owners, and replaced magic +attribute forwarding with explicit ports and test hooks. Compatibility aliases +or delegates remain only where in-repository callers demonstrate the stable +facade; they do not own duplicated mutable state. + +The upstream retry work is owned by `template_artifacts`, `tip_refresh`, and +`block_candidates`; the upstream health correction is owned by `job_delivery`, +`observability`, and `metrics`. The upstream hot-path correction is owned by +`stratum_session`, `job_delivery`, `share_submission`, `vardiff_service`, +`block_candidates`, `share_ledger`, and `metrics`: client vardiff state has a +per-client lock, share accounting has a dedicated lock, normal submission uses +one coordinator control snapshot, and candidate retry state remains observable +without making intentional waits look healthy. Coordinator code only wires +those ports and forwards stable compatibility calls. + +The three cumulative milestones—durability/submission, +concurrency/finalization, and observability/cleanup—are complete. The final +tree satisfies the [invariants](invariants.md), the structural result is +recorded in [Structure](structure.md), and the full evidence is recorded in +[Validation](validation.md). + +The implementation is organized into the review stack in +[Stacked PRs](stacked-prs.md). This document intentionally avoids commit IDs so +restacking cannot make the completion record stale. diff --git a/docs/prism-coordinator-refactor/stacked-prs.md b/docs/prism-coordinator-refactor/stacked-prs.md new file mode 100644 index 0000000..43d6ab5 --- /dev/null +++ b/docs/prism-coordinator-refactor/stacked-prs.md @@ -0,0 +1,33 @@ +# Stacked PR Review Plan + +The refactor is published as nine bottom-up PRs. Every published stack commit +must be GPG-signed. PR 1 targets `1.x.x`; each later PR targets the branch +immediately above it so reviewers see only that slice. + +| PR | Branch | Scope | +| --- | --- | --- | +| [#73](https://github.com/Qbit-Org/qbit-mining-bootstrap/pull/73) | `prism-test-shards` | mechanical test sharding and shared fixtures; no production behavior | +| [#74](https://github.com/Qbit-Org/qbit-mining-bootstrap/pull/74) | `prism-coordinator-core-owners` | configuration/RPC, lifecycle, payout, templates, bundles, refresh, sessions, delivery, share writing, same-tip template reuse, retry pacing, observed coordinator locking, and per-client vardiff lock wiring | +| [#75](https://github.com/Qbit-Org/qbit-mining-bootstrap/pull/75) | `prism-audit-artifact-owner` | PostgreSQL publication order, audit filesystem authority, compilation/verification, replay, retention, and migration/process gates | +| [#76](https://github.com/Qbit-Org/qbit-mining-bootstrap/pull/76) | `prism-block-candidate-submission` | durable candidate replay, attempt marking, bounded retry heartbeats/backoff state, finalize-only replay pacing, terminalization, and coordinator ports | +| [#77](https://github.com/Qbit-Org/qbit-mining-bootstrap/pull/77) | `prism-share-submission` | single-snapshot share classification, bounded duplicate tracking, and dedicated share-accounting synchronization | +| [#78](https://github.com/Qbit-Org/qbit-mining-bootstrap/pull/78) | `prism-vardiff-finalization` | per-client vardiff/idle-retarget synchronization, named accepted-block finalization phases, and the documented B3 decision | +| [#79](https://github.com/Qbit-Org/qbit-mining-bootstrap/pull/79) | `prism-observability-http` | cached health, separate first-job/coverage-loss grace, complete cached metrics, and audit/public HTTP facade | +| [#80](https://github.com/Qbit-Org/qbit-mining-bootstrap/pull/80) | `prism-final-ownership-cleanup` | reorg reconciler, synchronized metrics snapshots including hot-lock/candidate gauges, watchdog owner, compatibility cleanup, and temporary-seam removal | +| [#81](https://github.com/Qbit-Org/qbit-mining-bootstrap/pull/81) | `prism-refactor-documentation` | final roadmap, ownership audit, validation evidence, stack guide, and ledger migration operations | + +Each PR targets the branch immediately above it; PR 1 targets `1.x.x`. Put the +stack position, base branch, dependency, focused validation, and any operator +impact in every PR body. PR 3 must call out the required existing-database +migration and maintenance-window lock; PR 9 carries the durable operator guide. + +Address review feedback from the bottom of the stack upward. Amend each fix +into its owning signed commit, then rebase and sign every descendant so each PR +continues to contain one intentional slice. Push rewritten branches together +with explicit force-with-lease expectations; stop if any remote tip changed. + +Validate focused behavior per PR. Run cumulative PRISM discovery after PR 2, +PostgreSQL/Rust after PR 3, the image build after PR 6, and the complete final +matrix on PR 9. Merge bottom-up. If the repository squash-merges a lower PR, +rebase the remaining branches onto the new `1.x.x` tip and force-push only with +lease. diff --git a/docs/prism-coordinator-refactor/structure.md b/docs/prism-coordinator-refactor/structure.md new file mode 100644 index 0000000..dceb90e --- /dev/null +++ b/docs/prism-coordinator-refactor/structure.md @@ -0,0 +1,69 @@ +# Final Structure + +## Coordinator boundary + +`PrismCoordinator` constructs owners, wires explicit ports, exposes the stable +facade used by existing tests/tools, binds listeners, handles signals, and +orders top-level shutdown. It does not own a background worker loop or a large +domain state machine. + +| Concern | Owner modules | +| --- | --- | +| configuration and RPC | `coordinator_config`, `rpc` | +| lifecycle and bounded execution | `background_services`, `bounded_executor`, `coordinator_shutdown` | +| payout, reorg, and CTV | `payout_state`, `reorg_reconciler`, `ctv_runtime` | +| templates, bundles, refresh, delivery | `template_artifacts`, `bundle_compiler`, `job_bundle`, `tip_refresh`, `job_delivery` | +| sessions, shares, candidates, finalization | `stratum_session`, `share_submission`, `share_writer`, `block_candidates`, `block_finalization` | +| vardiff | `vardiff_service` | +| audit, health, metrics, HTTP | `audit_artifacts`, `observability`, `progress_health`, `metrics`, `audit_http` | + +Owners hold their own mutable state, locks, bounded queues/executors, counters, +and lifecycle. Cross-owner calls use explicit immutable inputs or narrow ports; +there is no broad replacement context object. + +## Structural audit + +- The coordinator fell from 19,799 lines on the integrated base to 8,900 lines, + a 55% reduction. +- No coordinator method at or over 200 lines is a domain state machine. The two + such methods are the 244-line constructor and 202-line delivery-owner factory; + other methods at or over 100 lines are configuration, listener startup, port + assembly, or factory wiring. +- Loop-shaped coordinator facade methods are narrow delegates. The only local + loop is bounded listener-startup retry, not a background worker. +- No leaf service imports the coordinator. `job_build_benchmark.py` is an + executable composition-level benchmark and intentionally constructs it. +- No PRISM module defines magic `__getattr__` or `__setattr__` forwarding. +- Docker Ruff reports no unused imports. Remaining compatibility exports and + descriptors have direct in-repository callers and delegate to one owner. + +The earlier approximate 3,000-line aspiration is not a release requirement. +The remaining size is an explicit compatibility/composition exception: 522 +mostly narrow facade and wiring methods preserve existing private integration +fixtures and public behavior. Moving those methods to mixins would only hide +the composition surface, while deleting demonstrated callers would be a +breaking API/test migration rather than ownership extraction. Future changes +should remove a facade only when its callers can migrate without weakening the +compatibility contract. + +## Optional further decomposition + +There is still useful follow-up work, but it is separate from domain ownership: + +- About 900 lines of `_Coordinator*` port adapters can move to a + `coordinator_adapters` module. This is a low-risk file-size improvement, not a + new ownership boundary. +- The constructor plus 35 lazy `_ensure_*` factories can become a typed + component builder. This would clarify the dependency graph, but it must + preserve lazy startup, shutdown order, and partial test construction. +- Remaining cohesive orchestration can move to existing owners: live chain and + template validation, payout/CTV policy construction, synchronous candidate + credit, idle-bundle cache validation, and progress eligibility assembly. +- The largest reduction requires migrating the many test/tool construction + sites and then deleting their private compatibility delegates and + descriptors. Do this as an explicit compatibility cleanup, not with mixins or + magic forwarding. + +Those changes are optional because current domain state already has one owner. +They should be proposed after the review stack lands so they do not obscure the +behavior-preserving extraction. diff --git a/docs/prism-coordinator-refactor/validation.md b/docs/prism-coordinator-refactor/validation.md new file mode 100644 index 0000000..6df3fe6 --- /dev/null +++ b/docs/prism-coordinator-refactor/validation.md @@ -0,0 +1,122 @@ +# Validation Strategy + +Validation follows risk and invalidation scope. Focused checks run per slice; +broad checks run at cumulative milestones and once more on the final tree. + +## Per slice + +Run: + +```sh +python3 -m py_compile +python3 -m unittest +git diff --check +``` + +Inspect the complete slice diff and confirm no leaf module imports +`lab.prism.prism_coordinator`. Add PostgreSQL, Rust, or process tests immediately +when the slice changes those boundaries; do not defer a directly relevant +failure to a milestone. + +## Cumulative PRISM regression + +Use the current shard names discovered in `tests/`. At minimum include the +candidate, vardiff, retained-job, share-writer, payout, job-builder, metrics, +tip-refresh, publication-boundary, initial/reconnect, hot-path, shutdown, CTV, +progress-health, audit, and public-API suites. The exact module list may grow as +new direct owner suites are added. + +## Full Python discovery + +Full discovery includes fixtures that create temporary Git repositories, so +scope the global-config override to this command only: + +```sh +GIT_CONFIG_GLOBAL=/dev/null \ + python3 -m unittest discover -s tests -p 'test_*.py' +``` + +Never export `GIT_CONFIG_GLOBAL` and never use it for repository commits. + +## PostgreSQL, Rust, and live validation + +Run when a milestone touches persistence, payout, shares, audit, candidates, +vardiff, or finalization: + +```sh +make test-prism-postgres-ledger +cargo test -p qbit-prism +``` + +At the concurrency/finalization and final milestones, run: + +```sh +make test-prism-stratum-regtest-live +make test-prism-stratum-postgres-regtest-live +``` + +If qbitd, qbit-cli, PostgreSQL, Docker, Rust, credentials, or another real +prerequisite is unavailable, record `UNAVAILABLE`, the missing prerequisite, +and the exact unrun command. Do not call it a pass. + +## Docker compile, lint, and build + +Lint only in Docker: + +```sh +docker run --rm \ + -e PYTHONPYCACHEPREFIX=/tmp/pycache \ + -v "$PWD:/work:ro" -w /work \ + python:3.12-slim \ + python -m compileall -q docker lab tests examples scripts + +docker run --rm \ + -v "$PWD:/work" -w /work \ + ghcr.io/astral-sh/ruff:0.12.5 \ + check --select E4,E7,E9,F lab/prism tests + +docker build -f lab/prism/Dockerfile -t qbit-prism-refactor-check . +``` + +Focused slices may lint only their changed Python paths. Milestones lint the +complete PRISM paths. The image build is required after B2 and on the final +tree. + +## Final structural and diff audit + +Before declaring completion: + +- no leaf service imports the coordinator; +- mutable domain state has one owner and no drifting scalar mirror; +- coordinator worker loops and large domain state machines have moved; +- compatibility re-exports/delegates have a demonstrated caller or are gone; +- no blocking I/O occurs under coordinator/session locks; +- queues, executors, cancellation, and shutdown joins are bounded; +- public schemas, metrics, environment defaults, and wire payloads are stable; +- `git diff --check`, status, target-branch diff, and generated-artifact hygiene + are clean; +- cumulative regression, full discovery, PostgreSQL/Rust, Docker lint/build, + and live-or-unavailable evidence all reflect the final tree. + +## Final evidence + +Recorded on 2026-07-22 after X1, integration through `origin/1.x.x` at +`b002caa`, stack reconstruction, upstream hot-path reconciliation, and a +thread-aware review audit showing no unresolved comments: + +| Check | Result | +| --- | --- | +| PRISM Python discovery | 1,251 passed | +| full Python discovery | 1,585 passed; `GIT_CONFIG_GLOBAL=/dev/null` was scoped only to this command | +| PostgreSQL ledger integration | `UNAVAILABLE`; `make test-prism-postgres-ledger` could not reach the stopped OrbStack Docker daemon | +| Rust `qbit-prism` | 182 passed | +| Docker Python compile | `UNAVAILABLE`; Docker daemon stopped | +| Docker Ruff | `UNAVAILABLE`; Docker daemon stopped; lint was not run outside Docker | +| PRISM image | `UNAVAILABLE`; Docker daemon stopped | +| Compose | PRISM target and permissionless, real-miner-smoke, auxpow, and prism profiles passed configuration validation | +| structural and target diff audit | passed; see [Structure](structure.md) | + +`make test-prism-stratum-regtest-live` and +`make test-prism-stratum-postgres-regtest-live` are `UNAVAILABLE`: their doctor +reported that the Docker daemon is not reachable before either live test could +run. `qbitd` and `qbit-cli` are also absent from the host path. diff --git a/docs/prism-ledger-ops.md b/docs/prism-ledger-ops.md index 3454794..5b4bb25 100644 --- a/docs/prism-ledger-ops.md +++ b/docs/prism-ledger-ops.md @@ -106,6 +106,11 @@ exponential delay starting at 250 milliseconds and capped at 30 seconds. They do not increment terminal abandonment counters. Replay carries the database row's block hash separately from candidate JSON, so malformed payloads can be quarantined by their authoritative outbox key instead of replaying forever. +If qbit has already returned the candidate outcome but durable outbox +finalization fails, replay resumes only that finalization step with the same +bounded pacing. It does not call `submitblock` again, recount an accepted block, +rebuild or republish audit evidence, or reacquire a share-writer floor already +released after the known outcome. When a network-valid hash is below a listener's advertised share target, the coordinator first stores a candidate-only intent, submits it synchronously, and @@ -144,9 +149,45 @@ shares. Audit bundles containing a credited row use enable stale-grace crediting. Deployments that run with `PRISM_POSTGRES_INIT_SCHEMA=0` must apply -`crates/qbit-prism/sql/001_share_ledger.sql` before starting upgraded -coordinators. Otherwise share inserts will fail because the `credit_policy` -column and updated window function signatures are missing. +`crates/qbit-prism/sql/001_share_ledger.sql` before starting any upgraded +coordinator. The file is the cumulative, idempotent schema initializer and +migration path, despite its `001` name. Skipping it can break share inserts, +reward-window calls, pool-block confirmation/reactivation, and audit evidence +publication because required columns, functions, and the durable publication +ordinal will be missing. + +### Audit publication ordering migration + +Existing databases must receive the new `audit_publication_sequence` migration. +It creates and validates a bigint sequence, adds the nullable pool-block column, +deterministically backfills confirmed and inactive rows by `found_at` and +`block_hash`, adds a unique index and state constraint, advances the allocator +beyond every retained ordinal, and replaces confirmation/reactivation functions +so each new durable confirmation receives an ordinal. Exact confirmation replay +preserves its prior ordinal. Historical inactive rows are backfilled so later +reactivation can retain that already-published ordinal without allocating a new +audit publication. + +The migration includes `ALTER TABLE` operations that require PostgreSQL's +`ACCESS EXCLUSIVE` table lock. They wait for existing readers and writers and +can interrupt new reads as well as writes while held. A later serialized phase +takes a transaction-scoped advisory lock and a `SHARE ROW EXCLUSIVE` lock on +`qbit_pool_blocks`, and the unique index is built non-concurrently. Treat the +whole migration as read-impacting: stop the old coordinator or use a reviewed +maintenance window, take the normal database backup, and apply the file with +`ON_ERROR_STOP` using the same database role and schema search path as PRISM. +For example: + +```sh +psql "$PRISM_DATABASE_URL" -v ON_ERROR_STOP=1 \ + -f crates/qbit-prism/sql/001_share_ledger.sql +``` + +With `PRISM_POSTGRES_INIT_SCHEMA=1`, coordinator construction applies the same +script before listeners open. The migration is rerunnable and tested across +fresh, legacy, partial, concurrent, malformed, and bigint-boundary states; it +fails closed instead of accepting a conflicting sequence, column, index, or +constraint definition. `qbit_shares_since_template_height(min_template_height)` supports operational replay and frontend recovery. It returns accepted shares at or above the diff --git a/lab/prism/block_finalization.py b/lab/prism/block_finalization.py index 19df4b5..7cb6588 100644 --- a/lab/prism/block_finalization.py +++ b/lab/prism/block_finalization.py @@ -975,8 +975,9 @@ def _publish_finalization_evidence( # before A1 publication. Confirmation-time state or a raw # sequence value cannot fence rollback gaps and restart # replays. P1's local serializer plus A1's process guard - # prevent another confirmation/reactivation from allocating - # between this read and the durable publication decision. + # prevent another confirmation from allocating between this + # read and the durable publication decision. Reactivation + # preserves its block's already-published ordinal. publication_floor_sequence = publication_floor_reader() else: # Compatibility-only ledgers used by legacy embeddings/tests