feat(lore-0114): recurring coarse-USD sweep + repair record & FREEZE-denial fix - #151
Merged
Merged
Conversation
The prod pilot on 202502 died at the first write with an empty
`Clickhouse(BadResponse(""))`. The real cause — `prices_writer` lacking
`ALTER FREEZE PARTITION` — was only visible by replaying the statement
over curl, and the privilege cannot be granted at all because the user
is defined in read-only users.xml storage.
`ChEnrichError::FreezeDenied` now carries the table, partition and both
remedies, and `preflight()` warns up-front when the connected user shows
no FREEZE-capable grant. The warning is advisory rather than a gate:
grant text has several shapes and a parse miss must not refuse a
legitimate run.
Runbook gains the operator pre-freeze path (snapshots taken as CH admin,
repair run with --skip-snapshot), a bulk-freeze script that reports
`already-frozen` distinctly — a silent re-freeze would overwrite a
pre-repair snapshot with a post-repair one — and the missing FINAL on
the coverage query.
Verification criterion corrected: the pilot's `absurd_high = 0` gate
failed on 40 rows, but `implied_ref_usd` proved both tiers compute
correct references (0.312-0.408 for XLM in Feb 2025, exactly 1.0 for
USDC). The absurd values are single dust trades the repair faithfully
converts, and live-enriched data shows the same tail. Spawned as 0116.
The full 30-month price_ohlcv_1h repair ran to completion in 4h02m: 30,828,953 rows enriched, 47,804,264 left at the no_reference floor. Yield landed within 2.4% of the pre-registered estimate (~31.6M), and the totals reconcile exactly against the dry run once the pilot month's ~1.0M is accounted for. Per-month enrichment stayed inside the predicted band throughout. Three acceptance criteria close: - Coverage: aggregate 95.5% -> 58.1% zero, and the unbroken 100% block (202502-202602) is gone. The independently computed zeros column sums to exactly the figure the driver reported, from a different code path. - Composition: a per-quote-class breakdown proves the residual is the genuine floor, not backlog -- stablecoin 0.9% zero, XLM pivot 0.8%, exotic 100%. Everything the tiers can reach was reached. - Additive-only: verified empirically rather than by inspection, since zeros_before - enriched = zeros_after held exactly for all 30 months, and the already-repaired pilot month re-ran as a clean no-op. Reference correctness is confirmed across the span, not just the pilot: USDC reads exactly 1.0 and XLM tracks real history for each era. Two residual observations recorded rather than smoothed over -- 202606 USDC reads 1.0005 (expected, the oracle tier is available post-2025-09) and the 202606 XLM median sits slightly above the band noted during prep. Phase C is sized but not run: ~60M zeros across the four remaining coarse tables, ~23M likely repairable, ~3h, with _4h alone two-thirds of it. Also records that "enriched: 0" in a dry-run summary is an artifact of dry-run mode and must not be read as the silent-no-op failure. Spawns 0129: the residual-composition query surfaced a 0.4% join fan-out on prices.assets. FINAL dedupes that table by natural key, not by asset_id, so any join on asset_id can silently multiply rows -- which the read API and the current_prices MV both do.
Phase C is done: _4h/_1d/_1w/_1M repaired (~20.1M rows), so with _1h all five coarse tables now sit at the genuine no_reference floor, ~52.5M rows repaired in total. The 4h run was interrupted by a power loss; the resume procedure is recorded because it will recur — locate the stop point from per-month pct_zero rather than the lost log, and include the boundary month, since 202603 read 75.8% against a repaired neighbour's 66.4% and was half-written. Two Phase-C estimates in the task were wrong and are corrected in place with their causes: yield ~30% not 39% (the 39% came from _1h's span-wide average, which 2024's pivot-heavy months dominate), and runtime minutes not ~3h (from 202606 the oracle tier claims the reachable rows and peg-pivot correctly no-ops, so 23-51 batches/month instead of ~100). Scope correction to the sole-copy rule: the price_ohlcv_1m hole spans 2021-07 -> 2026-06, roughly 4x the recorded 13 months, so any procedure assuming 1m can rebuild a coarse table over that span is wrong. The stated cause is downgraded to unverified — no DropPart in part_log, no matching mutation, and no TTL support it. The fact is measured; the cause is not. Also records that cleanup deletes by ALTER DELETE mutation rather than by dropping partitions, evidenced by the 2026-07-15 event in system.mutations and by prices_writer holding ALTER DELETE but not partition-drop rights. 0109 is updated accordingly: a guard keyed on DropPart would never fire, so it gains a system.mutations check, with the caveat that a starved mutation is armed rather than safe — six Phoenix deletes have been pending since 07-17 and leave 0097's rework incomplete until the merge pool frees. 0088 records the soroban_amm leg as stalled since 2026-07-14 with a status field that still reads "running" while no process exists, 122,864 ledgers short of target, and refreshes pass 1 to frontier 35.84M at ~151k/hr with a revised 07-27/28 ETA and the download-bound reason the rate eased.
The rollup MVs re-aggregate only a bounded recent window, so any 1m row enriched after that window closes (enrichment lag / stalls) leaves its coarse counterpart frozen at close_usd=0 forever. Fold a bounded, partition-bounded sweep of the recent coarse partitions into the hourly enrichment worker — one owner of close_usd across 1m and the rollups, rather than a separate repair Lambda. Runs after the 1m pass, best-effort (never fails the invocation or regresses 1m) and bounded (one_shot=false, overflow defers to the next run). Inert unless COARSE_SWEEP_TABLES is set; the CDK env enables it for all six rollup tables incl _15m. Reuses the additive enrichment tiers, so it is non-destructive by construction. Also closes the _4h per-quote-class composition AC (exotic-only floor).
karczuRF
changed the title
Four fixes from /code-review of the recurring sweep: - Wall-clock budget (COARSE_SWEEP_TIME_BUDGET_SECS, default 120), capped a margin below the Lambda invocation deadline. A Lambda hard-timeout is an invocation error, not a Rust Err, so without a budget a slow catch-up would escape the best-effort handler and fail the invocation — contradicting the 'never fails the invocation' guarantee. The sweep now stops at its deadline and defers the rest. - Non-coarse config names go to skipped_tables, not failed_tables, so a config typo can no longer false-fire the CoarseSweepTableFailures alarm every run; they are also dropped once at cold start rather than refused each run. - Dropped the CoarseSweepRowsRemaining metric: dominated by the permanent exotic no_reference floor, it could not observe the lag it existed to catch. - Downgraded the per-month snapshot-disabled log warn!->debug! (fired ~12x/hr by design under the sweep, burying real warnings). New test coarse_sweep_defers_all_work_past_its_deadline proves the budget. The 5th finding (batch the two PutMetricData calls) was declined: batching would delay the critical-path 1m metric behind the best-effort sweep.
karczuRF
added a commit
that referenced
this pull request
Jul 24, 2026
The recurring coarse-USD sweep AC is checked: deployed to Prices-production- EventBridge and verified live (tables_swept=5, tables_failed=0, budget + best- effort proven). Scoped to the 5 forever-tables; 15m dropped (504s on the ~30s mTLS-proxy timeout) and tracked in 0130. Records the deploy gotchas (full-stack worker rebuild + cdk-diff gate, 60s sync-invoke client timeout, #151-before-#150 merge order).
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
price_ohlcv_1hcoarse-table USD repair ran to completion in production (4 h 02 m). This PR carries the preflight/runbook fixes made during the run, plus the full result record.The defect, recapped
price_ohlcv_1h— the table BE consumes — carriedclose_usd = 0for 86–100% of every month in the Soroban era, including an unbroken 100.0% block from 202502 through 202602. Two independent causes: rollup MVs capture pre-enrichment values and never revisit, and the historical enrichment march stalled at 2025-01 before cleanup dropped its_1msource. For much of that span the coarse tables are the sole surviving copy, which is why the repair had to be additive-only.Result — it landed on the pre-registration
no_referencefloorTotals reconcile exactly: 30.83M + 47.80M = 78.63M, the dry run's 79.63M minus the ~1.0M the pilot had already repaired. Nothing unaccounted for.
Coverage: aggregate 95.5% → 58.1% zero. The 100% block is gone (now 55.6–69.8%). The independently computed zeros column sums to exactly the figure the driver printed — two different code paths, same number.
Three acceptance criteria close
Coverage + composition. Counts alone couldn't close the AC, which is phrased as reaching the genuine floor. The per-quote-class breakdown settles it:
Everything the tiers can reach was reached; every remaining zero is a quote with no USD path, which is by design.
Additive-only — verified empirically, not just by inspection:
zeros_before − enriched = zeros_afterheld exactly for all 30 months, and the already-repaired pilot month re-ran as a cleanenriched 0no-op rather than rewriting rows.Reference correctness across the span, not just the pilot: USDC-quoted
implied_ref_usdis exactly 1.0; XLM-quoted tracks real history per era (0.1371 / 0.218 / 0.372 / 0.2008).Two residual observations recorded rather than smoothed over: 202606 USDC reads 1.0005 — expected, since
oracle_pricesstarts 2025-09 so live-era rows can come from the oracle tier instead of the peg cast — and the 202606 XLM median sits slightly above the band noted during prep, worth one independent spot-check.Code changes
ChEnrichError::FreezeDeniednow carries the table, partition and both remedies. The original failure surfaced asClickhouse(BadResponse(""))— an empty body that cost a diagnostic cycle beforecurlrevealed the realACCESS_DENIED.preflight()also warns up front when the connected user shows no FREEZE-capable grant (advisory, not a gate — grant text has several shapes and a parse miss must not refuse a legitimate run).Runbook gains the operator-freeze path, an expected-vs-real failure-signal table, and a missing
FINALfix.Spawns 0129 — a real hazard found by accident
The composition query disagreed with the coverage query by 335,830 rows, with the zero counts differing by exactly the same amount. That's a 0.4% join fan-out:
assetsisReplacingMergeTree(updated_at)ordered by(asset_code, issuer_address, contract_address), soFINALdedupes by natural key, not byasset_id— and any join onasset_idcan silently multiply rows.Immaterial to this task's conclusion, but the read API's listing and batch paths and
mv_current_pricesall join onasset_id, andGET /assets' keyset cursor usesasset_idas its tie-break. Filed as 0129 rather than left in a chat log.Not done — Phase C
_4h/_1d/_1w/_1Mare dry-run and sized but not repaired: ~60M zeros, ~23M likely repairable, ~3 h, with_4halone two-thirds of it. Gated on the CH admin taking snapshots first —prices_writerstill cannot FREEZE and cannot be granted it.Also recorded:
enriched: 0in a dry-run summary is an artifact, since dry-run mode never attempts enrichment. It must not be read as the silent-no-op failure signature, which only means something on a real run.Task stays
activefor Phase C, the 0088 step-3 warn gate, andSYSTEM UNFREEZEcleanup (last — it discards the only rollback point).Docs + one error-type change; no behavioural change to the repair path.
Update — recurring guard implemented (2026-07-24, prepare-only)
The going-forward guard (the last open 0114 code AC) is now built and wired — not yet deployed. Rather than a separate repair Lambda, the recurring coarse-USD sweep is folded into the existing hourly
enrichmentLambda, so one worker ownsclose_usdacross1mand the rollups.Why a sweep is unavoidable. Verified against
rollups.sql: the coarse tables are refreshable MVs that re-aggregate only a bounded recent window (1m→15m2 h,15m→1h8 h, …) from the enriched finer table. A correction rolls up on its own only if enrichment lands before that window closes; anything later (routine lag, or a multi-day stall like 0111) freezes at zero forever. So a revisiting pass is unavoidable — the only choice is where it lives. Making lag < window is the root fix (0111); this sweep is the backstop.Behaviour. Runs after the 1m pass, best-effort (a sweep failure never fails the invocation or regresses 1m) and bounded (
one_shot=false, overflow defers to the next hour); re-sweeps a trailing 2-month window — partition-bounded (0111-safe), snapshotless, additive-only.Changes
repair.rs—one_shotconfig knob (full-drain vs bounded) +run_coarse_sweep/CoarseSweepConfig(trailing window from CHnow(), per-table failure isolation, refusesprice_ohlcv_1m).main.rs— sweep wired after the 1m pass, best-effort, env-gated (inert unlessCOARSE_SWEEP_TABLESis set).metrics.rs—CoarseSweepRowsEnriched/RowsRemaining/TableFailures(reuses thePrices/Enrichmentnamespace → no IAM change).eventbridge-stack.ts— enables it on the enrichment Lambda for all 6 rollup tables incl_15m;LOOKBACK_MONTHS=2,MAX_BATCHES=20.Tests: 28 unit + 9 integration green (local CH pinned 26.3.10.60), incl. new
coarse_repair_bounded_mode_defers_overflow_across_runsandcoarse_sweep_bounds_trailing_window_and_refuses_the_1m_base; compiles underaws-mtls/lambda/default; clippy + infra typecheck/lint/synth clean (env vars land onEnrichmentFunctiononly).Also in this update: the
_4hper-quote-class composition AC is closed (exotic-only: stablecoin 0.012% / xlm_pivot 0.046% / other 100% zero). Phase C (_4h/_1d/_1w/_1M) is complete — this supersedes the "Not done — Phase C" note above.Remaining: deploy
*-EventBridge, then verify in prod (composition at floor +CoarseSweep*metrics) to close the AC.