Skip past un-scannable blob names so bloated named-blob containers still make progress - #3291
Conversation
…ogress When a container's stale-blob page scan is killed by the DB long-transaction limit even after shrinking the page, the offending blob name has more live versions than can be sorted under the limit, and shrinking the page does not help: the scan filesorts the whole [cursor,end] range before LIMIT, so its cost is independent of page size. Previously the container was deferred at the same cursor and made zero forward progress across runs. Add NamedBlobDb.getFirstBlobName(container, from): a cheap index-only lookup (blob_name >= ? ORDER BY blob_name, version LIMIT 1). On a shrunk-page kill the runner uses it to advance the cursor just past the stuck blob name and resumes after it, so the rest of the container is still cleaned. The skipped blob name is left to a targeted reap / TTL / version cap. Skips per container per run are bounded (MAX_BLOB_NAME_SKIPS_PER_RUN) so a broadly bloated container cannot spin on many killed scans in one run; remaining skips happen on later runs, which resume from the saved cursor. Adds a BlobNameSkippedCount metric and a test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3291 +/- ##
=============================================
- Coverage 64.24% 50.74% -13.51%
+ Complexity 10398 8687 -1711
=============================================
Files 840 938 +98
Lines 71755 80532 +8777
Branches 8611 9697 +1086
=============================================
- Hits 46099 40865 -5234
- Misses 23004 36275 +13271
- Partials 2652 3392 +740 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| throw new IllegalStateException("Stale-blob scan for container " + container.getId() | ||
| + " was killed even at the shrunk page size; deferring after " + blobNameSkips + " skips this run", e); | ||
| } | ||
| String stuckBlobName = |
There was a problem hiding this comment.
nit: better to have some log identify if exception happened, like time out exception
There was a problem hiding this comment.
Good call — since this PR was already merged, I put the fix in a follow-up: #3292. The getFirstBlobName().get(timeout) lookup is now wrapped so a TimeoutException/ExecutionException is logged at WARN with the specific cause (plus container id and cursor) before the container is deferred to the next run.
Addresses review feedback on linkedin#3291: the getFirstBlobName().get(timeout) call that finds the next blob name to skip to could fail (e.g. a query timeout) without any log identifying what happened. Wrap it and log the specific cause (TimeoutException / ExecutionException) at WARN before deferring the container to the next scheduled run. InterruptedException still propagates for clean shutdown handling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses review feedback on #3291: the getFirstBlobName().get(timeout) call that finds the next blob name to skip to could fail (e.g. a query timeout) without any log identifying what happened. Wrap it and log the specific cause (TimeoutException / ExecutionException) at WARN before deferring the container to the next scheduled run. InterruptedException still propagates for clean shutdown handling. Co-authored-by: Zicheng Liu <zicliu@linkedin.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Lets the named-blob stale-data cleanup runner make forward progress on a container that has a single blob name with more live versions than can be scanned under the database long-transaction limit.
Problem
The cleaner's page query pages by blob name (
... blob_name >= ? ORDER BY blob_name ASC, version DESC LIMIT ?). That mixed-order sort cannot be served by the PK index(account_id, container_id, blob_name, version), so MySQL filesorts the entire[cursor, end-of-container]range beforeLIMITapplies — its cost is O(range), independent of page size. If one blob name has enough live versions, the scan is killed before returning any row, the cursor never advances, and the container makes zero progress across runs. Shrinking the page (already in place) doesn't help, because the sort reads the whole range regardless ofLIMIT.Change
NamedBlobDb.getFirstBlobName(container, from)— a cheap index-only point lookup (blob_name >= ? ORDER BY blob_name, version LIMIT 1) implemented for MySQL and the in-memory DB.stuckBlobName + "\0"), resuming after it. The rest of the container is still cleaned; the skipped blob name is left to a targeted reap / TTL / version cap (a blob too bloated to scan won't be fixed by scanning it again).MAX_BLOB_NAME_SKIPS_PER_RUN) so a broadly bloated container can't spin on many killed scans in one run; remaining skips happen on later runs, which resume from the saved cursor. If there's nothing to skip to, the container is deferred (cursor preserved) rather than dropped.BlobNameSkippedCountmetric.Testing
NamedBlobsCleanupRunnerTest: a container whose first blob name is fatally bloated (full + shrunk scans killed) is skipped past viagetFirstBlobName, and the region after it is scanned. Existing kill/defer/resume tests are unchanged (when there is nothing to skip to, the runner still defers and preserves the cursor).