Skip to content

Skip past un-scannable blob names so bloated named-blob containers still make progress - #3291

Merged
zichengl merged 1 commit into
linkedin:masterfrom
zichengl:zichengl/named-blob-cleaner-skip-ahead
Aug 20, 2026
Merged

Skip past un-scannable blob names so bloated named-blob containers still make progress#3291
zichengl merged 1 commit into
linkedin:masterfrom
zichengl:zichengl/named-blob-cleaner-skip-ahead

Conversation

@zichengl

Copy link
Copy Markdown
Contributor

Summary

Lets the named-blob stale-data cleanup runner make forward progress on a container that has a single blob name with more live versions than can be scanned under the database long-transaction limit.

Problem

The cleaner's page query pages by blob name (... blob_name >= ? ORDER BY blob_name ASC, version DESC LIMIT ?). That mixed-order sort cannot be served by the PK index (account_id, container_id, blob_name, version), so MySQL filesorts the entire [cursor, end-of-container] range before LIMIT applies — its cost is O(range), independent of page size. If one blob name has enough live versions, the scan is killed before returning any row, the cursor never advances, and the container makes zero progress across runs. Shrinking the page (already in place) doesn't help, because the sort reads the whole range regardless of LIMIT.

Change

  • Add NamedBlobDb.getFirstBlobName(container, from) — a cheap index-only point lookup (blob_name >= ? ORDER BY blob_name, version LIMIT 1) implemented for MySQL and the in-memory DB.
  • On a shrunk-page kill, the runner looks up the offending blob name and advances the cursor just past it (stuckBlobName + "\0"), resuming after it. The rest of the container is still cleaned; the skipped blob name is left to a targeted reap / TTL / version cap (a blob too bloated to scan won't be fixed by scanning it again).
  • Skips per container per run are bounded (MAX_BLOB_NAME_SKIPS_PER_RUN) so a broadly bloated container can't spin on many killed scans in one run; remaining skips happen on later runs, which resume from the saved cursor. If there's nothing to skip to, the container is deferred (cursor preserved) rather than dropped.
  • New BlobNameSkippedCount metric.

Testing

NamedBlobsCleanupRunnerTest: a container whose first blob name is fatally bloated (full + shrunk scans killed) is skipped past via getFirstBlobName, and the region after it is scanned. Existing kill/defer/resume tests are unchanged (when there is nothing to skip to, the runner still defers and preserves the cursor).

…ogress

When a container's stale-blob page scan is killed by the DB long-transaction
limit even after shrinking the page, the offending blob name has more live
versions than can be sorted under the limit, and shrinking the page does not
help: the scan filesorts the whole [cursor,end] range before LIMIT, so its cost
is independent of page size. Previously the container was deferred at the same
cursor and made zero forward progress across runs.

Add NamedBlobDb.getFirstBlobName(container, from): a cheap index-only lookup
(blob_name >= ? ORDER BY blob_name, version LIMIT 1). On a shrunk-page kill the
runner uses it to advance the cursor just past the stuck blob name and resumes
after it, so the rest of the container is still cleaned. The skipped blob name
is left to a targeted reap / TTL / version cap. Skips per container per run are
bounded (MAX_BLOB_NAME_SKIPS_PER_RUN) so a broadly bloated container cannot spin
on many killed scans in one run; remaining skips happen on later runs, which
resume from the saved cursor. Adds a BlobNameSkippedCount metric and a test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov-commenter

codecov-commenter commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 6.66667% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.74%. Comparing base (52ba813) to head (394badb).
⚠️ Report is 412 commits behind head on master.

Files with missing lines Patch % Lines
...github/ambry/frontend/NamedBlobsCleanupRunner.java 9.52% 19 Missing ⚠️
.../java/com/github/ambry/named/MySqlNamedBlobDb.java 7.69% 12 Missing ⚠️
...ava/com/github/ambry/commons/InMemNamedBlobDb.java 0.00% 11 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3291       +/-   ##
=============================================
- Coverage     64.24%   50.74%   -13.51%     
+ Complexity    10398     8687     -1711     
=============================================
  Files           840      938       +98     
  Lines         71755    80532     +8777     
  Branches       8611     9697     +1086     
=============================================
- Hits          46099    40865     -5234     
- Misses        23004    36275    +13271     
- Partials       2652     3392      +740     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

throw new IllegalStateException("Stale-blob scan for container " + container.getId()
+ " was killed even at the shrunk page size; deferring after " + blobNameSkips + " skips this run", e);
}
String stuckBlobName =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: better to have some log identify if exception happened, like time out exception

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — since this PR was already merged, I put the fix in a follow-up: #3292. The getFirstBlobName().get(timeout) lookup is now wrapped so a TimeoutException/ExecutionException is logged at WARN with the specific cause (plus container id and cursor) before the container is deferred to the next run.

@zichengl
zichengl merged commit 4a7057f into linkedin:master Aug 20, 2026
11 checks passed
zichengl pushed a commit to zichengl/ambry that referenced this pull request Aug 20, 2026
Addresses review feedback on linkedin#3291: the getFirstBlobName().get(timeout) call
that finds the next blob name to skip to could fail (e.g. a query timeout)
without any log identifying what happened. Wrap it and log the specific cause
(TimeoutException / ExecutionException) at WARN before deferring the container
to the next scheduled run. InterruptedException still propagates for clean
shutdown handling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
zichengl added a commit that referenced this pull request Aug 21, 2026
Addresses review feedback on #3291: the getFirstBlobName().get(timeout) call
that finds the next blob name to skip to could fail (e.g. a query timeout)
without any log identifying what happened. Wrap it and log the specific cause
(TimeoutException / ExecutionException) at WARN before deferring the container
to the next scheduled run. InterruptedException still propagates for clean
shutdown handling.

Co-authored-by: Zicheng Liu <zicliu@linkedin.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants