Stop preloading per-adapter Prometheus label values - #134
Merged
Conversation
preloadLabelValues walked CoreBidderNames() and called .With() on every adapter metric, so all ~340 bidders compiled into the binary -- 263 in the core list plus 77 aliases registered at config load -- got a full series set created at zero and exported forever, whether or not they ever saw a request. Measured against a real msp-prod registry that is 9,180 of 9,429 series per pod, 97.4%, times 16-140 replicas depending on the deployment. Only 39 bidders are referenced anywhere in our stored requests. SRE is carrying 792,360 series and this is very nearly all of it. Note that disabling the unused adapters in config does not help: preload never consults BidderInfos, so enabled and disabled bidders are preloaded alike. Commented out rather than deleted so a rebase onto upstream keeps the blocks visible and one edit puts them back. The four locals that only fed adapter preloads, and the openrtb_ext import behind CoreBidderNames(), go with them, as do the three metricsDisabled.Adapter* wrappers that would otherwise be no-op conditionals around nothing. What this gives up is the zero baseline, and it is worth being clear that the baseline was allocated backwards: it existed for the ~340 upstream bidders nobody calls and was missing for every msp_* bidder, because those were never in CoreBidderNames(). A bidder that is configured but never called now reads as no data instead of 0. Adapter series for bidders that actually serve are created lazily on first use, as they always were for ours. TestMetricCountGatekeeping divided adapter cardinality by the bidder count, so with nothing preloaded it would pass on zero and guard nothing. Replaced with an assertion that nothing per-adapter is preloaded, which fails if a block is switched back on. Verified it fails by re-enabling adapter_panics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fushouli
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
preloadLabelValueswalksCoreBidderNames()and calls.With()on every adapter metric, so all ~340 bidders compiled into the binary (263 in the core list + 77 aliases registered at config load) get a full series set created at zero and exported forever — whether or not they ever see a request.Measured against a real
msp-prodregistry:Times 16–140 replicas depending on deployment. Only 39 bidders are referenced anywhere in our stored requests. SRE is currently carrying 792,360 series and this is very nearly all of it.
Worth noting:
adapters.<x>.disabled: truedoes not help. Preload never consultsBidderInfos, so enabled and disabled bidders are preloaded alike.Commented, not deleted
So a rebase onto upstream keeps the blocks visible and one edit puts them back. Along with them go the four locals that only fed adapter preloads (
adapterValues,adapterErrorValues,bidTypeValues,cookieValues), theopenrtb_extimport behindCoreBidderNames(), and the threemetricsDisabled.Adapter*wrappers that would otherwise be no-op conditionals around nothing.What this gives up
The zero baseline — and the baseline was allocated backwards. It existed for the ~340 upstream bidders nobody calls, and was missing for every
msp_*bidder, because those were never inCoreBidderNames(). So we're dropping a property we only had where it was worthless.After this, a bidder that is configured but never called reads as no data rather than
0. Bidders that actually serve get their series lazily on first use, exactly as ours always did.sum(rate(...))is unaffected. Worth a skim of the adapter dashboards for panels that assume a series exists.Test
TestMetricCountGatekeepingdivided adapter cardinality by the bidder count, so with nothing preloaded it would pass on a count of zero and guard nothing. Replaced with an assertion that nothing per-adapter is preloaded, which fails loudly if a block is switched back on. Verified it bites: temporarily re-enabling theadapter_panicspreload fails with "Adapter metrics should not be preloaded".go build ./...andgo vet ./...clean;./validate.sh --nofmtpasses except a pre-existing local./exchangefailure (msp_nova: builder not registered— plugin builders aren't registered under plaingo test; fails identically with this change stashed).Follow-up
The parent
msprepo needs a submodule pointer bump to pick this up, then a canary → promote deploy. This also makes ParticleMedia/msp#3077 (a scrape-time relabel doing the same job at ingest) redundant — closing that in favor of this.🤖 Generated with Claude Code