If your app holds more than one LLM API key, something has to decide which key each request uses and which keys are currently rate limited. This is that piece.
dotnet add package LlmKeyPoolI hit this building a batch translation pipeline. Ten Gemini keys, a hundred thousand chapters to push through them. The first version retried on 429 with exponential backoff and got nowhere, because a key that returned "quota exceeded: requests per day" is not going to succeed 8 seconds later. It needs to sit out until Google's daily reset, and the request needs a different key right now.
That turns out to be a different job from retrying:
retry-only: key A → 429 → 1s → 429 → 2s → 429 → 4s → 429 → give up
key pool: key A → 429 "RPD" → cool A until 00:00 PT → key B → 200
Polly does the first job well. It has no opinion about credentials, which is correct for Polly and leaves this gap. The two compose: the pool picks the key, Polly wraps the call.
Register a pool:
builder.Services.AddLlmKeyPool("gemini", pool =>
{
pool.Provider = "gemini";
pool.AddKeys(builder.Configuration["Gemini:ApiKeys"]!); // CSV, newlines or semicolons
pool.FailurePolicy = KeyFailurePolicies.Gemini;
pool.Selector = KeySelectors.LeastRecentlyUsed;
pool.MaxConcurrencyPerKey = 4;
});Then hook it to an HttpClient and you are done. Every request gets a key, and a rate
limited response is retried on a different one:
builder.Services.AddHttpClient("gemini", c => c.BaseAddress = new Uri("https://generativelanguage.googleapis.com/v1beta/"))
.AddLlmKeyPool("gemini", o =>
{
o.Placement = KeyPlacement.Header("x-goog-api-key"); // or BearerToken(), QueryString("key")
o.MaxKeyAttempts = 3;
})
.AddStandardResilienceHandler(); // optional: Polly, inside each key attemptCalling code does not change and does not know the pool exists:
var response = await http.PostAsJsonAsync("models/gemini-2.5-flash:generateContent", body, ct);The handler returns the last failed response instead of throwing, which is what you expect
from HttpClient. Note that with MaxKeyAttempts > 1 the request body is buffered so it
can be replayed on the next key.
If you are not on HttpClient, or you want control over what counts as a failure:
var text = await pool.ExecuteAsync(async (lease, ct) =>
{
using var request = new HttpRequestMessage(HttpMethod.Post, endpoint)
{
Content = JsonContent.Create(payload),
};
request.Headers.Add("x-goog-api-key", lease.Key.Value);
using var response = await http.SendAsync(request, ct);
if (!response.IsSuccessStatusCode)
{
var body = await response.Content.ReadAsStringAsync(ct);
lease.ReportFailure(response, body);
response.EnsureSuccessStatusCode(); // throwing is what triggers the rotation
}
var result = await response.Content.ReadFromJsonAsync<Result>(ct);
lease.ReportSuccess(result!.Usage.TotalTokens);
return result.Text;
}, cancellationToken: ct);If the delegate throws, the pool classifies the error, cools the key for however long that particular failure warrants, and runs the delegate again on the next key. Returning a value ends the call, so throw when you want to rotate.
Pass the response body if you have it. For several providers the quota dimension only appears in the error text, and without it every 429 looks the same.
Most of the value is here, in IKeyFailurePolicy. The order is: Retry-After if present,
then provider reset headers, then a wait hint in the error text, then a default for whichever
dimension the message named.
The dimensions are treated differently on purpose:
- Per minute. Trust the provider's number. When Gemini says "Please retry in 12.5s" that is accurate and rounding it up to a minute wastes the key.
- Per day. Ignore a short hint and wait for the actual reset.
Retry-After: 30on a spent daily budget is wrong, and Gemini's daily quota resets at midnight Pacific, not UTC and not your local time. - Not stated. Start conservative and double on a streak. Guessing short here is how you end up hammering a dead key every 60 seconds for an hour.
Other cases: insufficient_quota disables the key instead of cooling it, because waiting
does not refill a prepaid balance. 401 disables. A quota-shaped 403 cools, since some
providers report a spent budget that way. 5xx and transport faults rotate without blaming
the key. 400 and 404 fail immediately, because a malformed request fails the same way on
every key and there is no reason to burn the pool rediscovering your own bug.
Profiles ship for Generic, Gemini and OpenAI. Write your own with QuotaPolicyProfile,
or mix providers in one pool with CompositeFailurePolicy.
KeySelectors.RoundRobin is the default. LeastRecentlyUsed spaces out hits on any one key,
which is what you want against RPM-limited free tiers. LeastLoaded goes by in-flight count.
One implementation note, because I got it wrong the first time: the round-robin cursor is reserved when the candidate order is built, not when a response comes back. If you advance it on completion, every concurrent caller reads the same value before any response lands and they all pile onto the same key while the rest of the pool sits idle.
app.MapGet("/keypool", (IKeyPoolProvider pools) => pools.Get("gemini").Inspect());key status quota resumes in ok/err tokens
AIza~4f2c1d9a cooling RPD 6h 12m 412/3 1.2M
AIza~9a30bb17 cooling TPM 18s 388/1 980K
AIza~7c1e0d55 available - - 401/0 1.1M
AQ.A~d1b93f60 disabled Auth - 0/1 0
Keys are identified by a fingerprint: four characters of prefix, enough to tell the provider
apart, plus a hash. KeyState has no secret in it and ApiKey.ToString() is masked, so a
status endpoint or a stray log line cannot leak a credential.
A cooldown that lasts until tomorrow is useless if a redeploy wipes it, since the next request walks straight back into an empty budget.
pool.PersistTo(new FileKeyStateStore("state/gemini-keys.json"));Snapshots are keyed by fingerprint and carry no key material, so the file is safe on a mounted volume. Writes land on a temp file and are moved into place, so a crash mid-write leaves the previous snapshot intact. Restoring can extend a cooldown but never shorten one.
A Redis store for sharing health across processes is next.
The pool emits under the meter and activity source LlmKeyPool:
builder.Services.AddOpenTelemetry()
.WithMetrics(m => m.AddMeter(KeyPoolTelemetry.Name))
.WithTracing(t => t.AddSource(KeyPoolTelemetry.Name));Counters for rents, cooldowns (tagged by quota class) and disables, a histogram of how long a rent waited, and gauges for available keys and in-flight requests. Key ids stay out of metric tags, since 50 keys would mean 50 time series per instrument. The id goes on the trace span instead.
Every time calculation goes through TimeProvider, so tests drive the clock instead of
sleeping:
var time = new FakeTimeProvider(start);
// cool a key for 8 hours
time.Advance(TimeSpan.FromHours(8));
Assert.Equal(KeyStatus.Available, pool.Inspect()[0].Status);123 tests covering day-long windows and DST transitions run in about 250 ms.
Two runnable samples in samples/, both working against a stub provider so they need no API key and no network:
- Worker — batch job that deliberately runs the pool dry, showing rotation, a revoked key
being disabled, and
EarliestAvailableAttelling the job when to wake up. - API — minimal API exposing
GET /keypoolfor key health and returning503with aRetry-Afterheader when every key is cooling.
It does not send requests or parse responses. Your provider SDK does that. It does not retry inside a single key, since Polly already does. It does not track cost or route between models. It is a library, not a gateway process.
Requires .NET 8 or 9. Only depends on Microsoft.Extensions.* abstractions.
This is for apps that legitimately hold several keys: one per tenant, separate dev and prod projects, multiple Azure OpenAI deployments, a paid key with a free backup. Check your provider's terms before pooling keys. Nothing here makes a quota bigger.
MIT