client, server: let callers opt out of the claim on queue RPCs - #120
Merged
Conversation
A queue subscription selects exactly one server before any claim exists, so on a queue RPC the handshake only ratifies a decision the bus already made. It costs two extra fire-and-forget publishes per call, and the claim response is broadcast to every server registered for the topic. Request gains skip_claim. The caller sets it when the RPC is queue-routed and no selection option depends on affinity; a server honors it only when it also sees a queue method with no affinity function, so a version skew over queue-ness cannot let every server run the handler. The caller still registers its claim channel and still answers a claim if one arrives, so a server that predates the field keeps working and no deploy ordering is required. selectServer returns the response that arrives instead of a claim rather than discarding it -- previously a successful response reaching that select was dropped and the call ended in ErrNoResponse. No generator change, so generated services are untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The routing type is either QUEUE or AFFINITY, so generated code never pairs them, but RegisterHandler is exported and nothing checked. The pairing is incoherent: the queue has already chosen the server, so there is nothing to arbitrate, and an affinity function returning < 0 drops the request with no response. newRPCHandler is the single choke point for rpc handlers, so validating there covers every path. Stream handlers never use queue subscriptions and are unaffected. This makes the affinityFunc conjunct in the skip_claim guard dead, so drop it: a queue method can no longer carry an affinity function. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A queue rpc has one candidate bidding a hardcoded 1, so SelectionFunc, MinimumAffinity and MaximumAffinity have nothing to select on -- the same incoherence as an affinity function, but set per request rather than at registration. Reject them instead of silently declining to skip the claim, which leaves skip_claim as just i.Queue. AcceptFirstAvailable and AffinityTimeout are excluded: getRequestOpts defaults both in for every non-affinity method. No call site in the livekit tree sets the rejected fields on a queue rpc. Comments throughout the change reduced to one or two lines stating why. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Skipping emitted nothing, so a queue rpc looked identical to no traffic and the rollout had no positive signal. ClaimSkipped keeps that in the existing claim_wait_time_ms series, where granted decaying to zero against skipped rising is the migration. Reported only where RequireClaim is set, so methods that never claimed do not inflate the count, and with a zero wait since nothing was negotiated. protocol needs no change: its OnClaim already labels by outcome.String(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
skip_claim is only sound if SubscribeQueue delivers to exactly one subscriber, which is a property of the bus, not of psrpc. A bus written elsewhere cannot be assumed to provide it, and silently skipping there would run the handler on every subscriber. ExclusiveQueuer is an optional interface, so a bus that does not know about it keeps the claim by default rather than opting out of safety. The three built-in buses declare it; a bus that wraps another must forward it, which testBus now does. Client and server each consult their own bus, and WithClientAlwaysClaim forces the handshake back on for a bus that declares more than it delivers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Construction-time config is not a kill switch: turning it off meant redeploying every client. Both options now take a func consulted per request, so it can be wired to whatever dynamic config the caller already has and revoked without a deploy. Unset means claim, so upgrading psrpc changes nothing until a caller opts in. Client and server hold independent switches; because a caller that asked to skip still answers a claim, revoking either side is safe at any time and in any order. The bus capability is still ANDed in -- it is a safety property, not a policy knob, so no flag can enable skipping on a bus that has not declared an exclusive queue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The opt-in flag already carries the requirement: whoever enables skipping is asserting their bus delivers a queue subscription to one subscriber, so detecting it separately was redundant machinery. Revoking on the client is enough to stop every server skipping, since a server only honors a flag the caller sets, so the server option was a second lever for the same switch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
paulwe
approved these changes
Aug 14, 2026
Skipping the handshake entirely was wrong: selectServer bounds the wait by the selection timeout, 1s by default, while a request may run to the full timeout. Any queue handler slower than that failed with ErrNoResponse despite succeeding, which TestSkipClaimSlowHandler now covers. The server still publishes a ClaimRequest, marked Handling, so the caller can tell a slow handler from a request nobody received. It no longer waits to be granted, and the caller no longer sends the grant, so this costs three hops rather than four. A publish failure fails the request rather than handling it anyway, so the caller cannot time out and retry work this server already ran. selectServer returns a selection now, which drops the skipClaim parameter: an announcement is recognized from the wire, and a successful response arriving during selection can only come from a server that never waited to be granted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
handleRequest is back to its original shape, and the claim protocol lives in one function: whether to announce is a boolean, it selects the field on the outgoing ClaimRequest, and the announcing path returns early instead of registering a response channel and waiting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A queue subscription selects exactly one server before any claim exists, so on a queue RPC the claim handshake only ratifies a decision the bus already made. The server bids a hardcoded
affinity = 1, and the caller setsAcceptFirstAvailablefor every non-affinity method, so nothing can change the outcome. It costs two extra fire-and-forget publishes per call, and the claim response goes toSRV.<service>.<topic>— no.Q— so every server registered for the topic receives and discards one per request.This is bus-independent:
SubscribeQueueis exclusive on NATS (queue group), Redis (SetNXon the payload hash), and local (round-robin), so the redundancy is not specific to any broker.1.
skip_claimRequestgainsskip_claim(field 9), set by the caller whenever the RPC is queue-routed. The server honors it only when its ownRequestInfoalso says queue:That re-check matters on the redis path specifically.
Channel.Serverembeds.QiffQueue, so on NATS a client/server disagreement over queue-ness puts them on different subjects and the request simply never arrives — fail-closed.Channel.Legacy, which redis uses, has no queue component, so both land on the same channel and a skew would otherwise let several plain subscribers each honorskip_claimand run the handler.2. Affinity is rejected on a queue RPC, not worked around
Trying to set affinity on a queue is a broken config. Only 1 server will ever get the request. Trying to configure affinity on a queue now results in an error.
Registration (
newRPCHandler, the single choke point for RPC handlers): an affinity function on a queue method is rejected withInvalidArgument. The generator emits one only forRouting_AFFINITY, so generated code cannot produce the pair, butRegisterHandleris exported and nothing checked. Its decline path (affinity < 0) would drop the request with no response. Stream handlers never use queue subscriptions and are unaffected.Per request (
newRPC):SelectionFunc,MinimumAffinityandMaximumAffinityon a queue method are rejected the same way.AcceptFirstAvailableandAffinityTimeoutare excluded, sincegetRequestOptsdefaults both in for every non-affinity method.Note this also surfaces an existing silent failure:
MinimumAffinity > 1on a queue method makes the claim unacceptable forever, so the RPC always fails with no explanation. That is now an error at the call.No call site in the livekit tree is affected: handlers are registered only from generated code, and of the five
WithSelectionOptscall sites, two are onStartEgress(affinity,queue=false) and three set onlyAcceptFirstAvailable/AffinityTimeout.3. A skipped claim is still observable
Skipping emitted no lifecycle event, so a migrated queue RPC looked identical to no traffic at all.
ClaimOutcomegainsClaimSkipped, reported with a zero wait since nothing was negotiated, and only whereRequireClaimis set so methods that never claimed do not inflate the count.It rides the existing
claim_wait_time_ms{service,method,outcome}series from livekit/protocol#1687 —granteddecaying to zero againstskippedrising is the migration, per method. protocol needs no change:PSRPCMetricsObserver.OnClaimalready labels byoutcome.String().Mixed fleets need no deploy ordering
The caller still registers its claim channel and still answers a claim if one arrives, so:
skip_claimdefaults false, server claims — unchangedThe permissiveness is what makes this safe. Not registering the claim channel when
skip_claimis set would reintroduce a break where the server times out at request expiry and silently returns nil.Incidental fix, and one behavior change
selectServerpreviously discarded a successful response that arrived during selection — the arm was commentedwill only happen with malformed requestsand only extracted an error. A response reaching it was consumed and the call ended inErrNoResponse. It now returns that response to the caller.Related behavior change, scoped to
skip_claim: an error response now returns immediately rather than being held until the selection timeout. The only response that can precede a claim isMalformedRequest; the old accumulate-and-wait exists so that on a broadcast RPC one server failing to deserialize does not lose a good response from another. A queue RPC has one recipient, so the first response is the final answer. Broadcast behavior is unchanged.Tests
TestSkipClaimasserts via the observer that a queue RPC recordsClaimSkippedand runs its handler once, while a broadcast RPC on the same client still recordsClaimGranted. It runs on all three buses, since the property comes fromSubscribeQueuerather than any one broker.TestQueueRejectsAffinityFuncandTestQueueRejectsAffinitySelectioncover the two validations, including that the broadcast case still accepts an affinity function and that the defaulted-in selection options are not rejected.Two
selectServerunit tests cover both mixed-fleet directions:skip_claimset with a claim arriving must honor the claim;skip_claimset with a response arriving must hand it back.Worth noting the existing suite never registered a
queue=truemethod — everyRegisterMethodcall passesfalsefor the queue argument — so the queue request path had no end-to-end coverage before this.Reviewer notes
internal/internal.pb.gocarries a protoc header change (v4.23.4→v5.28.2). Regenerated with the repo's pinnedprotocandprotoc-gen-gov1.36.4 to match the existing file; the pinned protoc has moved since the file was last generated. Worth regenerating with whatever CI pins.getRequireClaimis untouched, so generated services need no regeneration. Making the skip a codegen decision instead would break both mixed-fleet directions, one of them silently.🤖 Generated with Claude Code