Skip to content

client, server: let callers opt out of the claim on queue RPCs - #120

Merged
erikhortsch merged 10 commits into
mainfrom
erik/skip-claim
Aug 14, 2026
Merged

client, server: let callers opt out of the claim on queue RPCs#120
erikhortsch merged 10 commits into
mainfrom
erik/skip-claim

Conversation

@erikhortsch

@erikhortsch erikhortsch commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

A queue subscription selects exactly one server before any claim exists, so on a queue RPC the claim handshake only ratifies a decision the bus already made. The server bids a hardcoded affinity = 1, and the caller sets AcceptFirstAvailable for every non-affinity method, so nothing can change the outcome. It costs two extra fire-and-forget publishes per call, and the claim response goes to SRV.<service>.<topic> — no .Q — so every server registered for the topic receives and discards one per request.

This is bus-independent: SubscribeQueue is exclusive on NATS (queue group), Redis (SetNX on the payload hash), and local (round-robin), so the redundancy is not specific to any broker.

1. skip_claim

Request gains skip_claim (field 9), set by the caller whenever the RPC is queue-routed. The server honors it only when its own RequestInfo also says queue:

if h.i.RequireClaim {
    if ir.SkipClaim && h.i.Queue {
        ...
    } else if claimed, err := h.claimRequest(s, ctx, ir, req); ...
}

That re-check matters on the redis path specifically. Channel.Server embeds .Q iff Queue, so on NATS a client/server disagreement over queue-ness puts them on different subjects and the request simply never arrives — fail-closed. Channel.Legacy, which redis uses, has no queue component, so both land on the same channel and a skew would otherwise let several plain subscribers each honor skip_claim and run the handler.

2. Affinity is rejected on a queue RPC, not worked around

Trying to set affinity on a queue is a broken config. Only 1 server will ever get the request. Trying to configure affinity on a queue now results in an error.

Registration (newRPCHandler, the single choke point for RPC handlers): an affinity function on a queue method is rejected with InvalidArgument. The generator emits one only for Routing_AFFINITY, so generated code cannot produce the pair, but RegisterHandler is exported and nothing checked. Its decline path (affinity < 0) would drop the request with no response. Stream handlers never use queue subscriptions and are unaffected.

Per request (newRPC): SelectionFunc, MinimumAffinity and MaximumAffinity on a queue method are rejected the same way. AcceptFirstAvailable and AffinityTimeout are excluded, since getRequestOpts defaults both in for every non-affinity method.

Note this also surfaces an existing silent failure: MinimumAffinity > 1 on a queue method makes the claim unacceptable forever, so the RPC always fails with no explanation. That is now an error at the call.

No call site in the livekit tree is affected: handlers are registered only from generated code, and of the five WithSelectionOpts call sites, two are on StartEgress (affinity, queue=false) and three set only AcceptFirstAvailable/AffinityTimeout.

3. A skipped claim is still observable

Skipping emitted no lifecycle event, so a migrated queue RPC looked identical to no traffic at all. ClaimOutcome gains ClaimSkipped, reported with a zero wait since nothing was negotiated, and only where RequireClaim is set so methods that never claimed do not inflate the count.

It rides the existing claim_wait_time_ms{service,method,outcome} series from livekit/protocol#1687granted decaying to zero against skipped rising is the migration, per method. protocol needs no change: PSRPCMetricsObserver.OnClaim already labels by outcome.String().

Mixed fleets need no deploy ordering

The caller still registers its claim channel and still answers a claim if one arrives, so:

behavior
old caller + new server skip_claim defaults false, server claims — unchanged
new caller + old server field ignored, server claims, caller answers it — unchanged
new caller + new server claim skipped, 4 hops become 2

The permissiveness is what makes this safe. Not registering the claim channel when skip_claim is set would reintroduce a break where the server times out at request expiry and silently returns nil.

Incidental fix, and one behavior change

selectServer previously discarded a successful response that arrived during selection — the arm was commented will only happen with malformed requests and only extracted an error. A response reaching it was consumed and the call ended in ErrNoResponse. It now returns that response to the caller.

Related behavior change, scoped to skip_claim: an error response now returns immediately rather than being held until the selection timeout. The only response that can precede a claim is MalformedRequest; the old accumulate-and-wait exists so that on a broadcast RPC one server failing to deserialize does not lose a good response from another. A queue RPC has one recipient, so the first response is the final answer. Broadcast behavior is unchanged.

Tests

TestSkipClaim asserts via the observer that a queue RPC records ClaimSkipped and runs its handler once, while a broadcast RPC on the same client still records ClaimGranted. It runs on all three buses, since the property comes from SubscribeQueue rather than any one broker.

TestQueueRejectsAffinityFunc and TestQueueRejectsAffinitySelection cover the two validations, including that the broadcast case still accepts an affinity function and that the defaulted-in selection options are not rejected.

Two selectServer unit tests cover both mixed-fleet directions: skip_claim set with a claim arriving must honor the claim; skip_claim set with a response arriving must hand it back.

Worth noting the existing suite never registered a queue=true method — every RegisterMethod call passes false for the queue argument — so the queue request path had no end-to-end coverage before this.

Reviewer notes

  • internal/internal.pb.go carries a protoc header change (v4.23.4v5.28.2). Regenerated with the repo's pinned protoc and protoc-gen-go v1.36.4 to match the existing file; the pinned protoc has moved since the file was last generated. Worth regenerating with whatever CI pins.
  • No generator change — getRequireClaim is untouched, so generated services need no regeneration. Making the skip a codegen decision instead would break both mixed-fleet directions, one of them silently.

🤖 Generated with Claude Code

erikhortsch and others added 5 commits August 11, 2026 15:20
A queue subscription selects exactly one server before any claim exists,
so on a queue RPC the handshake only ratifies a decision the bus already
made. It costs two extra fire-and-forget publishes per call, and the
claim response is broadcast to every server registered for the topic.

Request gains skip_claim. The caller sets it when the RPC is queue-routed
and no selection option depends on affinity; a server honors it only when
it also sees a queue method with no affinity function, so a version skew
over queue-ness cannot let every server run the handler.

The caller still registers its claim channel and still answers a claim if
one arrives, so a server that predates the field keeps working and no
deploy ordering is required. selectServer returns the response that
arrives instead of a claim rather than discarding it -- previously a
successful response reaching that select was dropped and the call ended
in ErrNoResponse.

No generator change, so generated services are untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The routing type is either QUEUE or AFFINITY, so generated code never
pairs them, but RegisterHandler is exported and nothing checked. The
pairing is incoherent: the queue has already chosen the server, so there
is nothing to arbitrate, and an affinity function returning < 0 drops the
request with no response.

newRPCHandler is the single choke point for rpc handlers, so validating
there covers every path. Stream handlers never use queue subscriptions
and are unaffected.

This makes the affinityFunc conjunct in the skip_claim guard dead, so
drop it: a queue method can no longer carry an affinity function.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A queue rpc has one candidate bidding a hardcoded 1, so SelectionFunc,
MinimumAffinity and MaximumAffinity have nothing to select on -- the same
incoherence as an affinity function, but set per request rather than at
registration. Reject them instead of silently declining to skip the
claim, which leaves skip_claim as just i.Queue.

AcceptFirstAvailable and AffinityTimeout are excluded: getRequestOpts
defaults both in for every non-affinity method. No call site in the
livekit tree sets the rejected fields on a queue rpc.

Comments throughout the change reduced to one or two lines stating why.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Skipping emitted nothing, so a queue rpc looked identical to no traffic
and the rollout had no positive signal. ClaimSkipped keeps that in the
existing claim_wait_time_ms series, where granted decaying to zero
against skipped rising is the migration.

Reported only where RequireClaim is set, so methods that never claimed
do not inflate the count, and with a zero wait since nothing was
negotiated. protocol needs no change: its OnClaim already labels by
outcome.String().

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@erikhortsch
erikhortsch requested a review from a team August 11, 2026 22:59
@erikhortsch
erikhortsch marked this pull request as ready for review August 11, 2026 22:59
erikhortsch and others added 3 commits August 12, 2026 08:07
skip_claim is only sound if SubscribeQueue delivers to exactly one
subscriber, which is a property of the bus, not of psrpc. A bus written
elsewhere cannot be assumed to provide it, and silently skipping there
would run the handler on every subscriber.

ExclusiveQueuer is an optional interface, so a bus that does not know
about it keeps the claim by default rather than opting out of safety.
The three built-in buses declare it; a bus that wraps another must
forward it, which testBus now does.

Client and server each consult their own bus, and
WithClientAlwaysClaim forces the handshake back on for a bus that
declares more than it delivers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Construction-time config is not a kill switch: turning it off meant
redeploying every client. Both options now take a func consulted per
request, so it can be wired to whatever dynamic config the caller
already has and revoked without a deploy.

Unset means claim, so upgrading psrpc changes nothing until a caller
opts in. Client and server hold independent switches; because a caller
that asked to skip still answers a claim, revoking either side is safe
at any time and in any order.

The bus capability is still ANDed in -- it is a safety property, not a
policy knob, so no flag can enable skipping on a bus that has not
declared an exclusive queue.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The opt-in flag already carries the requirement: whoever enables skipping
is asserting their bus delivers a queue subscription to one subscriber,
so detecting it separately was redundant machinery.

Revoking on the client is enough to stop every server skipping, since a
server only honors a flag the caller sets, so the server option was a
second lever for the same switch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
erikhortsch and others added 2 commits August 14, 2026 08:16
Skipping the handshake entirely was wrong: selectServer bounds the wait
by the selection timeout, 1s by default, while a request may run to the
full timeout. Any queue handler slower than that failed with
ErrNoResponse despite succeeding, which TestSkipClaimSlowHandler now
covers.

The server still publishes a ClaimRequest, marked Handling, so the caller
can tell a slow handler from a request nobody received. It no longer
waits to be granted, and the caller no longer sends the grant, so this
costs three hops rather than four. A publish failure fails the request
rather than handling it anyway, so the caller cannot time out and retry
work this server already ran.

selectServer returns a selection now, which drops the skipClaim
parameter: an announcement is recognized from the wire, and a successful
response arriving during selection can only come from a server that never
waited to be granted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
handleRequest is back to its original shape, and the claim protocol lives
in one function: whether to announce is a boolean, it selects the field on
the outgoing ClaimRequest, and the announcing path returns early instead
of registering a response channel and waiting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@erikhortsch
erikhortsch merged commit c7c1207 into main Aug 14, 2026
5 checks passed
@erikhortsch
erikhortsch deleted the erik/skip-claim branch August 14, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants