fix(inference): keep a stream retryable until it emits generation - #2219
Open
u9g wants to merge 1 commit into
Open
fix(inference): keep a stream retryable until it emits generation#2219u9g wants to merge 1 commit into
u9g wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: ba1d5ba The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
toubatbrian
approved these changes
Aug 4, 2026
Port of the python fix. retryable was cleared by any chunk reaching the caller, including ones that carry no output: a usage block, or provider metadata such as the gateway's deployment stamp or a Gemini thought signature. Since the gateway stamps its leading (contentless) delta, every streamed response went unretryable from its first chunk -- so a mid-stream stall failed the turn outright rather than retrying, with nothing generated and nothing for the retry to duplicate. Clear retryable only on text or a tool call, the output a retry would actually repeat.
u9g
force-pushed
the
fix/llm-retryable-content-only
branch
from
August 7, 2026 17:55
eaf251f to
ba1d5ba
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of livekit/agents#6697 —
agents/src/inference/llm.tsis a line-for-line equivalent of the Python file and carries the identical bug.What
retryablewas cleared by any chunk reaching the caller, including chunks that carry no output — a usage block, or provider metadata such as the inference gateway's deployment/tier stamp or a Gemini thought signature.The gateway stamps
extra_content.livekitonto every delta it forwards, including the leading contentless one, andparseChoicereturns a realChatChunkfor a delta whose only payload isextra_content(theif (!content && !deltaExtra) return undefinedguard passes ondeltaExtra). So every streamed response through the gateway went unretryable from its first chunk. Only failures landing before that first chunk could still recover.Diagnosed from a Python simulation run, where a mid-stream stall failed the turn outright with nothing generated and nothing a retry could have duplicated. The JS path has had no reported incident, but the code is the same.
Change
retryableis now cleared on text or a tool call — the output a retry would actually repeat. Theretryable = falseon the usage chunk is dropped for the same reason: token counts aren't output either.Tests
agents/src/inference/llm.test.ts, using the existingclient.chat.completions.createstubbing pattern, with a stub that yields chunks then throws. The metadata chunk in the fixtures is the literalextra_content.livekitpayload observed in production.maxRetry: 2)Confirmed failing without the fix (
expected 1 to be 3). Full suite: 1518 passed across 111 files;tsc --noEmitclean.Not ported
Python also gained an
httpx.TimeoutExceptionclause, so a timeout waiting on the stream body is reported asAPITimeoutErrorrather thanAPIConnectionError. The same structural gap exists here —OpenAI.APIConnectionTimeoutErroronly catches what the OpenAI client mapped — but I could not establish which concrete error class a mid-stream stall produces in Node, and there is no precedent in this repo to follow. A guessed class name would be a check that silently never matches, so it needs a reproduction first.