Add optional token-budget embedding batches - #76
Merged
Conversation
Count-only batching can exceed a provider's aggregate token limit even when each input fits the model. Shared callers need one way to lower request width from a conservative per-input upper bound. Keep the budget opt-in so existing callers retain their current request shape. Callers still own model limits and token estimates; the vector package only enforces the declared bound.
roborev: Combined Review (
|
Providers can cap the combined input tokens separately from the number of inputs. Callers need API documentation that identifies when the option applies and how a conservative per-input bound affects request capacity. Generated with OpenAI Codex Co-authored-by: OpenAI Codex <noreply@openai.com>
roborev: Combined Review (
|
Callers should configure batching on the operation instead of constructing temporary option structs. A single functional-option path also makes the token budget available to both direct encoding and document fills. Shared fill batches must use the effective token-bound width when they split work. Otherwise they can still exceed the provider limit even though direct EncodeBatched calls enforce it. Generated with OpenAI Codex Co-authored-by: OpenAI Codex <noreply@openai.com>
roborev: Combined Review (
|
Per-document fills discovered an invalid token budget inside EncodeBatched. The document error handler could then treat that configuration failure as bad content and stamp-skip a pending document without vectors. Validate the batch configuration at the Fill boundary so configuration errors cannot enter document-level recovery. Generated with OpenAI Codex Co-authored-by: OpenAI Codex <noreply@openai.com>
roborev: Combined Review (
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Embedding providers can cap the combined input tokens in a request independently of input count.
EncodeBatchedandFillnow use functional options;WithBatchTokenBudgetderives the effective batch width from a caller-supplied total budget and conservative per-input bound, and applies it to direct encoding and shared fill batches.This replaces the public
BatchOptionsandFillOptionsstructs before the new API ships. Omitting the token budget preserves count-only batching. Callers remain responsible for model limits and token estimates;kit/vectordoes not tokenize or alter input text.