fix(llm): raise EmptyCompletionError instead of crashing on empty completions - #520
Open
virajsabhaya23 wants to merge 1 commit into
Open
fix(llm): raise EmptyCompletionError instead of crashing on empty completions#520virajsabhaya23 wants to merge 1 commit into
virajsabhaya23 wants to merge 1 commit into
Conversation
…pletions When a model returned no visible content, `LLMResult.text` was set to `None` and the failure only surfaced four frames later in `parse_plan`, where `_TASK_RE.finditer(raw)` raised `TypeError: expected string or bytes-like object, got 'NoneType'`. The traceback blamed the regex and said nothing about `max_tokens`, `finish_reason`, or the empty completion. Reasoning models charge hidden `reasoning_content` against the same completion cap, so a demanding question can exhaust a 2048-token budget and return `content: null` with `finish_reason: "length"` regardless of prompt size. - Validate the completion where it arrives, before `LLMResult` is constructed, so a `None` never enters the system. `EmptyCompletionError` names the model, `finish_reason`, tokens consumed and the cap. - Make the cap overridable via `AOB_LLM_MAX_TOKENS`, defaulting to the current 2048 so behaviour is unchanged unless the variable is set. - Apply both to `LiteLLMBackend` and `OpenAICompatBackend` via a shared `result_from_response` helper; both had the same unchecked `.content` read and the same hardcoded cap. Closes IBM#511 Signed-off-by: Viraj <77448246+virajsabhaya23@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
An empty model completion took the whole run down with a
TypeErrorthat blameda regex four frames from the cause.
LLMResult.textwas set toNoneand thefailure only surfaced in
parse_planat_TASK_RE.finditer(raw), with nomention of
max_tokens,finish_reason, or the empty completion.Fix Details
Reasoning models charge hidden
reasoning_contentagainst the same completioncap, so a demanding question can exhaust a 2048-token budget and return
content: nullwithfinish_reason: "length"regardless of prompt size. Therun reported in #511 had a 256-token prompt and hit the 2048 cap exactly.
LLMResultis constructed,so a
Nonenever enters the system.EmptyCompletionErrornames the model,finish_reason, tokens consumed and the cap.AOB_LLM_MAX_TOKENS, defaulting to the current2048 so nothing changes unless it is set. Documented in
INSTRUCTIONS.md.LiteLLMBackendandOpenAICompatBackendthrough a sharedresult_from_responsehelper. The issue named only the litellm path, butopenai_compathad the identical unchecked.contentread and the samehardcoded cap.
Scope note: no retry/backoff here. #511 distinguishes a null completion
(possibly retryable) from a 429 (back off); that is a policy decision worth its
own PR.
Impact on Benchmarking
Previously-passing runs are unaffected: the default cap is unchanged at 2048 and
a non-empty completion follows the same path. Runs that previously crashed now
raise a named, diagnostic error instead of a misleading
TypeError.Related Issues
plan_executewith aTypeErrorthat blames the regex #511Verification Steps
uv run pytest src/llm/tests— 19 passing onmain, 34 on this branch.uv run ruff format --check src/llm && uv run ruff check src/llm— clean.None/ empty / whitespace-only content, the recorded256-prompt/2048-completion failure,
AOB_LLM_MAX_TOKENSreaching therequest, and rejection of invalid override values.
Note: the template's
uv run pytest tests/integrationpath does not exist inthis repo; tests live under
src/*/tests.Checklist