Skip to content

fix(llm): raise EmptyCompletionError instead of crashing on empty completions - #520

Open
virajsabhaya23 wants to merge 1 commit into
IBM:mainfrom
virajsabhaya23:bugfix/empty-completion-crash
Open

fix(llm): raise EmptyCompletionError instead of crashing on empty completions#520
virajsabhaya23 wants to merge 1 commit into
IBM:mainfrom
virajsabhaya23:bugfix/empty-completion-crash

Conversation

@virajsabhaya23

Copy link
Copy Markdown

Description

An empty model completion took the whole run down with a TypeError that blamed
a regex four frames from the cause. LLMResult.text was set to None and the
failure only surfaced in parse_plan at _TASK_RE.finditer(raw), with no
mention of max_tokens, finish_reason, or the empty completion.

Fix Details

Reasoning models charge hidden reasoning_content against the same completion
cap, so a demanding question can exhaust a 2048-token budget and return
content: null with finish_reason: "length" regardless of prompt size. The
run reported in #511 had a 256-token prompt and hit the 2048 cap exactly.

  • Validate the completion where it arrives, before LLMResult is constructed,
    so a None never enters the system. EmptyCompletionError names the model,
    finish_reason, tokens consumed and the cap.
  • Make the cap overridable via AOB_LLM_MAX_TOKENS, defaulting to the current
    2048 so nothing changes unless it is set. Documented in INSTRUCTIONS.md.
  • Applied to both LiteLLMBackend and OpenAICompatBackend through a shared
    result_from_response helper. The issue named only the litellm path, but
    openai_compat had the identical unchecked .content read and the same
    hardcoded cap.

Scope note: no retry/backoff here. #511 distinguishes a null completion
(possibly retryable) from a 429 (back off); that is a policy decision worth its
own PR.

Impact on Benchmarking

  • No change to baselines: This fix only improves stability/performance.
  • Baseline change

Previously-passing runs are unaffected: the default cap is unchanged at 2048 and
a non-empty completion follows the same path. Runs that previously crashed now
raise a named, diagnostic error instead of a misleading TypeError.

Related Issues

Verification Steps

  1. uv run pytest src/llm/tests — 19 passing on main, 34 on this branch.
  2. uv run ruff format --check src/llm && uv run ruff check src/llm — clean.
  3. New tests cover None / empty / whitespace-only content, the recorded
    256-prompt/2048-completion failure, AOB_LLM_MAX_TOKENS reaching the
    request, and rejection of invalid override values.

Note: the template's uv run pytest tests/integration path does not exist in
this repo; tests live under src/*/tests.

Checklist

  • I have added tests that prove my fix is effective.
  • My code follows the project's Ruff formatting and linting rules.
  • I have signed off my commits (DCO).

…pletions

When a model returned no visible content, `LLMResult.text` was set to `None`
and the failure only surfaced four frames later in `parse_plan`, where
`_TASK_RE.finditer(raw)` raised `TypeError: expected string or bytes-like
object, got 'NoneType'`. The traceback blamed the regex and said nothing about
`max_tokens`, `finish_reason`, or the empty completion.

Reasoning models charge hidden `reasoning_content` against the same completion
cap, so a demanding question can exhaust a 2048-token budget and return
`content: null` with `finish_reason: "length"` regardless of prompt size.

- Validate the completion where it arrives, before `LLMResult` is constructed,
  so a `None` never enters the system. `EmptyCompletionError` names the model,
  `finish_reason`, tokens consumed and the cap.
- Make the cap overridable via `AOB_LLM_MAX_TOKENS`, defaulting to the current
  2048 so behaviour is unchanged unless the variable is set.
- Apply both to `LiteLLMBackend` and `OpenAICompatBackend` via a shared
  `result_from_response` helper; both had the same unchecked `.content` read
  and the same hardcoded cap.

Closes IBM#511

Signed-off-by: Viraj <77448246+virajsabhaya23@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An empty completion crashes plan_execute with a TypeError that blames the regex

1 participant