An empty completion crashes `plan_execute` with a `TypeError` that blames the regex

[arthapraha]

Repository: IBM/AssetOpsBench · Observed at: e11d1c1 (current main)
Source: finding 7 of our defect report · follows #503, the last of the six

When the model returns no visible content, the run does not degrade, retry or
report. It dies — and the traceback names a regex four frames from the cause.

What happens

src/llm/litellm.py:46 sends a fixed cap and takes the content unchecked:

"max_tokens": 2048,
...
return LLMResult(text=response.choices[0].message.content, ...)   # may be None

src/agent/plan_execute/planner.py:131 passes that value straight on:

raw = self._llm.generate(prompt)
return parse_plan(raw)

and parse_plan line 61 hands it to a regex:

tasks = {int(m.group(1)): m.group(2).strip() for m in _TASK_RE.finditer(raw)}

giving:

TypeError: expected string or bytes-like object, got 'NoneType'

The traceback indicts _TASK_RE.finditer. Nothing in the failure output
mentions max_tokens, finish_reason, or an empty completion. We spent a
substantial part of an evening on four wrong hypotheses — a prompt-shape effect,
a rate limit, an endpoint outage, a backend fault — before instrumenting
litellm.completion and reading what actually arrived.

Why it happens, and why it is ordinary

Reasoning models emit reasoning_content alongside content, and the cap
applies to both together. A model that thinks hard about a demanding question
spends the budget reasoning and returns content: null with
finish_reason: "length".

Prompt size is not the trigger, and one recorded run shows it:

prompt_tokens	256
completion_tokens	2048 — exactly the cap
outcome	crashed

A 256-token prompt exhausted a 2048-token budget. That run had no MCP
servers mounted, so there was no tool catalogue in the prompt and nothing to
look the answer up in — the model had to reason its way there. The cap binds
on reasoning demand, not prompt length, which means raising it for large tool
catalogues alone would not close this.

Rate, measured rather than impressionistic

Across a twelve-run probe (two arms, three scenarios, two passes, five-second
pacing): 2 failures in 12.

They were not the same failure. One was this budget exhaustion; the other was
an HTTP 429 on a run whose model call had already returned normally
(finish_reason: "stop"). We separate them because the right response differs —
a null completion may be worth retrying, a rate limit is worth backing off from —
and an aggregate "17% failure rate" would have hidden that.

Suggested fix

Attached PR, in the order we would apply it:

1. Check the value where it arrives. EmptyCompletionError, raised in
   LiteLLMBackend.generate_with_usage — after the completion returns and
   before LLMResult is constructed, so a None never enters the system at
   all. It names the model, finish_reason, tokens consumed and the cap. A
   regex is not the right place to discover a completion was empty.
2. Make the cap overridable — AOB_LLM_MAX_TOKENS, defaulting to the current
   2048, so nothing changes unless you change it.

Test suite: 19 passing on main → 25 on the branch (+6), 3 files, +160/−4.

What we are not claiming

We provoked it ourselves. We were injecting an answer-format instruction
that contradicted the planner's own output-format instruction, and the model
spent its budget deliberating the conflict. That was our bug and we fixed it.
But the defect is not that our prompt was bad — it is that an empty
completion, which any sufficiently demanding question can produce against a
reasoning model, takes the whole run down and reports the wrong line. We reached
it by an unusual road; the road is not the finding.

We did not audit the other runners. claude_agent, deep_agent,
openai_agent, stirrup_agent and opencode_agent may share this class of
problem by a different mechanism. We have no evidence either way and would
rather say so than imply a sweep we did not run.

The model is not the point. We saw this on a reasoning model reached through
a hosted inference endpoint; any reasoning model on a sufficiently demanding
question reaches the same place. Nothing here is specific to one backend.

---

Findings 1–6 reached the maintainers by email on 10 August. This continues that
exchange, and is the last of the six issues announced in #503.