fix: treat CrewAI's empty final answer as a finished turn, not a failure - #621
Merged
AlexanderZ-Band merged 4 commits intoSep 7, 2026
Merged
Conversation
CrewAI raises ValueError("Invalid response from LLM call - None or empty.")
whenever the model returns no final text. This agent answers only through
band_send_message and its system prompt says so, so the model has nothing
left to say once its tools have run: kickoff_async raised on 61 of 61 turns
in the last crewai lane run. The success path after it was unreachable.
Every turn's outcome was therefore decided by the except handler, which
guessed from tool bookkeeping whether the failure was benign and re-raised
when no terminal tool had run. A turn asked to do only read work ("call
band_list_tasks ... do not call any other tool") flips neither replied nor
tool_executed, since fetching state is not terminal -- so the adapter marked
the delivery failed for a turn that did exactly what it was asked. The test
passed only when the model disobeyed and sent a message anyway, which is why
it looked flaky and why it "passed on windows": that job needed all three
attempts and only survived the one where the model called band_send_message.
Catch the empty final answer where it happens and normalize it to "no final
text", then let the post-turn logic run for real. The missing-reply policy
that already existed 20 lines above -- unreachable until now -- becomes the
single place that judges a silent turn, and it keeps the is_terminal_success
semantics the swallow condition used. Genuine errors still report and raise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W8z6z1muD8yg9Y3pre92ec
Code review on #621 found the swallow was unconditional: a turn where the very first LLM call came back empty -- no tool call at all, indistinguishable from a genuine provider failure -- was silently marked processed instead of failing the delivery for a retry. That also widened the missing-reply gate on the clean-return path with no coverage, left the exception's own detail untraced at default log level, and typed the same error string in five places. ReplyTracker gets any_tool_ran, flipped on any tool call this turn (success or failure, terminal or not) -- coarser than tool_executed/replied, which only count terminal work. The empty-answer swallow now requires it: a turn that ran some tool (even read-only) before going quiet is a finished turn; a turn that ran nothing at all keeps failing the delivery, same as before this PR. Also: EMPTY_LLM_RESPONSE_MARKER is now the one definition the matcher and all four test call sites reference; ReplyTracker.did_productive_work replaces the inline replied-or-tool_executed check and is now a required parameter; the missing-reply log moved to WARNING with the exception text kept at DEBUG; the crewai/pydantic_ai cross-reference and the memory-rehydration E2E exclusion reason (post-#621, the turn now finishes -- it just never replies) are updated to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W8z6z1muD8yg9Y3pre92ec
test_suppresses_empty_final_answer_after_reply and ..._after_tool_only_turn hand-set ReplyTracker.replied/.tool_executed/.any_tool_ran directly, baking in the assumption that SEND_MESSAGE and a terminal tool always set any_tool_ran alongside them -- true today, but nothing would catch it breaking. Route both through the real _mark_productive_work, like the read-only sibling test already does. Mutation-verified: all three now fail if any_tool_ran is removed from _mark_productive_work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W8z6z1muD8yg9Y3pre92ec
The test now exercises the real _mark_productive_work path rather than reporting a bug, so the regression-report framing no longer fits. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W8z6z1muD8yg9Y3pre92ec
AlexanderZ-Band
requested review from
a team
and
a lite review from Copilot
and removed request for
Copilot
September 7, 2026 18:05
AlexanderZ-Band
deleted the
fix/crewai-empty-final-answer-is-normal-turn-end
branch
September 7, 2026 18:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CrewAI raises
ValueError("Invalid response from LLM call - None or empty.")on every turn (this agent replies only viaband_send_message, so the model has nothing left to say once its tools ran — measured 61/61 turns in one lane run). Theexcepthandler guessed from tool bookkeeping whether that was benign, and re-raised when no terminal (write) tool had run — failing the whole delivery for a turn instructed to do read-only work only (test_task_lifecycle_across_task_adapters's second turn: "call band_list_tasks ... do not call any other tool").Looked like flakiness / a platform difference, but wasn't: both
crewai/ubuntuandcrewai/windowsran the same 3 attempts; ubuntu's model obeyed every time and failed every time, windows only passed the attempt where the model disobeyed and sent a message anyway.Fix
Catch the empty final answer, normalize it to "no final text", and let the existing missing-reply policy (previously unreachable) decide the outcome instead of the
excepthandler guessing. No change tois_terminal_successor read-only classification. Genuine errors still report and raise.Verification
crewaiE2E green on both ubuntu and windows, first attempt, no retry needed for the target test (runs 34147284182, 34148069857).🤖 Generated with Claude Code
https://claude.ai/code/session_01W8z6z1muD8yg9Y3pre92ec