runStructuredCompletion reports truncated responses as schema failures (finishReason never read)

[JohnD-EE]

runStructuredCompletion never looks at finishReason. When a response is truncated at the token cap but still carries partial content, the parse fails, both attempts burn, and the caller is told the model broke the schema contract. The real cause — an exhausted output budget — is nowhere in the error or the logs.

Where it's dropped

lib/orchestration/llm/structured-completion.ts

• const firstParsed = opts.parse(first.content) (line 201) and const retryParsed = opts.parse(retry.content) (line 245) look only at the content string. first.finishReason / retry.finishReason are on the response and are simply not read.
• onFinalFailure (line 121) takes no arguments (line 247), so a caller cannot tell a truncation from a contract violation even if it wanted to.

lib/orchestration/llm/openai-compatible.ts

• The truncation guard at line 214 requires empty content:

  if (choice.finish_reason === 'length' && content.length === 0 && toolCalls.length === 0) {

  It catches the case where reasoning consumed the entire budget and nothing visible came back. It cannot catch the far more common case: reasoning consumed most of the budget, the model emitted a few hundred tokens of a JSON object, and got cut off mid-string. finish_reason is 'length', content is non-empty, no error is raised.

lib/orchestration/llm/anthropic.ts:318 gets this right for the forced-tool path (isStructuredExtraction && finishReason === 'length' → truncated_no_output), which is a useful model for the fix — but it only covers Anthropic structured extraction, not the shared runner.

Impact

The symptom is a misleading error, which is worse than a missing one. In ConQuest a design-time LLM judge failed in production with:

UnknownError: {"dimension":"clarity","model":"gpt-5.4","provider":"openai","issuePaths":[],
"error":"Judge response was not valid against the schema after one retry"}

Nothing there points at the token budget. It reads as a schema/contract bug, and the empty issuePaths (our own Zod-issue capture, which never ran because JSON.parse threw) reads as "no schema problems found" rather than "we never got JSON". The actual fault was maxTokens: 2048 on an openai-reasoning model, where the cap is sent as max_completion_tokens and covers hidden reasoning tokens plus visible output.

Two things make this worth fixing in the runner rather than per-caller:

1. Every caller of runStructuredCompletion inherits it, and each one rediscovers it the hard way. A caller cannot detect it itself — StructuredCompletionResult doesn't expose finishReason, and the failure path throws before returning anything.
2. The retry is knowingly futile. When attempt one truncates at maxTokens, attempt two runs at the same cap with a longer prompt (the retry message is appended), so it truncates too — a second paid call whose outcome is predictable from data the runner already has.

Suggested fix

Read finishReason on the parse-failure path. Two pieces, both backward-compatible:

// 1. Don't spend a second call that cannot succeed.
if (firstParsed === null && first.finishReason === 'length') {
  throw new Error(
    `Structured completion truncated at maxTokens (${maxTokens}) before the response could ` +
      `be parsed. Raise maxTokens — on OpenAI reasoning models this cap is sent as ` +
      `max_completion_tokens and covers reasoning tokens as well as visible output.`
  );
}

// 2. Let the caller phrase its own error correctly.
onFinalFailure?: (info?: { finishReason: LlmFinishReason }) => Error;

Adding an optional parameter keeps every existing zero-arg onFinalFailure assignable. Alternatively, or additionally, surface finishReason on StructuredCompletionResult so successful-but-truncated responses are visible too.

Worth considering alongside: relaxing the openai-compatible.ts:214 guard so finish_reason === 'length' is at least reported (a truncated finish reason already maps through mapFinishReason), rather than only raising when the content is empty.

---

Surfaced in the ConQuest fork by a production judge failure. The fork carries an app-side workaround (a sawParseableJson flag so its own error message can say "most likely truncated" instead of "invalid against the schema"), which every other caller of this runner would have to duplicate. Filing upstream so it can be fixed once.