fix(llm): tell the model the field names, not just "match the schema" - #132
Merged
Merged
Conversation
Verifying the viewpoint fix on the live server, the draft came back with `proposal_source: deterministic_fallback` — i.e. the AI path had failed and the product quietly degraded. The server log said 「Invalid structured LLM output; retrying」 four times. Root cause: `generate_structured` instructs the model to "Respond only with valid JSON matching the requested schema" and never sends the schema. The model has to guess field names and types. Five prompts in the repo ask for structured output without writing the shape out themselves (viewpoint, series, opportunity, intent drafts, publish check); the three that do carry an explicit example are the ones fixed in earlier rounds. This is the same failure that made the digest silently degrade before (#85) — the lesson was applied per-prompt instead of at the boundary. The instruction now carries a skeleton generated from the pydantic model itself: field names, types, first literal value, and nested models, e.g. {"statement": "", "rationale": "", "limitations": [""]} which every caller benefits from, without touching any prompt. The skeleton is built by walking `model_fields` (with a depth guard for self-referential models), so it cannot drift from the contract the response is validated against. Test: `generate_structured` must put the field names in the system prompt — a stub client captures the prompt and asserts both the schema fields and the caller's system prompt are present. Hand-written examples in the digest / project-start / anchor prompts stay as they are; they describe semantics, and they now agree with the skeleton.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
在线上验证「提炼候选」修复时发现的第二层问题：视图虽然 201 了，但
proposal_source是 deterministic_fallback——AI 路径失败后静默降级。服务器日志：连续 4 次「Invalid structured LLM output; retrying」。根因
generate_structured只告诉模型「Respond only with valid JSON matching the requested schema」，但从没把 schema 发给模型——字段名、类型全靠猜。仓库里有 5 个提示词属于这种情况（观点、系列、机会、意图草案、发布前检查），而带显式 JSON 示例的 3 个正是前几轮修过的（#85 让消化期静默降级的就是同一个病）。当时是逐提示词补示例，没有在边界上修。修法
generate_structured现在把 pydantic 模型自身的字段骨架写进系统提示（字段名、类型、字面量取值、嵌套模型），例如：{"statement": "", "rationale": "", "limitations": [""]}骨架是从
model_fields走出来的（带自引用深度保护），所以不可能与"校验用的契约"漂移；所有调用方一起受益，不用动任何提示词。消化期 / 起点 / 锚点提示词里手写的示例保留（它们描述语义），且现在与骨架一致。测试与验证
新增测试：结构化调用必须把字段名写进系统提示（桩客户端捕获提示内容，同时断言调用方自己的 system prompt 还在）。后端全量 + ruff 待跑完确认；部署后再打一次同一个线上调用，期望
proposal_source变成ai（这是这次修复的判据）。