test(runtime): send more tools than the composition names - #4780
Conversation
The chain test sent a turn with no tools, so the only thing it said about the tool rows was that there were at most 64 of them -- true of a list with none. The cap, the ranking and the remainder crossed the backend, the ledger and the SQLite round trip untested; only the unit fold covered them, on a composition built in memory. It now sends 69, and asserts the same rows the panel reads: 64 named, the five largest-last carried as a counted remainder, and the two adding up to the tool bytes the segment claims -- warm off the committed row, and again cold after reopening the store. Generated-by: Claude Code
jackwener
left a comment
There was a problem hiding this comment.
Approving at exact head ee25aa94. No findings. The gap this closes is one I raised on #4722, so the check that mattered was not whether the new assertions can fail — it was whether they catch the specific mutation that escaped before. They do.
Verified against the original escape
The finding on #4722 was that inserting a 65th named row after the helper's correct fold left 102/102 relevant tests green, because this chain test ran with an empty toolset and asserted only tools.length <= 64 — which holds for a list with none.
Rebuilt on this head and mutated production prompt-composition.ts three ways:
| Mutation | Result |
|---|---|
remainder.reduce(...) → 0 (the one the PR cites) |
fails — byte conservation |
ranked.slice(0, MAX) → slice(0, MAX + 1) — the escape from #4722 |
fails |
ranked.slice(0, MAX) → slice(-MAX) — keep the smallest instead of the largest |
fails |
Baseline passes 2/2. The second row is the one that counts: that mutation is what the old assertion could not see, and it is now caught at the sealed row rather than only in the in-memory fold.
The assertions are stronger than what was suggested
The change I proposed was 65 or 69 real tools with assertions on the named count, the exact remainder, byte conservation and a warm/cold read-back. What landed adds one more thing:
assert.deepEqual(tools.map((tool) => tool.name), largest);That pins the ranking, not just the count — the named 64 must be exactly the 64 largest, in descending order. Ranking is what makes the row actionable ("the top of the list is what a reader could remove"), and a cap that kept an arbitrary 64 would satisfy every count-and-bytes assertion while making the panel misleading. Mutation 3 above exists only because that assertion does.
Two smaller things done right: tools.length <= MAX became === MAX, which removes the vacuous-truth hole that caused this in the first place; and assertToolsAccountedFor runs on both the warm row committed by the send and the cold row after reopening the store, so the round trip is covered rather than assumed.
Test-only, no production change. label and test are green on this head. The PR is still a draft, so this approval covers the change rather than clearing it to merge, and the merge decision is a human's.
简体中文
在 ee25aa94 上批准。没有发现问题。 这次关闭的缺口是我在 #4722 上提的,所以要紧的检验不是「新断言能不能失败」,而是它能不能抓住当初逃掉的那个具体变异。能。
针对原逃逸变异的验证
#4722 上的发现是:在 helper 完成正确折叠之后塞入第 65 个 named row,相关测试仍然 102/102 全绿——因为这条链路测试跑的是空工具集,而它唯一的断言是 tools.length <= 64,这对一个空列表天然成立。
在这个 head 上重新构建,并对生产 prompt-composition.ts 做了三种变异:
| 变异 | 结果 |
|---|---|
remainder.reduce(...) → 0(PR 自己引用的那个) |
失败——字节守恒 |
ranked.slice(0, MAX) → slice(0, MAX + 1)——#4722 的那个逃逸变异 |
失败 |
ranked.slice(0, MAX) → slice(-MAX)——留最小的而不是最大的 |
失败 |
基线 2/2 通过。第二行才是关键:那个变异正是旧断言看不见的,现在它在落账那一行就被抓住了,而不再只在内存里的 fold 上被抓。
断言比当初建议的更强
我提的补法是用 65 或 69 个真实工具,断言 named 数量、精确 remainder、字节守恒,以及冷热读回一致。实际落地的多了一样:
assert.deepEqual(tools.map((tool) => tool.name), largest);它钉住的是排序,不只是数量——被命名的 64 个必须恰好是最大的 64 个,且降序排列。排序正是这一行之所以可操作的原因(「列表顶端就是读者可以删掉的东西」),而一个保留任意 64 个的 cap,能满足所有关于数量和字节的断言,却会让面板产生误导。 上面第三种变异之所以能被抓,只因为有这条断言。
另外两处小改也做对了:tools.length <= MAX 改成了 === MAX,堵掉了当初正是它造成问题的那个「空真」漏洞;而 assertToolsAccountedFor 在发送提交的热行与重开存储后的冷行上都跑了一遍,所以那趟往返是被覆盖的,而不是被假定的。
纯测试,无生产改动。这个 head 上 label 与 test 均为绿。PR 仍是 draft,所以本批准针对的是这次改动,不构成放行合并;合并与否由人决定。
Automated review notice: This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
Summary
Follow-up to a review point on #4722: the chain test that seals a real send into SQLite sent a turn with no tools, so its only assertion about the tool rows was
tools.length <= 64— which holds for a list with none. The cap, the ranking and the remainder crossed the backend, the AgentRun ledger and the SQLite round trip completely untested; the only coverage was the unit fold inrequest-shape.test.ts, on a composition built in memory.The turn now carries 69 tools, and the test asserts the rows the context panel actually reads:
tool_definitionssegment's own byte count, so nothing is dropped by the cap without being accounted for;Test-only. No production code changes.
✅ Verification
format,lintclean.test:diston@maka/runtime: 3,219 tests, 0 failures.remainder.reduce(...)→0inprompt-composition.ts) fails the test; reverting passes it. Thetools.length === 64assertion is itself the proof the toolset now reaches the sealed row.AI use
Select exactly one:
Tool(s) and scope: Claude Code — wrote the test changes and ran the verification. Reviewed by me.
Checklist
Does this PR entail a change in behavior?