Skip to content

test(runtime): send more tools than the composition names - #4780

Merged
Astro-Han merged 1 commit into
apache:mainfrom
Astro-Han:test/prompt-composition-tool-cap
Sep 4, 2026
Merged

test(runtime): send more tools than the composition names#4780
Astro-Han merged 1 commit into
apache:mainfrom
Astro-Han:test/prompt-composition-tool-cap

Conversation

@Astro-Han

Copy link
Copy Markdown
Contributor

Summary

Follow-up to a review point on #4722: the chain test that seals a real send into SQLite sent a turn with no tools, so its only assertion about the tool rows was tools.length <= 64 — which holds for a list with none. The cap, the ranking and the remainder crossed the backend, the AgentRun ledger and the SQLite round trip completely untested; the only coverage was the unit fold in request-shape.test.ts, on a composition built in memory.

The turn now carries 69 tools, and the test asserts the rows the context panel actually reads:

  • 64 named, and they are the 64 largest;
  • the remaining 5 carried as a counted remainder;
  • named bytes + remainder bytes = the tool_definitions segment's own byte count, so nothing is dropped by the cap without being accounted for;
  • all of it twice — warm, off the row the send committed, and cold, after reopening the store and folding the ledger.

Test-only. No production code changes.

✅ Verification

  • format, lint clean. test:dist on @maka/runtime: 3,219 tests, 0 failures.
  • The new assertions bite: making the fold report zero remainder bytes (remainder.reduce(...)0 in prompt-composition.ts) fails the test; reverting passes it. The tools.length === 64 assertion is itself the proof the toolset now reaches the sealed row.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code — wrote the test changes and ran the verification. Reviewed by me.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes
  • No — test-only

The chain test sent a turn with no tools, so the only thing it said about the
tool rows was that there were at most 64 of them -- true of a list with none.
The cap, the ranking and the remainder crossed the backend, the ledger and the
SQLite round trip untested; only the unit fold covered them, on a composition
built in memory.

It now sends 69, and asserts the same rows the panel reads: 64 named, the five
largest-last carried as a counted remainder, and the two adding up to the tool
bytes the segment claims -- warm off the committed row, and again cold after
reopening the store.

Generated-by: Claude Code
@github-actions github-actions Bot added the effort/S Under 100 readable lines label Sep 4, 2026
@Astro-Han
Astro-Han marked this pull request as ready for review September 4, 2026 12:28

@jackwener jackwener left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at exact head ee25aa94. No findings. The gap this closes is one I raised on #4722, so the check that mattered was not whether the new assertions can fail — it was whether they catch the specific mutation that escaped before. They do.

Verified against the original escape

The finding on #4722 was that inserting a 65th named row after the helper's correct fold left 102/102 relevant tests green, because this chain test ran with an empty toolset and asserted only tools.length <= 64 — which holds for a list with none.

Rebuilt on this head and mutated production prompt-composition.ts three ways:

Mutation Result
remainder.reduce(...)0 (the one the PR cites) fails — byte conservation
ranked.slice(0, MAX)slice(0, MAX + 1)the escape from #4722 fails
ranked.slice(0, MAX)slice(-MAX) — keep the smallest instead of the largest fails

Baseline passes 2/2. The second row is the one that counts: that mutation is what the old assertion could not see, and it is now caught at the sealed row rather than only in the in-memory fold.

The assertions are stronger than what was suggested

The change I proposed was 65 or 69 real tools with assertions on the named count, the exact remainder, byte conservation and a warm/cold read-back. What landed adds one more thing:

assert.deepEqual(tools.map((tool) => tool.name), largest);

That pins the ranking, not just the count — the named 64 must be exactly the 64 largest, in descending order. Ranking is what makes the row actionable ("the top of the list is what a reader could remove"), and a cap that kept an arbitrary 64 would satisfy every count-and-bytes assertion while making the panel misleading. Mutation 3 above exists only because that assertion does.

Two smaller things done right: tools.length <= MAX became === MAX, which removes the vacuous-truth hole that caused this in the first place; and assertToolsAccountedFor runs on both the warm row committed by the send and the cold row after reopening the store, so the round trip is covered rather than assumed.

Test-only, no production change. label and test are green on this head. The PR is still a draft, so this approval covers the change rather than clearing it to merge, and the merge decision is a human's.

简体中文

ee25aa94 上批准。没有发现问题。 这次关闭的缺口是我在 #4722 上提的,所以要紧的检验不是「新断言能不能失败」,而是它能不能抓住当初逃掉的那个具体变异能。

针对原逃逸变异的验证

#4722 上的发现是:在 helper 完成正确折叠之后塞入第 65 个 named row,相关测试仍然 102/102 全绿——因为这条链路测试跑的是空工具集,而它唯一的断言是 tools.length <= 64,这对一个空列表天然成立

在这个 head 上重新构建,并对生产 prompt-composition.ts 做了三种变异:

变异 结果
remainder.reduce(...)0(PR 自己引用的那个) 失败——字节守恒
ranked.slice(0, MAX)slice(0, MAX + 1)——#4722 的那个逃逸变异 失败
ranked.slice(0, MAX)slice(-MAX)——留最小的而不是最大的 失败

基线 2/2 通过。第二行才是关键:那个变异正是旧断言看不见的,现在它在落账那一行就被抓住了,而不再只在内存里的 fold 上被抓。

断言比当初建议的更强

我提的补法是用 65 或 69 个真实工具,断言 named 数量、精确 remainder、字节守恒,以及冷热读回一致。实际落地的多了一样:

assert.deepEqual(tools.map((tool) => tool.name), largest);

它钉住的是排序,不只是数量——被命名的 64 个必须恰好是最大的 64 个,且降序排列。排序正是这一行之所以可操作的原因(「列表顶端就是读者可以删掉的东西」),而一个保留任意 64 个的 cap,能满足所有关于数量和字节的断言,却会让面板产生误导。 上面第三种变异之所以能被抓,只因为有这条断言。

另外两处小改也做对了:tools.length <= MAX 改成了 === MAX,堵掉了当初正是它造成问题的那个「空真」漏洞;而 assertToolsAccountedFor发送提交的热行重开存储后的冷行上都跑了一遍,所以那趟往返是被覆盖的,而不是被假定的。

纯测试,无生产改动。这个 head 上 labeltest 均为绿。PR 仍是 draft,所以本批准针对的是这次改动,不构成放行合并;合并与否由人决定。


Automated review notice: This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.

@Astro-Han
Astro-Han merged commit 96de2c9 into apache:main Sep 4, 2026
5 checks passed
@Astro-Han
Astro-Han deleted the test/prompt-composition-tool-cap branch September 4, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/S Under 100 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants