Give the model the tools and the guardrails it was missing, without changing the model

[devbadya]

Everything here is a deterministic function over text, so all of it is verified by pnpm check — 1,066 tests — and none of it needed a GPU. What the model then does with any of it is what /?eval is for; that is stated per change below rather than claimed.

The model itself is unchanged, deliberately: no new weights, no sampling change, no strategy change, and nothing added to SYSTEM_PROMPT, which was measured making tool use worse.

1. The calculator refused ten of eighteen expressions a 0.8B model plausibly writes

A refused expression spends the tool round and sends the model back to the mental arithmetic the tool exists to replace. 98,765 * 4,321, 18% of 2450, 18 Prozent von 2450, (17 * 23) / 4 =, 2 x 3, $1200 * 1.19 are all read now, because each has exactly one arithmetic meaning. Genuine ambiguity is still refused: 18% off 2450 is a discount, 12 % 5 is still modulo, and a unit conversion is not arithmetic. The two failure messages end in a clause the next round can act on.

2. Two skills were taking questions that belong elsewhere

Was ist heute für ein Tag? went to a web search, Was bedeutet TLDR? to the page summariser, which asked which page was meant. The first is worth more than the two questions it fixes: a trigger anywhere in the catalogue is matched before any keyword, so was ist heute in research-question at priority 10 was beating welcher tag in current-date's index at priority 25. A German phrasing left to the index is only safe until some other skill writes a trigger over it.

3. A conversion had no tool at all

convert is new: length, mass, temperature, volume, speed, area, data and duration, with every spelling either language uses, since the model passes the user's own word through as the argument. It was the one question in the app with nothing behind it — lookup-term excludes it by hand, the calculator refuses it correctly, so the model did it in its head.

What it refuses matters as much as what it converts. Grams to cups depends on what is in the cup; a month and a year are not fixed durations; Mb and MB differ by a factor of eight and by one letter's case. The convert-units skill sits above arithmetic, and its 5 miles in km trigger anchors the target on a known unit so 20 minutes to Berlin is not read as a conversion into a city. The answer check follows the tool: a converted number is now held to the same standard as a calculated one.

4. A figure, a date and an attribution had nowhere to go

Three quarters of ordinary questions reached no skill. Some belong to nobody, but one group did not — How many people live in Tokyo?, Wie alt ist Angela Merkel?, When was the Eiffel Tower built?, Wer hat die Bundestagswahl 2025 gewonnen? — and it is exactly what this model produces plausibly and unverifiably. All three shapes are research-question triggers now, in both languages, each excluding the version of itself about the user or the assistant, since how old are you and when is my flight are not on the web.

Growing that skill cost another one two keywords, which is the mechanism worth remembering: chance of rain and regenwahrscheinlichkeit were weather keywords and how high is is a research trigger, so the forecast question moved to a search engine the moment the trigger was written. Both are weather triggers now.

Coverage over the same 32 ordinary questions: 24 reached no skill, now 15.

5. read_page handed over the top of the page, not the part that answers

The cap is 8,000 characters — roughly 2,000 tokens, the largest thing in this model's context — and it used to be the first 8,000, which is a bet that the answer was printed at the top. What is at the top is the navigation, the cookie notice and the related links. src/tools/extract.ts strips that furniture and, only if the page is still too long, keeps the passages about the question: blocks with each heading glued to its paragraph, scored by the question's terms weighted by how rare each is in that page, with leftover budget spent on the paragraphs either side of the best match and on the page's own opening. Skipped passages are marked […].

The question comes from the agent loop, not from the model: runAgent passes the user's turn to every tool as an optional ToolContext, because asking the model for a fourth argument would spend tool-calling accuracy on something already known. With no question, or one sharing no word with the page, it falls back to the head.

Measured against the real Wikipedia article on the Eiffel Tower (44,988 characters), four of five answers were unreachable before and are selected now:

  head: no    selected: yes   at  36134   "every 7 years" — How often is the Eiffel Tower repainted?
  head: no    selected: yes   at  26158   "7,300 tonnes" — How much does the Eiffel Tower weigh?
  head: no    selected: yes   at  28544   "9 cm" — How far does the Eiffel Tower sway in the wind?
  head: no    selected: yes   at  14770   "1,710" — How many steps are there to the top?
  head: yes   selected: yes   at    980   "330 metres" — How tall is the Eiffel Tower?   (the control)

6. Nothing checked that the answer was in the language of the question

The system prompt asks for the user's language and this model drifts back to English. It is the most visible way it is wrong and the one thing settleable without asking the model anything, since the evidence is the user's own message. wrong-language joins the three deterministic checks, and it is as shy as they are: both sides have to be recognisable, so 330 Meter. has no language, a German answer quoting an English sentence is still German, and a question in a third language leaves the check silent. The correction is written in the language it asks for.

7. Two caps that were claimed and not enforced

The transcript now has a budget in toHistory — 8,000 characters, oldest whole turns dropped, the turn being answered kept whatever it costs. Not because the window is full: Qwen3.5 carries 262,144 positions. It is re-prefilled on every round of every turn, twice for the capped strategies, and length dilutes attention.

And an MCP result is capped at the same 8,000 characters as a page read. The README already said it was; nothing did it, so one verbose server could take the whole prompt.

This is the least certain change in the set and the one to point the harness at first: a transcript cap trades context for focus, and only measurement can say whether 8,000 is the right side of that trade.

What was not done

A bigger model. It is the only change that raises general reasoning rather than a specific behaviour, and it was ruled out for this branch. For the record, there is no text-only 2B export; onnx-community/Qwen3.5-2B-ONNX at q4f16 is a 1,039 MiB decoder plus 280 MiB of embeddings against 448 MiB today.

skill-coverage-before-after.log
read-page-finds-the-answer.log
converter-and-language-check.log
calculator-and-routing-before-after.log

_{To show artifacts inline, enable in settings.}