Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,9 @@ Search and `read_page` share the reader's budget of 20 requests a minute per IP,

**The search itself now carries the facts a 0.8B model would otherwise spend a round guessing at.** Every `web_search` result is stamped with today's local date, so "current" and "today's news" have a date without calling `current_time`. A German question searches German Wikipedia and, on DuckDuckGo, prefers German results (`kl=de-de`); English _who was Ada Lovelace_ is not mistaken for German because bare `was` is also English. German Wikipedia is smaller, so an empty result there falls through to English rather than telling the model the subject does not exist.

The `research-question` skill does not offer `web_search`. It offers `research`, which searches the live web and Wikipedia together, picks three **different sites** (Wikipedia first when it appeared, because MediaWiki is free and the lead paragraph usually answers; `investor.nvidia.com` and `nvidianews.nvidia.com` count as one), ranks the rest by whether their snippet already bears on the question, and reads them in parallel. A page that will not open, or that comes back as a login wall or a firewall interstitial, is replaced from the remaining hits rather than quoted; only when nothing else is left does the search snippet stand in. Inflected forms of a word still match — _Bundeskanzlers_ answers _Bundeskanzler_ — and a one-line Wikipedia claim that names the incumbent (_Amtsträger ist seit dem 6. Mai 2025 Friedrich Merz_) is kept even when the lead is a definition of the office. If nothing readable comes back the tool throws rather than telling the model the subject does not exist. `lookup-term` still searches and optionally reads one page — a name does not need three sources.
The `research-question` skill does not offer `web_search`. It offers `research`, which searches the live web and Wikipedia together, picks three **different sites** (Wikipedia first when it appeared, because MediaWiki is free and the lead paragraph usually answers; `investor.nvidia.com` and `nvidianews.nvidia.com` count as one), ranks the rest by whether their snippet already bears on the question, and reads them in parallel. A page that will not open, or that comes back as a login wall or a firewall interstitial, is replaced from the remaining hits rather than quoted; only when nothing else is left does the search snippet stand in. Inflected forms of a word still match — _Bundeskanzlers_ answers _Bundeskanzler_ — so a German page is not silent on a German question. The search query is narrowed the way the weather tool narrows a place: _What is the capital of France?_ is searched as `capital of France`, which ranks Paris rather than a page that uses the sentence as an example of a question; _why_ and _how_ keep the shell, because _sky blue_ ranks a colour swatch and _why is the sky blue_ ranks the physics. Wikipedia's own ranking still puts a list and _capital punishment_ ahead of Paris; the article whose short description _is_ the query takes the free first slot, and a Wikimedia list is pushed behind it. A one-line Wikipedia claim that names the incumbent (_Amtsträger ist seit dem 6. Mai 2025 Friedrich Merz_) is kept even when the lead is a definition of the office. If nothing readable comes back the tool throws rather than telling the model the subject does not exist.

A factual question that no other skill claimed — _What is the capital of France?_, _Warum ist der Himmel blau?_ — still reaches this skill, as a fourth routing stage rather than a broader trigger. Greetings do not: _wie geht's dir_ and _how are you_ stay with the model. `lookup-term` still searches and optionally reads one page — a name does not need three sources.

**LangSearch is the way off that shared budget without paying for one.** `api.langsearch.com` is a search API rather than a results page, its free tier allows 1,000 searches a day and one a second, and a key needs no card — so a search stops competing with `read_page` for the same 20 requests a minute. Two things about it are worth knowing before choosing it. Its snippets are index text rather than prose, lower-cased and with spaces around the punctuation, which a 0.8B model reads less confidently than a sentence. And it answers in an envelope: a refusal it decides to report with a 200 arrives as a `msg` and no result set, so `searchLangSearch` raises that rather than passing an empty list to a model that would relay it as "this does not exist". Long summaries are available per result and are switched off — each is the whole page behind the result, which would leave a 0.8B context with no room for the answer.

Expand Down Expand Up @@ -497,7 +499,7 @@ The first four, and the two pronoun cases at the bottom, are the same mistake: a
| _Fasse das zusammen_ | nothing | The object is usually a pronoun, and `seite zusammen` isn't |
| _Welchen Wochentag haben wir?_ | nothing | `wochentag` was not among the clock's keywords |

Widening a claim is where a skill starts stealing again, so each of these is anchored as tightly as the shape allows: `tell me about` and `erkläre mir` end at the name, so _tell me about the trip we planned_ is three tokens past matching; the optional article cannot widen the bare-name shape because the name is still the last token, which is what keeps _what is the capital of France?_ out; `add` needs both operands, so _add 3 more rows_ is not a sum; and the two-word `aktuelle …` fragment excludes the subjects the clock and the thermometer own, because answering _aktuelle Uhrzeit_ with a web search is worse than not routing it at all. `\b(regen)?schirm\b` does not fire on _Bildschirm_, since there is no word boundary inside a compound.
Widening a claim is where a skill starts stealing again, so each of these is anchored as tightly as the shape allows: `tell me about` and `erkläre mir` end at the name, so _tell me about the trip we planned_ is three tokens past matching; the optional article cannot widen the bare-name shape because the name is still the last token, which is what keeps _what is the capital of France?_ out of `lookup-term` (it is researched instead, as a leftover factual question); `add` needs both operands, so _add 3 more rows_ is not a sum; and the two-word `aktuelle …` fragment excludes the subjects the clock and the thermometer own, because answering _aktuelle Uhrzeit_ with a web search is worse than not routing it at all. `\b(regen)?schirm\b` does not fire on _Bildschirm_, since there is no word boundary inside a compound.

Two known misses are left deliberately. _GDP of Germany 2024_ needs a bare year to be a trigger, and that is exactly the rule that sent _I was born in 2024_ to a search engine. _What's 2 plus two?_ needs the spelled-out operand, which buys one phrasing and a new class of false positive.

Expand All @@ -521,11 +523,12 @@ What is loaded, and when:

`loadCatalog` reads only the frontmatter, so a skill's exemplars are parsed the first time that skill wins a turn and never for a skill that does not. A test proves it: a catalogue entry whose exemplar cannot parse routes perfectly well, and only throws when something asks for its body.

Routing runs three stages, cheapest and most certain first:
Routing runs four stages, cheapest and most certain first:

1. **Triggers** — the author's regexes, matched against the shape of a request. Precise, free, unable to hallucinate.
2. **Search** — an inverted index over curated `keywords`, ranked by inverse document frequency. This is what catches phrasings no trigger anticipated, including the languages the triggers are not written in. The app answers in the language it is asked in, and a German shape is written out as a trigger only where it needs something a keyword cannot say — an exclusion, like _wie spät ist es_ having to stand down when a city follows. _Zusammenfassung bitte_ and _Quadratwurzel von 144_ reach their skill through the index instead, and used to reach none.
3. **Carry-over** — _and in Lisbon?_ matches nothing by itself, and the skill that answered the question it continues is exactly the one it needs.
4. **Question** — a leftover factual question, like _What is the capital of France?_ or _Warum ist der Himmel blau?_, is researched rather than answered from training data. This is `isResearchable` in code, not a broader trigger: greetings (_wie geht's dir_, _how are you_), small talk, creative work and questions about the user or the assistant never reach it. A skill that already matched in stages 1–3 still wins.

Retrieval is lexical rather than semantic on purpose. RAG-MCP shows semantic retrieval of tool schemas beating a flat prompt three to one, 43.1% against 13.6% ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1)), and it is the right shape for hundreds of entries; for a handful of short ones, BM25-style scoring is where sparse retrieval is strongest, and a dense retriever would mean shipping a second model — 22 MB and up — into an app whose premise is one download and no server. The seam is in `retrieve.ts` if that changes: anything that can score an entry against a message can replace `search`.

Expand All @@ -535,7 +538,7 @@ Retrieval is lexical rather than semantic on purpose. RAG-MCP shows semantic ret

A skill that keeps applying to turns it has nothing to do with is worse than no skill: it spends context and narrows the tool list on a request that needed neither. So carry-over is deliberately hard to enter and easy to leave.

- A continuation has to either **say so** (`and`, `und`, `what about`) or be **too short to be asking anything of its own**. Length alone is not enough, and this is where the mechanism would turn harmful: _what is the capital of France?_ is six words, and answering it with the weather skill's exemplars resident would send the model searching for a fact it already knows.
- A continuation has to either **say so** (`and`, `und`, `what about`) or be **too short to be asking anything of its own**. Length alone is not enough, and this is where the mechanism would turn harmful: _what is the capital of France?_ is six words, and answering it with the weather skill's exemplars resident would send the model searching for a fact under the wrong tool list.
- It survives **two turns** on carry-over alone. Past that it has stopped being a continuation and become a default.
- It is dropped the moment another skill matches, the message asks something fresh, the turn closes the exchange (_thanks_), or a new chat starts.

Expand Down
13 changes: 13 additions & 0 deletions src/components/MessageItem.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,19 @@ describe('MessageItem', () => {
expect(screen.getByText('weather skill · matched “wetter”')).toBeInTheDocument()
})

it('names the skill a leftover factual question was researched with', () => {
render(
<MessageItem
message={message({
content: 'Paris.',
skill: { name: 'research-question', reason: 'question', matched: [] },
})}
/>,
)

expect(screen.getByText('research-question skill · researched')).toBeInTheDocument()
})

it('admits when a skill was carried over rather than matched', () => {
render(
<MessageItem
Expand Down
1 change: 1 addition & 0 deletions src/components/MessageItem.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ function describeReview(found: ReviewCheck[]): string {
*/
function describeSkill(applied: AppliedSkill): string {
if (applied.reason === 'carried-over') return `${applied.name} skill · carried over`
if (applied.reason === 'question') return `${applied.name} skill · researched`
const [matched] = applied.matched
if (applied.reason === 'search' && matched) return `${applied.name} skill · matched “${matched}”`
return `${applied.name} skill`
Expand Down
24 changes: 21 additions & 3 deletions src/eval/scenarios.ts
Original file line number Diff line number Diff line change
Expand Up @@ -300,11 +300,29 @@ export const SCENARIOS: Scenario[] = [
accept: (answer) => answer.trim().length > 10,
},
{
id: 'no-tool-capital',
category: 'no-tool',
id: 'web-factual-capital',
category: 'web',
// A leftover factual question used to be left to the model, which answered
// Paris from training data and would invent the ones it did not know. The
// question stage now researches it; greetings still do not.
prompt: 'What is the capital of France?',
expectTool: null,
expectTool: 'research',
accept: matches(/paris/i),
online: true,
},
{
id: 'no-tool-greeting',
category: 'no-tool',
prompt: "Wie geht's dir?",
expectTool: null,
accept: (answer) => answer.trim().length > 2,
},
{
id: 'no-tool-how-are-you',
category: 'no-tool',
prompt: 'How are you?',
expectTool: null,
accept: (answer) => answer.trim().length > 2,
},
{
id: 'no-tool-haiku',
Expand Down
10 changes: 9 additions & 1 deletion src/skills/activate.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -163,9 +163,17 @@ describe('the shipped skills', () => {
expect(activation?.tools.map((tool) => tool.schema.function.name)).toEqual(['weather'])
})

it('leaves the full tool list to a turn no skill routed', () => {
it('researches a leftover factual question instead of leaving the full tool list', () => {
const { activation } = activate('What is the capital of France?', shipped, builtinTools)

expect(activation?.skill.name).toBe('research-question')
expect(activation?.reason).toBe('question')
expect(activation?.tools.map((tool) => tool.schema.function.name)).toEqual(['research'])
})

it('leaves the full tool list to a turn no skill routed', () => {
const { activation } = activate('How are you?', shipped, builtinTools)

expect(activation).toBeNull()
})

Expand Down
12 changes: 9 additions & 3 deletions src/skills/library.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -322,6 +322,12 @@ describe('research-question', () => {
['Schau nach, was daraus geworden ist', 'search'],
['Finde heraus, wer das geschrieben hat', 'search'],
['aktuelle Nachrichten bitte', 'search'],
['What is the capital of France?', 'question'],
['What temperature does water boil at?', 'question'],
['Why is the sky blue?', 'question'],
['Warum ist der Himmel blau?', 'question'],
['What is 32 fahrenheit in celsius', 'question'],
['Was ist die Hauptstadt von Frankreich?', 'question'],
])('takes %j by %s', (message, how) => {
expect(routed(message)).toBe('research-question')
expect(reason(message)).toBe(how)
Expand Down Expand Up @@ -402,15 +408,12 @@ describe('priority and near misses', () => {

it.each([
'Write a two-line rhyme about rain.',
'What is the capital of France?',
'What is my favourite colour?',
'What temperature does water boil at?',
"I can't remember the capital of Peru.",
'Erzähl mir einen Witz',
'I was born in 2024',
'I currently live in Berlin',
'Was machst du heute?',
'What is 32 fahrenheit in celsius',
'What is that?',
'What is this?',
'What is it?',
Expand All @@ -427,6 +430,9 @@ describe('priority and near misses', () => {
'Wer ist das?',
'Wer ist es?',
'Hello there',
'How are you?',
'wie gehts dir',
"Wie geht's dir?",
'ok thanks',
// The instruction shapes lookup-term gained. Each one is a name away from
// matching, and none of them is a name.
Expand Down
16 changes: 16 additions & 0 deletions src/skills/research-question/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,22 @@ jarvis:
Friedrich Merz, seit Mai 2025.

Source: https://de.wikipedia.org/wiki/Bundeskanzler_(Deutschland) https://www.bundeskanzler.de
- user: What is the capital of France?
steps:
- tool: research
arguments:
query: capital of France
result: |
Researched 2026-08-31 for "capital of France" across 2 sources, all read in full.

1. Paris - Wikipedia — https://en.wikipedia.org/wiki/Paris
"Paris is the capital and most populous city of France."
2. France - Wikipedia — https://en.wikipedia.org/wiki/France
"The capital is Paris, a major European city and a global centre of art."
answer: |
Paris.

Source: https://en.wikipedia.org/wiki/Paris
---

Call `research` once. Answer from the quoted passages, in the language you were asked. Cite more than one source URL when several came back.
63 changes: 63 additions & 0 deletions src/skills/researchable.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
import { describe, expect, it } from 'vitest'
import { isResearchable } from './researchable'

describe('isResearchable', () => {
it.each([
'What is the capital of France?',
'What temperature does water boil at?',
'Why is the sky blue?',
'Warum ist der Himmel blau?',
'How does photosynthesis work?',
'What is 32 fahrenheit in celsius',
'Explain the French Revolution',
'Tell me about the history of Rome',
'Was ist die Hauptstadt von Frankreich?',
'Wie funktioniert Photosynthese?',
])('takes the factual question %j', (message) => {
expect(isResearchable(message)).toBe(true)
})

it.each([
'How are you?',
"How's it going?",
'wie gehts dir',
"Wie geht's dir?",
'Wie geht es dir?',
'wie gehts',
"what's up?",
"What's up",
'Hallo',
'Hello there',
'Was machst du heute?',
'Who are you?',
'What is your name?',
'What is my favourite colour?',
"What's my favourite colour?",
'Was ist mein Lieblingsessen?',
'Who is that?',
'What is this?',
'Was ist das?',
'Was ist los?',
'Write a two-line rhyme about rain.',
'Erzähl mir einen Witz',
'Can you write a poem?',
'thanks',
'I was born in 2024',
'I currently live in Berlin',
"I can't remember the capital of Peru.",
'ok cool',
'Tell me about yourself',
'Tell me about the trip we planned',
'Tell me about it',
'and in Lisbon?',
'Erklär mir das nochmal',
'Erkläre mir warum das nicht geht',
'Erkläre mir, wie das funktioniert',
])('leaves %j alone', (message) => {
expect(isResearchable(message)).toBe(false)
})

it('still researches news that only looks like a greeting at the start', () => {
expect(isResearchable("What's up in France?")).toBe(true)
})
})
Loading