From c605d127e04eb022bc0c07917c626ea84a024154 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 31 Aug 2026 10:54:05 +0000 Subject: [PATCH 1/3] Research leftover questions, and open pages that actually answer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A factual question no skill claimed was left to the 0.8B model, which answered from training data. Routing now researches those — capital of France, why the sky is blue — and still leaves greetings like wie geht's dir alone. The tool itself now searches Wikipedia alongside the web, ranks by snippet, replaces a blocked page from the remaining hits, and treats Bundeskanzler/Bundeskanzlers as one term. Co-authored-by: Sebastian --- README.md | 11 +- src/components/MessageItem.test.tsx | 13 + src/components/MessageItem.tsx | 1 + src/eval/scenarios.ts | 30 +- src/skills/activate.test.ts | 10 +- src/skills/library.test.ts | 12 +- src/skills/research-question/SKILL.md | 16 ++ src/skills/researchable.test.ts | 63 +++++ src/skills/researchable.ts | 99 +++++++ src/skills/route.test.ts | 57 +++- src/skills/route.ts | 17 +- src/tools/builtins.ts | 2 +- src/tools/research.test.ts | 238 +++++++++++++++- src/tools/research.ts | 382 +++++++++++++++++++++----- 14 files changed, 848 insertions(+), 103 deletions(-) create mode 100644 src/skills/researchable.test.ts create mode 100644 src/skills/researchable.ts diff --git a/README.md b/README.md index 9d010de..6ae1458 100644 --- a/README.md +++ b/README.md @@ -239,7 +239,9 @@ Search and `read_page` share the reader's budget of 20 requests a minute per IP, **The search itself now carries the facts a 0.8B model would otherwise spend a round guessing at.** Every `web_search` result is stamped with today's local date, so "current" and "today's news" have a date without calling `current_time`. A German question searches German Wikipedia and, on DuckDuckGo, prefers German results (`kl=de-de`); English _who was Ada Lovelace_ is not mistaken for German because bare `was` is also English. German Wikipedia is smaller, so an empty result there falls through to English rather than telling the model the subject does not exist. -The `research-question` skill does not offer `web_search`. It offers `research`, which searches, picks three **different sites** (`investor.nvidia.com` and `nvidianews.nvidia.com` count as one), reads them in parallel, and returns the passages that bear on the question. Wikipedia pages go through MediaWiki, so a typical call spends one reader request on the search and two on the other sites rather than six. A page that will not open becomes its search snippet; if nothing readable comes back the tool throws rather than telling the model the subject does not exist. `lookup-term` still searches and optionally reads one page — a name does not need three sources. +The `research-question` skill does not offer `web_search`. It offers `research`, which searches the live web and Wikipedia together, picks three **different sites** (Wikipedia first when it appeared, because MediaWiki is free and the lead paragraph usually answers; `investor.nvidia.com` and `nvidianews.nvidia.com` count as one), ranks the rest by whether their snippet already bears on the question, and reads them in parallel. A page that will not open, or that comes back as a login wall or a firewall interstitial, is replaced from the remaining hits rather than quoted; only when nothing else is left does the search snippet stand in. Inflected forms of a word still match — _Bundeskanzlers_ answers _Bundeskanzler_ — so a German page is not silent on a German question. If nothing readable comes back the tool throws rather than telling the model the subject does not exist. + +A factual question that no other skill claimed — _What is the capital of France?_, _Warum ist der Himmel blau?_ — still reaches this skill, as a fourth routing stage rather than a broader trigger. Greetings do not: _wie geht's dir_ and _how are you_ stay with the model. `lookup-term` still searches and optionally reads one page — a name does not need three sources. **LangSearch is the way off that shared budget without paying for one.** `api.langsearch.com` is a search API rather than a results page, its free tier allows 1,000 searches a day and one a second, and a key needs no card — so a search stops competing with `read_page` for the same 20 requests a minute. Two things about it are worth knowing before choosing it. Its snippets are index text rather than prose, lower-cased and with spaces around the punctuation, which a 0.8B model reads less confidently than a sentence. And it answers in an envelope: a refusal it decides to report with a 200 arrives as a `msg` and no result set, so `searchLangSearch` raises that rather than passing an empty list to a model that would relay it as "this does not exist". Long summaries are available per result and are switched off — each is the whole page behind the result, which would leave a 0.8B context with no room for the answer. @@ -497,7 +499,7 @@ The first four, and the two pronoun cases at the bottom, are the same mistake: a | _Fasse das zusammen_ | nothing | The object is usually a pronoun, and `seite zusammen` isn't | | _Welchen Wochentag haben wir?_ | nothing | `wochentag` was not among the clock's keywords | -Widening a claim is where a skill starts stealing again, so each of these is anchored as tightly as the shape allows: `tell me about` and `erkläre mir` end at the name, so _tell me about the trip we planned_ is three tokens past matching; the optional article cannot widen the bare-name shape because the name is still the last token, which is what keeps _what is the capital of France?_ out; `add` needs both operands, so _add 3 more rows_ is not a sum; and the two-word `aktuelle …` fragment excludes the subjects the clock and the thermometer own, because answering _aktuelle Uhrzeit_ with a web search is worse than not routing it at all. `\b(regen)?schirm\b` does not fire on _Bildschirm_, since there is no word boundary inside a compound. +Widening a claim is where a skill starts stealing again, so each of these is anchored as tightly as the shape allows: `tell me about` and `erkläre mir` end at the name, so _tell me about the trip we planned_ is three tokens past matching; the optional article cannot widen the bare-name shape because the name is still the last token, which is what keeps _what is the capital of France?_ out of `lookup-term` (it is researched instead, as a leftover factual question); `add` needs both operands, so _add 3 more rows_ is not a sum; and the two-word `aktuelle …` fragment excludes the subjects the clock and the thermometer own, because answering _aktuelle Uhrzeit_ with a web search is worse than not routing it at all. `\b(regen)?schirm\b` does not fire on _Bildschirm_, since there is no word boundary inside a compound. Two known misses are left deliberately. _GDP of Germany 2024_ needs a bare year to be a trigger, and that is exactly the rule that sent _I was born in 2024_ to a search engine. _What's 2 plus two?_ needs the spelled-out operand, which buys one phrasing and a new class of false positive. @@ -521,11 +523,12 @@ What is loaded, and when: `loadCatalog` reads only the frontmatter, so a skill's exemplars are parsed the first time that skill wins a turn and never for a skill that does not. A test proves it: a catalogue entry whose exemplar cannot parse routes perfectly well, and only throws when something asks for its body. -Routing runs three stages, cheapest and most certain first: +Routing runs four stages, cheapest and most certain first: 1. **Triggers** — the author's regexes, matched against the shape of a request. Precise, free, unable to hallucinate. 2. **Search** — an inverted index over curated `keywords`, ranked by inverse document frequency. This is what catches phrasings no trigger anticipated, including the languages the triggers are not written in. The app answers in the language it is asked in, and a German shape is written out as a trigger only where it needs something a keyword cannot say — an exclusion, like _wie spät ist es_ having to stand down when a city follows. _Zusammenfassung bitte_ and _Quadratwurzel von 144_ reach their skill through the index instead, and used to reach none. 3. **Carry-over** — _and in Lisbon?_ matches nothing by itself, and the skill that answered the question it continues is exactly the one it needs. +4. **Question** — a leftover factual question, like _What is the capital of France?_ or _Warum ist der Himmel blau?_, is researched rather than answered from training data. This is `isResearchable` in code, not a broader trigger: greetings (_wie geht's dir_, _how are you_), small talk, creative work and questions about the user or the assistant never reach it. A skill that already matched in stages 1–3 still wins. Retrieval is lexical rather than semantic on purpose. RAG-MCP shows semantic retrieval of tool schemas beating a flat prompt three to one, 43.1% against 13.6% ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1)), and it is the right shape for hundreds of entries; for a handful of short ones, BM25-style scoring is where sparse retrieval is strongest, and a dense retriever would mean shipping a second model — 22 MB and up — into an app whose premise is one download and no server. The seam is in `retrieve.ts` if that changes: anything that can score an entry against a message can replace `search`. @@ -535,7 +538,7 @@ Retrieval is lexical rather than semantic on purpose. RAG-MCP shows semantic ret A skill that keeps applying to turns it has nothing to do with is worse than no skill: it spends context and narrows the tool list on a request that needed neither. So carry-over is deliberately hard to enter and easy to leave. -- A continuation has to either **say so** (`and`, `und`, `what about`) or be **too short to be asking anything of its own**. Length alone is not enough, and this is where the mechanism would turn harmful: _what is the capital of France?_ is six words, and answering it with the weather skill's exemplars resident would send the model searching for a fact it already knows. +- A continuation has to either **say so** (`and`, `und`, `what about`) or be **too short to be asking anything of its own**. Length alone is not enough, and this is where the mechanism would turn harmful: _what is the capital of France?_ is six words, and answering it with the weather skill's exemplars resident would send the model searching for a fact under the wrong tool list. - It survives **two turns** on carry-over alone. Past that it has stopped being a continuation and become a default. - It is dropped the moment another skill matches, the message asks something fresh, the turn closes the exchange (_thanks_), or a new chat starts. diff --git a/src/components/MessageItem.test.tsx b/src/components/MessageItem.test.tsx index 528f94e..51a0b03 100644 --- a/src/components/MessageItem.test.tsx +++ b/src/components/MessageItem.test.tsx @@ -228,6 +228,19 @@ describe('MessageItem', () => { expect(screen.getByText('weather skill · matched “wetter”')).toBeInTheDocument() }) + it('names the skill a leftover factual question was researched with', () => { + render( + , + ) + + expect(screen.getByText('research-question skill · researched')).toBeInTheDocument() + }) + it('admits when a skill was carried over rather than matched', () => { render( answer.trim().length > 10, }, { - id: 'no-tool-capital', - category: 'no-tool', + id: 'web-factual-capital', + category: 'web', + // A leftover factual question used to be left to the model, which answered + // Paris from training data and would invent the ones it did not know. The + // question stage now researches it; greetings still do not. prompt: 'What is the capital of France?', - expectTool: null, + expectTool: 'research', accept: matches(/paris/i), + online: true, + }, + { + id: 'no-tool-greeting', + category: 'no-tool', + prompt: "Wie geht's dir?", + expectTool: null, + accept: (answer) => answer.trim().length > 2, + }, + { + id: 'no-tool-how-are-you', + category: 'no-tool', + prompt: 'How are you?', + expectTool: null, + accept: (answer) => answer.trim().length > 2, }, { id: 'no-tool-haiku', @@ -380,7 +398,8 @@ export const SCENARIOS: Scenario[] = [ // `who is` and `who won` were triggers and authorship was not, so the one // question a search engine answers best reached nothing. prompt: 'Who wrote Dune?', - expectTool: 'web_search', + expectTool: 'research', + acceptCall: (calls) => /dune/i.test(searchQuery(calls) ?? ''), accept: matches(/herbert/i), online: true, }, @@ -390,7 +409,8 @@ export const SCENARIOS: Scenario[] = [ // A figure a 0.8B model will otherwise invent, confidently and to three // significant figures. prompt: "What's the population of Tokyo?", - expectTool: 'web_search', + expectTool: 'research', + acceptCall: (calls) => /tokyo|tokio/i.test(searchQuery(calls) ?? ''), accept: (answer) => /\d/.test(answer), online: true, }, diff --git a/src/skills/activate.test.ts b/src/skills/activate.test.ts index 0a71e2f..69c37bf 100644 --- a/src/skills/activate.test.ts +++ b/src/skills/activate.test.ts @@ -163,9 +163,17 @@ describe('the shipped skills', () => { expect(activation?.tools.map((tool) => tool.schema.function.name)).toEqual(['weather']) }) - it('leaves the full tool list to a turn no skill routed', () => { + it('researches a leftover factual question instead of leaving the full tool list', () => { const { activation } = activate('What is the capital of France?', shipped, builtinTools) + expect(activation?.skill.name).toBe('research-question') + expect(activation?.reason).toBe('question') + expect(activation?.tools.map((tool) => tool.schema.function.name)).toEqual(['research']) + }) + + it('leaves the full tool list to a turn no skill routed', () => { + const { activation } = activate('How are you?', shipped, builtinTools) + expect(activation).toBeNull() }) diff --git a/src/skills/library.test.ts b/src/skills/library.test.ts index 31fa730..323a0d0 100644 --- a/src/skills/library.test.ts +++ b/src/skills/library.test.ts @@ -322,6 +322,12 @@ describe('research-question', () => { ['Schau nach, was daraus geworden ist', 'search'], ['Finde heraus, wer das geschrieben hat', 'search'], ['aktuelle Nachrichten bitte', 'search'], + ['What is the capital of France?', 'question'], + ['What temperature does water boil at?', 'question'], + ['Why is the sky blue?', 'question'], + ['Warum ist der Himmel blau?', 'question'], + ['What is 32 fahrenheit in celsius', 'question'], + ['Was ist die Hauptstadt von Frankreich?', 'question'], ])('takes %j by %s', (message, how) => { expect(routed(message)).toBe('research-question') expect(reason(message)).toBe(how) @@ -402,15 +408,12 @@ describe('priority and near misses', () => { it.each([ 'Write a two-line rhyme about rain.', - 'What is the capital of France?', 'What is my favourite colour?', - 'What temperature does water boil at?', "I can't remember the capital of Peru.", 'Erzähl mir einen Witz', 'I was born in 2024', 'I currently live in Berlin', 'Was machst du heute?', - 'What is 32 fahrenheit in celsius', 'What is that?', 'What is this?', 'What is it?', @@ -427,6 +430,9 @@ describe('priority and near misses', () => { 'Wer ist das?', 'Wer ist es?', 'Hello there', + 'How are you?', + 'wie gehts dir', + "Wie geht's dir?", 'ok thanks', // The instruction shapes lookup-term gained. Each one is a name away from // matching, and none of them is a name. diff --git a/src/skills/research-question/SKILL.md b/src/skills/research-question/SKILL.md index 2e58bfd..af6f06a 100644 --- a/src/skills/research-question/SKILL.md +++ b/src/skills/research-question/SKILL.md @@ -84,6 +84,22 @@ jarvis: Friedrich Merz, seit Mai 2025. Source: https://de.wikipedia.org/wiki/Bundeskanzler_(Deutschland) https://www.bundeskanzler.de + - user: What is the capital of France? + steps: + - tool: research + arguments: + query: capital of France + result: | + Researched 2026-08-31 for "capital of France" across 2 sources, all read in full. + + 1. Paris - Wikipedia — https://en.wikipedia.org/wiki/Paris + "Paris is the capital and most populous city of France." + 2. France - Wikipedia — https://en.wikipedia.org/wiki/France + "The capital is Paris, a major European city and a global centre of art." + answer: | + Paris. + + Source: https://en.wikipedia.org/wiki/Paris --- Call `research` once. Answer from the quoted passages, in the language you were asked. Cite more than one source URL when several came back. diff --git a/src/skills/researchable.test.ts b/src/skills/researchable.test.ts new file mode 100644 index 0000000..a6c0993 --- /dev/null +++ b/src/skills/researchable.test.ts @@ -0,0 +1,63 @@ +import { describe, expect, it } from 'vitest' +import { isResearchable } from './researchable' + +describe('isResearchable', () => { + it.each([ + 'What is the capital of France?', + 'What temperature does water boil at?', + 'Why is the sky blue?', + 'Warum ist der Himmel blau?', + 'How does photosynthesis work?', + 'What is 32 fahrenheit in celsius', + 'Explain the French Revolution', + 'Tell me about the history of Rome', + 'Was ist die Hauptstadt von Frankreich?', + 'Wie funktioniert Photosynthese?', + ])('takes the factual question %j', (message) => { + expect(isResearchable(message)).toBe(true) + }) + + it.each([ + 'How are you?', + "How's it going?", + 'wie gehts dir', + "Wie geht's dir?", + 'Wie geht es dir?', + 'wie gehts', + "what's up?", + "What's up", + 'Hallo', + 'Hello there', + 'Was machst du heute?', + 'Who are you?', + 'What is your name?', + 'What is my favourite colour?', + "What's my favourite colour?", + 'Was ist mein Lieblingsessen?', + 'Who is that?', + 'What is this?', + 'Was ist das?', + 'Was ist los?', + 'Write a two-line rhyme about rain.', + 'Erzähl mir einen Witz', + 'Can you write a poem?', + 'thanks', + 'I was born in 2024', + 'I currently live in Berlin', + "I can't remember the capital of Peru.", + 'ok cool', + 'Tell me about yourself', + 'Tell me about the trip we planned', + 'Tell me about it', + 'and in Lisbon?', + 'Erklär mir das nochmal', + 'Erkläre mir warum das nicht geht', + 'Erkläre mir, wie das funktioniert', + ])('leaves %j alone', (message) => { + expect(isResearchable(message)).toBe(false) + }) + + it('still researches news that only looks like a greeting at the start', () => { + expect(isResearchable("What's up in France?")).toBe(true) + }) +}) diff --git a/src/skills/researchable.ts b/src/skills/researchable.ts new file mode 100644 index 0000000..82132d8 --- /dev/null +++ b/src/skills/researchable.ts @@ -0,0 +1,99 @@ +import { contentTerms } from './retrieve' +import type { SkillEntry } from './types' + +/** + * Whether a message that matched no skill is still a question worth researching. + * + * Triggers and the keyword index catch the shapes the author wrote. What they + * miss is an ordinary factual question — *What is the capital of France?*, + * *Warum ist der Himmel blau?* — which a 0.8B model will otherwise answer from + * training data, confidently and often wrongly. Greeting, small talk, creative + * work and questions about the user or the assistant are not that: researching + * *wie geht's dir* is the failure this exists to avoid. + * + * This runs in code on purpose. Asking the model whether to research spends the + * generation the skill exists to skip, and a regex on the skill itself cannot + * hold the exclusions without stealing greetings the way `who is` once stole + * *Who is that?*. + */ +export const RESEARCH_SKILL = 'research-question' + +/** A question word at the start, in either language the app is asked in. */ +const INTERROGATIVE = + /^\s*(what|whats|which|who|when|where|why|how|is|are|does|do|did|can|could|should|would|wer|was|wann|wo|warum|wieso|weshalb|welche[rsn]?|ist|sind|hat|haben|kannst|k(ö|oe)nntest|erkl(ä|ae)r)/i + +/** Asked as an instruction to look something up, which is the same job. */ +const LOOKUP = /^\s*(tell me about|explain|look up|find out|search for|google|schau nach|finde heraus)\b/i + +/** + * *Erkläre mir warum das nicht geht* is shaped like a lookup and is about this + * conversation. The object has to be a public subject, or this becomes a search + * for a pronoun. + */ +const IN_CONTEXT = + /^\s*erkl(ä|ae)r(?:e|st)?\s+(?:mir|uns)[,\s]+(?:warum|wieso|wie|das|dies|es)\b|^\s*explain\s+(?:why|how)\s+(?:this|that|it)\b/i + +/** + * *Tell me about the trip we planned* is shaped like a lookup and is about us. + * The object has to be a public subject, or this becomes a search for a holiday. + */ +const PERSONAL_OBJECT = + /\b(we|our|us|my|mine|it|that|this|wir|unser|unsere[nms]?|mich|mir|ich|dir|dich|du)\b/i + +/** A follow-up fragment, not a question of its own. Carry-over owns these. */ +const CONTINUES = /^\s*(and|und|auch|also|plus|what about|how about|was ist mit|oh and)\b/i + +/** + * Small talk, including the German greeting that is shaped exactly like a + * question. Anchored where a longer message with the same words is actually + * asking — *what's up in France* is news; *what's up?* is not. + */ +const GREETING = + /^\s*(?:hi|hey|hello|hallo|moin|servus|hiya|yo|howdy)\b|^\s*(?:how are you|how're you|how are ya|how'?s it going|how do you do|how have you been)\b|^\s*(?:what'?s up|was geht(?:\s+ab)?|na)\s*\??\s*$|^\s*wie\s+geht(?:'?s|\s+es)\b|^\s*alles\s+klar\s*\??\s*$/i + +/** The assistant, not a public fact. Whole-message shapes, so *what are you supposed to do if…* survives. */ +const ABOUT_ASSISTANT = + /^\s*(?:who are you|what are you|what('?s| is) your name|what can you (?:do|help)|how (?:do|can) you help|can you help(?: me)?|tell me about yourself)\s*\??\s*$|^\s*(?:wer bist du|was bist du|wie hei(ß|ss)t du|was kannst du|was machst du|kannst du (?:mir )?helfen|erz(ä|ae)hl(?: e)? (?:mir )?von dir)\b/i + +/** The user's own attributes. A favourite colour is recall, not a search. */ +const ABOUT_USER = + /^\s*(?:what('?s| is)|was ist)\s+(?:my|mein|meine)\b|\b(?:my favourite|mein lieblings)|^\s*(?:wie hei(ß|ss)e ich|what's my|what is my)\b/i + +/** + * A pronoun where a subject should be. *Who is that?* used to fire `who is` and + * send a search engine a word with no referent. + */ +const PRONOUN_QUESTION = + /^\s*(?:who|what|which|wer|was|welche[rsn]?)\s+(?:is|are|was|were|ist|sind|war|waren)\s+(?:that|this|it|these|those|they|them|there|das|dies|es|los)\s*\??\s*$/i + +/** + * Creative work asked as a question. *Can you write a poem?* starts with `can` + * and would otherwise look like a request to look something up. A factual + * *what is a haiku?* still researches, because it does not start with write. + */ +const CREATIVE = + /^\s*(?:write|schreib(?:e|en)?|erz(ä|ae)hl(?:e)?|dichte|make up|compose|tell me|can you|could you|kannst du)\b[\s\S]*\b(?:haiku|rhyme|poem|limerick|joke|witz|gedicht|geschichte|lied|song)\b/i + +/** The exchange is closing, not asking. Same list `isFollowUp` already uses. */ +const CLOSES = /^\s*(thanks|thank you|thx|cheers|ok|okay|cool|nice|great|danke|dankesch(ö|oe)n|bye|ciao)\b/i + +function isQuestion(message: string): boolean { + return /[?？]/.test(message) || INTERROGATIVE.test(message) || LOOKUP.test(message) +} + +export function isResearchable(message: string): boolean { + const text = message.trim() + if (!text) return false + if (CLOSES.test(text) || GREETING.test(text) || ABOUT_ASSISTANT.test(text) || ABOUT_USER.test(text)) { + return false + } + if (CONTINUES.test(text)) return false + if (PRONOUN_QUESTION.test(text) || CREATIVE.test(text) || IN_CONTEXT.test(text)) return false + if (!isQuestion(text)) return false + if (LOOKUP.test(text) && PERSONAL_OBJECT.test(text.replace(LOOKUP, ''))) return false + return contentTerms(text).length > 0 +} + +export function researchSkill(catalog: SkillEntry[]): SkillEntry | undefined { + return catalog.find((entry) => entry.name === RESEARCH_SKILL) +} diff --git a/src/skills/route.test.ts b/src/skills/route.test.ts index cf4eb5f..73b1533 100644 --- a/src/skills/route.test.ts +++ b/src/skills/route.test.ts @@ -178,11 +178,11 @@ describe('routing by search', () => { describe('routing nothing at all', () => { it.each([ 'Write a two-line rhyme about rain.', - 'What is the capital of France?', // Answered from what recall put in the prompt, with no tool round spent. 'What is my favourite colour?', // Physics, not this afternoon: the word alone must not pull in the weather. - 'What temperature does water boil at?', + // It *is* a question with a public subject, so research takes it; the + // weather skill must still not. // The user's own recall, not the app's: neither asks for anything stored. "I can't remember the capital of Peru.", 'Erzähl mir einen Witz', @@ -190,11 +190,8 @@ describe('routing nothing at all', () => { 'I was born in 2024', 'I currently live in Berlin', // `heute` was a current-date keyword, which turned every mention of today - // into a question about the date. + // into a question about the date. It is also about the assistant. 'Was machst du heute?', - // A conversion, not a name: the digit-bearing token lookup-term matches has - // to carry letters too, or `1inch` and `32` are the same shape to it. - 'What is 32 fahrenheit in celsius', // A continuation that names a city is not itself a clock question: it is // carried only while a clock skill is already resident. 'and in germany', @@ -209,6 +206,9 @@ describe('routing nothing at all', () => { 'Wer ist das?', 'Wer ist es?', 'Was ist los?', + 'How are you?', + 'wie gehts dir', + "Wie geht's dir?", ])('leaves %j to the model', (message) => { // Firing a tool-shaped skill on plain conversation is the failure mode that // makes a small model reach for tools it does not need. @@ -216,6 +216,24 @@ describe('routing nothing at all', () => { }) }) +describe('routing a leftover factual question', () => { + it.each([ + 'What is the capital of France?', + 'What temperature does water boil at?', + 'Why is the sky blue?', + 'Warum ist der Himmel blau?', + 'How does photosynthesis work?', + // A conversion, not a name: lookup-term's digit-bearing token has to carry + // letters too, or `1inch` and `32` are the same shape to it. Nothing else + // owns it, and a 0.8B model will invent the number. + 'What is 32 fahrenheit in celsius', + 'Was ist die Hauptstadt von Frankreich?', + ])('routes %j to research-question', (message) => { + expect(routed(message)).toBe('research-question') + expect(reason(message)).toBe('question') + }) +}) + describe('keeping a skill across a follow-up', () => { const resident: SkillMemory = { name: 'weather', carried: 0 } @@ -243,14 +261,23 @@ describe('keeping a skill across a follow-up', () => { expect(route('and in Lisbon?', catalog, stale).memory).toBeNull() }) - it.each([ - 'What is the capital of France?', - 'Write me a haiku about trains', - 'thanks!', - 'Danke, das war alles', - ])('evicts it when %j asks something of its own', (message) => { - expect(routed(message, resident)).toBeNull() - expect(route(message, catalog, resident).memory).toBeNull() + it.each(['Write me a haiku about trains', 'thanks!', 'Danke, das war alles'])( + 'evicts it when %j asks something of its own', + (message) => { + expect(routed(message, resident)).toBeNull() + expect(route(message, catalog, resident).memory).toBeNull() + }, + ) + + it('replaces a resident skill when a leftover question is worth researching', () => { + // Not a continuation: it names its own subject. Researching it is the + // point of the question stage; answering it with the weather skill would + // send the model searching for a fact under the wrong exemplars. + const routing = route('What is the capital of France?', catalog, resident) + + expect(routing.route?.entry.name).toBe('research-question') + expect(routing.route?.reason).toBe('question') + expect(routing.memory).toEqual({ name: 'research-question', carried: 0 }) }) it('replaces it when another skill matches outright', () => { @@ -289,7 +316,7 @@ describe('isFollowUp', () => { it.each([ // Six words, and answering it with another skill's exemplars resident would - // send the model searching for a fact it already knows. + // send the model searching for a fact under the wrong tool list. 'What is the capital of France?', 'Who wrote Dune?', 'thanks', diff --git a/src/skills/route.ts b/src/skills/route.ts index ff942d6..dee747d 100644 --- a/src/skills/route.ts +++ b/src/skills/route.ts @@ -1,3 +1,4 @@ +import { isResearchable, researchSkill } from './researchable' import { search } from './retrieve' import type { SkillEntry } from './types' @@ -5,7 +6,7 @@ import type { SkillEntry } from './types' * Choosing a skill, keeping it while it is useful, and dropping it when it is * not. * - * Three stages, cheapest and most certain first: + * Four stages, cheapest and most certain first: * * 1. **Triggers.** A regex the author wrote for the shape of a request. Precise, * free, and unable to hallucinate. @@ -14,13 +15,17 @@ import type { SkillEntry } from './types' * 3. **Carry-over.** A follow-up like *and in Lisbon?* matches nothing on its * own, and the skill that answered the question it continues is exactly the * one it needs. + * 4. **Question.** A factual question that still matched nothing — *What is the + * capital of France?* — is researched rather than answered from training + * data. Greetings, small talk and questions about the user or the assistant + * never reach this stage. * * Stage 3 is the only stateful part, and it is deliberately hard to enter and * easy to leave. A skill that keeps applying to turns it has nothing to do with * is worse than no skill: it spends context and narrows the tool list on a * request that needed neither. */ -export type RouteReason = 'trigger' | 'search' | 'carried-over' +export type RouteReason = 'trigger' | 'search' | 'carried-over' | 'question' export interface Route { entry: SkillEntry @@ -117,6 +122,14 @@ export function route(message: string, catalog: SkillEntry[], memory: SkillMemor } } + const research = researchSkill(catalog) + if (research && isResearchable(message)) { + return { + route: { entry: research, reason: 'question', matched: [] }, + memory: { name: research.name, carried: 0 }, + } + } + // Nothing matched and nothing is worth keeping resident, so the skill is // dropped here rather than lingering into a conversation it left behind. return { route: null, memory: null } diff --git a/src/tools/builtins.ts b/src/tools/builtins.ts index 4c39080..e743e3c 100644 --- a/src/tools/builtins.ts +++ b/src/tools/builtins.ts @@ -146,7 +146,7 @@ export const currentTime = defineTool( function createResearch(config: WebAccessConfig): Tool { return defineTool( 'research', - 'Search the web, read the three most independent results and return quoted passages from each. Use for current events, people, organisations, or anything you would otherwise be guessing at.', + 'Search the web, read the three most independent results and return quoted passages from each. Use for current events, people, organisations, facts you would otherwise be guessing at, or any question whose answer can be looked up.', { type: 'object', properties: { query: { type: 'string', description: 'The question to research' } }, diff --git a/src/tools/research.test.ts b/src/tools/research.test.ts index 4b37bad..4bfc2d4 100644 --- a/src/tools/research.test.ts +++ b/src/tools/research.test.ts @@ -1,5 +1,15 @@ import { afterEach, describe, expect, it, vi } from 'vitest' -import { digest, diverseFirst, looksBlocked, paragraphsOf, passagesFor, researchQuestion } from './research' +import { + digest, + diverseFirst, + isUnreadableUrl, + looksBlocked, + paragraphsOf, + passagesFor, + pickCandidates, + related, + researchQuestion, +} from './research' import type { SearchResult, WebAccessConfig } from './web' function result(url: string, title = 'Title', snippet = 'A snippet.'): SearchResult { @@ -82,6 +92,85 @@ describe('diverseFirst', () => { }) }) +describe('related', () => { + it('treats a German inflection as the same word', () => { + expect(related('bundeskanzler', 'bundeskanzlers')).toBe(true) + expect(related('kanzler', 'kanzlerin')).toBe(true) + }) + + it('does not treat a short stem as every word that starts with it', () => { + expect(related('news', 'newspaper')).toBe(false) + expect(related('the', 'there')).toBe(false) + }) +}) + +describe('isUnreadableUrl', () => { + it('flags a login wall and a PDF gallery', () => { + expect(isUnreadableUrl('https://example.com/login')).toBe(true) + expect(isUnreadableUrl('https://nvidianews.nvidia.com/_gallery/download_pdf/68af/')).toBe(true) + }) + + it('leaves an ordinary article alone', () => { + expect(isUnreadableUrl('https://investor.nvidia.com/news/q2/')).toBe(false) + }) +}) + +describe('pickCandidates', () => { + const question = 'Who is the chief executive of Nvidia?' + + it('puts a Wikipedia article ahead of other sites, because MediaWiki is free', () => { + const chosen = pickCandidates(question, [ + result('https://www.reuters.com/nvidia', 'Reuters', 'Nvidia named a new chief executive.'), + result('https://en.wikipedia.org/wiki/Nvidia', 'Nvidia', 'Nvidia is a computing company.'), + result('https://www.bbc.co.uk/nvidia', 'BBC', 'Nvidia chief executive spoke today.'), + ]) + + expect(chosen[0]?.url).toBe('https://en.wikipedia.org/wiki/Nvidia') + }) + + it('ranks other sites by whether the snippet already answers the question', () => { + const chosen = pickCandidates(question, [ + result('https://flights.example/routes', 'Routes', 'The airline flies to thirty cities.'), + result( + 'https://fictionalairways.example/leadership', + 'Leadership', + 'Ama Osei was appointed chief executive in March 2023.', + ), + ]) + + expect(chosen[0]?.url).toBe('https://fictionalairways.example/leadership') + }) + + it('demotes a login wall behind pages that can actually be read', () => { + const chosen = pickCandidates(question, [ + result('https://example.com/login', 'Sign in', 'Sign in to read about the chief executive.'), + result('https://news.example/nvidia', 'News', 'Nvidia chief executive Jensen Huang.'), + ]) + + expect(chosen.map((entry) => entry.url)).toEqual([ + 'https://news.example/nvidia', + 'https://example.com/login', + ]) + }) + + it('does not let a second Wikipedia article crowd out another site', () => { + const chosen = pickCandidates(question, [ + result('https://en.wikipedia.org/wiki/Nvidia', 'Nvidia', 'Nvidia is a computing company.'), + result( + 'https://en.wikipedia.org/wiki/Jensen_Huang', + 'Huang', + 'Jensen Huang is Nvidia chief executive.', + ), + result('https://www.reuters.com/nvidia', 'Reuters', 'Nvidia chief executive spoke today.'), + ]) + + expect(chosen.slice(0, 2).map((entry) => entry.url)).toEqual([ + 'https://en.wikipedia.org/wiki/Nvidia', + 'https://www.reuters.com/nvidia', + ]) + }) +}) + describe('paragraphsOf', () => { it('keeps link text and drops the address', () => { const [paragraph] = paragraphsOf( @@ -286,6 +375,17 @@ describe('passagesFor', () => { expect(first).toContain('Ama Osei') }) + it('matches a German inflection, so the answering sentence is not silent', () => { + const page = [ + 'Die Bundesrepublik hat eine parlamentarische Demokratie mit einem Bundestag.', + 'Das Amt des Bundeskanzlers übt Friedrich Merz seit dem 6. Mai 2025 aus.', + ].join('\n\n') + + const [first] = forOnePage('Wer ist der Bundeskanzler?', page) + + expect(first).toContain('Friedrich Merz') + }) + it('returns at most two passages, however long the page is', () => { expect(forOnePage('airline chief executive founded Accra 1974', PAGE)).toHaveLength(2) }) @@ -439,6 +539,7 @@ describe('digest', () => { /** * The whole call, over a stubbed network: LangSearch answers the search and the * reader answers each page, which is the shape every provider reduces to. + * Wikipedia is searched alongside and answers empty unless a test fills it. */ describe('researchQuestion', () => { const config: WebAccessConfig = { provider: 'langsearch', langsearchApiKey: 'key' } @@ -446,16 +547,28 @@ describe('researchQuestion', () => { const LEADERSHIP = 'https://fictionalairways.example/leadership' const AIRTIMES = 'https://airtimes.example/osei' + const GAZETTE = 'https://gazette.example/osei' + const WIKI = 'https://en.wikipedia.org/wiki/Fictional_Airways' const HITS = [ { name: 'Leadership', url: LEADERSHIP, snippet: 'Ama Osei leads the airline.' }, { name: 'Airtimes', url: AIRTIMES, snippet: 'The board appointed Ama Osei in 2023.' }, ] + const LEADERSHIP_PAGE = 'Ama Osei has led Fictional Airways as chief executive since March 2023 in Accra.' + const AIRTIMES_PAGE = + 'The board appointed Ama Osei as chief executive in March 2023, succeeding Piet Hendriks.' + const WIKI_EXTRACT = + 'Fictional Airways is an airline based in Accra. Ama Osei has been its chief executive since 2023.' + /** A page body, or the status the reader should refuse it with. */ type Reply = string | number - function stubNetwork(pages: Record, hits = HITS) { + function stubNetwork( + pages: Record, + hits = HITS, + wiki: { title: string; url: string; extract: string }[] = [], + ) { const calls: string[] = [] vi.stubGlobal( @@ -468,6 +581,53 @@ describe('researchQuestion', () => { return { ok: true, status: 200, json: async () => ({ data: { webPages: { value: hits } } }) } } + if (url.includes('wikipedia.org')) { + const parsed = new URL(url) + if (parsed.searchParams.has('gsrsearch')) { + return { + ok: true, + status: 200, + json: async () => ({ + query: { + pages: Object.fromEntries( + wiki.map((entry, at) => [ + String(at + 1), + { + pageid: at + 1, + title: entry.title, + index: at + 1, + extract: entry.extract, + fullurl: entry.url, + }, + ]), + ), + }, + }), + } + } + const requested = (parsed.searchParams.get('titles') ?? '').replace(/_/g, ' ') + const match = wiki.find((entry) => entry.title.replace(/_/g, ' ') === requested) + if (match) { + return { + ok: true, + status: 200, + json: async () => ({ + query: { + pages: { + '1': { + pageid: 1, + title: match.title, + extract: match.extract, + fullurl: match.url, + }, + }, + }, + }), + } + } + return { ok: true, status: 200, json: async () => ({ batchcomplete: '' }) } + } + const target = url.replace('https://r.jina.ai/', '') const reply = pages[target] if (reply === undefined || typeof reply === 'number') { @@ -491,8 +651,8 @@ describe('researchQuestion', () => { it('quotes every source it could read, each against its own URL', async () => { stubNetwork({ - [LEADERSHIP]: 'Ama Osei has led Fictional Airways as chief executive since March 2023 in Accra.', - [AIRTIMES]: 'The board appointed Ama Osei as chief executive in March 2023, succeeding Piet Hendriks.', + [LEADERSHIP]: LEADERSHIP_PAGE, + [AIRTIMES]: AIRTIMES_PAGE, }) const result = await researchQuestion(question, config) @@ -508,15 +668,18 @@ describe('researchQuestion', () => { it('reads the pages at the same time rather than one after another', async () => { const calls = stubNetwork({ - [LEADERSHIP]: 'Ama Osei has led Fictional Airways as chief executive since March 2023 in Accra.', - [AIRTIMES]: 'The board appointed Ama Osei as chief executive in March 2023, succeeding Piet Hendriks.', + [LEADERSHIP]: LEADERSHIP_PAGE, + [AIRTIMES]: AIRTIMES_PAGE, }) await researchQuestion(question, config) - // One search and one read per source: the cost of the call is what the - // reader's per-minute budget is spent on, so it is worth pinning. - expect(calls).toHaveLength(3) + const searches = calls.filter( + (url) => url.startsWith('https://api.langsearch.com') || url.includes('gsrsearch'), + ) + const reads = calls.filter((url) => url.startsWith('https://r.jina.ai/')) + expect(searches).toHaveLength(2) + expect(reads).toHaveLength(2) }) /** @@ -526,7 +689,7 @@ describe('researchQuestion', () => { */ it('stands a page that would not open in as its search snippet', async () => { stubNetwork({ - [LEADERSHIP]: 'Ama Osei has led Fictional Airways as chief executive since March 2023 in Accra.', + [LEADERSHIP]: LEADERSHIP_PAGE, [AIRTIMES]: 429, }) @@ -536,6 +699,61 @@ describe('researchQuestion', () => { expect(result).toContain('"The board appointed Ama Osei in 2023."') }) + it('replaces a blocked page from the remaining hits rather than quoting the firewall', async () => { + const fourth = 'https://profile.example/osei' + stubNetwork( + { + [LEADERSHIP]: 'Sucuri WebSite Firewall - Access Denied. Time: 2026-08-31.', + [AIRTIMES]: AIRTIMES_PAGE, + [GAZETTE]: 'Ama Osei took office as chief executive of Fictional Airways in Accra in 2023.', + [fourth]: 'Ama Osei joined Fictional Airways from the civil aviation authority in 2014.', + }, + [ + { + name: 'Leadership', + url: LEADERSHIP, + snippet: 'Ama Osei is the chief executive of Fictional Airways.', + }, + ...HITS.slice(1), + { name: 'Gazette', url: GAZETTE, snippet: 'Ama Osei took office in Accra.' }, + { name: 'Profile', url: fourth, snippet: 'A profile of the airline.' }, + ], + ) + + const result = await researchQuestion(question, config) + + expect(result).not.toContain('Access Denied') + expect(result).toContain(fourth) + expect(result).toContain('all read in full') + }) + + it('searches Wikipedia alongside the web and reads the article through MediaWiki', async () => { + stubNetwork( + { + [LEADERSHIP]: LEADERSHIP_PAGE, + [AIRTIMES]: AIRTIMES_PAGE, + }, + HITS, + [{ title: 'Fictional Airways', url: WIKI, extract: WIKI_EXTRACT }], + ) + + const result = await researchQuestion(question, config) + + expect(result).toContain(WIKI) + expect(result).toContain('Ama Osei has been its chief executive since 2023') + expect(result).toContain('across 3 sources, all read in full') + }) + + it('does not search Wikipedia twice when Wikipedia is already the provider', async () => { + const calls = stubNetwork({}, [], [{ title: 'Fictional Airways', url: WIKI, extract: WIKI_EXTRACT }]) + + await researchQuestion(question, { provider: 'wikipedia' }) + + const searches = calls.filter((url) => url.includes('gsrsearch')) + expect(searches).toHaveLength(1) + expect(calls.some((url) => url.startsWith('https://api.langsearch.com'))).toBe(false) + }) + it('says so when no page opened and only snippets are left', async () => { stubNetwork({ [LEADERSHIP]: 500, [AIRTIMES]: 429 }) diff --git a/src/tools/research.ts b/src/tools/research.ts index 9d7b6ef..55fa667 100644 --- a/src/tools/research.ts +++ b/src/tools/research.ts @@ -14,12 +14,16 @@ * so a typical call is one search and two reads. Five sources was six requests * and three questions in a minute before the rate limit. * - * The pages are read in full and quoted in part. Selection is lexical: paragraphs - * are scored by the question's terms, each weighted by how rare it is across - * every paragraph the turn fetched. + * Wikipedia is fetched alongside the web, because MediaWiki is free and the + * lead paragraph is usually the sentence that names the person. A page that + * comes back as a firewall, a login wall or empty prose is replaced from the + * remaining hits rather than quoted; the search snippet only stands in when + * nothing else could be opened. Passages are scored lexically against the + * question, with inflected forms of a word counting as the same term, so a + * German page is not silent on a German question. */ -import { readPage, searchWeb, type SearchResult, type WebAccessConfig } from './web' +import { readPage, searchWeb, wikipediaPage, type SearchResult, type WebAccessConfig } from './web' /** * How many results to ask for before narrowing them. Larger than `MAX_SOURCES` @@ -31,6 +35,19 @@ const SEARCH_LIMIT = 8 /** Three independent sites. A fourth is usually the same claim from a mirror. */ const MAX_SOURCES = 3 +/** + * How many page-reads a turn may spend filling those three slots. + * + * The first wave is `MAX_SOURCES` in parallel. A blocked or empty page spends + * one of the remainder on a replacement rather than quoting its snippet while + * unread hits sit unused. Five is one search-plus-three plus two retries, still + * inside the reader's 20-a-minute budget for a single question. + */ +const MAX_READ_ATTEMPTS = 5 + +/** Enough Wikipedia hits to have a lead article after disambiguations are demoted. */ +const WIKI_SEARCH_LIMIT = 3 + /** Two passages carry a claim and its context. A third is usually the same claim again. */ const MAX_PASSAGES_PER_SOURCE = 2 @@ -55,6 +72,32 @@ function words(text: string): string[] { return [...text.toLowerCase().matchAll(WORD)].map((match) => match[0]) } +/** + * Whether two tokens are the same word in different clothes. + * + * German office titles inflect: a question about the *Bundeskanzler* is + * answered by a sentence about the *Bundeskanzlers* Amt, and treating those as + * unrelated is what made a German page silent on a German question. A shared + * prefix of five letters, with only a short suffix on either side, catches the + * inflections without treating *news* as *newspaper*. + */ +export function related(a: string, b: string): boolean { + if (a === b) return true + if (a.length < 5 || b.length < 5) return false + const n = Math.min(a.length, b.length) + let i = 0 + while (i < n && a[i] === b[i]) i += 1 + return i >= 5 && a.length - i <= 4 && b.length - i <= 4 +} + +function holds(haystack: Set, term: string): boolean { + if (haystack.has(term)) return true + for (const word of haystack) { + if (related(term, word)) return true + } + return false +} + function collapse(value: string): string { return value.replace(/\s+/g, ' ').trim() } @@ -76,6 +119,35 @@ function hostOf(url: string): string { } } +function isWikipediaUrl(url: string): boolean { + try { + return wikipediaPage(new URL(url)) !== null + } catch { + return false + } +} + +/** + * A URL that will not yield a page worth quoting. + * + * Login walls survive the reader as a 200 with prose on them, and then score + * against the question because they mention the site. `download_pdf` is the + * NVIDIA gallery that used to occupy a source slot with a binary rather than + * an article. Demoted rather than dropped, so a search that returned nothing + * else still has something to try. + */ +export function isUnreadableUrl(url: string): boolean { + try { + const path = new URL(url).pathname.toLowerCase() + return ( + /\/(?:login|log-in|signin|sign-in|sign-up|signup|register|consent)(?:\/|$)/.test(path) || + path.includes('download_pdf') + ) + } catch { + return true + } +} + /** * Compound public suffixes where the last two labels are not the site. * @@ -285,37 +357,168 @@ function inverseFrequency(documentFrequency: number, total: number): number { } /** - * What each word of the question is worth, measured over every paragraph the - * turn fetched rather than over one page at a time. - * - * Pooling is what makes a stop list unnecessary, and it has to be pooled to - * work: across a hundred paragraphs *the* appears in nearly all of them and ends - * up worth about two per cent of *executive*, but within a single four-paragraph - * page it can be exactly as rare and score just as high. Measured on one page, - * "who is the chief executive of the airline" ranked the paragraph containing - * *the airline* level with the one naming the chief executive. + * Question-shell words. Pooling idf across a hundred paragraphs makes a stop + * list unnecessary — *the* appears everywhere and ends up worth about two per + * cent of *executive*. Across the three pages a research call actually reads, + * *the* can be exactly as rare as *executive* and rank the wrong paragraph + * first. Dropping the shell is what keeps *who is the chief executive* from + * quoting the paragraph that only says *the airline*. * * A length floor would be the cheap way to drop *of* and *is*, and it is the * wrong one: *UN*, *EU* and *AI* are two letters and are the whole question. */ +const SHELL = new Set([ + 'who', + 'what', + 'which', + 'when', + 'where', + 'why', + 'how', + 'wer', + 'was', + 'wann', + 'wo', + 'warum', + 'wieso', + 'weshalb', + 'welche', + 'welcher', + 'welches', + 'welchen', + 'is', + 'are', + 'were', + 'the', + 'a', + 'an', + 'of', + 'in', + 'on', + 'at', + 'to', + 'for', + 'and', + 'or', + 'ist', + 'sind', + 'der', + 'die', + 'das', + 'ein', + 'eine', + 'und', + 'oder', + 'von', + 'im', + 'für', + 'zu', + 'does', + 'do', + 'did', + 'can', + 'could', + 'please', + 'tell', + 'me', + 'you', + 'your', + 'hat', + 'haben', + 'dem', + 'den', + 'des', +]) + +function questionTerms(question: string): Set { + const all = new Set(words(question).filter((term) => term.length > 1)) + const focused = new Set([...all].filter((term) => !SHELL.has(term))) + return focused.size > 0 ? focused : all +} + function weigh(question: string, corpus: string[]): Map { - const terms = new Set(words(question).filter((term) => term.length > 1)) + const terms = questionTerms(question) const tokenized = corpus.map((paragraph) => new Set(words(paragraph))) const weights = new Map() for (const term of terms) { - const seen = tokenized.filter((paragraph) => paragraph.has(term)).length + const seen = tokenized.filter((paragraph) => holds(paragraph, term)).length if (seen > 0) weights.set(term, inverseFrequency(seen, tokenized.length)) } return weights } function score(text: string, weights: Map): number { + const present = new Set(words(text)) let total = 0 - for (const term of new Set(words(text))) total += weights.get(term) ?? 0 + for (const [term, weight] of weights) { + if (holds(present, term)) total += weight + } return total } +function rankBySnippet(question: string, results: SearchResult[]): SearchResult[] { + if (results.length <= 1) return results + const weights = weigh( + question, + results.map((result) => `${result.title} ${result.snippet}`), + ) + return results + .map((result, at) => ({ + result, + at, + score: score(`${result.title} ${result.snippet}`, weights), + })) + .sort((a, b) => b.score - a.score || a.at - b.at) + .map((entry) => entry.result) +} + +/** + * The results worth reading, in the order they should be tried. + * + * Wikipedia first, because MediaWiki is free and the lead paragraph usually + * names the person. Then one hit per remaining site, ranked by whether the + * snippet already bears on the question, so a PDF gallery sitting at rank two + * does not spend a reader request ahead of the article that answers it. Extra + * pages from a site already chosen, and URLs that will not yield a page, come + * last — they fill a slot only when nothing else is left. + */ +export function pickCandidates(question: string, results: SearchResult[]): SearchResult[] { + const ordered = diverseFirst(results, results.length) + const seen = new Set() + const primary: SearchResult[] = [] + const extra: SearchResult[] = [] + + for (const entry of ordered) { + const site = siteOf(entry.url) + if (seen.has(site)) extra.push(entry) + else { + seen.add(site) + primary.push(entry) + } + } + + const wiki: SearchResult[] = [] + const other: SearchResult[] = [] + const junk: SearchResult[] = [] + for (const entry of primary) { + if (isUnreadableUrl(entry.url)) junk.push(entry) + else if (isWikipediaUrl(entry.url)) wiki.push(entry) + else other.push(entry) + } + + const extraReadable = extra.filter((entry) => !isUnreadableUrl(entry.url)) + const extraJunk = extra.filter((entry) => isUnreadableUrl(entry.url)) + + return [ + ...wiki, + ...rankBySnippet(question, other), + ...rankBySnippet(question, extraReadable), + ...junk, + ...extraJunk, + ] +} + const SENTENCE_END = /(?<=[.!?…])\s+/ /** @@ -451,71 +654,126 @@ export function digest(question: string, sources: Source[]): string { return `${body.slice(0, MAX_DIGEST_CHARS)}\n\n[Truncated: further sources were dropped.]` } -function reasonsFrom(settled: PromiseSettledResult[]): string { - return settled - .flatMap((outcome) => - outcome.status === 'rejected' - ? [outcome.reason instanceof Error ? outcome.reason.message : String(outcome.reason)] - : [], - ) - .join('; ') +/** + * Wikipedia alongside the web, never instead of it. + * + * MediaWiki is free of the reader's budget and the lead paragraph usually + * names the person, so a DuckDuckGo or LangSearch turn that never returned + * Wikipedia used to spend three reader requests on news pages and still miss + * the sentence that answered the question. A failure here is swallowed: the + * web results are still an answer, and a thrown encyclopedia search would + * take them down with it. + */ +async function encyclopediaHits(question: string, config: WebAccessConfig): Promise { + if (config.provider === 'wikipedia') return [] + try { + return await searchWeb(question, WIKI_SEARCH_LIMIT, { provider: 'wikipedia' }) + } catch { + return [] + } +} + +interface Opened { + url: string + title: string + paragraphs: string[] } /** - * Searches, reads the most promising results in parallel and returns the - * passages that bear on the question, each with the URL it came from. + * Reads until three pages have prose on them, or the attempt budget is gone. * - * A page that fails is replaced by its search snippet rather than allowed to - * take the answer down with it — the reader's per-minute budget is shared, so a - * 429 on the third page is an ordinary event and not a reason to abandon the - * two that arrived. + * The first wave is parallel. A blocked, empty or refused page does not keep + * its slot: the next unread candidate is tried, up to `MAX_READ_ATTEMPTS`. + * Snippets from the failed hits only stand in once nothing else can be opened, + * and a firewall body is never quoted — the search snippet for that URL at + * least came from the index. */ -export async function researchQuestion(question: string, config: WebAccessConfig): Promise { - const results = await searchWeb(question, SEARCH_LIMIT, config) - if (results.length === 0) return `Researched ${todayStamp()} for "${question}". No results.` - - const selected = diverseFirst(results, MAX_SOURCES) - const settled = await Promise.allSettled(selected.map((result) => readPage(result.url, config))) +async function readBest( + question: string, + candidates: SearchResult[], + config: WebAccessConfig, +): Promise<{ sources: Source[]; reasons: string[] }> { + const opened: Opened[] = [] + const fallbacks: SearchResult[] = [] + const reasons: string[] = [] + let next = 0 + let attempts = 0 + + while (opened.length < MAX_SOURCES && next < candidates.length && attempts < MAX_READ_ATTEMPTS) { + const take = Math.min(MAX_SOURCES - opened.length, MAX_READ_ATTEMPTS - attempts, candidates.length - next) + const batch = candidates.slice(next, next + take) + next += batch.length + attempts += batch.length + + const settled = await Promise.allSettled(batch.map((entry) => readPage(entry.url, config))) + + for (const [at, result] of batch.entries()) { + const outcome = settled[at] + if (outcome?.status === 'fulfilled') { + const { title, text, url } = outcome.value + if (!looksBlocked(title, text)) { + const paragraphs = paragraphsOf(text) + if (paragraphs.length > 0) { + opened.push({ url, title: title || result.title, paragraphs }) + continue + } + } + } else if (outcome?.status === 'rejected') { + reasons.push(outcome.reason instanceof Error ? outcome.reason.message : String(outcome.reason)) + } + fallbacks.push(result) + } + } const chosen = passagesFor( question, - settled.map((outcome) => { - if (outcome.status !== 'fulfilled') return [] - const { title, text } = outcome.value - // A page the site refused to serve arrives as a 200 with prose on it. Left - // in, it is a source that says nothing and cannot be told from one that - // does; the search snippet for the same URL at least came from the index. - return looksBlocked(title, text) ? [] : paragraphsOf(text) - }), + opened.map((entry) => entry.paragraphs), ) - const sources = selected.flatMap((result, at): Source[] => { - const outcome = settled[at] + const sources: Source[] = [] + for (const [at, entry] of opened.entries()) { const passages = chosen[at] ?? [] + if (passages.length === 0) continue + sources.push({ url: entry.url, title: entry.title, passages, read: true }) + } - if (outcome?.status === 'fulfilled' && passages.length > 0) { - return [ - { - url: outcome.value.url, - title: outcome.value.title || result.title, - passages, - read: true, - }, - ] - } - + for (const result of fallbacks) { + if (sources.length >= MAX_SOURCES) break const snippet = collapse(result.snippet) - if (!snippet) return [] - return [{ url: result.url, title: result.title, passages: [snippet], read: false }] - }) + if (!snippet) continue + sources.push({ url: result.url, title: result.title, passages: [snippet], read: false }) + } + + return { sources, reasons } +} + +/** + * Searches the web and Wikipedia, reads the most promising results in parallel + * and returns the passages that bear on the question, each with the URL it + * came from. + * + * A page that fails is replaced from the remaining hits rather than allowed to + * take the answer down with it — the reader's per-minute budget is shared, so a + * 429 on the third page is an ordinary event and not a reason to abandon the + * two that arrived. + */ +export async function researchQuestion(question: string, config: WebAccessConfig): Promise { + const [webResults, wikiResults] = await Promise.all([ + searchWeb(question, SEARCH_LIMIT, config), + encyclopediaHits(question, config), + ]) + + const combined = [...wikiResults, ...webResults] + if (combined.length === 0) return `Researched ${todayStamp()} for "${question}". No results.` + + const { sources, reasons } = await readBest(question, pickCandidates(question, combined), config) // Every source silent means the search found pages and nothing could be read // off any of them. Reporting that as a result would have the model relay it as // "there is nothing on this", which is the one thing it must not say. if (sources.length === 0) { - const reasons = reasonsFrom(settled) throw new Error( - `Found ${results.length} results for "${question}" but could not read any of them${reasons ? `: ${reasons}` : '.'}`, + `Found ${combined.length} results for "${question}" but could not read any of them${reasons.length > 0 ? `: ${reasons.join('; ')}` : '.'}`, ) } From 45c200fe2642da2b5dfe362c481c636e5a98b456 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 31 Aug 2026 11:04:15 +0000 Subject: [PATCH 2/3] Search the subject, not the question shell A 0.8B model forwards the whole question to research. Searching "What is the capital of France?" ranked a page about questions ahead of Paris. The query is now narrowed the way weather narrows a place, while why/how keep their shell so "sky blue" does not beat Rayleigh scattering. Language is taken from the original, so a stripped German who-question still hits de.wiki. Co-authored-by: Sebastian --- README.md | 2 +- src/tools/research.test.ts | 30 +++++++++++++++++++++ src/tools/research.ts | 55 +++++++++++++++++++++++++++++++++++--- src/tools/web.ts | 32 +++++++++++++--------- 4 files changed, 102 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 6ae1458..96a78e9 100644 --- a/README.md +++ b/README.md @@ -239,7 +239,7 @@ Search and `read_page` share the reader's budget of 20 requests a minute per IP, **The search itself now carries the facts a 0.8B model would otherwise spend a round guessing at.** Every `web_search` result is stamped with today's local date, so "current" and "today's news" have a date without calling `current_time`. A German question searches German Wikipedia and, on DuckDuckGo, prefers German results (`kl=de-de`); English _who was Ada Lovelace_ is not mistaken for German because bare `was` is also English. German Wikipedia is smaller, so an empty result there falls through to English rather than telling the model the subject does not exist. -The `research-question` skill does not offer `web_search`. It offers `research`, which searches the live web and Wikipedia together, picks three **different sites** (Wikipedia first when it appeared, because MediaWiki is free and the lead paragraph usually answers; `investor.nvidia.com` and `nvidianews.nvidia.com` count as one), ranks the rest by whether their snippet already bears on the question, and reads them in parallel. A page that will not open, or that comes back as a login wall or a firewall interstitial, is replaced from the remaining hits rather than quoted; only when nothing else is left does the search snippet stand in. Inflected forms of a word still match — _Bundeskanzlers_ answers _Bundeskanzler_ — so a German page is not silent on a German question. If nothing readable comes back the tool throws rather than telling the model the subject does not exist. +The `research-question` skill does not offer `web_search`. It offers `research`, which searches the live web and Wikipedia together, picks three **different sites** (Wikipedia first when it appeared, because MediaWiki is free and the lead paragraph usually answers; `investor.nvidia.com` and `nvidianews.nvidia.com` count as one), ranks the rest by whether their snippet already bears on the question, and reads them in parallel. A page that will not open, or that comes back as a login wall or a firewall interstitial, is replaced from the remaining hits rather than quoted; only when nothing else is left does the search snippet stand in. Inflected forms of a word still match — _Bundeskanzlers_ answers _Bundeskanzler_ — so a German page is not silent on a German question. The search query is narrowed the way the weather tool narrows a place: _What is the capital of France?_ is searched as `capital of France`, which ranks Paris rather than a page that uses the sentence as an example of a question; _why_ and _how_ keep the shell, because _sky blue_ ranks a colour swatch and _why is the sky blue_ ranks the physics. If nothing readable comes back the tool throws rather than telling the model the subject does not exist. A factual question that no other skill claimed — _What is the capital of France?_, _Warum ist der Himmel blau?_ — still reaches this skill, as a fourth routing stage rather than a broader trigger. Greetings do not: _wie geht's dir_ and _how are you_ stay with the model. `lookup-term` still searches and optionally reads one page — a name does not need three sources. diff --git a/src/tools/research.test.ts b/src/tools/research.test.ts index 4bfc2d4..29929ca 100644 --- a/src/tools/research.test.ts +++ b/src/tools/research.test.ts @@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { digest, diverseFirst, + focusQuery, isUnreadableUrl, looksBlocked, paragraphsOf, @@ -104,6 +105,22 @@ describe('related', () => { }) }) +describe('focusQuery', () => { + it.each([ + ['What is the capital of France?', 'capital of France'], + ['Who is the current secretary-general of the UN?', 'current secretary-general of the UN'], + ['Warum ist der Himmel blau?', 'Warum ist der Himmel blau'], + ['Why is the sky blue?', 'Why is the sky blue'], + ['How does photosynthesis work?', 'How does photosynthesis work'], + ['Wer ist der Bundeskanzler?', 'Bundeskanzler'], + ["What's the population of Tokyo", 'population of Tokyo'], + ['How much is a Big Mac in Japan?', 'Big Mac in Japan'], + ['capital of France', 'capital of France'], + ])('narrows %j to %j', (raw, expected) => { + expect(focusQuery(raw)).toBe(expected) + }) +}) + describe('isUnreadableUrl', () => { it('flags a login wall and a PDF gallery', () => { expect(isUnreadableUrl('https://example.com/login')).toBe(true) @@ -754,6 +771,19 @@ describe('researchQuestion', () => { expect(calls.some((url) => url.startsWith('https://api.langsearch.com'))).toBe(false) }) + it('searches for the subject rather than the question shell', async () => { + const calls = stubNetwork({ + [LEADERSHIP]: LEADERSHIP_PAGE, + [AIRTIMES]: AIRTIMES_PAGE, + }) + + await researchQuestion('What is the capital of France?', config) + + const wiki = calls.find((url) => url.includes('gsrsearch')) + const asked = wiki ? new URL(wiki).searchParams.get('gsrsearch') : null + expect(asked).toBe('capital of France') + }) + it('says so when no page opened and only snippets are left', async () => { stubNetwork({ [LEADERSHIP]: 500, [AIRTIMES]: 429 }) diff --git a/src/tools/research.ts b/src/tools/research.ts index 55fa667..bd0a0af 100644 --- a/src/tools/research.ts +++ b/src/tools/research.ts @@ -23,7 +23,14 @@ * German page is not silent on a German question. */ -import { readPage, searchWeb, wikipediaPage, type SearchResult, type WebAccessConfig } from './web' +import { + queryLanguage, + readPage, + searchWeb, + wikipediaPage, + type SearchResult, + type WebAccessConfig, +} from './web' /** * How many results to ask for before narrowing them. Larger than `MAX_SOURCES` @@ -102,6 +109,41 @@ function collapse(value: string): string { return value.replace(/\s+/g, ' ').trim() } +/** + * The query to send the search engine, given what the model passed. + * + * A 0.8B model often forwards the whole question: *What is the capital of + * France?* Searching that verbatim ranks a page *about questions* that uses + * the sentence as an example, ahead of Paris. Stripping the interrogative + * shell is the same narrowing `placeCandidates` does for weather. *Why* and + * *how* questions keep the shell: *sky blue* ranks a colour swatch, *why is the + * sky blue* ranks diffuse sky radiation. The language of the original is what + * `searchWeb` still uses, so a stripped German *who* question does not flip to + * English Wikipedia. + */ +export function focusQuery(raw: string): string { + const original = collapse(raw.replace(/[?!？]+$/g, '')) + if (!original) return raw.trim() + + // *Why is the sky blue?* is a better Wikipedia search than *sky blue*, which + // ranks a colour and a football club ahead of Rayleigh scattering. Prices and + // *who/what* questions still want the shell gone. + if (/^(why|warum|wieso|weshalb|how|wie)\b/i.test(original) && !/^(how much|wie viel)/i.test(original)) { + return original + } + + let text = original.replace( + /^(how much|wie viel(?:e)?)\s+(is|are|does|do|did|kostet|kosten)\s+(?:a|an|the|ein|eine|der|die|das)?\s*/i, + '', + ) + text = text.replace( + /^(what's|whats|what|which|who|when|where|why|how|wer|was|wann|wo|warum|wieso|weshalb|welche[rsn]?)\s+(?:(?:is|are|was|were|do|does|did|ist|sind|war|waren|hat|haben)\s+)?(?:(?:the|a|an|der|die|das|ein|eine|den|dem)\s+)?/i, + '', + ) + text = collapse(text) + return text.length >= 2 ? text : original +} + /** * Closes the gap the separators above open in front of punctuation. A quote is * only known to be a closing one when punctuation follows it, so that is the @@ -667,7 +709,12 @@ export function digest(question: string, sources: Source[]): string { async function encyclopediaHits(question: string, config: WebAccessConfig): Promise { if (config.provider === 'wikipedia') return [] try { - return await searchWeb(question, WIKI_SEARCH_LIMIT, { provider: 'wikipedia' }) + return await searchWeb( + focusQuery(question), + WIKI_SEARCH_LIMIT, + { provider: 'wikipedia' }, + queryLanguage(question), + ) } catch { return [] } @@ -758,8 +805,10 @@ async function readBest( * two that arrived. */ export async function researchQuestion(question: string, config: WebAccessConfig): Promise { + const query = focusQuery(question) + const language = queryLanguage(question) const [webResults, wikiResults] = await Promise.all([ - searchWeb(question, SEARCH_LIMIT, config), + searchWeb(query, SEARCH_LIMIT, config, language), encyclopediaHits(question, config), ]) diff --git a/src/tools/web.ts b/src/tools/web.ts index 2bbdfdd..2f04bad 100644 --- a/src/tools/web.ts +++ b/src/tools/web.ts @@ -179,7 +179,7 @@ const JINA_SEARCH_ENDPOINT = 'https://s.jina.ai/' const LANGSEARCH_ENDPOINT = 'https://api.langsearch.com/v1/web-search' const READER_ENDPOINT = 'https://r.jina.ai/' -type WikiLang = 'de' | 'en' +export type WikiLang = 'de' | 'en' /** * Which Wikipedia edition a query should search, and which DuckDuckGo region it @@ -304,11 +304,16 @@ async function proxyRequest( return payload } -async function searchViaProxy(base: string, query: string, limit: number): Promise { +async function searchViaProxy( + base: string, + query: string, + limit: number, + language: WikiLang, +): Promise { const payload = await proxyRequest<{ results?: SearchResult[]; error?: string }>( base, '/api/search', - { query, limit, region: queryLanguage(query) === 'de' ? 'de-de' : undefined }, + { query, limit, region: language === 'de' ? 'de-de' : undefined }, 'The tool proxy', ) if (!payload.results) { @@ -426,12 +431,11 @@ async function searchWikipediaEdition(lang: WikiLang, query: string, limit: numb })) } -async function searchWikipedia(query: string, limit: number): Promise { - const lang = queryLanguage(query) - const results = await searchWikipediaEdition(lang, query, limit) +async function searchWikipedia(query: string, limit: number, language: WikiLang): Promise { + const results = await searchWikipediaEdition(language, query, limit) // German Wikipedia is smaller. An empty result there is often a missing // article, not a missing subject, and English still has one. - if (results.length > 0 || lang === 'en') return results + if (results.length > 0 || language === 'en') return results return searchWikipediaEdition('en', query, limit) } @@ -648,11 +652,11 @@ export function parseDuckDuckGoResults(markdown: string): SearchResult[] { const UNREADABLE = 'DuckDuckGo returned nothing this parser could read. It may have refused the reader — try again in a moment.' -function duckDuckGoTarget(endpoint: string, query: string): string { +function duckDuckGoTarget(endpoint: string, query: string, language: WikiLang): string { const target = `${endpoint}?q=${encodeURIComponent(query)}` // `kl` is DDG's region. English questions keep the unregionalised URL the // parser and the tests already know; German ones prefer German sites. - return queryLanguage(query) === 'de' ? `${target}&kl=de-de` : target + return language === 'de' ? `${target}&kl=de-de` : target } /** @@ -667,11 +671,12 @@ async function searchDuckDuckGo( query: string, limit: number, config: WebAccessConfig, + language: WikiLang, ): Promise { let failure: Error | undefined for (const endpoint of DUCKDUCKGO_ENDPOINTS) { - const target = duckDuckGoTarget(endpoint, query) + const target = duckDuckGoTarget(endpoint, query, language) let content: string try { @@ -702,6 +707,7 @@ export async function searchWeb( query: string, limit: number, config: WebAccessConfig, + language: WikiLang = queryLanguage(query), ): Promise { if (config.provider === 'jina') { return searchJina(query, limit, requireKey(config.jinaApiKey, 'Jina search needs a Jina API key')) @@ -714,12 +720,12 @@ export async function searchWeb( ) } if (config.provider === 'wikipedia') { - return searchWikipedia(query, limit) + return searchWikipedia(query, limit, language) } const proxy = configuredProxyBase(config) if (proxy !== undefined) { try { - return await searchViaProxy(proxy, query, limit) + return await searchViaProxy(proxy, query, limit, language) } catch { // The proxy is an optimisation, not a dependency. A hosted build points // every visitor at one process, so an outage, a spent budget or an @@ -727,7 +733,7 @@ export async function searchWeb( // the answer. Browser-direct is what this build did before the proxy. } } - return searchDuckDuckGo(query, limit, config) + return searchDuckDuckGo(query, limit, config, language) } /** Literal private hosts only: a page has no resolver, so a name cannot be checked here. */ From c1df257f12251ebab04fadab1603e5a3d3d94c17 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 31 Aug 2026 11:14:09 +0000 Subject: [PATCH 3/3] Prefer the Wikipedia article whose description is the query MediaWiki still ranks "capital of France" as a list, then capital punishment, then Paris. Paris's short description is the query; the others are a list article and the death penalty. That article takes the free first slot, Wikimedia lists are pushed back, and a second Wikipedia hit no longer crowds out another site. Co-authored-by: Sebastian --- README.md | 2 +- src/tools/research.test.ts | 28 ++++++++++++++-- src/tools/research.ts | 34 ++++++++++++++------ src/tools/web.test.ts | 54 +++++++++++++++++++++++++++++++ src/tools/web.ts | 66 +++++++++++++++++++++++++++++++++++--- 5 files changed, 168 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 96a78e9..f00ee6e 100644 --- a/README.md +++ b/README.md @@ -239,7 +239,7 @@ Search and `read_page` share the reader's budget of 20 requests a minute per IP, **The search itself now carries the facts a 0.8B model would otherwise spend a round guessing at.** Every `web_search` result is stamped with today's local date, so "current" and "today's news" have a date without calling `current_time`. A German question searches German Wikipedia and, on DuckDuckGo, prefers German results (`kl=de-de`); English _who was Ada Lovelace_ is not mistaken for German because bare `was` is also English. German Wikipedia is smaller, so an empty result there falls through to English rather than telling the model the subject does not exist. -The `research-question` skill does not offer `web_search`. It offers `research`, which searches the live web and Wikipedia together, picks three **different sites** (Wikipedia first when it appeared, because MediaWiki is free and the lead paragraph usually answers; `investor.nvidia.com` and `nvidianews.nvidia.com` count as one), ranks the rest by whether their snippet already bears on the question, and reads them in parallel. A page that will not open, or that comes back as a login wall or a firewall interstitial, is replaced from the remaining hits rather than quoted; only when nothing else is left does the search snippet stand in. Inflected forms of a word still match — _Bundeskanzlers_ answers _Bundeskanzler_ — so a German page is not silent on a German question. The search query is narrowed the way the weather tool narrows a place: _What is the capital of France?_ is searched as `capital of France`, which ranks Paris rather than a page that uses the sentence as an example of a question; _why_ and _how_ keep the shell, because _sky blue_ ranks a colour swatch and _why is the sky blue_ ranks the physics. If nothing readable comes back the tool throws rather than telling the model the subject does not exist. +The `research-question` skill does not offer `web_search`. It offers `research`, which searches the live web and Wikipedia together, picks three **different sites** (Wikipedia first when it appeared, because MediaWiki is free and the lead paragraph usually answers; `investor.nvidia.com` and `nvidianews.nvidia.com` count as one), ranks the rest by whether their snippet already bears on the question, and reads them in parallel. A page that will not open, or that comes back as a login wall or a firewall interstitial, is replaced from the remaining hits rather than quoted; only when nothing else is left does the search snippet stand in. Inflected forms of a word still match — _Bundeskanzlers_ answers _Bundeskanzler_ — so a German page is not silent on a German question. The search query is narrowed the way the weather tool narrows a place: _What is the capital of France?_ is searched as `capital of France`, which ranks Paris rather than a page that uses the sentence as an example of a question; _why_ and _how_ keep the shell, because _sky blue_ ranks a colour swatch and _why is the sky blue_ ranks the physics. Wikipedia's own ranking still puts a list and _capital punishment_ ahead of Paris; the article whose short description _is_ the query takes the free first slot, and a Wikimedia list is pushed behind it. If nothing readable comes back the tool throws rather than telling the model the subject does not exist. A factual question that no other skill claimed — _What is the capital of France?_, _Warum ist der Himmel blau?_ — still reaches this skill, as a fourth routing stage rather than a broader trigger. Greetings do not: _wie geht's dir_ and _how are you_ stay with the model. `lookup-term` still searches and optionally reads one page — a name does not need three sources. diff --git a/src/tools/research.test.ts b/src/tools/research.test.ts index 29929ca..1314e51 100644 --- a/src/tools/research.test.ts +++ b/src/tools/research.test.ts @@ -115,7 +115,7 @@ describe('focusQuery', () => { ['Wer ist der Bundeskanzler?', 'Bundeskanzler'], ["What's the population of Tokyo", 'population of Tokyo'], ['How much is a Big Mac in Japan?', 'Big Mac in Japan'], - ['capital of France', 'capital of France'], + ['Was ist die Hauptstadt von Frankreich?', 'Hauptstadt von Frankreich'], ])('narrows %j to %j', (raw, expected) => { expect(focusQuery(raw)).toBe(expected) }) @@ -182,10 +182,34 @@ describe('pickCandidates', () => { ]) expect(chosen.slice(0, 2).map((entry) => entry.url)).toEqual([ - 'https://en.wikipedia.org/wiki/Nvidia', + 'https://en.wikipedia.org/wiki/Jensen_Huang', 'https://www.reuters.com/nvidia', ]) }) + + it('gives the Wikipedia slot to the article whose snippet is the question, not a same-word trap', () => { + const chosen = pickCandidates('What is the capital of France?', [ + result( + 'https://en.wikipedia.org/wiki/List_of_capitals_of_France', + 'List of capitals of France', + 'A chronological list of former seats of government.', + ), + result( + 'https://en.wikipedia.org/wiki/Capital_punishment_in_France', + 'Capital punishment in France', + 'Capital punishment in France is banned by the constitution.', + ), + result( + 'https://en.wikipedia.org/wiki/Paris', + 'Paris', + 'Capital of France. Paris is the capital and largest city of France.', + ), + result('https://www.britannica.com/paris', 'Britannica', 'Paris is the capital of France.'), + ]) + + expect(chosen[0]?.url).toBe('https://en.wikipedia.org/wiki/Paris') + expect(chosen[1]?.url).toBe('https://www.britannica.com/paris') + }) }) describe('paragraphsOf', () => { diff --git a/src/tools/research.ts b/src/tools/research.ts index bd0a0af..f6e9f47 100644 --- a/src/tools/research.ts +++ b/src/tools/research.ts @@ -52,8 +52,11 @@ const MAX_SOURCES = 3 */ const MAX_READ_ATTEMPTS = 5 -/** Enough Wikipedia hits to have a lead article after disambiguations are demoted. */ -const WIKI_SEARCH_LIMIT = 3 +/** + * Enough Wikipedia hits to rerank after a list and a same-word trap (capital + * punishment) occupy the top of the index. + */ +const WIKI_SEARCH_LIMIT = 5 /** Two passages carry a claim and its context. A third is usually the same claim again. */ const MAX_PASSAGES_PER_SOURCE = 2 @@ -505,12 +508,13 @@ function rankBySnippet(question: string, results: SearchResult[]): SearchResult[ question, results.map((result) => `${result.title} ${result.snippet}`), ) + const focused = focusQuery(question).toLowerCase() return results - .map((result, at) => ({ - result, - at, - score: score(`${result.title} ${result.snippet}`, weights), - })) + .map((result, at) => { + const hay = `${result.title} ${result.snippet}`.toLowerCase() + const phrase = focused.length >= 8 && hay.includes(focused) ? 1 : 0 + return { result, at, score: score(hay, weights) + phrase } + }) .sort((a, b) => b.score - a.score || a.at - b.at) .map((entry) => entry.result) } @@ -551,11 +555,23 @@ export function pickCandidates(question: string, results: SearchResult[]): Searc const extraReadable = extra.filter((entry) => !isUnreadableUrl(entry.url)) const extraJunk = extra.filter((entry) => isUnreadableUrl(entry.url)) + const extraWiki: SearchResult[] = [] + const extraOther: SearchResult[] = [] + for (const entry of extraReadable) { + if (isWikipediaUrl(entry.url)) extraWiki.push(entry) + else extraOther.push(entry) + } + + // One Wikipedia slot, and it should be the article that answers — Huang on a + // CEO question, not the company page that happened to rank first. + const wikiRanked = rankBySnippet(question, [...wiki, ...extraWiki]) + const wikiLead = wikiRanked.slice(0, 1) + const wikiMore = wikiRanked.slice(1) return [ - ...wiki, + ...wikiLead, ...rankBySnippet(question, other), - ...rankBySnippet(question, extraReadable), + ...rankBySnippet(question, [...wikiMore, ...extraOther]), ...junk, ...extraJunk, ] diff --git a/src/tools/web.test.ts b/src/tools/web.test.ts index 18c0441..61f600a 100644 --- a/src/tools/web.test.ts +++ b/src/tools/web.test.ts @@ -1,6 +1,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { configuredProxyBase, + descriptionFit, missingSearchKey, normalizeWebAccess, parseDuckDuckGoResults, @@ -41,6 +42,14 @@ afterEach(() => { vi.unstubAllGlobals() }) +describe('descriptionFit', () => { + it('treats an exact Wikipedia short description as the question', () => { + expect(descriptionFit('capital of France', 'Capital of France')).toBe(3) + expect(descriptionFit('capital of France', 'overview of the capital punishment in France')).toBe(0) + expect(descriptionFit('capital of France', 'Wikimedia list article')).toBe(0) + }) +}) + describe('queryLanguage', () => { it.each([ ['webgpu', 'en'], @@ -82,6 +91,7 @@ describe('searchWeb with Wikipedia', () => { expect(url.searchParams.get('origin')).toBe('*') expect(url.searchParams.get('gsrsearch')).toBe('webgpu') expect(url.searchParams.get('gsrlimit')).toBe('2') + expect(url.searchParams.get('prop') ?? '').toContain('description') expect(results.map((result) => result.title)).toEqual(['First', 'Second']) expect(results[0]).toEqual({ title: 'First', url: 'https://w/1', snippet: 'Best match.' }) }) @@ -125,6 +135,50 @@ describe('searchWeb with Wikipedia', () => { expect(await searchWeb('zzzz', 5, wikipedia)).toEqual([]) }) + it('ranks the article whose short description is the query ahead of a list and a same-word trap', async () => { + stubFetch( + jsonResponse({ + query: { + pages: { + '1': { + pageid: 1, + title: 'List of capitals of France', + index: 1, + extract: 'This is a chronological list of capitals of France.', + terms: { description: ['Wikimedia list article'] }, + fullurl: 'https://en.wikipedia.org/wiki/List_of_capitals_of_France', + }, + '2': { + pageid: 2, + title: 'Capital punishment in France', + index: 2, + extract: 'Capital punishment in France is banned.', + terms: { description: ['overview of the capital punishment in France'] }, + fullurl: 'https://en.wikipedia.org/wiki/Capital_punishment_in_France', + }, + '3': { + pageid: 3, + title: 'Paris', + index: 3, + extract: 'Paris is the capital and largest city of France.', + description: 'Capital of France', + fullurl: 'https://en.wikipedia.org/wiki/Paris', + }, + }, + }, + }), + ) + + const results = await searchWeb('capital of France', 3, wikipedia) + + expect(results.map((result) => result.title)).toEqual([ + 'Paris', + 'Capital punishment in France', + 'List of capitals of France', + ]) + expect(results[0]?.snippet).toMatch(/^Capital of France\./) + }) + it('searches German Wikipedia for a German question', async () => { const fetchMock = stubFetch(jsonResponse(payload)) diff --git a/src/tools/web.ts b/src/tools/web.ts index 2f04bad..e43ba05 100644 --- a/src/tools/web.ts +++ b/src/tools/web.ts @@ -351,6 +351,10 @@ interface WikipediaPage { extract?: string fullurl?: string pageprops?: { disambiguation?: string } + /** Local short description, when the wiki has one. Paris: "Capital of France". */ + description?: string + /** Wikidata descriptions via `pageterms`. List articles say "Wikimedia list article". */ + terms?: { description?: string[] } } /** @@ -363,6 +367,53 @@ function isDisambiguation(page: WikipediaPage): boolean { return page.pageprops?.disambiguation !== undefined } +function wikipediaBlurbs(page: WikipediaPage): string[] { + const blurbs: string[] = [] + if (page.description?.trim()) blurbs.push(collapse(page.description)) + for (const term of page.terms?.description ?? []) { + if (term.trim()) blurbs.push(collapse(term)) + } + return blurbs +} + +/** + * How closely a Wikipedia short description is the thing that was asked. + * + * MediaWiki ranks *capital of France* as a list, then capital punishment, then + * Paris. Paris's description *is* "Capital of France"; the others are a list + * article and an overview of the death penalty. Boosting an exact match is + * what puts the article that answers in front of the same-word trap. + */ +export function descriptionFit(query: string, description: string): number { + const q = collapse(query).toLowerCase() + const d = collapse(description).toLowerCase() + if (!q || !d) return 0 + if (d === q) return 3 + const [shorter, longer] = d.length <= q.length ? [d, q] : [q, d] + if (shorter.includes(' ') && longer.includes(shorter)) return 2 + return 0 +} + +function wikiListPenalty(page: WikipediaPage): number { + const blob = wikipediaBlurbs(page).join(' ').toLowerCase() + return /wikimedia list|wikimedia-liste/.test(blob) ? 1 : 0 +} + +function wikiBoost(query: string, page: WikipediaPage): number { + return wikipediaBlurbs(page).reduce((best, blurb) => Math.max(best, descriptionFit(query, blurb)), 0) +} + +function wikiSnippet(page: WikipediaPage): string { + const extract = collapse(page.extract ?? '') + const blurb = wikipediaBlurbs(page).find( + (text) => !/^wikimedia\b/i.test(text) && !/^topics referred to by the same term$/i.test(text), + ) + if (blurb && !extract.toLowerCase().startsWith(blurb.toLowerCase())) { + return truncate(collapse(`${blurb}. ${extract}`), MAX_SNIPPET_CHARS) + } + return truncate(extract, MAX_SNIPPET_CHARS) +} + interface WikipediaResponse { query?: { pages?: Record } } @@ -401,13 +452,14 @@ async function searchWikipediaEdition(lang: WikiLang, query: string, limit: numb generator: 'search', gsrsearch: query, gsrlimit: String(limit), - prop: 'extracts|info|pageprops', + prop: 'extracts|info|pageprops|description|pageterms', exintro: '1', explaintext: '1', // Explicit so the number of extracts never rides on the API's default. exlimit: 'max', inprop: 'url', ppprop: 'disambiguation', + wbptterms: 'description', format: 'json', // The MediaWiki API withholds `Access-Control-Allow-Origin` unless the // request asks for anonymous cross-origin access by name. @@ -419,15 +471,21 @@ async function searchWikipediaEdition(lang: WikiLang, query: string, limit: numb label: 'Wikipedia', }) - // `generator=search` returns pages keyed by id, so ranking survives only in `index`. + // `generator=search` returns pages keyed by id, so ranking survives only in + // `index`. A short description that *is* the query (Paris: "Capital of France") + // is a better signal than that index, and a Wikimedia list is a worse one. return Object.values(payload.query?.pages ?? {}) .sort( - (a, b) => Number(isDisambiguation(a)) - Number(isDisambiguation(b)) || (a.index ?? 0) - (b.index ?? 0), + (a, b) => + Number(isDisambiguation(a)) - Number(isDisambiguation(b)) || + wikiListPenalty(a) - wikiListPenalty(b) || + wikiBoost(query, b) - wikiBoost(query, a) || + (a.index ?? 0) - (b.index ?? 0), ) .map((page) => ({ title: page.title, url: page.fullurl ?? `${origin}/?curid=${page.pageid}`, - snippet: truncate(collapse(page.extract ?? ''), MAX_SNIPPET_CHARS), + snippet: wikiSnippet(page), })) }