Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ Search and `read_page` share the reader's budget of 20 requests a minute per IP,

**The search itself now carries the facts a 0.8B model would otherwise spend a round guessing at.** Every `web_search` result is stamped with today's local date, so "current" and "today's news" have a date without calling `current_time`. A German question searches German Wikipedia and, on DuckDuckGo, prefers German results (`kl=de-de`); English _who was Ada Lovelace_ is not mistaken for German because bare `was` is also English. German Wikipedia is smaller, so an empty result there falls through to English rather than telling the model the subject does not exist.

The `research-question` skill does not offer `web_search`. It offers `research`, which searches, picks three **different sites** (`investor.nvidia.com` and `nvidianews.nvidia.com` count as one), reads them in parallel, and returns the passages that bear on the question. Wikipedia pages go through MediaWiki, so a typical call spends one reader request on the search and two on the other sites rather than six. A page that will not open becomes its search snippet; if nothing readable comes back the tool throws rather than telling the model the subject does not exist. `lookup-term` still searches and optionally reads one page — a name does not need three sources.
The `research-question` skill does not offer `web_search`. It offers `research`, which searches the live web and Wikipedia together, picks three **different sites** (Wikipedia first when it appeared, because MediaWiki is free and the lead paragraph usually answers; `investor.nvidia.com` and `nvidianews.nvidia.com` count as one), ranks the rest by whether their snippet already bears on the question, and reads them in parallel. A page that will not open, or that comes back as a login wall or a firewall interstitial, is replaced from the remaining hits rather than quoted; only when nothing else is left does the search snippet stand in. Inflected forms of a word still match — _Bundeskanzlers_ answers _Bundeskanzler_ — and a one-line Wikipedia claim that names the incumbent (_Amtsträger ist seit dem 6. Mai 2025 Friedrich Merz_) is kept even when the lead is a definition of the office. If nothing readable comes back the tool throws rather than telling the model the subject does not exist. `lookup-term` still searches and optionally reads one page — a name does not need three sources.

**LangSearch is the way off that shared budget without paying for one.** `api.langsearch.com` is a search API rather than a results page, its free tier allows 1,000 searches a day and one a second, and a key needs no card — so a search stops competing with `read_page` for the same 20 requests a minute. Two things about it are worth knowing before choosing it. Its snippets are index text rather than prose, lower-cased and with spaces around the punctuation, which a 0.8B model reads less confidently than a sentence. And it answers in an envelope: a refusal it decides to report with a 200 arrives as a `msg` and no result set, so `searchLangSearch` raises that rather than passing an empty list to a model that would relay it as "this does not exist". Long summaries are available per result and are switched off — each is the whole page behind the result, which would leave a 0.8B context with no room for the answer.

Expand Down
6 changes: 4 additions & 2 deletions src/eval/scenarios.ts
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,8 @@ export const SCENARIOS: Scenario[] = [
// `who is` and `who won` were triggers and authorship was not, so the one
// question a search engine answers best reached nothing.
prompt: 'Who wrote Dune?',
expectTool: 'web_search',
expectTool: 'research',
acceptCall: (calls) => /dune/i.test(searchQuery(calls) ?? ''),
accept: matches(/herbert/i),
online: true,
},
Expand All @@ -390,7 +391,8 @@ export const SCENARIOS: Scenario[] = [
// A figure a 0.8B model will otherwise invent, confidently and to three
// significant figures.
prompt: "What's the population of Tokyo?",
expectTool: 'web_search',
expectTool: 'research',
acceptCall: (calls) => /tokyo|tokio/i.test(searchQuery(calls) ?? ''),
accept: (answer) => /\d/.test(answer),
online: true,
},
Expand Down
Loading