From 53195b518fa3d8b4cc3deb5ec4cccf29d383bdb4 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 26 Aug 2026 10:18:57 +0000 Subject: [PATCH 1/2] Compare several independent sites in one web_search A search returned a title, a URL and one line each, and left the model to open the promising ones and notice where they disagreed. A 0.8B model spends its whole tool budget there. web_search now searches, picks up to four results from different sites, reads them in parallel through the same reader read_page uses, and returns one brief under 4,000 characters. Wikipedia is held back unless nothing else can be read: its extract is a paragraph where a results page gives a line, so in rank order it decides every answer by itself. What the sources agree and disagree on is worked out deterministically, as weather.ts reconciles three forecasts rather than asking the model to. LangSearch sends its own text with each result, capped, so that provider compares sites on the one request it was already spending. Co-authored-by: Sebastian --- README.md | 51 ++- src/components/SettingsPanel.tsx | 7 +- src/skills/lookup-term/SKILL.md | 31 +- src/skills/research-question/SKILL.md | 18 +- src/tools/builtins.ts | 17 +- src/tools/search-brief.test.ts | 360 ++++++++++++++++ src/tools/search-brief.ts | 583 ++++++++++++++++++++++++++ src/tools/web.test.ts | 29 +- src/tools/web.ts | 63 ++- 9 files changed, 1104 insertions(+), 55 deletions(-) create mode 100644 src/tools/search-brief.test.ts create mode 100644 src/tools/search-brief.ts diff --git a/README.md b/README.md index 6d79abd..1730eb3 100644 --- a/README.md +++ b/README.md @@ -189,7 +189,7 @@ Three details make the app work from a repository sub-path rather than a domain | Tool | What it does | | -------------- | ------------------------------------------------------------------- | -| `web_search` | Full web search with no key; Wikipedia, LangSearch or Jina instead. | +| `web_search` | Searches, reads several independent sites, and compares them. | | `read_page` | Fetches a URL and returns its readable text. | | `calculator` | Exact arithmetic via a hand-written parser. | | `current_time` | Local date, time, and timezone. | @@ -202,7 +202,7 @@ A browser may only read a response whose origin opts in with CORS headers, which **`read_page`** goes through `r.jina.ai`, which reflects the requesting origin, needs no account, and returns extracted markdown rather than raw HTML. Anonymous use is capped at 20 requests per minute per IP; a Jina key raises that and is optional. -**`web_search`** has a provider choice under **Tools → Web access**: +**`web_search`** is a search, four page reads and a comparison in one call — see [comparing several sites in one call](#comparing-several-sites-in-one-call). It has a provider choice under **Tools → Web access**: | Provider | Key | Covers | | ---------- | ----- | ----------------------------------------------------------------------- | @@ -215,11 +215,48 @@ The default takes no key and no signup: `r.jina.ai` is pointed at a DuckDuckGo r That fragility has already been paid once, and not in the way it looked. `lite.duckduckgo.com` was the only page asked, and one afternoon the reader could not load it: it waited on the page and returned a 422, so the default provider could not search at all. The html page answered the same query in the same second, and an hour later both were fine. So the fault was never that one page died — it is that one page is enough for the search to work and not enough for it to keep working. Both are asked now, `duckduckgo.com/html/` first because it is what answered during the outage. They write a hit differently, `1.[Title](link)` against `## [Title](link)`, and the parser reads both. -Search and `read_page` share the reader's budget of 20 requests a minute per IP, so one search plus one page read spends two. A Jina key raises the ceiling for both and is what the Jina provider needs outright. +Search and `read_page` share the reader's budget of 20 requests a minute per IP, and a search now spends up to five of them: one for the results page and one for each site it compares. A Jina key raises the ceiling for both and is what the Jina provider needs outright. -**LangSearch is the way off that shared budget without paying for one.** `api.langsearch.com` is a search API rather than a results page, its free tier allows 1,000 searches a day and one a second, and a key needs no card — so a search stops competing with `read_page` for the same 20 requests a minute. Two things about it are worth knowing before choosing it. Its snippets are index text rather than prose, lower-cased and with spaces around the punctuation, which a 0.8B model reads less confidently than a sentence. And it answers in an envelope: a refusal it decides to report with a 200 arrives as a `msg` and no result set, so `searchLangSearch` raises that rather than passing an empty list to a model that would relay it as "this does not exist". Long summaries are available per result and are switched off — each is the whole page behind the result, which would leave a 0.8B context with no room for the answer. +**LangSearch is the way off that shared budget without paying for one.** `api.langsearch.com` is a search API rather than a results page, its free tier allows 1,000 searches a day and one a second, and a key needs no card — so a search stops competing with `read_page` for the same 20 requests a minute. It also sends text with each result, which is what lets it compare several sites on the one request it was already spending, where the other providers fetch each page. Two things about it are worth knowing before choosing it. Its snippets are index text rather than prose, lower-cased and with spaces around the punctuation, which a 0.8B model reads less confidently than a sentence. And it answers in an envelope: a refusal it decides to report with a 200 arrives as a `msg` and no result set, so `searchLangSearch` raises that rather than passing an empty list to a model that would relay it as "this does not exist". Its per-result summaries used to be switched off outright, because each is the whole page behind the result and several of those leave a 0.8B context with no room for the answer. They are capped at 800 characters now, which is the same budget a fetched page gets. -The tool description changes with the provider, so the model is told whether it is searching an encyclopedia or the web — without that it cheerfully asks Wikipedia for this morning's news. Wikipedia is worth keeping selected for definitions and biography: its extracts are full paragraphs where a results page gives a line. +The tool description changes with the provider, so the model is told whether it is searching an encyclopedia or the web — without that it cheerfully asks Wikipedia for this morning's news. Wikipedia is worth keeping selected for definitions and biography: its extracts are full paragraphs where a results page gives a line. It is the one provider that is not cross-checked, for the reason below. + +### Comparing several sites in one call + +A search used to return a title, a URL and one line each, and everything that made those lines worth anything — opening the promising ones, noticing that three of them say 2023 and the fourth says 2021 — was left to the model. A 0.8B model spends its whole [tool budget](#when-a-turn-runs-out-of-tool-rounds) there. The recorded failure is four searches in a row, none of them read. + +So `src/tools/search-brief.ts` takes the shape [`weather`](#tools) already had: the comparison happens in the tool, not in the conversation. One call searches, picks up to four results, reads them in parallel through the same reader `read_page` uses, and returns one brief under 4,000 characters. + +Two rules decide which four, and both exist to stop the brief being an encyclopedia lookup wearing four coats: + +- **One page per site.** Two pages of one publisher are one source, so a second hit on a domain is dropped and the search engine's ranking decides the rest. `siteOf` reads `investor.nvidia.com` and `nvidianews.nvidia.com` as one site, and knows that `bbc.co.uk` and `theguardian.co.uk` are two. +- **Wikipedia goes last.** Its extract is a paragraph where a results page gives a line, so left in rank order it decides every answer by itself — and a mirror of it is not a second opinion. It is used when nothing else could be read, and not before. + +What the sources agree and disagree on is worked out **deterministically**, for the same reason the three forecasts are reconciled in `weather.ts` rather than in the prompt: a second generation spent grading four extracts is exactly the capacity the answer needed, and intrinsic self-grading makes reasoning worse rather than better. Names and figures are what a rule can honestly compare, so they are all it claims to have compared: + +```text +Searched 2026-08-26 for "Fictional Airways chief executive" — 3 sources +1. Leadership (fictionalairways.example) + https://fictionalairways.example/leadership + Ama Osei has led Fictional Airways as chief executive since 2023. +2. Fictional Airways names a new chief (dailywire.example) + https://dailywire.example/fictional-airways-ceo + The airline confirmed Ama Osei as chief executive in 2023. +3. Fictional Airways profile (aviationweek.example) + https://aviationweek.example/fictional-airways + Chief executive: Jordan Hale, in post since 2021. +Agreed across sources: "Ama Osei" in 2/3; "2023" in 2/3 +Sources disagree: "2023" (dailywire.example, fictionalairways.example) vs "2021" (aviationweek.example) +``` + +Four details are what keep that honest: + +- **Every source keeps its own URL on its own line**, because that is what [`reviewAnswer`](#checking-the-answer-before-it-is-shown) grounds a citation against and what `splitSources` turns into pills. A comparison the model cannot cite is worse than no comparison. +- **A dead link costs one source, not the brief.** The pages are read with `allSettled`, and one that 404s, refuses the reader or spends the last of a rate limit falls back to its search snippet — labelled `snippet only`, so the model is not told a line was a page. +- **Naming both figures is not disagreeing.** A page that mentions last year's number alongside this year's is the ordinary way to write one, so a disagreement is only reported when a site names a value and _not_ the one the others agree on. A single reading nothing contradicts is reported as nothing, since hedging an unchallenged answer is its own error. +- **A figure is compared with figures of its own kind.** Years are their own kind, because that is where sources differ in a way worth reporting; otherwise the digit count stands in, so a revenue figure is never held against a percentage. `46,700` and `46.700` are one value; `46.7` is not. + +The skills teach the rest. [`research-question`](#skills) answers with the consensus and names the site that disagrees, and its exemplar ends with more than one URL. Keys are entered at runtime and kept in `localStorage`. None of this reads a build-time environment variable, deliberately: a key compiled into the bundle is a key published to every visitor. @@ -258,9 +295,9 @@ The calculator deliberately avoids `eval`. Expressions come from model output, w ### What leaves the browser -Inference does not: prompts, reasoning, and replies never leave the GPU, and neither do [memories](#memory), which are written to IndexedDB in this browser and read back into a prompt that goes no further than the GPU either. Tools are the exception, and always were. A `web_search` call sends the query to the chosen provider, a `read_page` call sends the URL to the reader, and a `weather` call sends the place name to Open-Meteo's geocoder and its coordinates to the two forecast services — the difference now is that these go direct, with no server of ours in the path to log them. +Inference does not: prompts, reasoning, and replies never leave the GPU, and neither do [memories](#memory), which are written to IndexedDB in this browser and read back into a prompt that goes no further than the GPU either. Tools are the exception, and always were. A `web_search` call sends the query to the chosen provider and then fetches the pages it compares through the reader, a `read_page` call sends the URL to the reader, and a `weather` call sends the place name to Open-Meteo's geocoder and its coordinates to the two forecast services — the difference now is that these go direct, with no server of ours in the path to log them. -Worth being precise about on the default provider: a search sends the query to `r.jina.ai`, which then sends it to DuckDuckGo. That is one more party than a search API like LangSearch or Jina involves, and one fewer than a proxy of ours would add. +Worth being precise about on the default provider: a search sends the query to `r.jina.ai`, which then sends it to DuckDuckGo, and then sends that reader the address of each page being compared. That is one more party than a search API like LangSearch or Jina involves, and one fewer than a proxy of ours would add. ### MCP servers diff --git a/src/components/SettingsPanel.tsx b/src/components/SettingsPanel.tsx index 4f53978..6a3335e 100644 --- a/src/components/SettingsPanel.tsx +++ b/src/components/SettingsPanel.tsx @@ -111,8 +111,9 @@ export function SettingsPanel() {

Web access

- Searches and page reads go straight from this page to the provider — there is no server in - between. Keys are stored in this browser only. + A search reads several independent sites and compares them. Those requests go straight from + this page to the provider — there is no server in between. Keys are stored in this browser + only.

{ const query = String(args.query ?? '').trim() if (!query) throw new Error('query must not be empty') - const limit = Math.min(Math.max(Number(args.limit ?? 5) || 5, 1), 10) - const results = await searchWeb(query, limit, config) - if (results.length === 0) return `No results for "${query}".` - return results - .map((result, index) => `${index + 1}. ${result.title}\n ${result.url}\n ${result.snippet}`) - .join('\n') + const limit = Math.min(Math.max(Number(args.limit ?? MAX_SOURCES) || MAX_SOURCES, 1), 6) + return searchBrief(query, limit, config) }, ) } @@ -59,7 +56,7 @@ function truncate(text: string): string { function createReadPage(config: WebAccessConfig): Tool { return defineTool( 'read_page', - 'Fetch a web page and return its readable text. Use after web_search when a snippet is not enough.', + 'Fetch a web page and return its readable text. Use after web_search when its extracts are not enough, or when the user gives you a URL.', { type: 'object', properties: { diff --git a/src/tools/search-brief.test.ts b/src/tools/search-brief.test.ts new file mode 100644 index 0000000..c1fc3a2 --- /dev/null +++ b/src/tools/search-brief.test.ts @@ -0,0 +1,360 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + compareSources, + formatBrief, + leadOf, + searchBrief, + selectDiverseSources, + siteOf, + type BriefSource, +} from './search-brief' +import type { SearchResult, WebAccessConfig } from './web' + +function hit(url: string, title = 'Title', snippet = 'A snippet.'): SearchResult { + return { title, url, snippet } +} + +function source(site: string, extract: string, read = true): BriefSource { + return { title: site, url: `https://${site}/page`, site, extract, read } +} + +const NOW = new Date('2026-08-26T09:00:00') + +describe('siteOf', () => { + it.each([ + ['https://www.reuters.com/business/', 'reuters.com'], + ['https://en.wikipedia.org/wiki/Arc', 'wikipedia.org'], + ['https://news.bbc.co.uk/story', 'bbc.co.uk'], + ['https://example.com', 'example.com'], + // Two pages of one publisher are one source however deep the subdomain goes. + ['https://investor.nvidia.com/news/q2/', 'nvidia.com'], + ])('reads %s as %s', (url, expected) => { + expect(siteOf(url)).toBe(expected) + }) +}) + +describe('selectDiverseSources', () => { + it('keeps the best result of each site and drops the rest', () => { + const selected = selectDiverseSources([ + hit('https://reuters.com/one'), + hit('https://reuters.com/two'), + hit('https://www.reuters.com/three'), + hit('https://bbc.com/story'), + ]) + + expect(selected.map((result) => result.url)).toEqual(['https://reuters.com/one', 'https://bbc.com/story']) + }) + + it('holds Wikipedia back when independent sites can be read instead', () => { + // Its extract is a paragraph where a results page gives a line, so left in + // rank order it decides the answer by itself and the rest are decoration. + const selected = selectDiverseSources( + [ + hit('https://en.wikipedia.org/wiki/Guterres'), + hit('https://un.org/sg'), + hit('https://reuters.com/un'), + hit('https://bbc.com/un'), + hit('https://ft.com/un'), + ], + 4, + ) + + expect(selected.map((result) => siteOf(result.url))).toEqual([ + 'un.org', + 'reuters.com', + 'bbc.com', + 'ft.com', + ]) + }) + + it('falls back to Wikipedia rather than returning nothing', () => { + const selected = selectDiverseSources([hit('https://de.wikipedia.org/wiki/Arc')], 4) + + expect(selected).toHaveLength(1) + }) + + it('ignores a result with no URL to read', () => { + expect(selectDiverseSources([hit(''), hit('https://bbc.com/story')])).toHaveLength(1) + }) +}) + +describe('leadOf', () => { + it('stops at the second heading rather than reading the whole page', () => { + const page = ['# Arc', 'Arc is a web browser.', '## Related articles', 'Ten other browsers.'].join('\n') + + expect(leadOf(page, 700)).toBe('Arc is a web browser.') + }) + + it('strips the markdown the reader emits', () => { + const page = + '![Image 3](https://x/i.png)\n**Arc** is a [browser](https://arc.net) from `The Browser Company`.' + + expect(leadOf(page, 700)).toBe('Arc is a browser from The Browser Company.') + }) + + it('cuts to the budget it was given', () => { + const lead = leadOf('word '.repeat(400), 200) + + expect(lead).toHaveLength(201) + expect(lead.endsWith('…')).toBe(true) + }) +}) + +describe('compareSources', () => { + it('reports a name and a year several sites share', () => { + const { overlap } = compareSources([ + source('a.example', 'Ama Osei has led the airline since 2023.'), + source('b.example', 'Ama Osei was confirmed in 2023.'), + source('c.example', 'The chief executive is Ama Osei.'), + ]) + + expect(overlap).toEqual([ + { term: 'Ama Osei', sites: 3 }, + { term: '2023', sites: 2 }, + ]) + }) + + it('names the site that disagrees with the others', () => { + const { conflicts } = compareSources([ + source('a.example', 'Chief executive since 2023.'), + source('b.example', 'In post since 2023.'), + source('c.example', 'In post since 2023.'), + source('d.example', 'Chief executive since 2021.'), + ]) + + expect(conflicts).toEqual([ + { + values: [ + { display: '2023', sites: ['a.example', 'b.example', 'c.example'] }, + { display: '2021', sites: ['d.example'] }, + ], + }, + ]) + }) + + it('says nothing when only one site names a figure', () => { + // One reading is not a disagreement, and reporting it as one would tell the + // model to hedge an answer nothing contradicted. + const { conflicts } = compareSources([ + source('a.example', 'Revenue was 46 billion.'), + source('b.example', 'The company is large.'), + ]) + + expect(conflicts).toEqual([]) + }) + + it('does not read a site that names both figures as contradicting either', () => { + const { conflicts } = compareSources([ + source('a.example', 'Revenue rose to 46 billion.'), + source('b.example', 'Revenue rose to 46 billion.'), + source('c.example', 'Revenue was 31 billion, and is now 46 billion.'), + ]) + + expect(conflicts).toEqual([]) + }) + + it('treats a figure written with either separator as one value', () => { + const { overlap } = compareSources([ + source('a.example', 'It reached 46,700 units.'), + source('b.example', 'It reached 46.700 units.'), + ]) + + expect(overlap).toEqual([{ term: '46,700', sites: 2 }]) + }) + + it('ignores a capitalised span that is only the start of a sentence', () => { + // Three pages all opening "The company" would otherwise read as agreement + // about a subject none of them named. + const { overlap } = compareSources([ + source('a.example', 'The company sells software.'), + source('b.example', 'The company sells software.'), + ]) + + expect(overlap).toEqual([]) + }) + + it('prefers the full name over a fragment of it', () => { + const { overlap } = compareSources([ + source('a.example', 'Ama Osei leads it.'), + source('b.example', 'Ama Osei leads it.'), + ]) + + expect(overlap).toEqual([{ term: 'Ama Osei', sites: 2 }]) + }) +}) + +describe('formatBrief', () => { + const sources = [ + source('un.org', 'António Guterres is the ninth Secretary-General, in post since 2017.'), + source('reuters.com', 'António Guterres was reappointed in 2021 for a term to 2026.'), + ] + + it('dates the brief, numbers the sources and keeps every URL', () => { + const brief = formatBrief('UN secretary-general', sources, NOW) + + expect(brief).toContain('Searched 2026-08-26 for "UN secretary-general" — 2 sources') + expect(brief).toContain('1. un.org (un.org)\n https://un.org/page') + expect(brief).toContain('2. reuters.com (reuters.com)\n https://reuters.com/page') + expect(brief).toContain('Agreed across sources: "António Guterres" in 2/2') + }) + + it('marks a source that could only be summarised from its search snippet', () => { + const brief = formatBrief('x', [source('a.example', 'A snippet.', false)], NOW) + + expect(brief).toContain('1 snippet only') + expect(brief).toContain('— snippet only') + }) + + it('says so rather than implying a cross-check that did not happen', () => { + const brief = formatBrief('x', [source('a.example', 'One reading.')], NOW) + + expect(brief).toContain('Only one source was readable, so nothing was cross-checked.') + }) + + it('stays inside the context it is allowed', () => { + const four = [1, 2, 3, 4].map((index) => source(`s${index}.example`, 'word '.repeat(400))) + + expect(formatBrief('x', four, NOW).length).toBeLessThanOrEqual(4001) + }) +}) + +/** Every response the reader and the providers answer with, in order. */ +function stubFetch(...bodies: unknown[]) { + const fetchMock = vi.fn(async () => { + const next = bodies.shift() + if (next instanceof Error) throw next + if (typeof next === 'number') return { ok: false, status: next, json: async () => ({}) } as Response + return { ok: true, status: 200, json: async () => next ?? {} } as Response + }) + vi.stubGlobal('fetch', fetchMock) + return fetchMock +} + +function readerPage(url: string, title: string, content: string) { + return { data: { url, title, content } } +} + +function duckDuckGoPage(...urls: string[]) { + const hits = urls.flatMap((url, index) => [ + `${index + 1}.[Result ${index + 1}](https://duckduckgo.com/l/?uddg=${encodeURIComponent(url)})`, + `Snippet ${index + 1}.`, + new URL(url).hostname, + '', + ]) + return { data: { content: hits.join('\n') } } +} + +const duckduckgo: WebAccessConfig = { provider: 'duckduckgo' } + +beforeEach(() => { + vi.stubGlobal('fetch', vi.fn()) +}) + +afterEach(() => { + vi.unstubAllGlobals() +}) + +describe('searchBrief', () => { + it('reads one page per site and compares what they say', async () => { + const fetchMock = stubFetch( + duckDuckGoPage('https://un.org/sg', 'https://reuters.com/un', 'https://un.org/other'), + readerPage('https://un.org/sg', 'Secretary-General', 'The office is held by António Guterres.'), + readerPage('https://reuters.com/un', 'At the UN', 'António Guterres was reappointed in 2021.'), + ) + + const brief = await searchBrief('who leads the UN', 4, duckduckgo, NOW) + + // One results page plus one page per selected site, and the second un.org + // result is not a second source. + expect(fetchMock).toHaveBeenCalledTimes(3) + expect(brief).toContain('2 sources') + expect(brief).toContain('The office is held by António Guterres.') + expect(brief).toContain('Agreed across sources: "António Guterres" in 2/2') + }) + + it('keeps the sources it could read when a page fails', async () => { + const fetchMock = stubFetch( + duckDuckGoPage('https://un.org/sg', 'https://reuters.com/un'), + readerPage('https://un.org/sg', 'Secretary-General', 'The office is held by António Guterres.'), + // The reader's per-IP budget is shared, and it runs out mid-brief. + 429, + ) + + const brief = await searchBrief('who leads the UN', 4, duckduckgo, NOW) + + expect(fetchMock).toHaveBeenCalledTimes(3) + expect(brief).toContain('1 snippet only') + expect(brief).toContain('The office is held by António Guterres.') + expect(brief).toContain('Snippet 2.') + }) + + it('falls back to the snippet rather than dropping a source with an unreadable page', async () => { + const brief = await searchBrief('anything', 4, duckduckgo, NOW).catch((error: Error) => error) + + // Nothing was stubbed beyond the default mock, so the search itself fails and + // the tool throws rather than returning something that reads like a result. + expect(brief).toBeInstanceOf(Error) + }) + + it('reports a query that matched nothing without reading any page', async () => { + const fetchMock = stubFetch({ data: { content: 'No results found for zzzz.' } }) + + expect(await searchBrief('zzzz', 4, duckduckgo, NOW)).toBe('No results for "zzzz".') + expect(fetchMock).toHaveBeenCalledTimes(1) + }) + + it('asks for more candidates than it needs, since duplicates are dropped after ranking', async () => { + const fetchMock = stubFetch( + duckDuckGoPage('https://a.example/1'), + readerPage('https://a.example/1', 'A', 'Text.'), + ) + + await searchBrief('anything', 2, duckduckgo, NOW) + + // The count is a parameter of the request the provider was answering anyway, + // so asking for headroom costs nothing. + const [first] = fetchMock.mock.calls[0] as unknown as [string] + expect(first).toContain('duckduckgo.com') + }) + + it('spends no reader request when the provider sends its own text', async () => { + const fetchMock = stubFetch({ + data: { + webPages: { + value: [ + { name: 'One', url: 'https://a.example/1', snippet: 'index text', summary: 'Ama Osei leads it.' }, + { name: 'Two', url: 'https://b.example/2', snippet: 'index text', summary: 'Ama Osei leads it.' }, + ], + }, + }, + }) + + const brief = await searchBrief( + 'who leads it', + 4, + { provider: 'langsearch', langsearchApiKey: 'sk-live' }, + NOW, + ) + + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(brief).not.toContain('snippet only') + expect(brief).toContain('Agreed across sources: "Ama Osei" in 2/2') + }) + + it('leaves a Wikipedia search as the article list it has always been', async () => { + const fetchMock = stubFetch({ + query: { + pages: { + '1': { pageid: 1, title: 'Arc', index: 1, extract: 'A browser.', fullurl: 'https://w/arc' }, + }, + }, + }) + + const brief = await searchBrief('Arc', 4, { provider: 'wikipedia' }, NOW) + + // One encyclopedia cannot be several independent sources, and its extracts + // are already paragraphs, so nothing is fetched and nothing is compared. + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(brief).toBe('1. Arc\n https://w/arc\n A browser.') + }) +}) diff --git a/src/tools/search-brief.ts b/src/tools/search-brief.ts new file mode 100644 index 0000000..b25bd31 --- /dev/null +++ b/src/tools/search-brief.ts @@ -0,0 +1,583 @@ +/** + * One search, several independent sites, compared before the model sees them. + * + * A search used to return a title, a URL and one line each, and everything that + * made those lines worth anything — opening the promising ones, noticing that + * three of them say 2023 and the fourth says 2021 — was left to the model. A + * 0.8B model spends its whole tool budget there: four searches in a row, none of + * them read, and `src/agent/budget.ts` exists because of it. + * + * So the shape is the one `weather.ts` already uses. The comparison happens here + * rather than in the conversation: the pages are fetched in parallel, their + * leads are cut to a budget, and what they agree and disagree on is worked out + * deterministically. What reaches the model is one brief it can answer from. + * + * Two rules keep it from being an encyclopedia lookup wearing four coats. + * Sources must come from *different* sites, because two pages of one publisher + * are one source; and Wikipedia is used only when nothing else could be read, + * since its extract is a paragraph where a results page gives a line and it + * would otherwise be the richest source in every brief. + * + * Nothing here needs a server. The pages go through the same reader `read_page` + * uses, which is the only fetch in this project verified to survive CORS from + * the deployed origin. + */ + +import { + collapse, + readPage, + searchWeb, + truncate, + type SearchProvider, + type SearchResult, + type WebAccessConfig, +} from './web' + +/** How many different sites are compared. Four fits the brief; more crowds it. */ +export const MAX_SOURCES = 4 + +/** + * Room for one source's text. + * + * Four of these plus the header, the URLs and the comparison stay inside + * `MAX_BRIEF_CHARS`, which is half of what `read_page` alone is allowed — + * comparing four pages must not cost more context than reading one. + */ +const MAX_EXTRACT_CHARS = 700 +const MIN_EXTRACT_CHARS = 200 + +/** Roughly 1,000 tokens for the whole brief. */ +const MAX_BRIEF_CHARS = 4000 + +/** What the header, the numbering and the comparison take before extracts do. */ +const BRIEF_OVERHEAD_CHARS = 600 + +/** At most this many agreements and disagreements are reported. */ +const MAX_OVERLAP = 4 +const MAX_CONFLICTS = 2 + +/** + * Providers that send usable text with each result, so the brief can be built + * from one request. Everything else is read page by page through the reader. + */ +const PROVIDERS_WITH_EXTRACTS: SearchProvider[] = ['langsearch'] + +/** + * Suffixes under which the label before them is still a registrant rather than + * a site. Without these `bbc.co.uk` and `theguardian.co.uk` look like one site. + */ +const TWO_LABEL_SUFFIXES = new Set([ + 'co.uk', + 'org.uk', + 'ac.uk', + 'gov.uk', + 'co.jp', + 'ne.jp', + 'or.jp', + 'com.au', + 'net.au', + 'org.au', + 'co.nz', + 'com.br', + 'com.cn', + 'com.tr', + 'co.in', + 'co.za', + 'com.mx', + 'com.ar', + 'co.kr', + 'com.sg', +]) + +/** + * Wikipedia and its siblings, held back rather than dropped. + * + * They are the best single page on many subjects and the worst way to answer a + * question about several: a lead paragraph beside three one-line snippets + * decides the answer by itself, and mirrors of it are not a second opinion. + */ +const ENCYCLOPEDIAS = new Set(['wikipedia.org', 'wikimedia.org', 'wikidata.org']) + +/** The site a URL belongs to, as a reader would name it. */ +export function siteOf(url: string): string { + let host: string + try { + host = new URL(url).hostname.toLowerCase().replace(/^www\./, '') + } catch { + return url + } + + const labels = host.split('.') + if (labels.length < 3) return host + + const lastTwo = labels.slice(-2).join('.') + return TWO_LABEL_SUFFIXES.has(lastTwo) ? labels.slice(-3).join('.') : lastTwo +} + +/** + * Picks the results worth reading: one per site, encyclopedias last. + * + * Rank order is kept within each group, so the best result of a site is the one + * that survives and the search engine's own judgement is not second-guessed + * beyond these two rules. + */ +export function selectDiverseSources(results: SearchResult[], limit = MAX_SOURCES): SearchResult[] { + const seen = new Set() + const independent: SearchResult[] = [] + const encyclopedic: SearchResult[] = [] + + for (const result of results) { + if (!result.url) continue + const site = siteOf(result.url) + if (seen.has(site)) continue + seen.add(site) + ;(ENCYCLOPEDIAS.has(site) ? encyclopedic : independent).push(result) + } + + return [...independent, ...encyclopedic].slice(0, Math.max(limit, 0)) +} + +/** A markdown link, image or emphasis carries no meaning once the page is a paragraph. */ +function plainText(line: string): string { + return ( + line + .replace(/!\[[^\]]*\]\([^)]*\)/g, '') + .replace(/\[([^\]]*)\]\([^)]*\)/g, '$1') + // A space rather than nothing: the reader drops the spaces around its own + // marks, so deleting them fuses words into ones that do not exist. The + // cost is a space before the punctuation, which the next line takes back. + .replace(/[*_`>]+/g, ' ') + .replace(/\s+([.,;:!?])/g, '$1') + ) +} + +/** + * The opening prose of a page, up to a budget. + * + * The reader returns the whole page as markdown, and the answer is almost always + * in its first section — so this stops at the second heading rather than reading + * a nav column, a cookie notice and a related-articles list into the context. + */ +export function leadOf(text: string, budget: number): string { + const kept: string[] = [] + let headings = 0 + let length = 0 + + for (const raw of text.split('\n')) { + const line = raw.trim() + if (!line) continue + + if (/^#{1,6}\s/.test(line)) { + headings += 1 + if (headings > 1 && kept.length > 0) break + continue + } + if (/^[-*+|=_]+$/.test(line)) continue + + const prose = collapse(plainText(line)) + if (!prose) continue + + kept.push(prose) + length += prose.length + 1 + if (length >= budget) break + } + + return truncate(collapse(kept.join(' ')), budget) +} + +export interface BriefSource { + title: string + url: string + /** The site, which is what the comparison and the reader both name it by. */ + site: string + extract: string + /** False when the page could not be read and its search snippet stood in. */ + read: boolean +} + +interface Mention { + display: string + sites: Set +} + +const NAME_SPAN = /\p{Lu}[\p{L}'’-]*(?:\s+\p{Lu}[\p{L}'’-]*){0,2}/gu + +/** + * Words that start a sentence far more often than they start a name. + * + * A capitalised span is only evidence of a subject if it is not just the first + * word of a sentence, and the sites a term appears on cannot tell the + * difference — three pages all beginning "The company" would otherwise read as + * agreement about something. + */ +const SENTENCE_STARTERS = new Set([ + 'a', + 'aber', + 'after', + 'also', + 'an', + 'and', + 'as', + 'at', + 'auch', + 'but', + 'by', + 'das', + 'der', + 'die', + 'ein', + 'eine', + 'er', + 'es', + 'for', + 'from', + 'für', + 'he', + 'his', + 'her', + 'however', + 'if', + 'im', + 'in', + 'is', + 'it', + 'its', + 'mit', + 'nach', + 'of', + 'on', + 'or', + 'seit', + 'she', + 'sie', + 'since', + 'that', + 'the', + 'their', + 'these', + 'they', + 'this', + 'those', + 'to', + 'und', + 'von', + 'was', + 'we', + 'when', + 'which', + 'who', + 'wir', + 'with', + 'you', +]) + +/** Drops the leading sentence word, so "The United Nations" is compared as "United Nations". */ +function trimLeadingStopwords(span: string): string { + const words = span.split(/\s+/) + while (words.length > 0 && SENTENCE_STARTERS.has((words[0] ?? '').toLowerCase())) words.shift() + return words.join(' ') +} + +function namesIn(text: string): string[] { + const found: string[] = [] + + for (const [span] of text.matchAll(NAME_SPAN)) { + const name = trimLeadingStopwords(span).replace(/[’'-]+$/, '') + if (name.length < 3) continue + if (name.split(/\s+/).every((word) => SENTENCE_STARTERS.has(word.toLowerCase()))) continue + found.push(name) + } + + return found +} + +const NUMBER_SPAN = /\d{4}-\d{2}-\d{2}|\d+(?:[.,]\d+)*/g + +/** + * One value however it was written, so `46,700` and `46.700` are the same figure + * and `46.7` is not. A final group of three digits is a thousands separator + * unless the number opens with a zero, which only a decimal does. + */ +function canonicalNumber(display: string): string { + const groups = display.split(/[.,]/) + if (groups.length === 1) return display + + const last = groups.at(-1) ?? '' + const decimal = last.length !== 3 || groups[0] === '0' + return decimal ? `${groups.slice(0, -1).join('')}.${last}` : groups.join('') +} + +/** + * What two numbers have to share to be answers to the same question. + * + * Years are their own kind because that is where sources disagree in a way worth + * reporting; otherwise the digit count stands in for it, so a revenue figure is + * never compared against a percentage. + */ +function numberKind(display: string): string { + if (/^\d{4}-\d{2}-\d{2}$/.test(display)) return 'date' + + const canonical = canonicalNumber(display) + const year = Number(canonical) + if (/^\d{4}$/.test(canonical) && year >= 1800 && year <= 2099) return 'year' + + const [integer = ''] = canonical.split('.') + return `d${integer.length}` +} + +/** Single digits are list markers, counts and footnotes far more often than facts. */ +function comparableKind(kind: string): boolean { + return kind !== 'd1' +} + +export interface Agreement { + term: string + sites: number +} + +export interface Conflict { + values: { display: string; sites: string[] }[] +} + +export interface Comparison { + overlap: Agreement[] + conflicts: Conflict[] +} + +function record(into: Map, key: string, display: string, site: string): void { + const existing = into.get(key) + if (existing) { + existing.sites.add(site) + return + } + into.set(key, { display, sites: new Set([site]) }) +} + +/** + * A term covered by more sites, and the longer of two terms covered by the same + * number, is the more useful thing to report. + */ +function byReach(left: Mention, right: Mention): number { + return right.sites.size - left.sites.size || right.display.length - left.display.length +} + +/** "Ama Osei" says everything "Ama" does, so the shorter one is not a second finding. */ +function subsumed(term: string, kept: Agreement[]): boolean { + const lower = term.toLowerCase() + return kept.some((entry) => { + const other = entry.term.toLowerCase() + return other !== lower && (other.includes(lower) || lower.includes(other)) + }) +} + +/** + * What the sources agree and disagree on. + * + * Deterministic on purpose, exactly as `weather.ts` reconciles three forecasts + * without asking the model to: a second generation spent grading the extracts + * is the capacity the answer needed. Names and figures are what a rule can + * honestly compare, so they are all it claims to have compared. + */ +export function compareSources(sources: BriefSource[]): Comparison { + const names = new Map() + const numbers = new Map>() + + for (const source of sources) { + for (const name of namesIn(source.extract)) { + record(names, name.toLowerCase(), name, source.site) + } + + for (const [display] of source.extract.matchAll(NUMBER_SPAN)) { + const kind = numberKind(display) + if (!comparableKind(kind)) continue + const byValue = numbers.get(kind) ?? new Map() + numbers.set(kind, byValue) + record(byValue, canonicalNumber(display), display, source.site) + } + } + + const shared = [...names.values(), ...[...numbers.values()].flatMap((byValue) => [...byValue.values()])] + .filter((mention) => mention.sites.size > 1) + .sort(byReach) + + const overlap: Agreement[] = [] + for (const mention of shared) { + if (overlap.length >= MAX_OVERLAP) break + if (subsumed(mention.display, overlap)) continue + overlap.push({ term: mention.display, sites: mention.sites.size }) + } + + const conflicts: Conflict[] = [] + for (const byValue of numbers.values()) { + const [leading, ...rest] = [...byValue.values()].sort(byReach) + // A disagreement is only reportable when there is a reading to disagree + // with: one site against one other says which pages differ, not which is + // out of date, and the extracts are in front of the model either way. + if (!leading || leading.sites.size < 2) continue + + // A site naming both figures is not contradicting anything — a page that + // mentions last year's number alongside this year's is the ordinary way to + // write one. Only a site that names a value and not the agreed one is. + const dissenting = rest.filter((mention) => [...mention.sites].some((site) => !leading.sites.has(site))) + if (dissenting.length === 0) continue + + conflicts.push({ + values: [leading, ...dissenting].slice(0, 3).map((mention) => ({ + display: mention.display, + sites: [...mention.sites].sort(), + })), + }) + } + + conflicts.sort((left, right) => (right.values[0]?.sites.length ?? 0) - (left.values[0]?.sites.length ?? 0)) + + return { overlap, conflicts: conflicts.slice(0, MAX_CONFLICTS) } +} + +/** `YYYY-MM-DD` in the user's own timezone, which is the day they are asking about. */ +function localDate(now: Date): string { + const month = String(now.getMonth() + 1).padStart(2, '0') + const day = String(now.getDate()).padStart(2, '0') + return `${now.getFullYear()}-${month}-${day}` +} + +function overlapLine(overlap: Agreement[], total: number): string { + const parts = overlap.map((entry) => `"${entry.term}" in ${entry.sites}/${total}`) + return `Agreed across sources: ${parts.join('; ')}` +} + +function conflictLine(conflict: Conflict): string { + const parts = conflict.values.map((value) => `"${value.display}" (${value.sites.join(', ')})`) + return `Sources disagree: ${parts.join(' vs ')}` +} + +/** + * The brief, in the shape the rest of the app already reads. + * + * Every source keeps its own URL on its own line, because that is what + * `reviewAnswer` grounds a citation against and what `splitSources` turns into + * pills. A comparison the model cannot cite is worse than no comparison. + */ +export function formatBrief(query: string, sources: BriefSource[], now = new Date()): string { + const snippetOnly = sources.filter((source) => !source.read).length + const header = [ + `Searched ${localDate(now)} for "${query}" — ${sources.length} ${sources.length === 1 ? 'source' : 'sources'}`, + snippetOnly > 0 ? `, ${snippetOnly} snippet only` : '', + ].join('') + + const entries = sources.map((source, index) => + [ + `${index + 1}. ${source.title || source.site} (${source.site})${source.read ? '' : ' — snippet only'}`, + ` ${source.url}`, + ` ${source.extract || 'No text could be read from this page.'}`, + ].join('\n'), + ) + + const { overlap, conflicts } = compareSources(sources) + const footer = + sources.length === 1 + ? ['Only one source was readable, so nothing was cross-checked.'] + : [ + ...(overlap.length > 0 ? [overlapLine(overlap, sources.length)] : []), + ...conflicts.map(conflictLine), + ] + + return truncate([header, ...entries, ...footer].join('\n'), MAX_BRIEF_CHARS) +} + +/** Splits the extract budget between the sources that will actually use it. */ +function extractBudget(count: number): number { + if (count === 0) return MAX_EXTRACT_CHARS + const share = Math.floor((MAX_BRIEF_CHARS - BRIEF_OVERHEAD_CHARS) / count) + return Math.max(MIN_EXTRACT_CHARS, Math.min(MAX_EXTRACT_CHARS, share)) +} + +function fromSnippet(result: SearchResult, budget: number): BriefSource { + const extract = truncate(collapse(result.extract ?? result.snippet), budget) + return { + title: result.title, + url: result.url, + site: siteOf(result.url), + extract, + // A provider that sends its own text has already been read once; a bare + // search snippet has not, and the brief says which of the two this is. + read: Boolean(result.extract), + } +} + +/** + * Reads the selected pages at once. + * + * `allSettled` rather than `all`: a page that 404s, refuses the reader or spends + * the last of a rate limit costs its own source and nothing else. Dropping the + * whole brief for one dead link would be the failure this tool exists to avoid. + */ +async function readSources( + selected: SearchResult[], + config: WebAccessConfig, + budget: number, +): Promise { + if (PROVIDERS_WITH_EXTRACTS.includes(config.provider)) { + return selected.map((result) => fromSnippet(result, budget)) + } + + const settled = await Promise.allSettled(selected.map((result) => readPage(result.url, config))) + + return selected.map((result, index) => { + const outcome = settled[index] + if (outcome?.status !== 'fulfilled') return fromSnippet(result, budget) + + const lead = leadOf(outcome.value.text, budget) + if (!lead) return fromSnippet(result, budget) + + return { + title: outcome.value.title || result.title, + url: outcome.value.url || result.url, + site: siteOf(outcome.value.url || result.url), + extract: lead, + read: true, + } + }) +} + +/** The list-of-snippets a Wikipedia search has always returned. */ +function formatArticles(results: SearchResult[]): string { + return results + .map((result, index) => `${index + 1}. ${result.title}\n ${result.url}\n ${result.snippet}`) + .join('\n') +} + +/** + * Everything one `web_search` call does. + * + * A turn has four tool rounds, so a tool that needs the model to search, then + * read, then read again to be useful is the wrong shape — the searching, the + * reading and the comparing all happen here, and the model spends its rounds on + * the answer instead. + * + * Wikipedia is exempt: comparing several independent sites is not something one + * encyclopedia can do, and its extracts are already paragraphs. That provider + * keeps returning articles, which is what the tool description promises there. + */ +export async function searchBrief( + query: string, + sourceLimit: number, + config: WebAccessConfig, + now = new Date(), +): Promise { + if (config.provider === 'wikipedia') { + const articles = await searchWeb(query, sourceLimit, config) + return articles.length === 0 ? `No results for "${query}".` : formatArticles(articles) + } + + // More candidates than sources, because same-site duplicates and encyclopedia + // mirrors are removed after ranking. It costs nothing: the count is a + // parameter of the one request the provider was going to answer anyway. + const candidates = await searchWeb(query, Math.min(sourceLimit + 3, 10), config) + if (candidates.length === 0) return `No results for "${query}".` + + const selected = selectDiverseSources(candidates, sourceLimit) + const sources = await readSources(selected, config, extractBudget(selected.length)) + + return formatBrief(query, sources, now) +} diff --git a/src/tools/web.test.ts b/src/tools/web.test.ts index cc3e654..3177e64 100644 --- a/src/tools/web.test.ts +++ b/src/tools/web.test.ts @@ -361,7 +361,7 @@ describe('searchWeb with LangSearch', () => { }, } - it('authenticates, suppresses the long summaries, and maps the results', async () => { + it('authenticates, asks for the summaries, and maps the results', async () => { const fetchMock = stubFetch(jsonResponse(payload)) const results = await searchWeb('chancellor of germany', 2, langsearch) @@ -369,19 +369,40 @@ describe('searchWeb with LangSearch', () => { const { url, headers, body } = lastRequest(fetchMock) expect(url.href).toBe('https://api.langsearch.com/v1/web-search') expect(headers.authorization).toBe('Bearer sk-live') - // A summary per result is the whole page behind it, which would crowd out - // the answer as well as the prompt. - expect(body).toEqual({ query: 'chancellor of germany', count: 2, summary: false }) + // The summaries are what let this provider compare several sites on the one + // request it was already spending, rather than a reader call per result. + expect(body).toEqual({ query: 'chancellor of germany', count: 2, summary: true }) expect(results).toEqual([ { title: 'Chancellor of Germany', url: 'https://en.wikipedia.org/wiki/Chancellor_of_Germany', snippet: 'the chancellor of germany is the head of government . friedrich merz holds the office .', + extract: 'A much longer text this provider only sends when asked.', }, + // No summary came back for this one, so nothing is invented for it. { title: 'Bundeskanzler', url: 'https://www.bundeskanzler.de/', snippet: 'der bundeskanzler' }, ]) }) + // A summary is the whole page behind the result. Several of them uncapped is + // the context the answer needed, which is why they used to be switched off. + it('caps a summary that arrives as a whole page', async () => { + stubFetch( + jsonResponse({ + data: { + webPages: { + value: [{ name: 'Long', url: 'https://example.com/', summary: 'word '.repeat(400) }], + }, + }, + }), + ) + + const [result] = await searchWeb('anything', 1, langsearch) + + expect(result?.extract).toHaveLength(801) + expect(result?.extract?.endsWith('…')).toBe(true) + }) + it('honours the limit even when the provider overshoots it', async () => { stubFetch(jsonResponse(payload)) diff --git a/src/tools/web.ts b/src/tools/web.ts index dfe5f7a..2f35980 100644 --- a/src/tools/web.ts +++ b/src/tools/web.ts @@ -31,6 +31,14 @@ export interface SearchResult { title: string url: string snippet: string + /** + * Longer text the provider sent alongside the snippet, where it offers one. + * + * Only LangSearch does, and it is what lets that provider be compared across + * several pages without spending a reader request per result. Capped here + * rather than where it is used, because the raw field is a whole page. + */ + extract?: string } export interface PageContent { @@ -68,7 +76,7 @@ export interface SearchProviderInfo { const DUCKDUCKGO_PROVIDER: SearchProviderInfo = { id: 'duckduckgo', label: 'DuckDuckGo', - note: 'Full web search including current events, with no key and no signup. Its results page is read through r.jina.ai, which allows 20 requests a minute without a key — a search and a page read spend one each.', + note: 'Full web search including current events, with no key and no signup. Its results page and the pages it compares are read through r.jina.ai, which allows 20 requests a minute without a key — so one search spends up to five of them.', } export const SEARCH_PROVIDERS: SearchProviderInfo[] = [ @@ -76,19 +84,19 @@ export const SEARCH_PROVIDERS: SearchProviderInfo[] = [ { id: 'wikipedia', label: 'Wikipedia', - note: 'Encyclopedic facts with a full lead paragraph each, straight from the MediaWiki API. Nothing about current events.', + note: 'Encyclopedic facts with a full lead paragraph each, straight from the MediaWiki API. Nothing about current events, and one encyclopedia cannot be several independent sources, so searches here are not cross-checked.', }, { id: 'langsearch', label: 'LangSearch', keyField: 'langsearchApiKey', - note: 'Full web search from a search API, on a key that costs nothing: the free tier allows 1,000 searches a day and one a second. Its snippets are index text rather than prose, so they read less cleanly than the others.', + note: 'Full web search from a search API, on a key that costs nothing: the free tier allows 1,000 searches a day and one a second. It sends text with every result, so several sites are compared without spending the reader’s allowance — but that text is index prose rather than a page, and reads less cleanly than the others.', }, { id: 'jina', label: 'Jina', keyField: 'jinaApiKey', - note: 'Full web search from a search API rather than a scraped results page. Needs a Jina key, which also raises the reader’s limits.', + note: 'Full web search from a search API rather than a scraped results page. Needs a Jina key, which also raises the limits of the reader that fetches the pages being compared.', }, ] @@ -131,6 +139,15 @@ export function normalizeWebAccess( const REQUEST_TIMEOUT_MS = 20_000 const MAX_SNIPPET_CHARS = 600 +/** + * How much of a provider-supplied extract is kept. + * + * LangSearch's summary is the whole page behind the result. Several of those + * uncapped is the entire context of a 0.8B model, which is what kept them + * switched off here; capped, they are the cheapest way to compare pages. + */ +const MAX_PROVIDER_EXTRACT_CHARS = 800 + const WIKIPEDIA_ENDPOINT = 'https://en.wikipedia.org/w/api.php' const JINA_SEARCH_ENDPOINT = 'https://s.jina.ai/' const LANGSEARCH_ENDPOINT = 'https://api.langsearch.com/v1/web-search' @@ -149,11 +166,12 @@ const READER_ENDPOINT = 'https://r.jina.ai/' */ const DUCKDUCKGO_ENDPOINTS = ['https://duckduckgo.com/html/', 'https://lite.duckduckgo.com/lite/'] -function collapse(value: string): string { +/** Shared with `search-brief.ts`, so provider text and page text are normalised the same way. */ +export function collapse(value: string): string { return value.replace(/\s+/g, ' ').trim() } -function truncate(value: string, limit: number): string { +export function truncate(value: string, limit: number): string { return value.length > limit ? `${value.slice(0, limit)}…` : value } @@ -297,7 +315,9 @@ async function searchJina(query: string, limit: number, apiKey: string): Promise interface LangSearchResponse { /** Set when the envelope carries a complaint rather than a result set. */ msg?: string | null - data?: { webPages?: { value?: { name?: string; url?: string; snippet?: string }[] } } + data?: { + webPages?: { value?: { name?: string; url?: string; snippet?: string; summary?: string }[] } + } } async function searchLangSearch(query: string, limit: number, apiKey: string): Promise { @@ -314,9 +334,12 @@ async function searchLangSearch(query: string, limit: number, apiKey: string): P accept: 'application/json', authorization: `Bearer ${apiKey}`, }, - // `summary: true` returns the whole page behind each result, which is the - // context a 0.8B model has for the answer as well as for the search. - body: JSON.stringify({ query, count: limit, summary: false }), + // `summary: true` returns the whole page behind each result. Uncapped + // that is the context a 0.8B model has for the answer as well as for the + // search, which is why it used to be off; capped at + // `MAX_PROVIDER_EXTRACT_CHARS` it buys several comparable pages for the + // one request this provider was already spending. + body: JSON.stringify({ query, count: limit, summary: true }), }, ) @@ -328,14 +351,18 @@ async function searchLangSearch(query: string, limit: number, apiKey: string): P throw new Error(payload.msg?.trim() || 'LangSearch returned no result set for this query.') } - return pages.slice(0, limit).map((page) => ({ - title: collapse(page.name ?? ''), - url: page.url ?? '', - // Snippets arrive as normalised index text — lower-cased, with spaces around - // the punctuation. Collapsing the newlines is as far as this goes; putting - // the prose back is not something a rule could do. - snippet: truncate(collapse(page.snippet ?? ''), MAX_SNIPPET_CHARS), - })) + return pages.slice(0, limit).map((page) => { + const summary = truncate(collapse(page.summary ?? ''), MAX_PROVIDER_EXTRACT_CHARS) + return { + title: collapse(page.name ?? ''), + url: page.url ?? '', + // Snippets arrive as normalised index text — lower-cased, with spaces around + // the punctuation. Collapsing the newlines is as far as this goes; putting + // the prose back is not something a rule could do. + snippet: truncate(collapse(page.snippet ?? ''), MAX_SNIPPET_CHARS), + ...(summary ? { extract: summary } : {}), + } + }) } interface ReaderResponse { From 4534ee34870f187b4322d175b21e93d2fabc79b5 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 26 Aug 2026 10:36:39 +0000 Subject: [PATCH 2/2] Read a page from its first sentence, not from its nav column MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified against the live web, the extracts were menus. un.org spent the whole 700-character budget on "Skip to main content · Welcome · English Français · Home · Biography", and the paragraph naming the office holder never reached the model. A menu is a list of labels and carries no sentence, so the lead now starts at the first line that ends one and stops at the heading after it. A page with no sentence in it at all is read from the top, because that is what it says. Flattened table rows are skipped for the same reason. The same run showed Wikipedia padding a brief that already had independent sources, which is what the change was meant to stop: an encyclopedia is now a way to have two readings instead of one, and is dropped once two independent sites have answered. Co-authored-by: Sebastian --- README.md | 4 +- src/tools/search-brief.test.ts | 47 ++++++++++++++++- src/tools/search-brief.ts | 92 ++++++++++++++++++++++++++-------- 3 files changed, 121 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 1730eb3..c015bad 100644 --- a/README.md +++ b/README.md @@ -230,7 +230,9 @@ So `src/tools/search-brief.ts` takes the shape [`weather`](#tools) already had: Two rules decide which four, and both exist to stop the brief being an encyclopedia lookup wearing four coats: - **One page per site.** Two pages of one publisher are one source, so a second hit on a domain is dropped and the search engine's ranking decides the rest. `siteOf` reads `investor.nvidia.com` and `nvidianews.nvidia.com` as one site, and knows that `bbc.co.uk` and `theguardian.co.uk` are two. -- **Wikipedia goes last.** Its extract is a paragraph where a results page gives a line, so left in rank order it decides every answer by itself — and a mirror of it is not a second opinion. It is used when nothing else could be read, and not before. +- **Wikipedia goes last.** Its extract is a paragraph where a results page gives a line, so left in rank order it decides every answer by itself — and a mirror of it is not a second opinion. It is a way to have two readings instead of one, never a way to fill a brief that already has independent ones: once two independent sites have answered, the encyclopedia is dropped. + +Each page is read from its **first real sentence**, not from the top. This is the rule that made the difference in practice: read from the top, the budget went on `Skip to main content · Welcome · English Français · Home · About`, and a 0.8B model handed 700 characters of nav column has been handed nothing. A menu is a list of labels and carries no sentence, so the first line that ends one is where the page starts talking; the section after it is where it stops. A page with no sentence in it at all — a price grid, a table — is read from the top instead, because that is what it says. What the sources agree and disagree on is worked out **deterministically**, for the same reason the three forecasts are reconciled in `weather.ts` rather than in the prompt: a second generation spent grading four extracts is exactly the capacity the answer needed, and intrinsic self-grading makes reasoning worse rather than better. Names and figures are what a rule can honestly compare, so they are all it claims to have compared: diff --git a/src/tools/search-brief.test.ts b/src/tools/search-brief.test.ts index c1fc3a2..25691eb 100644 --- a/src/tools/search-brief.test.ts +++ b/src/tools/search-brief.test.ts @@ -45,7 +45,7 @@ describe('selectDiverseSources', () => { expect(selected.map((result) => result.url)).toEqual(['https://reuters.com/one', 'https://bbc.com/story']) }) - it('holds Wikipedia back when independent sites can be read instead', () => { + it('drops Wikipedia when independent sites can be read instead', () => { // Its extract is a paragraph where a results page gives a line, so left in // rank order it decides the answer by itself and the rest are decoration. const selected = selectDiverseSources( @@ -67,6 +67,17 @@ describe('selectDiverseSources', () => { ]) }) + it('keeps Wikipedia rather than leaving one site to answer alone', () => { + // Two readings and one of them an encyclopedia beats a brief with nothing + // to cross-check against. Padding is the thing to avoid, not the mention. + const selected = selectDiverseSources( + [hit('https://un.org/sg'), hit('https://en.wikipedia.org/wiki/Guterres')], + 4, + ) + + expect(selected.map((result) => siteOf(result.url))).toEqual(['un.org', 'wikipedia.org']) + }) + it('falls back to Wikipedia rather than returning nothing', () => { const selected = selectDiverseSources([hit('https://de.wikipedia.org/wiki/Arc')], 4) @@ -79,6 +90,40 @@ describe('selectDiverseSources', () => { }) describe('leadOf', () => { + it('starts at the first real sentence rather than in the nav column', () => { + // Observed on un.org: read from the top, the whole budget went on the menu, + // and the paragraph naming the office holder never reached the model. + const page = [ + '# About the Secretary-General', + 'Skip to main content', + 'Welcome to the United Nations English Français Русский Español Search Home Biography Reports', + 'António Guterres, the ninth Secretary-General of the United Nations, took office on 1 January 2017.', + ].join('\n') + + expect(leadOf(page, 700)).toBe( + 'António Guterres, the ninth Secretary-General of the United Nations, took office on 1 January 2017.', + ) + }) + + it('reads a page with no sentence in it from the top', () => { + // A price grid says what it says. Finding no prose must not return nothing. + const page = ['# NVDA', 'Revenue (ttm) $253.49B', 'Employees 42,000'].join('\n') + + expect(leadOf(page, 700)).toBe('Revenue (ttm) $253.49B Employees 42,000') + }) + + it('skips a table flattened into a line of cells', () => { + const page = [ + '| Incumbent | António Guterres |', + '| --- | --- |', + 'The office is described in Chapter XV of the Charter of the United Nations.', + ].join('\n') + + expect(leadOf(page, 700)).toBe( + 'The office is described in Chapter XV of the Charter of the United Nations.', + ) + }) + it('stops at the second heading rather than reading the whole page', () => { const page = ['# Arc', 'Arc is a web browser.', '## Related articles', 'Ten other browsers.'].join('\n') diff --git a/src/tools/search-brief.ts b/src/tools/search-brief.ts index b25bd31..b1396c3 100644 --- a/src/tools/search-brief.ts +++ b/src/tools/search-brief.ts @@ -14,9 +14,10 @@ * * Two rules keep it from being an encyclopedia lookup wearing four coats. * Sources must come from *different* sites, because two pages of one publisher - * are one source; and Wikipedia is used only when nothing else could be read, - * since its extract is a paragraph where a results page gives a line and it - * would otherwise be the richest source in every brief. + * are one source; and an encyclopedia is only used to get off a single reading, + * never to fill a brief that has independent ones, since its extract is a + * paragraph where a results page gives a line and it would otherwise be the + * richest source in every brief. * * Nothing here needs a server. The pages go through the same reader `read_page` * uses, which is the only fetch in this project verified to survive CORS from @@ -134,7 +135,11 @@ export function selectDiverseSources(results: SearchResult[], limit = MAX_SOURCE ;(ENCYCLOPEDIAS.has(site) ? encyclopedic : independent).push(result) } - return [...independent, ...encyclopedic].slice(0, Math.max(limit, 0)) + // An encyclopedia is a way to have two readings rather than one, never a way + // to fill a brief that already has independent ones. Padding four slots with + // it is how a search of the whole web ends up answering out of Wikipedia. + const sources = independent.length > 1 ? independent : [...independent, ...encyclopedic] + return sources.slice(0, Math.max(limit, 0)) } /** A markdown link, image or emphasis carries no meaning once the page is a paragraph. */ @@ -151,34 +156,81 @@ function plainText(line: string): string { ) } +interface Line { + text: string + heading: boolean +} + +function readableLines(text: string): Line[] { + const lines: Line[] = [] + + for (const raw of text.split('\n')) { + const line = raw.trim() + if (!line) continue + + if (/^#{1,6}\s/.test(line)) { + lines.push({ text: '', heading: true }) + continue + } + // A horizontal rule, and a table row flattened into one line of cells. The + // second is not a paragraph however much text it holds. + if (/^[-*+|=_]+$/.test(line) || line.startsWith('|')) continue + + const prose = collapse(plainText(line)) + if (prose) lines.push({ text: prose, heading: false }) + } + + return lines +} + +/** A full stop, not a decimal point: `213.05` ends no sentence. */ +const SENTENCE_END = /[.!?]["'”’)\]]?(?:\s|$)/ + +/** + * Whether a line is written prose rather than furniture. + * + * This is the rule that made the difference in practice. Read from the top, the + * budget went on "Skip to main content · Welcome · English Français · Home · + * About" — a nav column is long, and a 0.8B model handed 700 characters of it + * has been handed nothing. A menu is a list of labels and carries no sentence, + * so the first line that ends a sentence is where the page starts talking. + */ +function isProse(line: Line): boolean { + return ( + !line.heading && + line.text.length >= 60 && + line.text.split(' ').length >= 8 && + SENTENCE_END.test(line.text) + ) +} + /** * The opening prose of a page, up to a budget. * * The reader returns the whole page as markdown, and the answer is almost always - * in its first section — so this stops at the second heading rather than reading - * a nav column, a cookie notice and a related-articles list into the context. + * in its first section — so this starts at the first real sentence and stops at + * the heading after it, rather than reading a nav column, a cookie notice and a + * related-articles list into the context. + * + * A page with no sentence in it at all — a table, a price grid — is read from + * the top instead. It is worth less, and it is what the page says. */ export function leadOf(text: string, budget: number): string { + const lines = readableLines(text) + const opening = lines.findIndex(isProse) + const kept: string[] = [] - let headings = 0 let length = 0 - for (const raw of text.split('\n')) { - const line = raw.trim() - if (!line) continue - - if (/^#{1,6}\s/.test(line)) { - headings += 1 - if (headings > 1 && kept.length > 0) break + for (const line of lines.slice(opening === -1 ? 0 : opening)) { + if (line.heading) { + // A heading once something has been kept starts the next section. + if (kept.length > 0) break continue } - if (/^[-*+|=_]+$/.test(line)) continue - - const prose = collapse(plainText(line)) - if (!prose) continue - kept.push(prose) - length += prose.length + 1 + kept.push(line.text) + length += line.text.length + 1 if (length >= budget) break }