From 265a179a6060ed5b925fea277d624142ced7cdeb Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 26 Aug 2026 10:57:25 +0000 Subject: [PATCH 1/3] Say when an answer came from memory rather than from a source MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Asked about a historical figure, the model produced ages of 48, 142 and 200 years and a claim no source made. Three things let that reach the screen looking like research, and two of them are fixable. Routing was the larger half. Only `wer war X` reached research-question, so `wie alt wurde X`, `wann starb X`, `erzähl mir über X` and `was hat X gemacht` matched no skill at all — nothing suggested searching and no exemplar showed it. Those shapes route now, in both languages, and a joke still does not. Figures were only ever compared against the calculator. So the one finding on "he lived to 142" was that it cited nothing, and the correction appended a real encyclopedia URL to an invented number. They are compared against every tool result now, and only where there is evidence to compare with. And an answer produced with no tool call at all had nothing to check, so every check stayed silent. A turn whose tool list was narrowed to research tools and that fetched nothing is now labelled: answered from the model's own memory, not from a search. A label rather than a finding, because the correction round has no tools and could not fix it. Reference works also stop being dropped from the brief once two other sites answer. For history an encyclopedia is the most reliable thing the search returned, so it gets one seat of four: never the majority, never absent. Co-authored-by: Sebastian --- README.md | 32 +++++-- src/agent/budget.test.ts | 2 +- src/agent/loop.test.ts | 39 ++++++++- src/agent/loop.ts | 38 +++++++- src/agent/review.test.ts | 121 +++++++++++++++++++++++++- src/agent/review.ts | 121 ++++++++++++++++++++++++-- src/components/MessageItem.test.tsx | 22 +++++ src/components/MessageItem.tsx | 12 +++ src/eval/scenarios.ts | 13 +++ src/skills/library.test.ts | 28 ++++++ src/skills/research-question/SKILL.md | 17 ++++ src/tools/search-brief.test.ts | 18 ++-- src/tools/search-brief.ts | 54 +++++++----- 13 files changed, 466 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index c015bad..eed4480 100644 --- a/README.md +++ b/README.md @@ -230,7 +230,7 @@ So `src/tools/search-brief.ts` takes the shape [`weather`](#tools) already had: Two rules decide which four, and both exist to stop the brief being an encyclopedia lookup wearing four coats: - **One page per site.** Two pages of one publisher are one source, so a second hit on a domain is dropped and the search engine's ranking decides the rest. `siteOf` reads `investor.nvidia.com` and `nvidianews.nvidia.com` as one site, and knows that `bbc.co.uk` and `theguardian.co.uk` are two. -- **Wikipedia goes last.** Its extract is a paragraph where a results page gives a line, so left in rank order it decides every answer by itself — and a mirror of it is not a second opinion. It is a way to have two readings instead of one, never a way to fill a brief that already has independent ones: once two independent sites have answered, the encyclopedia is dropped. +- **A reference work gets one seat of the four.** Both extremes of this were wrong. Left to rank freely, Wikipedia and Britannica take several seats and decide the answer by themselves — a lead paragraph beside three one-line snippets is not a comparison, and a mirror of Wikipedia is not a second opinion. Dropped whenever two other sites answered, the brief for _who was X_ fills up with whatever happens to rank, and for history and biography an encyclopedia is the most reliable thing the search returned rather than the least. One seat is the rule that survives both: never the majority, never absent when the search found one. Each page is read from its **first real sentence**, not from the top. This is the rule that made the difference in practice: read from the top, the budget went on `Skip to main content · Welcome · English Français · Home · About`, and a 0.8B model handed 700 characters of nav column has been handed nothing. A menu is a list of labels and carries no sentence, so the first line that ends one is where the page starts talking; the section after it is where it stops. A page with no sentence in it at all — a price grid, a table — is read from the top instead, because that is what it says. @@ -497,11 +497,12 @@ A skill fires on some requests. This runs on all of them. Between the model settling on an answer and that answer reaching the screen, `src/agent/review.ts` reads it back against what the turn actually produced — the results the tools returned, and the URLs already in the conversation. Three things are checked: -| Check | Fires when | -| ----------------- | --------------------------------------------------------------------------- | -| `wrong-number` | The calculator returned a value the answer states nowhere, at any precision | -| `invented-source` | The answer cites a URL that no tool returned and nobody supplied | -| `missing-source` | Tools returned sources and the answer cites none | +| Check | Fires when | +| -------------------- | --------------------------------------------------------------------------- | +| `wrong-number` | The calculator returned a value the answer states nowhere, at any precision | +| `unsupported-figure` | The answer states a figure no source gave and the user never mentioned | +| `invented-source` | The answer cites a URL that no tool returned and nobody supplied | +| `missing-source` | Tools returned sources and the answer cites none | A failed check costs one further generation. The model is handed its own draft and told what to change — _The calculator returned 6748 \* 9 = 60732. Give that number, exactly as it came back._ — and the correction replaces the draft only if it leaves fewer problems behind. Otherwise the draft stands. That gate is the important half: the correction comes from the same 0.8B model, so a mechanism that could not tell an improvement from a regression would be a coin toss on every reply. @@ -509,6 +510,25 @@ A failed check costs one further generation. The model is handed its own draft a They are also deliberately shy. A clarifying question is asked for no citation; a long decimal quoted to fewer places counts as the calculator's number; citing the site when a page on it was read is close enough; a URL from an earlier reply is not an invention. Every check would rather miss a mistake than invent one, because a check that fires on a correct answer costs a generation and teaches you to ignore the whole mechanism. +**`unsupported-figure` exists because of one failure and is narrow because of the rule above.** Asked about a historical figure, the model produced ages of 48, 142 and 200 years, none of which any source gave. The only finding was that the answer cited nothing — so the correction dutifully appended a real encyclopedia URL to an invented number, which is worse than leaving it uncited. Figures are now compared against every tool result rather than against the calculator alone. + +What counts as a figure is deliberately almost nothing: a plain integer of three digits or more, and that is all. This check compares against a corpus rather than against one result, so every way a correct number can be written differently is a way for it to fire on a correct answer. Anything with a decimal point or a thousands separator is skipped, because `46,700` and `46.700` are one figure and `81.6` is a fair rounding of `81.62`. One and two digit numbers are skipped, which is the honest cost: a source gives 1889 and 1945 and never gives 56, so a correctly derived age appears in no evidence at all. `142` is not two digits, and that is the case this was built for. + +### When the answer came from nowhere + +Every check above needs evidence to fire. Turn that around and the gap is obvious: an answer produced with **no tool call at all** has nothing to check, so all three stay silent and the reply reaches the screen looking exactly like a researched one. That was the actual shape of the failure — a question about a life, answered out of a 0.8B model's recollection, indistinguishable from an answer that had been looked up. + +So a reply that was supposed to come from a source and did not is labelled `no source` · _answered from the model's own memory, not from a search_. Four things make that honest rather than noisy: + +- **It is a label, not a finding.** The correction round runs with the tools withheld, so asking the model to go and cite something would spend a generation to arrive back where it started. +- **"Was supposed to" is read off the tool list**, which already carries the decision. A [skill](#skills) narrows the list to what its kind of request needs, so a turn offered nothing but `web_search` and `read_page` is one some router decided was a lookup. The default list has six tools and never qualifies, which is what keeps the label off _write me a rhyme_. +- **Searching and then not citing is a different fault.** If a tool did fetch something, `missing-source` owns it and has a fix. Calling that turn "answered from memory" would simply be untrue. +- **An answer that claims nothing is not unsourced.** A clarifying question and a plain "I could not find it" are exempt, the same way they are exempt from `missing-source`. + +The other half of that failure was routing, and it was the larger half. Only _wer war X_ reached `research-question`; _wie alt wurde X_, _wann starb X_, _erzähl mir über X_ and _was hat X gemacht_ matched no skill at all, so nothing suggested searching and no exemplar showed it. Those shapes route now, in both languages. _Erzähl mir einen Witz_ deliberately does not, which is why the German pattern requires the preposition that turns the rest of the sentence into a subject. + +**None of this makes a 0.8B model truthful, and it is worth being plain about that.** A claim with no figure and no URL in it — the model asserting something about a person that is simply false — is caught by nothing here. Checking it would mean asking whether a sentence follows from a source, which needs a second model, and intrinsic self-grading is measured to make reasoning worse. What the app can do is refuse to present a guess as research, which is what the label is for. + The interface says what happened rather than quietly rewriting the reply. While the corrected answer streams in it is labelled with what is being fixed, and afterwards it carries `corrected` — claimed only for an answer that now passes every check — or `flagged`, naming what is still wrong with the text on screen. An answer half fixed and advertised as corrected would be worse than no check at all. ## When a turn runs out of tool rounds diff --git a/src/agent/budget.test.ts b/src/agent/budget.test.ts index 6f791cf..a42599c 100644 --- a/src/agent/budget.test.ts +++ b/src/agent/budget.test.ts @@ -5,7 +5,7 @@ import { budgetFallback, callFingerprint, repeatedCallNote, windDownNote } from import type { ReviewEvidence } from './review' function evidence(results: { tool: string; result: string }[] = []): ReviewEvidence { - return { toolResults: results, knownUrls: [] } + return { toolResults: results, knownUrls: [], knownFigures: [] } } const searchResult = { diff --git a/src/agent/loop.test.ts b/src/agent/loop.test.ts index 38e6f3b..d71b934 100644 --- a/src/agent/loop.test.ts +++ b/src/agent/loop.test.ts @@ -34,8 +34,12 @@ function toolCall(name: string, parameter: string, value: string): string { return `thinking${value}` } +/** + * The year is in here because the answers below state it. A figure the search + * did not return is its own finding now, and these cases are about citations. + */ const searchResult = - '1. Leadership — Fictional Airways\n https://fictionalairways.example/leadership\n Ama Osei leads it.' + '1. Leadership — Fictional Airways\n https://fictionalairways.example/leadership\n Ama Osei has led it since 2023.' describe('runAgent', () => { it('keeps reasoning separate when the model states an answer', async () => { @@ -380,6 +384,39 @@ describe('checking the answer before returning it', () => { expect(hooks.onCorrection).toHaveBeenCalledWith(['missing-source']) }) + it('labels a lookup that was answered without looking anything up', async () => { + // The tool list is the signal: a skill narrowed this turn to research tools, + // so a reply that fetched nothing is the model's recollection. Nothing else + // in the review can see this case, because every other check needs evidence. + const client = fakeClient(['I know thisHitler lived from 1889 to 1945.']) + + const result = await runAgent(client, turns, [search], callbacks()) + + expect(client.generate).toHaveBeenCalledTimes(1) + expect(result.review).toEqual({ found: [], corrected: false, unsourced: true }) + }) + + it('does not label a turn that was never a lookup', async () => { + // Six tools means nobody decided this was a lookup, and "write me a rhyme" + // must not come back marked as unsourced research. + const client = fakeClient(['doneRain on the roof, and a tap on the proof.']) + + const result = await runAgent(client, turns, [search, calculator], callbacks()) + + expect(result.review).toEqual({ found: [], corrected: false }) + }) + + it('does not label a lookup that did search', async () => { + const client = fakeClient([ + toolCall('web_search', 'query', 'Fictional Airways chief executive'), + 'read itAma Osei, since 2023.\n\nSource: https://fictionalairways.example/leadership', + ]) + + const result = await runAgent(client, turns, [search], callbacks()) + + expect(result.review).toEqual({ found: [], corrected: false }) + }) + it('can be switched off so the eval can measure what it is worth', async () => { const client = fakeClient([ toolCall('web_search', 'query', 'Fictional Airways chief executive'), diff --git a/src/agent/loop.ts b/src/agent/loop.ts index bea2da0..5b7e445 100644 --- a/src/agent/loop.ts +++ b/src/agent/loop.ts @@ -20,8 +20,10 @@ import { parseModelOutput, parsePartial, type ParsedToolCall } from './parse' import { collectEvidence, correctionPrompt, + isUnsourced, reviewAnswer, type ReviewCheck, + type ReviewEvidence, type ReviewOutcome, } from './review' @@ -78,6 +80,33 @@ interface Draft { found: ReviewCheck[] } +/** The tools whose whole purpose is to fetch something the model does not know. */ +const RESEARCH_TOOLS = new Set(['web_search', 'read_page']) + +/** + * Whether this turn was supposed to answer out of a source. + * + * Read off the tool list rather than plumbed in, because the tool list already + * carries the decision: a skill narrows it to what that kind of request needs, + * so a turn offered nothing but research tools is a turn some router decided was + * a lookup. The default list has six tools in it and never qualifies, which is + * what keeps the label off "write me a rhyme". + */ +function expectsSource(tools: Tool[]): boolean { + return tools.length > 0 && tools.every((tool) => RESEARCH_TOOLS.has(tool.schema.function.name)) +} + +/** + * Marks a finished answer that was meant to come from a source and did not. + * + * Applied last, to the text that is actually going out — including the wind-down + * fallback, which is nothing but sources and so is never unsourced. + */ +function labelSourcing(answer: AgentResult, evidence: ReviewEvidence): AgentResult { + if (!isUnsourced(answer.content, evidence)) return answer + return { ...answer, review: { ...(answer.review ?? { found: [], corrected: false }), unsourced: true } } +} + /** * Decides which of the two answers the user gets, and what to say about it. * @@ -130,6 +159,7 @@ export async function runAgent( const strategy = options.strategy ?? DEFAULT_STRATEGY const checking = options.review ?? true const evidence = collectEvidence(turns) + const sourcing = checking && expectsSource(tools) let last: AgentResult = { content: '', @@ -185,14 +215,16 @@ export async function runAgent( const found = findings.map((finding) => finding.check) if (findings.length === 0 || corrections >= MAX_CORRECTIONS) { - const answer = settle(last, found, draft) - if (!windDown) return answer + const settled = settle(last, found, draft) // The round that had to answer produced nothing, and `settle` found no // earlier draft to fall back on. What the tools did return is worth // more than the apology this used to end on. Substituted after the // check rather than before it, because deterministic text assembled // from tool results has nothing for a correction round to fix. - return { ...answer, content: answer.content || budgetFallback(evidence), windDown: true } + const answer = windDown + ? { ...settled, content: settled.content || budgetFallback(evidence), windDown: true } + : settled + return sourcing ? labelSourcing(answer, evidence) : answer } corrections += 1 diff --git a/src/agent/review.test.ts b/src/agent/review.test.ts index b7c6654..8020af8 100644 --- a/src/agent/review.test.ts +++ b/src/agent/review.test.ts @@ -1,14 +1,14 @@ import { describe, expect, it } from 'vitest' -import { collectEvidence, correctionPrompt, reviewAnswer, type ReviewEvidence } from './review' +import { collectEvidence, correctionPrompt, isUnsourced, reviewAnswer, type ReviewEvidence } from './review' function evidence(overrides: Partial = {}): ReviewEvidence { - return { toolResults: [], knownUrls: [], ...overrides } + return { toolResults: [], knownUrls: [], knownFigures: [], ...overrides } } const searchResult = { tool: 'web_search', result: - '1. Leadership — Fictional Airways\n https://fictionalairways.example/leadership\n Ama Osei leads it.', + '1. Leadership — Fictional Airways\n https://fictionalairways.example/leadership\n Ama Osei has led it since 2023.', } function checks(answer: string, given: ReviewEvidence): string[] { @@ -73,6 +73,66 @@ describe('reviewAnswer', () => { }) }) + describe('figures', () => { + const searched = evidence({ toolResults: [searchResult] }) + + it('catches a figure no source gave', () => { + // The reported failure: an answer about a life that stated an age nothing + // had returned. Its only finding used to be that it cited no source, so + // the correction attached a real URL to an invented number. + expect( + checks('He lived to 142.\n\nSource: https://fictionalairways.example/leadership', searched), + ).toEqual(['unsupported-figure']) + }) + + it('names the figure to drop', () => { + const [finding] = reviewAnswer('He was 200 years old.', searched) + + expect(finding?.instruction).toContain('No source gives 200') + }) + + it('accepts a figure the search returned', () => { + expect(checks('Since 2023.\n\nSource: https://fictionalairways.example/leadership', searched)).toEqual( + [], + ) + }) + + it('accepts a figure the user supplied', () => { + const asked = evidence({ toolResults: [searchResult], knownFigures: ['1889'] }) + + expect( + checks('Yes, 1889 is right.\n\nSource: https://fictionalairways.example/leadership', asked), + ).toEqual([]) + }) + + it.each([ + ['an age that is two digits', 'He was 56 when he died.'], + ['a percentage', 'Revenue grew 85%.'], + ['a figure with a decimal point', 'Revenue was 81.62 billion.'], + ['a figure with a thousands separator', 'It reached 46,700 units.'], + ['a thousands group written with a space', 'It reached 46 700 units.'], + ])('is too shy to challenge %s', (_case, answer) => { + // Every one of these is a way a correct number can be written that the + // evidence does not contain verbatim. A check that fires on a correct + // answer costs a generation and teaches the reader to ignore the label. + expect(checks(`${answer}\n\nSource: https://fictionalairways.example/leadership`, searched)).toEqual([]) + }) + + it('ignores digits that are part of a URL', () => { + const linked = evidence({ + toolResults: [{ tool: 'web_search', result: 'https://example.com/2026/08/report' }], + }) + + expect(checks('See https://example.com/2026/08/report', linked)).toEqual([]) + }) + + it('says nothing about figures when no tool returned anything', () => { + // With no evidence every figure is unsupported, and reporting that would + // be telling the model off for answering. `isUnsourced` covers this case. + expect(checks('He lived to 142.', evidence())).toEqual([]) + }) + }) + describe('sources', () => { const searched = evidence({ toolResults: [searchResult] }) @@ -164,6 +224,48 @@ describe('reviewAnswer', () => { }) }) +describe('isUnsourced', () => { + const searched = evidence({ toolResults: [searchResult] }) + + it('is true for a factual answer no tool contributed to', () => { + // The whole reason this exists: every other check needs evidence to fire, so + // the answer that consulted nothing was the one nothing was said about. + expect(isUnsourced('Hitler lived from 1889 to 1945.', evidence())).toBe(true) + }) + + it('is false once a tool returned a source, cited or not', () => { + // Not citing it is `missing-source`, which has a fix. Calling a turn that + // searched "answered from memory" would simply be untrue. + expect(isUnsourced('Ama Osei runs it.', searched)).toBe(false) + }) + + it('is false when the answer cites a source from earlier in the conversation', () => { + // A follow-up runs no tools of its own, and the source is still on screen. + const followUp = evidence({ knownUrls: ['https://fictionalairways.example/leadership'] }) + + expect(isUnsourced('Since 2023.\n\nSource: https://fictionalairways.example/leadership', followUp)).toBe( + false, + ) + }) + + it('is false for a URL the model invented', () => { + // `invented-source` owns that one, and stacking both labels on one reply + // would say the same thing twice. + expect(isUnsourced('Ama Osei.\n\nSource: https://madeup.example/ceo', searched)).toBe(false) + }) + + it.each(['Which city do you mean?', 'I could not find out who runs it.'])( + 'is false for %j, which claims nothing', + (answer) => { + expect(isUnsourced(answer, evidence())).toBe(false) + }, + ) + + it('is false for an empty draft, which the reasoning promotion handles', () => { + expect(isUnsourced(' ', evidence())).toBe(false) + }) +}) + describe('collectEvidence', () => { it('takes the URLs already in the conversation and no tool results yet', () => { const collected = collectEvidence([ @@ -178,8 +280,21 @@ describe('collectEvidence', () => { expect(collected).toEqual({ toolResults: [], knownUrls: ['https://example.com/pricing', 'https://example.com/old'], + knownFigures: [], }) }) + + it('takes figures from the user and nowhere else', () => { + // A skill's worked example is an assistant turn. Reading figures out of it + // would whitelist the example's own numbers on every turn that skill wins. + const collected = collectEvidence([ + { role: 'user', content: 'Is 1889 right?' }, + { role: 'assistant', content: 'Earlier I said 1723.' }, + { role: 'tool', content: 'A tool once returned 1456.' }, + ]) + + expect(collected.knownFigures).toEqual(['1889']) + }) }) describe('correctionPrompt', () => { diff --git a/src/agent/review.ts b/src/agent/review.ts index e39460b..15ff344 100644 --- a/src/agent/review.ts +++ b/src/agent/review.ts @@ -15,7 +15,7 @@ import type { ChatTurn } from '@/llm/protocol' * costs a second generation and teaches the user to ignore the whole mechanism, * so every one of them prefers to miss a mistake over inventing one. */ -export type ReviewCheck = 'wrong-number' | 'invented-source' | 'missing-source' +export type ReviewCheck = 'wrong-number' | 'unsupported-figure' | 'invented-source' | 'missing-source' export interface ReviewFinding { check: ReviewCheck @@ -28,6 +28,12 @@ export interface ReviewOutcome { found: ReviewCheck[] /** True when a corrected answer replaced the draft. */ corrected: boolean + /** + * The turn was routed to a skill that answers out of a source, and the answer + * cites none. Not a finding, because no correction can fix it — a label, so + * the reply does not pass for research it never did. + */ + unsourced?: boolean } export interface ReviewEvidence { @@ -39,6 +45,12 @@ export interface ReviewEvidence { * answer above must not read as an invention. */ knownUrls: string[] + /** + * Figures the user themselves put in the conversation. A number they supplied + * is not the model's invention, and asking it to justify their own figure + * against a search result would fire on a correct answer. + */ + knownFigures: string[] } const URL_IN_TEXT = /https?:\/\/[^\s<>"'`)\]}]+/g @@ -50,11 +62,15 @@ export function findUrls(text: string): string[] { /** Evidence as it stands before the first tool has run. */ export function collectEvidence(turns: ChatTurn[]): ReviewEvidence { + const spoken = turns.filter((turn) => turn.role === 'user' || turn.role === 'assistant') + return { toolResults: [], - knownUrls: turns - .filter((turn) => turn.role === 'user' || turn.role === 'assistant') - .flatMap((turn) => findUrls(turn.content)), + knownUrls: spoken.flatMap((turn) => findUrls(turn.content)), + // Figures come from the user's turns alone, where the URLs come from both. + // A skill's worked example is an assistant turn, and taking figures from it + // would whitelist the example's own numbers on every turn that skill wins. + knownFigures: turns.filter((turn) => turn.role === 'user').flatMap((turn) => figuresIn(turn.content)), } } @@ -128,6 +144,41 @@ function statesNumber(answer: string, value: number): boolean { return renderings(value).some((rendering) => digits.includes(rendering)) } +const NUMBER_IN_TEXT = /\d[\d.,]*/g + +/** `60 732` is one number written with a space, not a 60 next to a 732. */ +const SPACED_THOUSANDS = /(\d)[\s\u00a0](?=\d{3}\b)/g + +/** + * Only a plain integer of three digits or more counts as a figure here, and + * that narrowness is the whole point. + * + * This check compares against a corpus rather than against one tool result, so + * every way a number can legitimately be written differently is a way for it to + * fire on a correct answer — which the repository treats as a bug rather than as + * a strict setting. So everything ambiguous is skipped: + * + * - **Anything with a separator or a decimal point.** `46,700` and `46.700` are + * the same figure, `81.6` is a fair rounding of `81.62`, and a rule that has + * to decide which is which will sometimes decide wrongly. A thousands group + * written with a space is joined up first, or `60 732` would arrive as a `60` + * and a `732` and the `732` would be reported as an invention. + * - **One and two digit numbers.** An age is the clearest case: a source gives + * 1889 and 1945 and never gives 56, so a correct age appears in no evidence. + * The cost is that a two-digit invention is missed; `142` is not. + * - **Anything inside a URL**, which is a path and not a claim. + * + * What is left is years and the large round figures a model invents when it is + * filling a gap, which is what this exists for. + */ +export function figuresIn(text: string): string[] { + const prose = text.replace(URL_IN_TEXT, ' ').replace(SPACED_THOUSANDS, '$1,') + + return [...prose.matchAll(NUMBER_IN_TEXT)] + .map((match) => match[0].replace(TRAILING_PUNCTUATION, '')) + .filter((figure) => /^\d{3,}$/.test(figure)) +} + /** A question back to the user, and a plain "I could not find it", cite nothing. */ const NOTHING_TO_CITE = /\?\s*$|\b(could ?n[o']t find|no results|don'?t know|do not know|unable to find)\b/i @@ -145,6 +196,43 @@ function preferredSource(evidence: ReviewEvidence): string | null { * * An empty list means the answer goes out as written, which is the common case. */ +/** Every URL the turn is entitled to cite, located for comparison. */ +function groundedUrls(evidence: ReviewEvidence): Located[] { + return [...evidence.knownUrls, ...evidence.toolResults.flatMap(({ result }) => findUrls(result))] + .map(locate) + .filter((entry): entry is Located => entry !== null) +} + +/** + * Whether an answer that was supposed to come from a source came from one. + * + * This is not a finding, and deliberately so: it cannot be corrected. The + * correction round runs with no tools, so asking the model to go and cite + * something would spend a generation to arrive back here. It is a label, and + * what it labels is the case every other check is blind to — a factual question + * answered out of the model's memory, which reaches the screen looking exactly + * like a researched one. + */ +export function isUnsourced(answer: string, evidence: ReviewEvidence): boolean { + const draft = answer.trim() + if (!draft) return false + + // A clarifying question and a plain "I could not find it" are answers that + // claim nothing, so there is nothing to have sourced. + if (NOTHING_TO_CITE.test(draft)) return false + + // Something was fetched. Whether the answer went on to cite it is + // `missing-source` — a different fault, with a fix, and saying "answered from + // memory" about a turn that ran a search would simply be untrue. + if (preferredSource(evidence) !== null) return false + + const known = groundedUrls(evidence) + return !findUrls(draft).some((url) => { + const cited = locate(url) + return cited !== null && isGrounded(cited, known) + }) +} + export function reviewAnswer(answer: string, evidence: ReviewEvidence): ReviewFinding[] { const draft = answer.trim() if (!draft) return [] @@ -161,10 +249,29 @@ export function reviewAnswer(answer: string, evidence: ReviewEvidence): ReviewFi }) } + // Only where there is something to check against. With no tool result every + // figure in the answer is unsupported, and saying so would amount to telling + // the model off for answering — which is what `isUnsourced` reports instead. + if (evidence.toolResults.length > 0) { + const supported = new Set([ + ...evidence.toolResults.flatMap(({ result }) => figuresIn(result)), + ...evidence.knownFigures, + ]) + const unsupported = figuresIn(draft).find((figure) => !supported.has(figure)) + + // Without this, the only finding on "he lived to 142" was that it cited + // nothing — so the correction appended a real source to a made-up number, + // which is worse than leaving it uncited. + if (unsupported) { + findings.push({ + check: 'unsupported-figure', + instruction: `No source gives ${unsupported}. Drop that figure, or replace it with one the results above state.`, + }) + } + } + const source = preferredSource(evidence) - const known = [...evidence.knownUrls, ...evidence.toolResults.flatMap(({ result }) => findUrls(result))] - .map(locate) - .filter((entry): entry is Located => entry !== null) + const known = groundedUrls(evidence) const invented = findUrls(draft).find((url) => { const cited = locate(url) diff --git a/src/components/MessageItem.test.tsx b/src/components/MessageItem.test.tsx index 528f94e..3fc5f87 100644 --- a/src/components/MessageItem.test.tsx +++ b/src/components/MessageItem.test.tsx @@ -144,6 +144,28 @@ describe('MessageItem', () => { expect(await navigator.clipboard.readText()).toBe('const a = 1') }) + it('says when a lookup was answered without looking anything up', () => { + // The failure this exists for looked exactly like a researched answer, so + // the difference has to be on the screen rather than inferable. + render( + , + ) + + expect(screen.getByText('no source')).toBeInTheDocument() + expect(screen.getByText(/from the model’s own memory/)).toBeInTheDocument() + }) + + it('says nothing about sourcing on an ordinary reply', () => { + render() + + expect(screen.queryByText('no source')).not.toBeInTheDocument() + }) + it('marks a failed turn as a failure rather than passing it off as an answer', () => { render() diff --git a/src/components/MessageItem.tsx b/src/components/MessageItem.tsx index ec751d2..ff272b3 100644 --- a/src/components/MessageItem.tsx +++ b/src/components/MessageItem.tsx @@ -34,6 +34,7 @@ function RetryButton({ children }: { children: string }) { const REVIEW_REASON: Record = { 'wrong-number': 'a number the calculator disagreed with', + 'unsupported-figure': 'a figure no source gave', 'invented-source': 'a source no tool returned', 'missing-source': 'a missing source', } @@ -144,6 +145,17 @@ export function MessageItem({ message, isLatest = false }: { message: Message; i )} + {/* This question was routed as a lookup and answered without one. + It is the model's recollection, and the one case the checks above + cannot see, so it has to be said rather than inferred. */} + {review?.unsourced && ( + <> + + no source + + answered from the model’s own memory, not from a search + + )} {/* The tools were taken away before this reply was written, so it is as good as what they had returned and no better. */} {message.windDown && ( diff --git a/src/eval/scenarios.ts b/src/eval/scenarios.ts index 67d90ca..4294527 100644 --- a/src/eval/scenarios.ts +++ b/src/eval/scenarios.ts @@ -347,6 +347,19 @@ export const SCENARIOS: Scenario[] = [ accept: matches(/example|illustrative|domain/i), online: true, }, + { + id: 'web-life-dates', + category: 'web', + // A life used to reach no skill at all, so it was answered out of the model's + // memory — which is where invented ages and dates come from. Routing it is + // half the fix; this measures whether the turn actually searches. + prompt: 'Wie alt wurde Ada Lovelace?', + expectTool: 'web_search', + // 1815 to 1852, so the answer is 36. Accepting either the age or the years + // keeps this about having looked rather than about the arithmetic. + accept: matches(/\b36\b|1815|1852/), + online: true, + }, { id: 'memory-update', category: 'memory', diff --git a/src/skills/library.test.ts b/src/skills/library.test.ts index d74fb86..1581e85 100644 --- a/src/skills/library.test.ts +++ b/src/skills/library.test.ts @@ -258,6 +258,28 @@ describe('research-question', () => { ['Schau nach, was daraus geworden ist', 'search'], ['Finde heraus, wer das geschrieben hat', 'search'], ['aktuelle Nachrichten bitte', 'search'], + // A life is looked up, not recalled. Only `wer war X` used to route, so + // every other way of asking about a person reached no skill at all and was + // answered out of the model's memory — which is where an invented age or + // date comes from, and it is the failure this family exists to stop. + ['How old is Elon Musk?', 'trigger'], + ['How old was Ada Lovelace?', 'trigger'], + ['Wie alt ist Elon Musk?', 'trigger'], + ['Wie alt wurde Ada Lovelace?', 'trigger'], + ['Wie alt war Kennedy?', 'trigger'], + ['When did Ada Lovelace die?', 'trigger'], + ['When was Ada Lovelace born?', 'trigger'], + ['Wann starb Ada Lovelace?', 'trigger'], + ['Wann wurde Ada Lovelace geboren?', 'trigger'], + ['Wann ist Ada Lovelace gestorben?', 'trigger'], + ['Tell me about Ada Lovelace', 'trigger'], + ['Erzähl mir über Ada Lovelace', 'trigger'], + ['Erzähle mir etwas über Ada Lovelace', 'trigger'], + ['Erzähl mir von Ada Lovelace', 'trigger'], + ['Was hat Ada Lovelace gemacht?', 'trigger'], + ['Was haben die Römer erreicht?', 'trigger'], + ['biography of Ada Lovelace', 'search'], + ['Biografie von Ada Lovelace', 'search'], ])('takes %j by %s', (message, how) => { expect(routed(message)).toBe('research-question') expect(reason(message)).toBe(how) @@ -328,7 +350,13 @@ describe('priority and near misses', () => { 'What is my favourite colour?', 'What temperature does water boil at?', "I can't remember the capital of Peru.", + // A joke is not a lookup, which is why the German shape needs the + // preposition that turns the rest of the sentence into a subject. 'Erzähl mir einen Witz', + 'Erzähl mir eine Geschichte', + // Second person: this is about the conversation, not about a subject. + 'Was haben wir gemacht?', + 'Was hast du gemacht?', 'I was born in 2024', 'I currently live in Berlin', 'Was machst du heute?', diff --git a/src/skills/research-question/SKILL.md b/src/skills/research-question/SKILL.md index 28e4a87..9afe073 100644 --- a/src/skills/research-question/SKILL.md +++ b/src/skills/research-question/SKILL.md @@ -16,6 +16,8 @@ jarvis: - finde heraus - wer hat gewonnen - aktuelle nachrichten + - biography of + - biografie von triggers: - '\b(latest|current|recent|news|today.s)\b' # `who is` without an exclusion also takes *Who is that?*, which is a @@ -40,6 +42,21 @@ jarvis: # something follows, the way *Was ist los in Frankreich?* does. - '\bwas (ist|passiert) (gerade|heute|aktuell)\b' - '\bwas ist los\b(?!\s*\??\s*$)' + # A life is looked up, never recalled. Only *wer war X* used to route, so + # *wie alt wurde X*, *wann starb X* and *erzähl mir über X* reached no skill + # at all and were answered out of the model's memory — which is where an + # invented age comes from. Routed here, the turn searches first. + - '\bhow old (is|was|were)\b' + - '\bwie alt (ist|war|wurde|sind|waren)\b' + - '\bwhen (did|was)\b[^?]{0,40}\b(die|died|born)\b' + - '\bwann (starb|gestorben)\b' + - '\bwann (wurde|ist)\b[^?]{0,40}\b(geboren|gestorben)\b' + - '\btell me about\b' + # *Erzähl mir einen Witz* is not a lookup, so this needs the preposition that + # makes the rest of the sentence a subject. + - '\berz(ä|ae)hl(e|s)?\s+(mir|uns)\s+(etwas\s+)?(über|ueber|von)\b' + # Second person excluded: *was haben wir gemacht* is about the conversation. + - '\bwas (hat|haben)\s+(?!wir\b|du\b|ihr\b)[^?]{0,40}\b(gemacht|getan|erreicht)\b' exemplars: - user: Who is the chief executive of Fictional Airways? steps: diff --git a/src/tools/search-brief.test.ts b/src/tools/search-brief.test.ts index 25691eb..8802b5f 100644 --- a/src/tools/search-brief.test.ts +++ b/src/tools/search-brief.test.ts @@ -45,31 +45,31 @@ describe('selectDiverseSources', () => { expect(selected.map((result) => result.url)).toEqual(['https://reuters.com/one', 'https://bbc.com/story']) }) - it('drops Wikipedia when independent sites can be read instead', () => { - // Its extract is a paragraph where a results page gives a line, so left in - // rank order it decides the answer by itself and the rest are decoration. + it('gives reference works one seat and no more', () => { + // A lead paragraph beside three one-line snippets decides the answer by + // itself, and a mirror of Wikipedia is not a second opinion. One seat keeps + // the best source on a historical question without letting it be the brief. const selected = selectDiverseSources( [ hit('https://en.wikipedia.org/wiki/Guterres'), + hit('https://www.britannica.com/biography/Guterres'), + hit('https://de.wikipedia.org/wiki/Guterres'), hit('https://un.org/sg'), hit('https://reuters.com/un'), hit('https://bbc.com/un'), - hit('https://ft.com/un'), ], 4, ) expect(selected.map((result) => siteOf(result.url))).toEqual([ + 'wikipedia.org', 'un.org', 'reuters.com', 'bbc.com', - 'ft.com', ]) }) - it('keeps Wikipedia rather than leaving one site to answer alone', () => { - // Two readings and one of them an encyclopedia beats a brief with nothing - // to cross-check against. Padding is the thing to avoid, not the mention. + it('keeps a reference work rather than leaving one site to answer alone', () => { const selected = selectDiverseSources( [hit('https://un.org/sg'), hit('https://en.wikipedia.org/wiki/Guterres')], 4, @@ -78,7 +78,7 @@ describe('selectDiverseSources', () => { expect(selected.map((result) => siteOf(result.url))).toEqual(['un.org', 'wikipedia.org']) }) - it('falls back to Wikipedia rather than returning nothing', () => { + it('returns the encyclopedia rather than nothing when it is all there is', () => { const selected = selectDiverseSources([hit('https://de.wikipedia.org/wiki/Arc')], 4) expect(selected).toHaveLength(1) diff --git a/src/tools/search-brief.ts b/src/tools/search-brief.ts index b1396c3..d5a837e 100644 --- a/src/tools/search-brief.ts +++ b/src/tools/search-brief.ts @@ -14,10 +14,10 @@ * * Two rules keep it from being an encyclopedia lookup wearing four coats. * Sources must come from *different* sites, because two pages of one publisher - * are one source; and an encyclopedia is only used to get off a single reading, - * never to fill a brief that has independent ones, since its extract is a - * paragraph where a results page gives a line and it would otherwise be the - * richest source in every brief. + * are one source; and a reference work gets one seat of the four, since its + * extract is a paragraph where a results page gives a line and it would + * otherwise be the richest voice in every brief — while dropping it outright + * would take the best source there is out of every question about history. * * Nothing here needs a server. The pages go through the same reader `read_page` * uses, which is the only fetch in this project verified to survive CORS from @@ -91,13 +91,21 @@ const TWO_LABEL_SUFFIXES = new Set([ ]) /** - * Wikipedia and its siblings, held back rather than dropped. + * Reference works, allowed exactly one seat. * - * They are the best single page on many subjects and the worst way to answer a - * question about several: a lead paragraph beside three one-line snippets - * decides the answer by itself, and mirrors of it are not a second opinion. + * Both extremes of this were wrong. Left to rank freely they take several seats + * and decide the answer by themselves — a lead paragraph beside three one-line + * snippets is not a comparison, and a mirror of Wikipedia is not a second + * opinion. Dropped whenever two other sites answered, the brief for *who was X* + * fills up with whatever happens to rank, and for history and biography an + * encyclopedia is the most reliable thing the search returned, not the least. + * + * One seat is the rule that survives both: never the majority, never absent + * when the search found one. */ -const ENCYCLOPEDIAS = new Set(['wikipedia.org', 'wikimedia.org', 'wikidata.org']) +const ENCYCLOPEDIAS = new Set(['wikipedia.org', 'wikimedia.org', 'wikidata.org', 'britannica.com']) + +const MAX_ENCYCLOPEDIAS = 1 /** The site a URL belongs to, as a reader would name it. */ export function siteOf(url: string): string { @@ -116,30 +124,34 @@ export function siteOf(url: string): string { } /** - * Picks the results worth reading: one per site, encyclopedias last. + * Picks the results worth reading: one per site, at most one reference work. * - * Rank order is kept within each group, so the best result of a site is the one - * that survives and the search engine's own judgement is not second-guessed - * beyond these two rules. + * Rank order is otherwise kept, so the best result of a site is the one that + * survives and the search engine's own judgement is not second-guessed beyond + * these two rules. */ export function selectDiverseSources(results: SearchResult[], limit = MAX_SOURCES): SearchResult[] { const seen = new Set() - const independent: SearchResult[] = [] - const encyclopedic: SearchResult[] = [] + const selected: SearchResult[] = [] + let encyclopedias = 0 for (const result of results) { + if (selected.length >= limit) break if (!result.url) continue + const site = siteOf(result.url) if (seen.has(site)) continue + + if (ENCYCLOPEDIAS.has(site)) { + if (encyclopedias >= MAX_ENCYCLOPEDIAS) continue + encyclopedias += 1 + } + seen.add(site) - ;(ENCYCLOPEDIAS.has(site) ? encyclopedic : independent).push(result) + selected.push(result) } - // An encyclopedia is a way to have two readings rather than one, never a way - // to fill a brief that already has independent ones. Padding four slots with - // it is how a search of the whole web ends up answering out of Wikipedia. - const sources = independent.length > 1 ? independent : [...independent, ...encyclopedic] - return sources.slice(0, Math.max(limit, 0)) + return selected } /** A markdown link, image or emphasis carries no meaning once the page is a paragraph. */ From c2e2e31ddb65a877e42263f0a9e63e46a3d680a9 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 26 Aug 2026 13:44:35 +0000 Subject: [PATCH 2/3] Answer a greeting with a sentence, not with a web search MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Typed on its own, `hallo` was answered with a search for "HALLO - German greetings for AI assistants" and a reply citing a Facebook video and a Reddit thread about how to say hello. Two faults stacked up. A greeting is one word, so `isFollowUp` read it as a fragment continuing the turn before — and that turn had been a research question, so `research-question` carried over, the tool list was narrowed to search and page reads, and every resident exemplar ends in a citation. Searching was the only thing left to do and citing was the shape it had been shown. An opener now drops whatever was resident, exactly as `thanks` does. `small-talk` then claims those messages with `tools: []` — no tools at all, so the chat template renders no tool block and there is no call format to imitate. That distinction between declaring nothing and declaring none is what makes "answer this without a search" expressible. Its triggers are anchored at both ends, which turned up a second gap: every shape `lookup-term` matches is anchored to the start of the message, so `Hallo, was ist Stripe?` reached no skill at all. The message as written is tried first, and only then the same message with the greeting taken off, so a bare `hallo` still reaches small talk and a greeting in front of a question no longer swallows it. Co-authored-by: Sebastian --- README.md | 14 ++++++- src/eval/scenarios.ts | 21 ++++++++++ src/skills/activate.ts | 15 +++++-- src/skills/library.test.ts | 73 +++++++++++++++++++++++++++++++++- src/skills/load.test.ts | 7 ++-- src/skills/load.ts | 15 ++++++- src/skills/route.ts | 67 ++++++++++++++++++++++++++----- src/skills/small-talk/SKILL.md | 28 +++++++++++++ src/skills/types.ts | 17 ++++++-- 9 files changed, 229 insertions(+), 28 deletions(-) create mode 100644 src/skills/small-talk/SKILL.md diff --git a/README.md b/README.md index eed4480..0e9ed9d 100644 --- a/README.md +++ b/README.md @@ -402,7 +402,17 @@ An exemplar can hold several steps, which is how a workflow gets taught. Split a Two further things a skill does. It **narrows the tool list** to what it declares, because tool-calling accuracy falls as the number of visible tools grows. And it can **override the reasoning budget** per skill. -Seven ship: `arithmetic`, `current-date`, `summarize-url`, `lookup-term`, `research-question`, `weather` and `memory`. +Declaring nothing and declaring **none** are different, and the difference is load-bearing. A skill with no `tools` key does not restrict the list. A skill that writes `tools: []` wants no tools at all and gets none — `runAgent` then hands the chat template no tool block, so there is no call format in front of the model to imitate. It is the same mechanism the [wind-down round](#when-a-turn-runs-out-of-tool-rounds) uses, and on a 0.8B model it is worth considerably more than a sentence asking for restraint. + +Eight ship: `small-talk`, `arithmetic`, `current-date`, `summarize-url`, `lookup-term`, `research-question`, `weather` and `memory`. + +### Why `small-talk` exists + +Typed on its own, `hallo` was answered with a web search for **`HALLO - German greetings for AI assistants`** and a reply citing two pages about how to say hello in German — one of them a Facebook video, one a Reddit thread. + +Two faults stacked up to produce that. A greeting is one word, so `isFollowUp` read it as a fragment continuing the previous turn, and the previous turn had been a research question — so `research-question` carried over, the tool list was narrowed to `web_search` and `read_page`, and the resident exemplars all end in `Source: https://…`. Searching was the only thing the model had been left able to do, and citing was the shape it had been shown. + +So an **opener is now as much a non-continuation as a closer**: `hallo`, `moin`, `guten Morgen`, `hi` and their kin drop whatever was resident, exactly as `thanks` does. And `small-talk` claims those messages itself, with `tools: []`, so the reply is a sentence rather than a search. Its triggers are anchored at both ends, which is what keeps _Hallo, was ist Stripe?_ with the research skill — a greeting in front of a question is a question. ### Which skill, and when @@ -485,7 +495,7 @@ A skill that keeps applying to turns it has nothing to do with is worse than no - A continuation has to either **say so** (`and`, `und`, `what about`) or be **too short to be asking anything of its own**. Length alone is not enough, and this is where the mechanism would turn harmful: _what is the capital of France?_ is six words, and answering it with the weather skill's exemplars resident would send the model searching for a fact it already knows. - It survives **two turns** on carry-over alone. Past that it has stopped being a continuation and become a default. -- It is dropped the moment another skill matches, the message asks something fresh, the turn closes the exchange (_thanks_), or a new chat starts. +- It is dropped the moment another skill matches, the message asks something fresh, the turn closes the exchange (_thanks_) or opens a new one (_hallo_), or a new chat starts. The resident skill is read back off the transcript rather than kept in a counter of its own, so rerunning a reply rewinds it too — a counter held to one side would still be carrying the turn it just discarded. diff --git a/src/eval/scenarios.ts b/src/eval/scenarios.ts index 4294527..1ed77f2 100644 --- a/src/eval/scenarios.ts +++ b/src/eval/scenarios.ts @@ -257,6 +257,27 @@ export const SCENARIOS: Scenario[] = [ // Only that it wrote something rather than reaching for a tool. accept: (answer) => answer.trim().length > 10, }, + { + id: 'no-tool-greeting', + category: 'no-tool', + // Observed: `hallo` inherited the previous turn's research skill, searched + // the web for "HALLO - German greetings for AI assistants", and cited two + // pages about how to say hello. A greeting is not a question about + // greetings, and the reply should be a sentence with no sources on it. + prompt: 'hallo', + expectTool: null, + accept: (answer) => answer.trim().length > 2 && !answer.includes('http'), + }, + { + id: 'no-tool-greeting-then-question', + category: 'lookup', + // The other half: a greeting in front of a real question must not switch + // the research off, or this stops being a fix and becomes a new bug. + prompt: 'Hallo, was ist Stripe?', + expectTool: 'web_search', + accept: matches(/payment|zahlung|checkout|billing|fintech|bezahl/i), + online: true, + }, { id: 'web-current-event', category: 'web', diff --git a/src/skills/activate.ts b/src/skills/activate.ts index cead224..89ec418 100644 --- a/src/skills/activate.ts +++ b/src/skills/activate.ts @@ -42,7 +42,12 @@ export interface ActivationResult { export function usableSkills(catalog: SkillEntry[], tools: Tool[]): SkillEntry[] { const available = new Set(tools.map((tool) => tool.schema.function.name)) return catalog.filter( - (entry) => entry.tools.length === 0 || entry.tools.some((name) => available.has(name)), + // Declaring nothing and declaring none are both always available: the first + // uses whatever is there, the second wants none of it. + (entry) => + entry.tools === undefined || + entry.tools.length === 0 || + entry.tools.some((name) => available.has(name)), ) } @@ -94,15 +99,17 @@ export function activate( const skill = routing.route.entry.load() const byName = new Map(tools.map((tool) => [tool.schema.function.name, tool])) - const selected = skill.tools + const selected = (skill.tools ?? []) .map((name) => byName.get(name)) .filter((tool): tool is Tool => tool !== undefined) return { activation: { skill, - // An empty declaration means the skill does not restrict the tool list. - tools: skill.tools.length === 0 ? tools : selected, + // No declaration at all means the skill does not restrict the list. An + // empty one means it wants no tools, and gets none: `runAgent` then hands + // the template no tool block, so there is no call format to imitate. + tools: skill.tools === undefined ? tools : selected, exemplars: withinBudget(skill), reason: routing.route.reason, matched: routing.route.matched, diff --git a/src/skills/library.test.ts b/src/skills/library.test.ts index 1581e85..d82a7a5 100644 --- a/src/skills/library.test.ts +++ b/src/skills/library.test.ts @@ -6,7 +6,7 @@ import { SYSTEM_PROMPT } from '@/llm/config' import { builtinTools } from '@/tools/builtins' import { activate, composeTurns } from './activate' import { loadCatalog } from './load' -import { route } from './route' +import { isFollowUp, route } from './route' const catalog = loadCatalog() @@ -25,8 +25,10 @@ function reason(message: string): string | null { * in `route.test.ts` is not finished. */ describe('the shipped library', () => { - it('is the seven skills the README names, highest priority first', () => { + it('is the eight skills the README names, highest priority first', () => { expect(catalog.map((entry) => [entry.name, entry.priority, entry.tools])).toEqual([ + // An empty list, not an absent one: small talk wants no tools at all. + ['small-talk', 40, []], ['memory', 35, ['memory']], ['arithmetic', 30, ['calculator']], ['weather', 28, ['weather']], @@ -286,6 +288,72 @@ describe('research-question', () => { }) }) +describe('small-talk', () => { + it.each([ + ['hallo', 'trigger'], + ['Hallo!', 'trigger'], + ['halloo', 'trigger'], + ['hi', 'trigger'], + ['Hi there', 'trigger'], + ['hey', 'trigger'], + ['moin', 'trigger'], + ['moin moin', 'trigger'], + ['servus', 'trigger'], + ['hello', 'trigger'], + ['Guten Morgen', 'trigger'], + ['Guten Abend!', 'trigger'], + ['Wie geht es dir?', 'trigger'], + ["wie geht's", 'trigger'], + ['How are you?', 'trigger'], + ['how are you doing', 'trigger'], + ])('takes %j by %s', (message, how) => { + expect(routed(message)).toBe('small-talk') + expect(reason(message)).toBe(how) + }) + + it('offers the model no tools at all', () => { + // Not a narrowed list: none. The chat template then renders no tool block, + // so there is no call format in front of a 0.8B model to imitate — which is + // what stops a greeting being answered with a web search and two citations. + const { activation } = activate('hallo', catalog, builtinTools) + + expect(activation?.skill.name).toBe('small-talk') + expect(activation?.tools).toEqual([]) + }) + + it.each([ + ['Hallo, was ist Stripe?', 'lookup-term'], + ['Hi, wie alt wurde Ada Lovelace?', 'research-question'], + ['Guten Morgen, wie ist das Wetter in Berlin?', 'weather'], + ['Hallo! Was ist 6748 * 9?', 'arithmetic'], + ['Hi, what is Notion?', 'lookup-term'], + ])('gives the real question in %j to %s', (message, expected) => { + // Every shape `lookup-term` matches is anchored to the start of the message, + // which a greeting had taken — so these reached no skill at all. The greeting + // is taken off and the rest offered up, but only after the message as + // written has been tried, or a bare `hallo` would stop reaching small talk. + expect(routed(message)).toBe(expected) + }) + + it('does not read a greeting that runs into its own sentence as a question', () => { + // No punctuation, so nothing is stripped: `hi there` is not about *there*. + expect(routed('hi there')).toBe('small-talk') + }) + + it.each(['hallo', 'hi', 'moin', 'Guten Morgen'])( + 'drops a resident research skill rather than letting %j inherit it', + (message) => { + // The reported failure: `hallo` is short enough to read as a fragment, so + // it carried over the previous turn's skill, was handed only the search + // tools, and dutifully searched for the word it had been greeted with. + const carried = { name: 'research-question', carried: 0 } + + expect(route(message, catalog, carried).route?.entry.name).toBe('small-talk') + expect(isFollowUp(message)).toBe(false) + }, + ) +}) + describe('memory', () => { it.each([ ['Remember that I prefer metric units.', 'trigger'], @@ -388,6 +456,7 @@ describe('priority and near misses', () => { describe('activating each shipped skill', () => { const cases: [string, string, string[]][] = [ + ['hallo', 'small-talk', []], ['Remember that I prefer metric units.', 'memory', ['memory']], ['What is 6748 * 9?', 'arithmetic', ['calculator']], ["What's the weather in Berlin?", 'weather', ['weather']], diff --git a/src/skills/load.test.ts b/src/skills/load.test.ts index 40527b4..d097ab4 100644 --- a/src/skills/load.test.ts +++ b/src/skills/load.test.ts @@ -19,8 +19,9 @@ describe('parseSkill', () => { expect(skill.name).toBe('example') expect(skill.guidance).toBe('Body.') - // Absent extension block means no restrictions rather than no skill. - expect(skill.tools).toEqual([]) + // Absent extension block means no restrictions rather than no skill, and + // that is why it stays undefined: an empty list means no tools at all. + expect(skill.tools).toBeUndefined() expect(skill.priority).toBe(0) }) @@ -238,7 +239,7 @@ describe('the shipped skills', () => { }) it.each(skills.map((skill) => [skill.name, skill] as const))('%s declares real tools', (_name, skill) => { - for (const tool of skill.tools) expect(builtinNames).toContain(tool) + for (const tool of skill.tools ?? []) expect(builtinNames).toContain(tool) }) it.each(skills.map((skill) => [skill.name, skill] as const))( diff --git a/src/skills/load.ts b/src/skills/load.ts index 6537900..33fdb7a 100644 --- a/src/skills/load.ts +++ b/src/skills/load.ts @@ -42,6 +42,17 @@ function stringArray(value: unknown, path: string, field: string): string[] { return value as string[] } +/** + * `tools`, keeping the difference between saying nothing and saying none. + * + * A skill that writes `tools: []` is asking for no tools, which is not the same + * as a skill that never mentioned them. Collapsing the two — which is what + * `stringArray` does — makes "answer this without a search" inexpressible. + */ +function toolList(value: unknown, path: string): string[] | undefined { + return value === undefined ? undefined : stringArray(value, path, 'tools') +} + function parseSteps(value: unknown, where: string): SkillStep[] { if (value === undefined) return [] if (!Array.isArray(value)) throw new Error(`${where}: "steps" must be a list`) @@ -129,7 +140,7 @@ interface SkillMetadata { keywords: string[] triggers: RegExp[] priority: number - tools: string[] + tools: string[] | undefined jarvis: Record body: string } @@ -167,7 +178,7 @@ function parseMetadata(source: string, path: string): SkillMetadata { keywords: checkKeywords(stringArray(jarvis.keywords, path, 'keywords'), path), triggers: compileTriggers(stringArray(jarvis.triggers, path, 'triggers'), path), priority: priority ?? 0, - tools: stringArray(jarvis.tools, path, 'tools'), + tools: toolList(jarvis.tools, path), jarvis, body, } diff --git a/src/skills/route.ts b/src/skills/route.ts index ff942d6..48e409d 100644 --- a/src/skills/route.ts +++ b/src/skills/route.ts @@ -67,6 +67,39 @@ const ASKS_AFRESH = /** Nothing to continue: the turn is closing the exchange, not extending it. */ const CLOSES = /^\s*(thanks|thank you|thx|cheers|ok|okay|cool|nice|great|danke|dankeschön|bye|ciao)\b/i +/** + * Nothing to continue for the opposite reason: the turn is starting over. + * + * An opener is as much a non-continuation as a closer, and this one had teeth. + * A greeting is short enough to read as a fragment, so `hallo` after a research + * question inherited that skill — narrowing the tool list to search and reading + * an exemplar that ends in a citation. The reply searched the web for the word + * it had been greeted with and cited two pages about how to say hello. + */ +const GREETS = + /^\s*(hallo+|hall(ö|oe)chen|hi+|hey+|hej|moin|servus|gr(ü|ue)(ß|ss) dich|gr(ü|ue)ezi|hello+|hiya|yo|guten (morgen|tag|abend)|na)\b/i + +/** + * The same openers, but only where punctuation shows the sentence carries on. + * + * *Hallo, was ist Stripe?* is a question with a greeting in front of it, and it + * reached no skill at all: every shape `lookup-term` matches is anchored to the + * start of the message, which the greeting had taken. Requiring the separator is + * what keeps `hi there` from being read as a question about *there*. + */ +const LEADING_GREETING = + /^\s*(hallo+|hall(ö|oe)chen|hi+|hey+|hej|moin( moin)?|servus|gr(ü|ue)(ß|ss) dich|gr(ü|ue)ezi|hello+|hiya|guten (morgen|tag|abend))\s*[,;:!.\u2013\u2014-]+\s*/i + +/** + * The message with a leading greeting taken off, where that leaves a question + * behind. A message that is *only* a greeting is returned untouched, so + * `small-talk` still gets it. + */ +export function withoutGreeting(message: string): string { + const stripped = message.replace(LEADING_GREETING, '') + return stripped.trim() ? stripped : message +} + const WORD = /[\p{L}\p{N}]+/gu /** @@ -79,7 +112,7 @@ const WORD = /[\p{L}\p{N}]+/gu * asking anything by itself. */ export function isFollowUp(message: string): boolean { - if (CLOSES.test(message)) return false + if (CLOSES.test(message) || GREETS.test(message)) return false if (CONTINUES.test(message)) return true if (ASKS_AFRESH.test(message)) return false @@ -93,19 +126,31 @@ export function matchTriggers(message: string, catalog: SkillEntry[]): SkillEntr } export function route(message: string, catalog: SkillEntry[], memory: SkillMemory | null = null): Routing { - const triggered = matchTriggers(message, catalog) - if (triggered) { - return { - route: { entry: triggered, reason: 'trigger', matched: [] }, - memory: { name: triggered.name, carried: 0 }, + // The message as written first, so a bare greeting still reaches `small-talk`, + // whose triggers are anchored at both ends. Only when nothing claims it is the + // greeting taken off and the rest offered up — *Hallo, was ist Stripe?* is a + // lookup, and every shape `lookup-term` matches starts at the message. + const candidates = [message, withoutGreeting(message)].filter( + (candidate, at, all) => all.indexOf(candidate) === at, + ) + + for (const candidate of candidates) { + const triggered = matchTriggers(candidate, catalog) + if (triggered) { + return { + route: { entry: triggered, reason: 'trigger', matched: [] }, + memory: { name: triggered.name, carried: 0 }, + } } } - const [best] = search(message, catalog) - if (best) { - return { - route: { entry: best.entry, reason: 'search', matched: best.matched }, - memory: { name: best.entry.name, carried: 0 }, + for (const candidate of candidates) { + const [best] = search(candidate, catalog) + if (best) { + return { + route: { entry: best.entry, reason: 'search', matched: best.matched }, + memory: { name: best.entry.name, carried: 0 }, + } } } diff --git a/src/skills/small-talk/SKILL.md b/src/skills/small-talk/SKILL.md new file mode 100644 index 0000000..7bd6be8 --- /dev/null +++ b/src/skills/small-talk/SKILL.md @@ -0,0 +1,28 @@ +--- +name: small-talk +description: Answers a greeting, a thank-you or a how-are-you in one sentence, without reaching for a tool. Use when the whole message is that and asks for nothing else. +jarvis: + priority: 40 + # Empty on purpose, which is not the same as absent: this skill wants no tools + # at all. The chat template then renders no tool block, so there is no call + # format in front of the model to imitate — the only thing that reliably stops + # a 0.8B model searching the web for the word it was greeted with. + tools: [] + triggers: + # The whole message, anchored at both ends. *Hallo, wie alt wurde Ada + # Lovelace?* is a question with a greeting attached and must reach the + # research skill instead. + - '^\s*(hallo+|hall(ö|oe)chen|hi+|hey+|hej|moin( moin)?|servus|gr(ü|ue)(ß|ss) dich|gr(ü|ue)ezi|hello+|hiya|yo)[\s!.,…]*$' + - '^\s*guten (morgen|tag|abend)[\s!.,…]*$' + - "^\\s*(wie geht('?s| es dir| es ihnen| es euch)|how are you( doing)?|how'?s it going)[\\s!?.,…]*$" + - '^\s*(na|nabend|servus zusammen|hallo zusammen|hi there|hey there)[\s!.,…]*$' + exemplars: + - user: hallo + answer: Hallo! Was kann ich für dich tun? + - user: how are you? + answer: Doing well, thanks. What can I help you with? +--- + +Answer in one short sentence and stop. + +A greeting is not a question about greetings. Do not search, do not cite anything, and do not list what you can do. diff --git a/src/skills/types.ts b/src/skills/types.ts index 6e68678..9bef756 100644 --- a/src/skills/types.ts +++ b/src/skills/types.ts @@ -45,8 +45,16 @@ export interface Skill { * honest and the exemplars do the real work. */ guidance: string - /** Tool names the model may see while this skill is active. */ - tools: string[] + /** + * Tool names the model may see while this skill is active. + * + * Three states, and the difference between the last two matters. Absent means + * the skill does not restrict the list. A list of names narrows it to those. + * An **empty** list means no tools at all — the chat template then renders no + * tool block, which is the only thing that reliably stops a 0.8B model + * reaching for a search it has no use for. + */ + tools?: string[] triggers: RegExp[] exemplars: SkillExemplar[] /** Higher wins when several skills match. */ @@ -74,9 +82,10 @@ export interface SkillEntry { * * A skill with none of its tools available teaches a call the model cannot * make, so it must not win the route — and answering that by loading every - * skill to look would give back what the catalogue is for. + * skill to look would give back what the catalogue is for. A skill that + * declares an empty list wants no tools and is always available. */ - tools: string[] + tools?: string[] /** Materialises the body and exemplars. Memoised, so calling it twice is free. */ load: () => Skill } From 58bc4cce469454f6a3d175beea4683669377ceea Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 26 Aug 2026 13:58:12 +0000 Subject: [PATCH 3/3] Read the spelling and the language off the sources Two things the search engine already does, which the brief was throwing away. It corrects spelling. `eln musk` returns "Elon Musk - Wikipedia" as its first hit and every source then spells the name properly, so there is nothing to look up and no dictionary to ship. `spellingFrom` compares the words of the query against the words the sources use and says where they differ. It reports rather than rewrites, because rewriting the query is the `1inch` failure: measured live, `was ist 1inch` produces no correction, since the term is in the sources verbatim. Three guards, each of them a false positive first. Two sources have to agree on a spelling. Question words are excluded by name, or `wer ist elon musk` reports that the sources spell it "der". And two edits need a word long enough to go wrong twice. And it localises by query language: `wer ist elon musk` returns de.wikipedia.org first, `who is elon musk` the English article. So German sources come back on their own as long as the query is not translated first, and the answer follows its sources. The German exemplar teaches exactly that. A live search also turned up a false conflict this fixes: a birth year, an election year and this year were reported as "1971 vs 2026 vs 2024", three sources contradicting each other. A biography dates several things and contradicts none of them, so past two distinct years the comparison is quiet. Co-authored-by: Sebastian --- README.md | 19 +++ src/eval/scenarios.ts | 22 +++ src/skills/research-question/SKILL.md | 26 +++- src/tools/search-brief.test.ts | 101 +++++++++++++ src/tools/search-brief.ts | 208 +++++++++++++++++++++++++- 5 files changed, 368 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 0e9ed9d..a4ed441 100644 --- a/README.md +++ b/README.md @@ -257,6 +257,25 @@ Four details are what keep that honest: - **A dead link costs one source, not the brief.** The pages are read with `allSettled`, and one that 404s, refuses the reader or spends the last of a rate limit falls back to its search snippet — labelled `snippet only`, so the model is not told a line was a page. - **Naming both figures is not disagreeing.** A page that mentions last year's number alongside this year's is the ordinary way to write one, so a disagreement is only reported when a site names a value and _not_ the one the others agree on. A single reading nothing contradicts is reported as nothing, since hedging an unchallenged answer is its own error. - **A figure is compared with figures of its own kind.** Years are their own kind, because that is where sources differ in a way worth reporting; otherwise the digit count stands in, so a revenue figure is never held against a percentage. `46,700` and `46.700` are one value; `46.7` is not. +- **More than two years between them is not a disagreement.** Measured on a live search for _wer ist elon musk_: a birth year, an election year and this year came back as `1971 vs 2026 vs 2024`, reported as three sources contradicting each other. A biography dates several things and contradicts none of them, so past two distinct years the comparison stays quiet. + +### Spelling, and the language of the answer + +Asked _wer ist eln musk_, the search returns "Elon Musk — Wikipedia" as its first hit and every source then spells the name properly. **The correcting has already happened**, upstream, for free — so there is no dictionary here and no spell-checker. `spellingFrom` compares the words of the query against the words the sources actually use, and where a query word appears nowhere but something two sources agree on sits one or two edits away, the brief says so: + +```text +The sources spell it "Elon" (the question wrote "eln"). Answer about that, and use their spelling. +``` + +It is **reported rather than acted on**, and that is the whole design. Rewriting the query would put back the failure [`lookup-term`](#why-lookup-term-exists) exists for, where the model decided `1inch` was a typo for `1 inch` and searched for a unit conversion. Measured against the live web, `was ist 1inch` produces no correction at all: the term is in the sources verbatim, so there is nothing to correct. + +Three guards keep it from inventing corrections, and each of them was a false positive first: + +- **Two sources have to agree on the spelling.** One page's own typo is not a correction. +- **Question words are excluded by name.** Without that, `wer ist elon musk` offers up `wer`, no source contains it, and `der` is one edit away on every German page — so the brief helpfully reported that the sources spell it "der". +- **Two edits are only allowed once a word is long enough to go wrong twice.** Otherwise short words turn into unrelated ones. + +The same measurement settled the language question. `wer ist elon musk` returns `de.wikipedia.org` first and the English article second; `who is elon musk` returns the English one. The engine localises by the language of the query, so the sources come back in the user's language **as long as the query is not translated first** — and the answer then follows its sources. That is what the German exemplar in `research-question` teaches: search with the words the question used, answer in the language it was asked in. It is the same discipline `lookup-term` already needed for `1inch`, arrived at from the other direction. The skills teach the rest. [`research-question`](#skills) answers with the consensus and names the site that disagrees, and its exemplar ends with more than one URL. diff --git a/src/eval/scenarios.ts b/src/eval/scenarios.ts index 1ed77f2..e0a4b65 100644 --- a/src/eval/scenarios.ts +++ b/src/eval/scenarios.ts @@ -368,6 +368,28 @@ export const SCENARIOS: Scenario[] = [ accept: matches(/example|illustrative|domain/i), online: true, }, + { + id: 'lookup-misspelled-name', + category: 'lookup', + // The query goes through untouched — rewriting it is the `1inch` failure — + // so the correcting is left to the search, which already does it: `eln musk` + // returns "Elon Musk" as its first hit and every source then spells it out. + prompt: 'wer ist eln musk', + expectTool: 'web_search', + acceptCall: keepsTermIntact('eln'), + accept: matches(/musk/i), + online: true, + }, + { + id: 'web-german-answer', + category: 'web', + // Asked in German, answered in German, from sources the German query found. + // Translating the query first is how this ends up answered off English pages. + prompt: 'Wer ist der Bundeskanzler von Deutschland?', + expectTool: 'web_search', + accept: matches(/kanzler|regierung|amt|seit/i), + online: true, + }, { id: 'web-life-dates', category: 'web', diff --git a/src/skills/research-question/SKILL.md b/src/skills/research-question/SKILL.md index 9afe073..9a4b5a9 100644 --- a/src/skills/research-question/SKILL.md +++ b/src/skills/research-question/SKILL.md @@ -81,6 +81,30 @@ jarvis: still lists Jordan Hale from 2021 and looks out of date. Source: https://fictionalairways.example/leadership https://dailywire.example/fictional-airways-ceo + # German, and the point is the query rather than the answer: searching with + # the words the question used is what brings German pages back, and the + # answer then follows its sources into the right language. Translating the + # question first is how a German question ends up answered from English + # pages. The misspelling is here for the same reason — the query is passed + # through untouched, and the correcting is left to the sources. + - user: wer ist eln musk + steps: + - tool: web_search + arguments: + query: wer ist eln musk + result: | + Searched 2026-08-26 for "wer ist eln musk" — 2 sources + 1. Elon Musk (de.example) + https://de.example/elon-musk + Elon Musk ist ein Unternehmer und führt Tesla und SpaceX. + 2. Elon Musk im Profil (nachrichten.example) + https://nachrichten.example/elon-musk + Der Unternehmer Elon Musk leitet Tesla, SpaceX und X. + The sources spell it "Elon" (the question wrote "eln"). Answer about that, and use their spelling. + answer: | + Du meinst Elon Musk. Er ist Unternehmer und führt Tesla, SpaceX und X. + + Source: https://de.example/elon-musk --- -Search once, then answer from the brief it returns. It carries several sites: give what they agree on, and say which site disagrees. Open one with `read_page` only when the brief is not enough. End with the URLs you used. +Search once with the words the question used — never translated — then answer from the brief it returns, in the language the question was asked in. It carries several sites: give what they agree on, and say which site disagrees. Open one with `read_page` only when the brief is not enough. End with the URLs you used. diff --git a/src/tools/search-brief.test.ts b/src/tools/search-brief.test.ts index 8802b5f..202ef84 100644 --- a/src/tools/search-brief.test.ts +++ b/src/tools/search-brief.test.ts @@ -6,6 +6,7 @@ import { searchBrief, selectDiverseSources, siteOf, + spellingFrom, type BriefSource, } from './search-brief' import type { SearchResult, WebAccessConfig } from './web' @@ -177,6 +178,28 @@ describe('compareSources', () => { ]) }) + it('does not read a spread of years as a disagreement about one of them', () => { + // Observed live on "wer ist elon musk": a birth year, an election year and + // this year were reported as three sources contradicting each other. + const { conflicts } = compareSources([ + source('a.example', 'Born 1971, backed a candidate in 2024.'), + source('b.example', 'Born 1971. As of 2026 the richest person alive.'), + source('c.example', 'In 2026 he became the first trillionaire, after 2024.'), + ]) + + expect(conflicts).toEqual([]) + }) + + it('still reports two years that are answers to the same question', () => { + const { conflicts } = compareSources([ + source('a.example', 'Chief executive since 2023.'), + source('b.example', 'In post since 2023.'), + source('c.example', 'Chief executive since 2021.'), + ]) + + expect(conflicts[0]?.values.map((value) => value.display)).toEqual(['2023', '2021']) + }) + it('says nothing when only one site names a figure', () => { // One reading is not a disagreement, and reporting it as one would tell the // model to hedge an answer nothing contradicted. @@ -228,6 +251,72 @@ describe('compareSources', () => { }) }) +describe('spellingFrom', () => { + const musk = [ + { ...source('de.example', 'Elon Musk ist ein Unternehmer und führt Tesla.'), title: 'Elon Musk' }, + { ...source('news.example', 'Der Unternehmer Elon Musk leitet SpaceX.'), title: 'Elon Musk im Profil' }, + ] + + it('reads the spelling out of the results the search already corrected', () => { + // `eln musk` returns "Elon Musk — Wikipedia" as its first hit, so the + // correction is in the data and nothing has to be looked up. + expect(spellingFrom('wer ist eln musk', musk)).toEqual([{ written: 'eln', found: 'Elon', sites: 2 }]) + }) + + it('says nothing when the query spelled it the way the sources do', () => { + expect(spellingFrom('wer ist elon musk', musk)).toEqual([]) + }) + + it('does not correct a question word into a word that happens to be near it', () => { + // Without a stoplist this reported that the sources spell `wer` as `der`, + // which is on every German page and one edit away. + const german = [ + source('a.example', 'Der Unternehmer leitet das Werk.'), + source('b.example', 'Der Unternehmer und das Werk.'), + ] + + expect(spellingFrom('wer ist das', german)).toEqual([]) + }) + + it('needs two sources to agree before it calls anything a misspelling', () => { + // One page's own typo is not a correction. + const single = [source('a.example', 'Elon Musk.'), source('b.example', 'Nothing relevant here.')] + + expect(spellingFrom('eln', single)).toEqual([]) + }) + + it('leaves a term that mixes letters and digits alone', () => { + // `1inch` is the failure `lookup-term` exists for: the model decided it was + // a typo for `1 inch` and searched for a unit conversion instead. + const defi = [ + source('a.example', 'The 1inch Network aggregates decentralised exchanges.'), + source('b.example', '1inch is a DEX aggregator.'), + ] + + expect(spellingFrom('1inch', defi)).toEqual([]) + }) + + it('does not turn a short word into an unrelated one', () => { + // Two edits are only allowed once a word is long enough to go wrong twice. + const sources = [source('a.example', 'Cats and dogs.'), source('b.example', 'Cats and dogs.')] + + expect(spellingFrom('rat', sources)).toEqual([]) + }) + + it('reports at most two, since more is a query to retype rather than annotate', () => { + const sources = [ + { ...source('a.example', 'Elon Musk founded SpaceX and Tesla.'), title: 'Elon Musk' }, + { ...source('b.example', 'Elon Musk founded SpaceX and Tesla.'), title: 'Elon Musk' }, + ] + + expect(spellingFrom('eln mask spacx tesln', sources).length).toBeLessThanOrEqual(2) + }) + + it('says nothing when there are no sources to read a spelling from', () => { + expect(spellingFrom('eln musk', [])).toEqual([]) + }) +}) + describe('formatBrief', () => { const sources = [ source('un.org', 'António Guterres is the ninth Secretary-General, in post since 2017.'), @@ -250,6 +339,18 @@ describe('formatBrief', () => { expect(brief).toContain('— snippet only') }) + it('carries the spelling the sources use, so a typo can still be answered', () => { + const musk = [ + { ...source('de.example', 'Elon Musk ist Unternehmer.'), title: 'Elon Musk' }, + { ...source('news.example', 'Elon Musk leitet SpaceX.'), title: 'Elon Musk im Profil' }, + ] + + const brief = formatBrief('wer ist eln musk', musk, NOW) + + expect(brief).toContain('The sources spell it "Elon" (the question wrote "eln").') + expect(brief).toContain('use their spelling') + }) + it('says so rather than implying a cross-check that did not happen', () => { const brief = formatBrief('x', [source('a.example', 'One reading.')], NOW) diff --git a/src/tools/search-brief.ts b/src/tools/search-brief.ts index d5a837e..1fb2b9c 100644 --- a/src/tools/search-brief.ts +++ b/src/tools/search-brief.ts @@ -394,6 +394,9 @@ function comparableKind(kind: string): boolean { return kind !== 'd1' } +/** Kinds where several different values are ordinary rather than contradictory. */ +const DATED_KINDS = new Set(['year', 'date']) + export interface Agreement { term: string sites: number @@ -403,9 +406,183 @@ export interface Conflict { values: { display: string; sites: string[] }[] } +export interface Spelling { + /** The word as the query wrote it. */ + written: string + /** How the sources write it instead. */ + found: string + sites: number +} + export interface Comparison { overlap: Agreement[] conflicts: Conflict[] + spelling: Spelling[] +} + +/** + * Words a query uses to ask rather than to name. + * + * This list is what stops the mechanism turning into nonsense. Without it, + * `wer ist elon musk` offers up `wer`, no source contains it, and `der` is one + * edit away on every German page — so the brief would helpfully report that the + * sources spell it "der". + */ +const ASKING_WORDS = new Set([ + 'about', + 'alt', + 'are', + 'aus', + 'can', + 'could', + 'dich', + 'did', + 'die', + 'dir', + 'does', + 'ein', + 'eine', + 'euch', + 'find', + 'für', + 'gibt', + 'hat', + 'haben', + 'has', + 'have', + 'how', + 'ist', + 'kann', + 'können', + 'me', + 'mich', + 'mir', + 'much', + 'muss', + 'must', + 'my', + 'out', + 'sind', + 'should', + 'soll', + 'tell', + 'the', + 'über', + 'uns', + 'viel', + 'von', + 'wann', + 'war', + 'waren', + 'warum', + 'was', + 'welche', + 'welcher', + 'welches', + 'wer', + 'were', + 'what', + 'when', + 'where', + 'which', + 'who', + 'why', + 'wie', + 'wieso', + 'wird', + 'werden', + 'will', + 'wo', + 'would', + 'your', +]) + +const WORD = /\p{L}[\p{L}'’-]*/gu + +function wordsIn(text: string): string[] { + return [...text.matchAll(WORD)].map(([word]) => word.toLowerCase()) +} + +/** How far apart two words are, giving up once they are further than `limit`. */ +function editDistance(left: string, right: string, limit: number): number { + if (Math.abs(left.length - right.length) > limit) return limit + 1 + + let previous = Array.from({ length: right.length + 1 }, (_, at) => at) + + for (let row = 1; row <= left.length; row += 1) { + const current = [row] + for (let column = 1; column <= right.length; column += 1) { + const substitution = (previous[column - 1] ?? 0) + (left[row - 1] === right[column - 1] ? 0 : 1) + current[column] = Math.min(substitution, (previous[column] ?? 0) + 1, (current[column - 1] ?? 0) + 1) + } + if (Math.min(...current) > limit) return limit + 1 + previous = current + } + + return previous[right.length] ?? limit + 1 +} + +/** One edit for a short word, two once there is enough of it to go wrong twice. */ +function allowedEdits(word: string): number { + return word.length >= 6 ? 2 : 1 +} + +/** + * How the sources spell a word the query got wrong. + * + * The search engine has already done the correcting: `eln musk` returns "Elon + * Musk — Wikipedia" as its first hit, and every source then writes the name + * properly. So there is nothing to look up and no dictionary to ship — the + * spelling is sitting in the results, and this only has to notice. + * + * Which is also why it is reported rather than acted on. Rewriting the query + * would put back the failure `lookup-term` exists for, where the model decided + * `1inch` was a typo for `1 inch` and searched for a unit conversion. + */ +export function spellingFrom(query: string, sources: BriefSource[]): Spelling[] { + if (sources.length === 0) return [] + + const candidates = new Map }>() + const present = new Set() + + for (const source of sources) { + for (const [word] of `${source.title} ${source.extract}`.matchAll(WORD)) { + const key = word.toLowerCase() + present.add(key) + const entry = candidates.get(key) ?? { display: word, sites: new Set() } + entry.sites.add(source.site) + candidates.set(key, entry) + } + } + + const found: Spelling[] = [] + + for (const word of new Set(wordsIn(query))) { + if (word.length < 3 || ASKING_WORDS.has(word) || present.has(word)) continue + + const limit = allowedEdits(word) + let best: { display: string; sites: number; distance: number } | undefined + + for (const [key, entry] of candidates) { + // Two sources spelling it the same way is the evidence. One page's typo is + // not a correction, and neither is a word from the site's own furniture. + if (entry.sites.size < 2 || key.length < 3) continue + const distance = editDistance(word, key, limit) + if (distance > limit) continue + if ( + !best || + distance < best.distance || + (distance === best.distance && entry.sites.size > best.sites) + ) { + best = { display: entry.display, sites: entry.sites.size, distance } + } + } + + if (best) found.push({ written: word, found: best.display, sites: best.sites }) + } + + // Two is already a query that needs retyping rather than annotating. + return found.slice(0, 2) } function record(into: Map, key: string, display: string, site: string): void { @@ -442,7 +619,7 @@ function subsumed(term: string, kept: Agreement[]): boolean { * is the capacity the answer needed. Names and figures are what a rule can * honestly compare, so they are all it claims to have compared. */ -export function compareSources(sources: BriefSource[]): Comparison { +export function compareSources(sources: BriefSource[], query = ''): Comparison { const names = new Map() const numbers = new Map>() @@ -472,7 +649,13 @@ export function compareSources(sources: BriefSource[]): Comparison { } const conflicts: Conflict[] = [] - for (const byValue of numbers.values()) { + for (const [kind, byValue] of numbers) { + // A biography names a birth year, an election year and this year, and none + // of them contradicts the others. Observed on a live search, which reported + // "1971 vs 2026 vs 2024" as a disagreement about one thing. Past two + // distinct years the sources are dating different events, not differing. + if (DATED_KINDS.has(kind) && byValue.size > 2) continue + const [leading, ...rest] = [...byValue.values()].sort(byReach) // A disagreement is only reportable when there is a reading to disagree // with: one site against one other says which pages differ, not which is @@ -495,7 +678,7 @@ export function compareSources(sources: BriefSource[]): Comparison { conflicts.sort((left, right) => (right.values[0]?.sites.length ?? 0) - (left.values[0]?.sites.length ?? 0)) - return { overlap, conflicts: conflicts.slice(0, MAX_CONFLICTS) } + return { overlap, conflicts: conflicts.slice(0, MAX_CONFLICTS), spelling: spellingFrom(query, sources) } } /** `YYYY-MM-DD` in the user's own timezone, which is the day they are asking about. */ @@ -515,6 +698,15 @@ function conflictLine(conflict: Conflict): string { return `Sources disagree: ${parts.join(' vs ')}` } +/** + * Said plainly, because the model has to be able to use the right spelling in an + * answer to a question that used the wrong one. + */ +function spellingLine(spelling: Spelling[]): string { + const parts = spelling.map((entry) => `"${entry.found}" (the question wrote "${entry.written}")`) + return `The sources spell it ${parts.join(' and ')}. Answer about that, and use their spelling.` +} + /** * The brief, in the shape the rest of the app already reads. * @@ -537,14 +729,16 @@ export function formatBrief(query: string, sources: BriefSource[], now = new Dat ].join('\n'), ) - const { overlap, conflicts } = compareSources(sources) - const footer = - sources.length === 1 + const { overlap, conflicts, spelling } = compareSources(sources, query) + const footer = [ + ...(spelling.length > 0 ? [spellingLine(spelling)] : []), + ...(sources.length === 1 ? ['Only one source was readable, so nothing was cross-checked.'] : [ ...(overlap.length > 0 ? [overlapLine(overlap, sources.length)] : []), ...conflicts.map(conflictLine), - ] + ]), + ] return truncate([header, ...entries, ...footer].join('\n'), MAX_BRIEF_CHARS) }