diff --git a/.cursor/rules/memory.mdc b/.cursor/rules/memory.mdc index 556ae49..41e1f77 100644 --- a/.cursor/rules/memory.mdc +++ b/.cursor/rules/memory.mdc @@ -7,7 +7,7 @@ Read `.cursor/skills/agent-memory/SKILL.md` before changing what Jarvis remember - `db.ts` is persistence, `manage.ts` is policy, `text.ts` is the one place text is compared. Nothing outside those touches IndexedDB. `topic.ts` is working memory from the transcript, not a - store: the last weather place is injected like recall and never written down. + store: the last weather place or research subject is injected like recall and never written down. - Deleting sets `deletedAt`; it does not remove the row. `readAllRecords` returns the trash too, so a new reader has to filter — the model can reach delete and clear through a tool. - Recall is injected into the system prompt, budgeted by `MAX_RECALL_CHARS`. Growing that budget is diff --git a/.cursor/skills/agent-memory/SKILL.md b/.cursor/skills/agent-memory/SKILL.md index 153452a..d9d2d06 100644 --- a/.cursor/skills/agent-memory/SKILL.md +++ b/.cursor/skills/agent-memory/SKILL.md @@ -54,10 +54,12 @@ what counts as a repeat, where a false positive would silently drop what the use - **No background extraction.** Writes are explicit — the model calls the tool, or the user types in the panel. ChatGPT's dreaming and mem0's extractor both spend a second model call per conversation; here that call runs on the user's own GPU and would double the cost of a turn. -- **The conversation topic is derived, never stored.** `topic.ts` reads the last weather place off - the transcript and injects one line, the same way recall is injected. It fires on a follow-up, - a weather turn with no place, or an anaphor (_der Bürgermeister_, _there_). It stays off a - fresh named subject — _Wer ist Elon Musk?_ after Frankfurt weather must not receive Frankfurt. +- **The conversation topic is derived, never stored.** `topic.ts` reads the last weather place or + the last research subject off the transcript and injects one line, the same way recall is + injected. Recency wins, so an older Frankfurt does not leak onto _und der von Frankreich?_. It + fires on a follow-up, a weather turn with no place, or an anaphor (_der Bürgermeister_, _there_). + It stays off a fresh named subject — _Wer ist Elon Musk?_ after Frankfurt weather must not + receive Frankfurt, and _und Elon Musk?_ after a chancellor turn must not receive Bundeskanzler. Memory off still pins it: that toggle is IndexedDB, not this chat. - **Contradictions are kept.** "Lives in Berlin" and "Lives in Lisbon" both stay; recall prefers the newest and the panel shows both. mem0 shipped the UPDATE/DELETE version of this and moved back to diff --git a/.cursor/skills/debug-model-output/SKILL.md b/.cursor/skills/debug-model-output/SKILL.md index 18381b4..2884052 100644 --- a/.cursor/skills/debug-model-output/SKILL.md +++ b/.cursor/skills/debug-model-output/SKILL.md @@ -60,7 +60,8 @@ Three things follow from that, and all three are easy to undo by accident: A check that cannot point at a tool result does not belong here. - **A check that fires on a correct answer is a bug**, not a strict setting. It costs a generation and trains the user to ignore the label. `review.test.ts` pins the shy cases — a clarifying - question, a rounded decimal, a source carried over from an earlier turn — and they are the point. + question, a rounded decimal, a researched surname standing in for the full name, a source + carried over from an earlier turn — and they are the point. - **Only successful tool results become evidence.** A failed fetch has nothing to check against, and demanding a citation for a page that never loaded is worse than saying nothing. diff --git a/README.md b/README.md index 26be74a..ed1fb06 100644 --- a/README.md +++ b/README.md @@ -342,7 +342,7 @@ Memories live in **IndexedDB**, in your browser, next to the model weights. Noth The [CoALA taxonomy](https://arxiv.org/abs/2309.02427) splits an agent's memory four ways: working memory is the live context window, and the three durable kinds are semantic, episodic and procedural. Only the durable three are stored — working memory is the transcript, which the tab already holds — and they are named in words the model can actually pick between. -The transcript is not enough on its own. A 0.8B model will answer tomorrow's weather without the city the last turn just resolved, so the last established place is pinned into the system prompt as one short line — _This conversation is about Frankfurt._ — the same way recall is. It is derived from the chat, never written to IndexedDB, and it stays off a fresh question that names its own subject. Switching memory off does not drop it: that toggle is for stored facts, not for this conversation. +The transcript is not enough on its own. A 0.8B model will answer tomorrow's weather without the city the last turn just resolved, so the last established place is pinned into the system prompt as one short line — _This conversation is about Frankfurt._ — the same way recall is. After a research turn the same line holds the office or subject — _This conversation is about Bundeskanzler._ — so _und der von Frankreich?_ is not searched as a fragment. Recency decides which one: an older Frankfurt does not leak onto a chancellor follow-up. It is derived from the chat, never written to IndexedDB, and it stays off a fresh question that names its own subject. Switching memory off does not drop it: that toggle is for stored facts, not for this conversation. | Kind | Is | Recalled | | ------------ | ---------------------------------- | ----------------------------- | @@ -550,19 +550,20 @@ Every reply says which skill answered it and how it was found: `weather skill · A skill fires on some requests. This runs on all of them. -Between the model settling on an answer and that answer reaching the screen, `src/agent/review.ts` reads it back against what the turn actually produced — the results the tools returned, and the URLs already in the conversation. Three things are checked: +Between the model settling on an answer and that answer reaching the screen, `src/agent/review.ts` reads it back against what the turn actually produced — the results the tools returned, and the URLs already in the conversation. Four things are checked: -| Check | Fires when | -| ----------------- | --------------------------------------------------------------------- | -| `wrong-number` | The calculator's value, or the clock's local HH:MM, is stated nowhere | -| `invented-source` | The answer cites a URL that no tool returned and nobody supplied | -| `missing-source` | Tools returned sources and the answer cites none | +| Check | Fires when | +| ----------------- | ---------------------------------------------------------------------------- | +| `wrong-number` | The calculator's value, or the clock's local HH:MM, is stated nowhere | +| `wrong-fact` | `research` opened with `Answer: …` and that name or figure is stated nowhere | +| `invented-source` | The answer cites a URL that no tool returned and nobody supplied | +| `missing-source` | Tools returned sources and the answer cites none | A failed check costs one further generation. The model is handed its own draft and told what to change — _The calculator returned 6748 \* 9 = 60732. Give that number, exactly as it came back._ — and the correction replaces the draft only if it leaves fewer problems behind. Otherwise the draft stands. That gate is the important half: the correction comes from the same 0.8B model, so a mechanism that could not tell an improvement from a regression would be a coin toss on every reply. **The checks are deterministic, and that is the design.** Asking the model to grade its own answer spends exactly the capacity the answer needed, and intrinsic self-correction — re-reading with nothing new to go on — degrades reasoning rather than improving it ([arXiv:2310.01798](https://arxiv.org/html/2310.01798)). What works is external feedback, so every check compares the draft against something already in the context, and the correction states the fix rather than inviting the model to hunt for one. -They are also deliberately shy. A clarifying question is asked for no citation; a long decimal quoted to fewer places counts as the calculator's number; citing the site when a page on it was read is close enough; a URL from an earlier reply is not an invention; a year-only clock answer is left alone, and a German date like `27.08.2026` is not a time. Every check would rather miss a mistake than invent one, because a check that fires on a correct answer costs a generation and teaches you to ignore the whole mechanism. +They are also deliberately shy. A clarifying question is asked for no citation; a long decimal quoted to fewer places counts as the calculator's number; a researched surname (`Merz`) counts as the full extract (`Friedrich Merz`); 14 million counts as 13.96 million, which is the same reading the extractor already accepted; citing the site when a page on it was read is close enough; a URL from an earlier reply is not an invention; a year-only clock answer is left alone, and a German date like `27.08.2026` is not a time. Every check would rather miss a mistake than invent one, because a check that fires on a correct answer costs a generation and teaches you to ignore the whole mechanism. The interface says what happened rather than quietly rewriting the reply. While the corrected answer streams in it is labelled with what is being fixed, and afterwards it carries `corrected` — claimed only for an answer that now passes every check — or `flagged`, naming what is still wrong with the text on screen. An answer half fixed and advertised as corrected would be worse than no check at all. diff --git a/src/agent/budget.test.ts b/src/agent/budget.test.ts index 24797be..e098acb 100644 --- a/src/agent/budget.test.ts +++ b/src/agent/budget.test.ts @@ -93,4 +93,26 @@ describe('budgetFallback', () => { expect(text).toContain('Try narrowing the question') expect(splitSources(text).sources).toEqual([]) }) + + it('hands over the researched one-liner when the wind-down came back empty', () => { + const text = budgetFallback( + evidence([ + { + tool: 'research', + result: [ + 'Answer: Friedrich Merz.', + '', + 'Researched 2026-09-10 for "Bundeskanzler" across 1 source, all read in full.', + '', + '1. Bundeskanzler — https://de.wikipedia.org/wiki/Bundeskanzler', + ' "Amtsträger ist Friedrich Merz."', + ].join('\n'), + }, + ]), + ) + + expect(text).toMatch(/^Friedrich Merz\./) + expect(text).not.toContain('could not settle') + expect(splitSources(text).sources).toEqual(['https://de.wikipedia.org/wiki/Bundeskanzler']) + }) }) diff --git a/src/agent/budget.ts b/src/agent/budget.ts index 552ecb7..b1ce502 100644 --- a/src/agent/budget.ts +++ b/src/agent/budget.ts @@ -1,5 +1,5 @@ import { MAX_TOOL_ROUNDS } from '@/llm/config' -import { findUrls, type ReviewEvidence } from './review' +import { findUrls, researchedAnswer, type ReviewEvidence } from './review' /** * What happens when a turn runs out of tool rounds. @@ -117,12 +117,19 @@ const FALLBACK_SOURCES = 3 * into citation pills — the pages are the part worth clicking. */ export function budgetFallback(evidence: ReviewEvidence): string { - const opening = `I could not settle on an answer within ${MAX_TOOL_ROUNDS} rounds of tool calls.` + const extracted = researchedAnswer(evidence) const sources = [...new Set(evidence.toolResults.flatMap(({ result }) => findUrls(result)))].slice( 0, FALLBACK_SOURCES, ) + // `research` already committed to a one-liner. Handing that over beats an + // apology: the wind-down round failed, not the search. + if (extracted) { + return sources.length > 0 ? `${extracted}.\n\nSource: ${sources.join(' ')}` : `${extracted}.` + } + + const opening = `I could not settle on an answer within ${MAX_TOOL_ROUNDS} rounds of tool calls.` if (sources.length === 0) return `${opening} Try narrowing the question.` return `${opening} These pages came up on the way, in case one of them helps.\n\nSource: ${sources.join(' ')}` } diff --git a/src/agent/loop.test.ts b/src/agent/loop.test.ts index 957ce41..31da8f5 100644 --- a/src/agent/loop.test.ts +++ b/src/agent/loop.test.ts @@ -315,6 +315,29 @@ describe('checking the answer before returning it', () => { ) }) + it('corrects a researched name the answer dropped', async () => { + const research = defineTool('research', 'research', { type: 'object', properties: {} }, async () => + [ + 'Answer: Ama Osei.', + '', + 'Researched 2026-09-10 for "who runs Fictional Airways" across 1 source, all read in full.', + '', + '1. Leadership — https://fictionalairways.example/leadership', + ' "Ama Osei has led the airline since 2023."', + ].join('\n'), + ) + const client = fakeClient([ + toolCall('research', 'query', 'who runs Fictional Airways'), + 'guessingPiet Hendriks runs it.\n\nSource: https://fictionalairways.example/leadership', + 'reading the digestAma Osei.\n\nSource: https://fictionalairways.example/leadership', + ]) + + const result = await runAgent(client, turns, [research], callbacks()) + + expect(result.content).toContain('Ama Osei') + expect(result.review).toEqual({ found: ['wrong-fact'], corrected: true }) + }) + it('corrects a number the answer did not take from the calculator', async () => { const client = fakeClient([ toolCall('calculator', 'expression', '6748 * 9'), diff --git a/src/agent/review.test.ts b/src/agent/review.test.ts index 61a763d..f75574f 100644 --- a/src/agent/review.test.ts +++ b/src/agent/review.test.ts @@ -1,5 +1,11 @@ import { describe, expect, it } from 'vitest' -import { collectEvidence, correctionPrompt, reviewAnswer, type ReviewEvidence } from './review' +import { + collectEvidence, + correctionPrompt, + researchedAnswer, + reviewAnswer, + type ReviewEvidence, +} from './review' function evidence(overrides: Partial = {}): ReviewEvidence { return { toolResults: [], knownUrls: [], ...overrides } @@ -223,6 +229,143 @@ describe('reviewAnswer', () => { ]) }) }) + + describe('researched facts', () => { + const researched = evidence({ + toolResults: [ + { + tool: 'research', + result: [ + 'Answer: Friedrich Merz.', + '', + 'Researched 2026-09-10 for "Bundeskanzler" across 2 sources, all read in full.', + '', + '1. Bundeskanzler — https://de.wikipedia.org/wiki/Bundeskanzler_(Deutschland)', + ' "Amtsträger ist seit dem 6. Mai 2025 Friedrich Merz (CDU)."', + ].join('\n'), + }, + ], + }) + + it('accepts the name the research digest opened with', () => { + expect( + checks( + 'Friedrich Merz, seit Mai 2025.\n\nSource: https://de.wikipedia.org/wiki/Bundeskanzler_(Deutschland)', + researched, + ), + ).toEqual([]) + }) + + it('accepts the surname when the full name was extracted', () => { + // A check that flagged "Merz" would fire on a correct German short answer. + expect( + checks('Merz.\n\nSource: https://de.wikipedia.org/wiki/Bundeskanzler_(Deutschland)', researched), + ).toEqual([]) + }) + + it('accepts a missing accent on a copied name', () => { + const un = evidence({ + toolResults: [ + { + tool: 'research', + result: + 'Answer: António Guterres.\n\nResearched 2026-09-10 for "UN" across 1 source, all read in full.\n\n1. UN — https://www.un.org/sg/en', + }, + ], + }) + + expect(checks('Antonio Guterres.\n\nSource: https://www.un.org/sg/en', un)).toEqual([]) + }) + + it('catches an invented name after research already answered', () => { + expect( + checks( + 'Olaf Scholz.\n\nSource: https://de.wikipedia.org/wiki/Bundeskanzler_(Deutschland)', + researched, + ), + ).toEqual(['wrong-fact']) + }) + + it('catches a reply that quotes the office and drops the incumbent', () => { + expect( + checks( + 'The chancellor is the head of government.\n\nSource: https://de.wikipedia.org/wiki/Bundeskanzler_(Deutschland)', + researched, + ), + ).toEqual(['wrong-fact']) + }) + + it('quotes the extract in the correction', () => { + const [finding] = reviewAnswer('Olaf Scholz.', researched) + + expect(finding?.instruction).toContain('Answer: Friedrich Merz') + }) + + it('accepts a figure the digest opened with, including German decimals', () => { + const population = evidence({ + toolResults: [ + { + tool: 'research', + result: + 'Answer: 13.96 million.\n\nResearched 2026-09-10 for "population of Tokyo" across 1 source, all read in full.\n\n1. Tokyo — https://en.wikipedia.org/wiki/Tokyo', + }, + ], + }) + + expect( + checks('About 13,96 Millionen.\n\nSource: https://en.wikipedia.org/wiki/Tokyo', population), + ).toEqual([]) + expect(checks('About 14 million.\n\nSource: https://en.wikipedia.org/wiki/Tokyo', population)).toEqual( + [], + ) + }) + + it('catches an invented figure after research already answered', () => { + const population = evidence({ + toolResults: [ + { + tool: 'research', + result: + 'Answer: 13.96 million.\n\nResearched 2026-09-10 for "population of Tokyo" across 1 source, all read in full.\n\n1. Tokyo — https://en.wikipedia.org/wiki/Tokyo', + }, + ], + }) + + expect( + checks('Tokyo has 11 million people.\n\nSource: https://en.wikipedia.org/wiki/Tokyo', population), + ).toEqual(['wrong-fact']) + }) + + it('leaves a biography digest alone when nothing was extracted', () => { + const bio = evidence({ + toolResults: [ + { + tool: 'research', + result: + 'Researched 2026-09-10 for "Who is Elon Musk" across 1 source, all read in full.\n\n1. Elon Musk — https://en.wikipedia.org/wiki/Elon_Musk\n "Elon Musk is a businessman."', + }, + ], + }) + + expect( + checks('Elon Musk is a businessman.\n\nSource: https://en.wikipedia.org/wiki/Elon_Musk', bio), + ).toEqual([]) + }) + + it('does not treat an Answer line inside a web_search snippet as an extract', () => { + const searched = evidence({ + toolResults: [ + { + tool: 'web_search', + result: '1. Quiz\n https://quiz.example\n Answer: Paris is a cheese.', + }, + ], + }) + + expect(researchedAnswer(searched)).toBeNull() + expect(checks('Lyon.\n\nSource: https://quiz.example', searched)).toEqual([]) + }) + }) }) describe('collectEvidence', () => { diff --git a/src/agent/review.ts b/src/agent/review.ts index 72d257d..ea0c628 100644 --- a/src/agent/review.ts +++ b/src/agent/review.ts @@ -16,7 +16,7 @@ import { localClockInResult } from '@/tools/clock' * costs a second generation and teaches the user to ignore the whole mechanism, * so every one of them prefers to miss a mistake over inventing one. */ -export type ReviewCheck = 'wrong-number' | 'invented-source' | 'missing-source' +export type ReviewCheck = 'wrong-number' | 'wrong-fact' | 'invented-source' | 'missing-source' export interface ReviewFinding { check: ReviewCheck @@ -196,9 +196,105 @@ function latestClock(evidence: ReviewEvidence): ClockTime | null { return null } +/** + * The one-liner `research` puts at the top of a digest when the sources agree. + * + * `digest` always writes `Answer: Friedrich Merz.` — a period after the extract, + * then a blank line. The period is the line ending, not part of the name. + */ +const RESEARCHED = /^Answer:\s+(.+)\.\s*$/m + /** A question back to the user, and a plain "I could not find it", cite nothing. */ const NOTHING_TO_CITE = /\?\s*$|\b(could ?n[o']t find|no results|don'?t know|do not know|unable to find)\b/i +/** + * The extract `research` already committed to, or `null` when the digest had + * nothing confident enough to put on the first line. A biography, a tie, or a + * failed call leaves this empty, and the check stays off — inventing a fact to + * demand would be the opposite of shy. + */ +export function researchedAnswer(evidence: ReviewEvidence): string | null { + for (const { tool, result } of [...evidence.toolResults].reverse()) { + if (tool !== 'research') continue + const match = RESEARCHED.exec(result) + const extracted = match?.[1]?.trim() + if (extracted) return extracted + } + return null +} + +function fold(value: string): string { + return value + .normalize('NFD') + .replace(/\p{M}+/gu, '') + .toLowerCase() +} + +/** + * The first number in an extract, accepting either decimal mark. + * + * `3,8 Millionen` and `13.96 million` are how the digest writes German and + * English figures. A thousands-grouped integer is left alone: `1,396` with + * three digits after the comma is thirteen hundred, not 1.396. + */ +function firstNumber(text: string): number | null { + const match = text.match(/-?\d+(?:[.,]\d+)?/) + if (!match?.[0]) return null + const raw = match[0] + const comma = raw.lastIndexOf(',') + const dot = raw.lastIndexOf('.') + let normalized = raw + if (comma >= 0 && dot < 0) { + const decimals = raw.length - comma - 1 + normalized = decimals > 0 && decimals <= 2 ? raw.replace(',', '.') : raw.replace(/,/g, '') + } else if (comma >= 0 && dot >= 0 && comma > dot) { + normalized = raw.replace(/\./g, '').replace(',', '.') + } else { + normalized = raw.replace(/,/g, '') + } + const value = Number(normalized) + return Number.isFinite(value) ? value : null +} + +/** + * Whether the draft already states the researched extract. + * + * A name passes when every token of it appears, so `Friedrich Merz, seit 2025` + * is enough and `Merz` alone is not a miss we invent — the last token of a + * multi-word name is the distinctive one, and asking for the rest costs a + * generation on an answer that is already right. A figure passes when the + * number is there with either decimal mark, or when the draft rounded it the + * way `extractFigure` already treats as the same reading (about ten per cent). + */ +function statesResearched(answer: string, extracted: string): boolean { + const foldedAnswer = fold(answer) + const foldedExtracted = fold(extracted) + if (foldedAnswer.includes(foldedExtracted)) return true + + const value = firstNumber(extracted) + if (value !== null && /\d/.test(extracted)) { + if (statesNumber(answer, value)) return true + const german = answer.replace(/(\d),(\d)/g, '$1.$2') + if (statesNumber(german, value)) return true + // The first number in the draft is often a year on the citation line, so + // every number is tried. Ten per cent is the same band `extractFigure` + // already treats as one reading — 13.96 million and 14 million pass, + // 11 million does not. + const scale = Math.max(Math.abs(value), 1) + for (const match of answer.matchAll(/-?\d+(?:[.,]\d+)?/g)) { + const stated = firstNumber(match[0] ?? '') + if (stated !== null && Math.abs(stated - value) / scale <= 0.1) return true + } + return false + } + + const tokens = foldedExtracted.match(/[\p{L}\p{N}]+/gu) ?? [] + if (tokens.length === 0) return false + if (tokens.every((token) => foldedAnswer.includes(token))) return true + const last = tokens[tokens.length - 1] + return tokens.length > 1 && last !== undefined && last.length >= 4 && foldedAnswer.includes(last) +} + /** The most recent URL a tool returned, which is the one worth citing. */ function preferredSource(evidence: ReviewEvidence): string | null { for (const { result } of [...evidence.toolResults].reverse()) { @@ -241,6 +337,14 @@ export function reviewAnswer(answer: string, evidence: ReviewEvidence): ReviewFi } } + const extracted = researchedAnswer(evidence) + if (extracted && !statesResearched(draft, extracted)) { + findings.push({ + check: 'wrong-fact', + instruction: `The research result opened with Answer: ${extracted}. Give that, in the language you were asked.`, + }) + } + const source = preferredSource(evidence) const known = [...evidence.knownUrls, ...evidence.toolResults.flatMap(({ result }) => findUrls(result))] .map(locate) diff --git a/src/components/Landing.tsx b/src/components/Landing.tsx index 3e60c4d..7ceaf48 100644 --- a/src/components/Landing.tsx +++ b/src/components/Landing.tsx @@ -68,7 +68,7 @@ const STEPS: { body: string; title: string }[] = [ }, { title: 'The answer is checked', - body: 'Before a reply is shown it is read back against what the tools returned. A number the tools disagree with, or a source nothing ever fetched, is corrected or flagged.', + body: 'Before a reply is shown it is read back against what the tools returned. A number the tools disagree with, a researched fact the reply dropped, or a source nothing ever fetched, is corrected or flagged.', }, ] diff --git a/src/components/MessageItem.test.tsx b/src/components/MessageItem.test.tsx index 51a0b03..15776de 100644 --- a/src/components/MessageItem.test.tsx +++ b/src/components/MessageItem.test.tsx @@ -205,6 +205,22 @@ describe('MessageItem', () => { expect(screen.getByText(/self-check found a source no tool returned/)).toBeInTheDocument() }) + it('names a researched fact the reply dropped', () => { + render( + , + ) + + expect(screen.getByText('corrected')).toBeInTheDocument() + expect( + screen.getByText(/self-check found a researched fact the reply dropped and fixed it/), + ).toBeInTheDocument() + }) + it('names the skill a reply was answered with', () => { render( = { 'wrong-number': 'a number the tools disagreed with', + 'wrong-fact': 'a researched fact the reply dropped', 'invented-source': 'a source no tool returned', 'missing-source': 'a missing source', } diff --git a/src/eval/runner.test.ts b/src/eval/runner.test.ts index 288ba12..1945dea 100644 --- a/src/eval/runner.test.ts +++ b/src/eval/runner.test.ts @@ -215,6 +215,46 @@ Use the calculator.`, ) }) + it('pins the last research subject onto a follow-up that only names a place', async () => { + const catalog = loadCatalog() + const client = fakeClient([ + 'Präsident Frankreich', + 'Emmanuel Macron.\n\nSource: https://de.wikipedia.org/wiki/Emmanuel_Macron', + ]) + const [attempt] = await runEval(client, { + scenarios: [ + { + id: 'research-follow-up', + category: 'web', + history: [ + { role: 'user', content: 'Wer ist der Bundeskanzler?' }, + { role: 'assistant', content: 'Friedrich Merz, seit Mai 2025.' }, + ], + prompt: 'und der von Frankreich?', + expectTool: 'research', + acceptCall: (calls) => + /frankreich|france|präsident|president/i.test(String(calls[0]?.arguments.query ?? '')), + accept: () => true, + }, + ], + arms: [{ id: 'baseline+skills', strategy: STRATEGIES.baseline, skills: catalog }], + repeats: 1, + tools: builtinTools, + }) + + expect(attempt?.skill).toBe('research-question') + expect(attempt?.skillReason).toBe('carried-over') + expect(attempt?.calledWell).toBe(true) + + const [turns] = vi.mocked(client.generate).mock.calls[0] ?? [] + expect(turns?.[0]).toEqual( + expect.objectContaining({ + role: 'system', + content: expect.stringContaining('This conversation is about Bundeskanzler.'), + }), + ) + }) + it('records what the answer check found and whether it fixed it', async () => { const [attempt] = await runEval(fakeClient([CALL, 'It comes to 5.', '2 + 2 = 4']), { scenarios: [arithmetic], diff --git a/src/eval/scenarios.ts b/src/eval/scenarios.ts index 3177d0c..74ed4bf 100644 --- a/src/eval/scenarios.ts +++ b/src/eval/scenarios.ts @@ -539,6 +539,24 @@ export const SCENARIOS: Scenario[] = [ accept: matches(/\b(ja|nein|regen|regnet|schauer|trocken|wolk|sonn|gewitter)/i), online: true, }, + { + id: 'web-follow-up-office', + category: 'web', + // After a chancellor question the office has to stay in the query. Searching + // the fragment *und der von Frankreich* is the 1inch failure again. + history: [ + { role: 'user', content: 'Wer ist der Bundeskanzler?' }, + { + role: 'assistant', + content: 'Friedrich Merz, seit Mai 2025.\n\nSource: https://de.wikipedia.org/wiki/Bundeskanzler', + }, + ], + prompt: 'und der von Frankreich?', + expectTool: 'research', + acceptCall: (calls) => /frankreich|france|präsident|president|macron/i.test(searchQuery(calls) ?? ''), + accept: matches(/macron/i), + online: true, + }, { id: 'no-tool-summarize-pronoun', category: 'no-tool', diff --git a/src/memory/topic.test.ts b/src/memory/topic.test.ts index f4528b1..d2f6633 100644 --- a/src/memory/topic.test.ts +++ b/src/memory/topic.test.ts @@ -2,7 +2,9 @@ import { describe, expect, it } from 'vitest' import { conversationTopic, joinPromptNotes, + lastEstablished, lastEstablishedPlace, + lastEstablishedSubject, renderTopicBlock, type TopicTurn, } from './topic' @@ -11,6 +13,25 @@ function turn(role: TopicTurn['role'], content: string, toolCalls?: TopicTurn['t return { role, content, toolCalls } } +const chancellorResearch: TopicTurn[] = [ + turn('user', 'Wer ist der Bundeskanzler?'), + turn('assistant', 'Friedrich Merz, seit Mai 2025.', [ + { + name: 'research', + arguments: { query: 'Bundeskanzler' }, + status: 'done', + result: [ + 'Answer: Friedrich Merz.', + '', + 'Researched 2026-09-10 for "Bundeskanzler" across 2 sources, all read in full.', + '', + '1. Bundeskanzler — https://de.wikipedia.org/wiki/Bundeskanzler', + ' "Amtsträger ist Friedrich Merz."', + ].join('\n'), + }, + ]), +] + const frankfurtWeather: TopicTurn[] = [ turn('user', 'Wie ist das Wetter in Frankfurt?'), turn('assistant', 'Frankfurt is around 18 °C and cloudy, with little wind.', [ @@ -120,6 +141,36 @@ describe('lastEstablishedPlace', () => { }) }) +describe('lastEstablished', () => { + it('reads the research subject off the digest header', () => { + expect(lastEstablished(chancellorResearch)).toEqual({ kind: 'subject', text: 'Bundeskanzler' }) + expect(lastEstablishedSubject(chancellorResearch)).toBe('Bundeskanzler') + }) + + it('falls back to the user question when no research call was recorded', () => { + expect( + lastEstablishedSubject([ + turn('user', 'Wer ist der Bundeskanzler?'), + turn('assistant', 'Friedrich Merz, seit Mai 2025.'), + ]), + ).toBe('Bundeskanzler') + }) + + it('prefers a later research turn over an older weather place', () => { + expect(lastEstablished([...frankfurtWeather, ...chancellorResearch])).toEqual({ + kind: 'subject', + text: 'Bundeskanzler', + }) + }) + + it('prefers a later weather place over an older research subject', () => { + expect(lastEstablished([...chancellorResearch, ...frankfurtWeather])).toEqual({ + kind: 'place', + text: 'Frankfurt', + }) + }) +}) + describe('conversationTopic', () => { it('pins the place onto a follow-up that names none', () => { expect(conversationTopic('Und morgen?', frankfurtWeather, { skill: 'weather' })).toBe( @@ -174,6 +225,39 @@ describe('conversationTopic', () => { it('stays silent when nothing was established', () => { expect(conversationTopic('Und morgen?', [], { skill: 'weather' })).toBe('') }) + + it('pins the last office onto a follow-up that only names a place', () => { + expect(conversationTopic('und der von Frankreich?', chancellorResearch)).toBe( + 'This conversation is about Bundeskanzler.', + ) + expect(conversationTopic('what about in France?', chancellorResearch)).toBe( + 'This conversation is about Bundeskanzler.', + ) + }) + + it('pins it when the eval history has no tool call, only the question', () => { + expect( + conversationTopic('und der von Frankreich?', [ + turn('user', 'Wer ist der Bundeskanzler?'), + turn('assistant', 'Friedrich Merz, seit Mai 2025.'), + ]), + ).toBe('This conversation is about Bundeskanzler.') + }) + + it('does not let an older city leak onto a research follow-up', () => { + expect(conversationTopic('und der von Frankreich?', [...frankfurtWeather, ...chancellorResearch])).toBe( + 'This conversation is about Bundeskanzler.', + ) + }) + + it('stays silent when the follow-up names a different person', () => { + expect(conversationTopic('und Elon Musk?', chancellorResearch)).toBe('') + }) + + it('stays silent on a fresh research question after a chancellor turn', () => { + expect(conversationTopic('Wer ist Elon Musk?', chancellorResearch)).toBe('') + expect(conversationTopic('What is the capital of France?', chancellorResearch)).toBe('') + }) }) describe('joinPromptNotes', () => { diff --git a/src/memory/topic.ts b/src/memory/topic.ts index da2e09f..adf3752 100644 --- a/src/memory/topic.ts +++ b/src/memory/topic.ts @@ -1,5 +1,7 @@ +import { isResearchable } from '@/skills/researchable' import { isFollowUp } from '@/skills/route' import { placeCandidates as clockPlaceCandidates, placeFromClockResult } from '@/tools/clock' +import { focusQuery } from '@/tools/research' import { placeCandidates as weatherPlaceCandidates } from '@/tools/weather' import { tokenize } from './text' @@ -8,8 +10,10 @@ import { tokenize } from './text' * * Durable memory is IndexedDB. Working memory is the transcript — except a * 0.8B model does not reliably read earlier turns, so the one fact worth - * keeping (the last place a weather or clock call resolved) is pinned into the - * system prompt the same way recall is. Nothing here is written down. + * keeping is pinned into the system prompt the same way recall is. For weather + * and the clock that is the last place. For research it is the last office or + * question — *Bundeskanzler* — so *und der von Frankreich?* is not searched as + * a fragment. Nothing here is written down. * * A fresh question that names its own subject — *Wer ist Elon Musk?* — must * not receive it. Mixing Frankfurt into that prompt is how the model answers @@ -27,6 +31,8 @@ export interface TopicTurn { }> } +export type Established = { kind: 'place'; text: string } | { kind: 'subject'; text: string } + /** A later question that is still about the place, without naming it again. */ const REFERS_BACK = /\b(dort|da|davon|darüber|dabei|hier|the (mayor|weather|forecast|city|airport)|der bürgermeister|die bürgermeisterin|die stadt|that (city|place|town)|there)\b/i @@ -43,12 +49,80 @@ const CLOCK_HINT = /\b(time|date|clock|uhrzeit|weltuhr|zeitzone|timezone|wie sp( const TEMPORAL = /^(today|tonight|tomorrow|now|right now|currently|heute|morgen|jetzt|gerade|aktuell|abend|nachmittag|later|dann|danach|the day after)$/i +/** + * *und der von Frankreich?* keeps the last office and only changes the place. + * Treating Frankreich as a new subject would drop *Bundeskanzler* and search + * a country name. + */ +const PLACE_ONLY_FOLLOW_UP = /(?:(?:der|die|das|the)\s+)?(?:von|of|in|aus|from)\s+[\p{L}\p{M}'’-]+/iu + +const RESEARCHED_FOR = /^Researched \d{4}-\d{2}-\d{2} for "([^"]+)"/m + +/** + * The most recently resolved place or research subject. + * + * Recency is the whole point: an older Frankfurt must not leak onto a chancellor + * follow-up, and an older Bundeskanzler must not leak onto tomorrow's weather. + * Tool results win over the user wording, the same way `lastEstablishedPlace` + * already prefers the city the weather tool actually resolved. + */ +export function lastEstablished(turns: readonly TopicTurn[]): Established | null { + for (const turn of [...turns].toReversed()) { + for (const call of (turn.toolCalls ?? []).toReversed()) { + if (call.status === 'error') continue + + if (call.name === 'weather') { + const fromResult = placeFromWeatherResult(call.result ?? '') + if (fromResult) return { kind: 'place', text: fromResult } + const fromArgs = isolatePlace(String(call.arguments?.place ?? ''), 'weather') + if (fromArgs && !TEMPORAL.test(fromArgs)) return { kind: 'place', text: fromArgs } + } + + if (call.name === 'current_time') { + const fromResult = placeFromClockResult(call.result ?? '') + if (fromResult) return { kind: 'place', text: fromResult } + const fromArgs = isolatePlace(String(call.arguments?.place ?? ''), 'clock') + if (fromArgs && !TEMPORAL.test(fromArgs)) return { kind: 'place', text: fromArgs } + } + + if (call.name === 'research') { + const fromResult = subjectFromResearchResult(call.result ?? '') + if (fromResult) return { kind: 'subject', text: fromResult } + const raw = String(call.arguments?.query ?? '').trim() + if (raw) { + const focused = focusQuery(raw) + if (focused) return { kind: 'subject', text: focused } + } + } + } + + if (turn.role !== 'user') continue + + if (WEATHER_HINT.test(turn.content) || CLOCK_HINT.test(turn.content)) { + const after = AFTER_PREPOSITION.exec(turn.content)?.[1] + if (after) { + const kind = WEATHER_HINT.test(turn.content) ? 'weather' : 'clock' + const isolated = isolatePlace(after, kind) + if (isolated && !TEMPORAL.test(isolated)) return { kind: 'place', text: isolated } + } + continue + } + + if (isResearchable(turn.content)) { + const focused = focusQuery(turn.content) + if (focused.length >= 2) return { kind: 'subject', text: focused } + } + } + + return null +} + /** * The city a weather or clock call actually resolved, or the one a question named. * - * Tool arguments win: they are what the model asked for. The result header is - * next — `Frankfurt, Germany — 15:45 local` — and a shaped user turn is last, - * so an eval history that only has prose still has something to pin. + * Scans for a place only, so existing callers that ask "where were we" still + * see Frankfurt after a later research turn. `conversationTopic` uses + * `lastEstablished` instead, which is recency across both kinds. */ export function lastEstablishedPlace(turns: readonly TopicTurn[]): string | null { for (const turn of [...turns].toReversed()) { @@ -80,33 +154,54 @@ export function lastEstablishedPlace(turns: readonly TopicTurn[]): string | null return null } +export function lastEstablishedSubject(turns: readonly TopicTurn[]): string | null { + const found = lastEstablished(turns) + return found?.kind === 'subject' ? found.text : null +} + +function subjectFromResearchResult(result: string): string | null { + const match = RESEARCHED_FOR.exec(result) + const subject = match?.[1]?.trim() + return subject ? subject : null +} + /** * One short line, or an empty string when this turn is not owed the place. * * Empty when nothing was established, when the question already names that - * place, when it names a different one, or when it is asking something new. + * subject, when it names a different one, or when it is asking something new. */ export function conversationTopic( query: string, prior: readonly TopicTurn[], options: { skill?: string | null } = {}, ): string { - const place = lastEstablishedPlace(prior) - if (!place) return '' - if (mentionsPlace(query, place)) return '' - - const named = placeNamedInQuery(query) - if (named && !samePlace(named, place)) return '' - - if ( - options.skill === 'weather' || - options.skill === 'world-clock' || - isFollowUp(query) || - REFERS_BACK.test(query) - ) { - return renderTopicBlock(place) + const established = lastEstablished(prior) + if (!established) return '' + if (mentionsTopic(query, established.text)) return '' + + if (established.kind === 'place') { + const named = placeNamedInQuery(query) + if (named && !samePlace(named, established.text)) return '' + + if ( + options.skill === 'weather' || + options.skill === 'world-clock' || + isFollowUp(query) || + REFERS_BACK.test(query) + ) { + return renderTopicBlock(established.text) + } + return '' } - return '' + + // A complete new question — *Wer ist Elon Musk?* after a chancellor turn — + // is research-question too. Pinning only on a follow-up is what keeps the + // last office out of that prompt. *und Elon Musk?* names a new person and + // is silent for the same reason; *und der von Frankreich?* is not a person. + if (!isFollowUp(query)) return '' + if (namesNewResearchSubject(query, established.text)) return '' + return renderTopicBlock(established.text) } export function renderTopicBlock(place: string): string { @@ -132,14 +227,25 @@ function placeFromWeatherResult(result: string): string | null { return city || null } -function mentionsPlace(text: string, place: string): boolean { +function mentionsTopic(text: string, topic: string): boolean { const asked = new Set(tokenize(text)) - const words = tokenize(place) + const words = tokenize(topic) return words.length > 0 && words.every((word) => asked.has(word)) } function samePlace(a: string, b: string): boolean { - return mentionsPlace(a, b) || mentionsPlace(b, a) + return mentionsTopic(a, b) || mentionsTopic(b, a) +} + +function namesNewResearchSubject(query: string, current: string): boolean { + if (PLACE_ONLY_FOLLOW_UP.test(query)) return false + const rest = query + .replace(FOLLOW_UP_PREFIX, '') + .replace(/^[?\s,]+/, '') + .replace(/[?!.]+$/g, '') + .trim() + if (!rest || mentionsTopic(rest, current)) return false + return tokenize(rest).length >= 2 } function placeNamedInQuery(query: string): string | null { diff --git a/src/memory/types.ts b/src/memory/types.ts index a2c8d76..0e59deb 100644 --- a/src/memory/types.ts +++ b/src/memory/types.ts @@ -6,8 +6,9 @@ * semantic (facts), episodic (things that happened) and procedural (how to * behave). Only the durable three are stored here — working memory is the * transcript, which this app already keeps in the tab. The last place a - * weather turn resolved is also pinned into the system prompt by `topic.ts`, - * because a 0.8B model does not reliably read earlier turns. + * weather turn resolved, or the last office a research turn resolved, is also + * pinned into the system prompt by `topic.ts`, because a 0.8B model does not + * reliably read earlier turns. * * The kinds are named in plain words rather than in the paper's terms because * the model has to pick one: `preference` is a word a 0.8B model uses correctly diff --git a/src/skills/research-question/SKILL.md b/src/skills/research-question/SKILL.md index 569b38e..1554b7b 100644 --- a/src/skills/research-question/SKILL.md +++ b/src/skills/research-question/SKILL.md @@ -101,7 +101,7 @@ jarvis: 1. Paris - Wikipedia — https://en.wikipedia.org/wiki/Paris "Paris is the capital and most populous city of France." 2. France - Wikipedia — https://en.wikipedia.org/wiki/France - "The capital is Paris, a major European city and a global centre of art." + "The capital is Paris." answer: | Paris. @@ -117,13 +117,31 @@ jarvis: Researched 2026-09-09 for "population of Tokyo" across 2 sources, all read in full. 1. Tokyo - Wikipedia — https://en.wikipedia.org/wiki/Tokyo - "Tokyo has a population of 13.96 million people in the 23 special wards." + "Tokyo has a population of 13.96 million people." 2. Statistics Bureau — https://www.stat.go.jp/english/ "The prefecture is home to 14 million residents." answer: | About 14 million. Source: https://en.wikipedia.org/wiki/Tokyo + - user: und der von Frankreich? + steps: + - tool: research + arguments: + query: Präsident Frankreich + result: | + Answer: Emmanuel Macron. + + Researched 2026-09-10 for "Präsident Frankreich" across 1 source, all read in full. + + 1. Macron — https://de.wikipedia.org/wiki/Emmanuel_Macron + "Emmanuel Macron ist seit 2017 Staatspräsident Frankreichs." + answer: | + Emmanuel Macron. + + Source: https://de.wikipedia.org/wiki/Emmanuel_Macron --- -Call `research` once. The first line is the answer — copy it, in the language you were asked. Cite the source URLs. +Call `research` once. Copy the first line, in the language you were asked. Cite the source. + +A follow-up like _und der von Frankreich?_ keeps the last office and adds the new place. diff --git a/src/skills/route.test.ts b/src/skills/route.test.ts index 73b1533..7fb6ead 100644 --- a/src/skills/route.test.ts +++ b/src/skills/route.test.ts @@ -301,6 +301,12 @@ describe('keeping a skill across a follow-up', () => { expect(reason('and in Lisbon?', world)).toBe('carried-over') }) + it('keeps research for a follow-up that only names a place', () => { + const research: SkillMemory = { name: 'research-question', carried: 0 } + expect(routed('und der von Frankreich?', research)).toBe('research-question') + expect(reason('und der von Frankreich?', research)).toBe('carried-over') + }) + it('forgets a skill that is no longer installed', () => { expect(routed('and in Lisbon?', { name: 'removed-skill', carried: 0 })).toBeNull() })