Skip to content

Sharpen the keyword index and stop the answer check inventing citations - #21

Draft
devbadya wants to merge 3 commits into
mainfrom
cursor/sharpen-skill-index-and-answer-check-0deb
Draft

Sharpen the keyword index and stop the answer check inventing citations#21
devbadya wants to merge 3 commits into
mainfrom
cursor/sharpen-skill-index-and-answer-check-0deb

Conversation

@devbadya

Copy link
Copy Markdown
Owner

A review of what shipped in #18, written as failing tests first. Nine of them failed, which is the whole content of this PR.

Routing: five everyday sentences reached a skill

Message Routed to Because
I work out every morning before breakfast arithmetic keyword work out
In summary, the trip was a success summarize-url keyword summary
Heute war ein wirklich schöner Tag current-date keyword heute
Ich rechne damit, dass es morgen klappt arithmetic keyword rechne
My sister will find out sooner or later research-question trigger find out

Four were keywords an ordinary sentence contains by accident. The last two came from triggers, not keywords, so work out now requires a number somewhere in the message and find out no longer matches after an auxiliary (will, might, to). All five are in the routes-to-nothing corpus.

Dead keywords, and why they were not merely dead. Four skills carried keywords their own triggers already matched — unreachable, and still counted in the document frequency, so they diluted the inverse document frequency of the terms that could be reached. A test now walks every shipped keyword past its skill's own triggers.

carry: false. summarize-url needed a way to decline carry-over: its job is a URL, and carried onto and this one? it handed the model a page reader and no page.

The retrieval index is built once per catalogue rather than rebuilt on every message.

The answer check: three false positives

  • A URL is only a citation when something was fetched. Asked what is Anthropic's website, the URL is the answer, and the check was flagging it as an invented source. Both citation checks now stand down when no tool returned a source.
  • An exemplar's URL was counted as evidence. The turns the model is sent begin with a skill's worked examples, so a model repeating https://1password.com out of an example instead of citing what it fetched was excused by the very check meant to catch it. Evidence now comes from the real conversation, passed in as options.evidence.
  • Exponent values and the typographic minus. 1e+21 has no plain-decimal rendering — toFixed keeps the e — so an answer writing the number out in full read as the wrong number; that check now stands down. And −5 (U+2212, which a model writing prose uses and Number never does) was flagged for disagreeing with -5.

Testing

pnpm check and pnpm build are green: 380 tests, 18 new. Every fix above started as a failing test.

messages the review caught routing somewhere they had no business going
message                                     was                   now
I work out every morning before breakfast   matched "work out"    — none —
In summary, the trip was a success          matched "summary"     — none —
Heute war ein wirklich schöner Tag          matched "heute"       — none —
Ich rechne damit, dass es morgen klappt     matched "rechne"      — none —
My sister will find out sooner or later     matched "find out"    — none —

requests that still have to reach their skill
Work out the tip on 45 euros                arithmetic (trigger)
Berechne 18 Prozent von 2450                arithmetic (search: berechne)
Fasse mir die Seite zusammen                summarize-url (search: seite zusammen)
Find out who won the league                 research-question (trigger)
Wie warm wird es morgen in Rom?             weather (search: wie warm)

carry-over, and the skill that refuses it
and in Lisbon?  (weather resident)          weather (carried-over)
and this one?   (summarize-url resident)    — none —

No UI change, so no new screenshots. Token generation is still not testable here: no WebGPU adapter on the VM and the model has no CPU fallback.

Open in Web Open in Cursor 

cursoragent and others added 3 commits August 24, 2026 16:10
Five everyday messages routed to a skill, and the tests now say so first:
'I work out every morning' reached the calculator, 'in summary, the trip was a
success' the page reader, 'heute war ein schöner Tag' the clock, 'ich rechne
damit' the calculator again, and 'my sister will find out sooner or later' the
web search. Four were keywords an ordinary sentence contains by accident; the
last two were triggers, so 'work out' now needs a number in the message and
'find out' no longer matches after an auxiliary.

Keywords a skill's own triggers already match are gone too, and a test keeps
them gone: unreachable, and they still dilute the idf of the terms that are not.

Two more from the same read-through. A skill can decline carry-over with
carry: false, which summarize-url does - carried onto a bare follow-up it offers
the model a page reader and no page. And the retrieval index is built once per
catalogue rather than once per message.

Co-authored-by: Sebastian <devbadya@users.noreply.github.com>
Three false positives found by re-reading it. A URL in an answer is only a
citation when something was actually fetched - asked for a website, the URL is
the answer - so both citation checks now stand down without a source. The
evidence comes from the real conversation rather than the composed turns, so an
exemplar's URL stays an invention, which is the citation this model is likeliest
to get wrong. And a value that only renders in exponent form is not checked at
all, where before an answer writing it out in full digits read as wrong.

Also the typographic minus, which a model writing prose uses and Number does
not: -5 was flagged for saying −5.

Co-authored-by: Sebastian <devbadya@users.noreply.github.com>
Co-authored-by: Sebastian <devbadya@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants