Sharpen the keyword index and stop the answer check inventing citations - #21
Draft
devbadya wants to merge 3 commits into
Draft
Sharpen the keyword index and stop the answer check inventing citations#21devbadya wants to merge 3 commits into
devbadya wants to merge 3 commits into
Conversation
Five everyday messages routed to a skill, and the tests now say so first: 'I work out every morning' reached the calculator, 'in summary, the trip was a success' the page reader, 'heute war ein schöner Tag' the clock, 'ich rechne damit' the calculator again, and 'my sister will find out sooner or later' the web search. Four were keywords an ordinary sentence contains by accident; the last two were triggers, so 'work out' now needs a number in the message and 'find out' no longer matches after an auxiliary. Keywords a skill's own triggers already match are gone too, and a test keeps them gone: unreachable, and they still dilute the idf of the terms that are not. Two more from the same read-through. A skill can decline carry-over with carry: false, which summarize-url does - carried onto a bare follow-up it offers the model a page reader and no page. And the retrieval index is built once per catalogue rather than once per message. Co-authored-by: Sebastian <devbadya@users.noreply.github.com>
Three false positives found by re-reading it. A URL in an answer is only a citation when something was actually fetched - asked for a website, the URL is the answer - so both citation checks now stand down without a source. The evidence comes from the real conversation rather than the composed turns, so an exemplar's URL stays an invention, which is the citation this model is likeliest to get wrong. And a value that only renders in exponent form is not checked at all, where before an answer writing it out in full digits read as wrong. Also the typographic minus, which a model writing prose uses and Number does not: -5 was flagged for saying −5. Co-authored-by: Sebastian <devbadya@users.noreply.github.com>
Co-authored-by: Sebastian <devbadya@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A review of what shipped in #18, written as failing tests first. Nine of them failed, which is the whole content of this PR.
Routing: five everyday sentences reached a skill
arithmeticwork outsummarize-urlsummarycurrent-dateheutearithmeticrechneresearch-questionfind outFour were keywords an ordinary sentence contains by accident. The last two came from triggers, not keywords, so
work outnow requires a number somewhere in the message andfind outno longer matches after an auxiliary (will,might,to). All five are in the routes-to-nothing corpus.Dead keywords, and why they were not merely dead. Four skills carried keywords their own triggers already matched — unreachable, and still counted in the document frequency, so they diluted the inverse document frequency of the terms that could be reached. A test now walks every shipped keyword past its skill's own triggers.
carry: false.summarize-urlneeded a way to decline carry-over: its job is a URL, and carried onto and this one? it handed the model a page reader and no page.The retrieval index is built once per catalogue rather than rebuilt on every message.
The answer check: three false positives
https://1password.comout of an example instead of citing what it fetched was excused by the very check meant to catch it. Evidence now comes from the real conversation, passed in asoptions.evidence.1e+21has no plain-decimal rendering —toFixedkeeps thee— so an answer writing the number out in full read as the wrong number; that check now stands down. And−5(U+2212, which a model writing prose uses andNumbernever does) was flagged for disagreeing with-5.Testing
pnpm checkandpnpm buildare green: 380 tests, 18 new. Every fix above started as a failing test.No UI change, so no new screenshots. Token generation is still not testable here: no WebGPU adapter on the VM and the model has no CPU fallback.