Skip to content

Search assistant replies and checkpoint bodies via the FTS5 index - #385

Merged
jongio merged 1 commit into
mainfrom
fix/deep-search-uses-fts-index
Aug 1, 2026
Merged

Search assistant replies and checkpoint bodies via the FTS5 index#385
jongio merged 1 commit into
mainfrom
fix/deep-search-uses-fts-index

Conversation

@jongio

@jongio jongio commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Searching the session list for a bug or PR number returned nothing when that number only appeared in Copilot's reply rather than in what the user typed.

What was wrong

The session store ships an FTS5 search_index table covering full turn text and every checkpoint field, but the list filter never touched it. Deep search built a LIKE predicate over turns.user_message, checkpoint title and overview, file paths, and refs. Assistant replies, checkpoint history, work done, technical details, and next steps were all unreachable.

SearchSessionsFTS did use the index, but only the Copilot SDK search tool ever called it.

The change

Deep search now also matches s.id IN (SELECT session_id FROM search_index WHERE content MATCH ?) when the index is present.

The index clause is additive rather than a replacement. FTS5 tokenizes, so it misses substrings inside a token: searching 1137 finds bug 1137 but not PR1137, which LIKE catches. Keeping both makes the result set a strict superset of the old behavior.

Stores without the index get an assistant_response LIKE clause instead, so assistant text is searchable there too.

Query terms are wrapped in double quotes before being passed to MATCH, so FTS5 operators (-, *, NEAR, ^) are treated as literals and can't turn a search into a syntax error that breaks the whole list. A whitespace-only query skips the MATCH clause rather than issuing an empty phrase.

Verification

Against a real 20k-session store, searching 1137:

sessions found
before 6
after 13

The recent session that prompted the report (the number appeared only in an assistant reply) is now in the results.

Tests

  • assistant-only match through the index
  • checkpoint body match (history, technical details, files)
  • no-match, so the index clause can't match everything
  • FTS5 operator characters as the whole query
  • grouped list path uses the same predicate
  • filter-builder shape for both the indexed and fallback branches

Searching the session list for a bug or PR number returned nothing when that
number only appeared in Copilot's reply rather than in what the user typed.

The session store ships an FTS5 `search_index` table covering full turn text
and every checkpoint field, but the list filter never touched it. Deep search
built a LIKE predicate over `turns.user_message`, checkpoint title and
overview, file paths, and refs. Assistant replies, checkpoint history, work
done, technical details, and next steps were all unreachable. `SearchSessionsFTS`
did use the index, but only the Copilot SDK search tool called it.

Deep search now also matches `s.id IN (SELECT session_id FROM search_index
WHERE content MATCH ?)` when the index is present.

The index clause is additive rather than a replacement. FTS5 tokenizes, so it
misses substrings inside a token: searching "1137" finds "bug 1137" but not
"PR1137", which LIKE catches. Keeping both makes the result set a strict
superset of the old behavior. Stores without the index get an
`assistant_response` LIKE clause instead, so assistant text is searchable
there too.

Query terms are wrapped in double quotes before being passed to MATCH, so
FTS5 operators (`-`, `*`, `NEAR`, `^`) are treated as literals and can't turn
a search into a syntax error that breaks the whole list. A whitespace-only
query skips the MATCH clause rather than issuing an empty phrase.

Verified against a real 20k-session store: searching "1137" went from 6
sessions to 13.

Tests: assistant-only match, checkpoint body match, no-match, FTS operator
characters, grouped list path, and filter-builder shape for both the indexed
and fallback branches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 345398e7-e955-4014-af51-893af8317955
@jongio
jongio merged commit 9557852 into main Aug 1, 2026
2 checks passed
@jongio
jongio deleted the fix/deep-search-uses-fts-index branch August 1, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant