docs(CC-548): FTS5 trigram-tokenizer spike — AMBER, defer - #538
Merged
Conversation
Per the ticket's own instruction ("不預設要做——先 spike，再決定"), investigated
whether switching context.db's FTS5 tokenizer from unicode61 to trigram is
worth implementing for CJK query quality. Three angles, all measured on the
real repo corpus rather than estimated:
- sqlite version floor (3.34.0) has no conflict with docs/platform-support.md
(which pins no sqlite minimum); a three-tier degraded path (trigram ->
unicode61 -> LIKE) would be structurally trivial on top of the existing
_ctx_fts5_available() probe/cache idiom.
- Rebuild migration cost is zero (the rebuild path is already an
unconditional full DROP+recreate), but rebuild is measurably ~4.9x slower
(0.87s -> 4.24s on this repo's 64MB corpus) and the index grows +32.6%.
- CJK query quality delta is real but narrow: unicode61 tokenizes contiguous
CJK text into large indivisible multi-character runs, silently missing a
query term that's a true substring of a longer run -- but this only
produced 1 extra hit out of ~68 sampled on this repo's actual
punctuation-delimited corpus. The ticket's original "unicode61 has no
ranking" premise did not hold under measurement -- bm25() ranks
non-degenerately today; the real defect is narrower than assumed.
Verdict: AMBER, defer -- benefit is real but small on this corpus today,
cost is real and non-trivial, neither side is a hard blocker. No
implementation slice opened.
Also surfaced (not fixed, not ticketed -- flagged in the spike's Open
risks): _ctx_fts_rebuild() wraps nothing in BEGIN/COMMIT, so a concurrent
query can already observe a window where content_fts doesn't exist or is
half-populated. Pre-existing, unrelated to this ticket's tokenizer question.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CC-548 was explicitly a spike ("不預設要做——先 spike，再決定"): decide whether switching
context.db's FTS5 tokenizer fromunicode61totrigramis worth implementing for CJK query quality. This PR is the spike result, not an implementation.Verdict: AMBER — defer, do not implement now.
All three measurements were made on the real repo corpus (64MB
context.db), not estimated:docs/platform-support.md(pins no sqlite minimum). A three-tier degraded path would be cheap to add on the existing_ctx_fts5_available()idiom.bm25()ranks non-degenerately today.No implementation slice opened, per the ticket's own instruction that a verdict color isn't an automatic green light.
Also surfaced, not fixed or ticketed (documented in the spike's Open risks):
_ctx_fts_rebuild()wraps nothing inBEGIN/COMMIT, so a concurrent query can observe a window wherecontent_ftsdoesn't exist or is half-populated. Pre-existing, unrelated to this ticket's tokenizer question — flagged for visibility, not scope-crept into a fix here.Verification
bash pm/scripts/validate.sh BACKLOG.md— exit 0tests/shell/test-pmctl-backlog.sh— 18 passed, 0 failed🤖 Generated with Claude Code
https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx