screenleon · screenleon · Aug 26, 2026 · Aug 26, 2026
diff --git a/BACKLOG.md b/BACKLOG.md
@@ -40,7 +40,7 @@ CC-001/CC-002 were consumed by PR #24 fix bundle inline, with no standalone entr
 | CC-560 | ✅ done | `_gate_scope_reference_index_collect` 每筆 reference 都以 `jq -nc` 建一個 JSON 物件（實測 4.9s×2），與 [[CC-557]] 已修掉的 `_gate_scope_expansion_append` 是同一類寫法；CC-557 未一併處理是因預算餘裕已足，非因不成立 | ops/gate | 2026-08-19 | pr:#511 | P3 | hygiene |
 | CC-554 | 🔵 active | 永久 regression test 缺少准入門檻：`/ship` 規範「修完每個 finding」但不規範修法形式，reviewer 每提一個邊界就永久長一個阻擋 case，case 又需要 meta-test 保護；QA 規則加六條准入條件＋五條替代路徑，ship.md 加對應例外（明確不設輪數上限，見 [[CC-544]]） | ops/gate | 2026-08-17 | — | P1 | hygiene |
 | CC-552 | 🔵 active | `test_default_worker_cap` 以 `sleep 0.1` 製造 worker 重疊窗口來驗證併發上限，違反 QA 規則的「不得以 sleep 同步」；主機負載會改變觀測到的重疊數，與 worker-cap 正確性無關（2026-08-17 CC-551 gate round 4 qa-tester，pre-existing） | ops/test | 2026-08-17 | — | P3 | hygiene |
-| CC-548 | 🔵 active | context.db FTS5 對 CJK 查詢無索引無排序（[[CC-465]] Requirement 3 殘留）：先 spike 驗 `tokenize='trigram'` 的 sqlite 版本下限與 index rebuild 成本，再決定是否實作 | memory | 2026-08-16 | — | P2 | retrieval |
+| CC-548 | ✅ closed 2026-08-26 | **[context.db FTS5 對 CJK 查詢無索引無排序]** Spike 判定 **AMBER — defer，暫不實作**：無 sqlite 版本下限硬衝突，rebuild 遷移成本為零，但實測 trigram rebuild 慢 ~4.9x、索引大 +32.6%，而品質增益在本 repo 實際語料上僅小幅（68 筆抽樣命中中多 1 筆），且原票「unicode61 無 ranking」前提經量測不成立。See `docs/spikes/CC-548.md`. | memory | 2026-08-16 | — | P2 | retrieval |
 | CC-466 | ⏸ deferred | 記憶卡片生命週期閉環：expires_at 執行 + 關窗式 supersede + usage sidecar 休眠偵測 + doctor→distill 接線；僅在 CC-467 證明 stale/dormant card 已形成實際問題時啟動 | memory | 2026-07-07 | feedback:2026-07-07 | P2 | retrieval |
 | CC-468 | ⏸ deferred | dispatch brief 帶 memory 約束與信任邊界；完成 CC-465→CC-467 後，僅在 usage evidence 證明有價值時啟動 | ops/memory | 2026-07-07 | — | P2 | retrieval |
 | CC-011 | 🟢 someday | sync-memory.sh + install 選項：symlink memory 到雲端資料夾實現跨裝置共用 | ux/memory | 2026-05-14 | — | — | — |
@@ -473,7 +473,7 @@ pre-existing 缺陷、與 ShellCheck 解析改動無關，且改法本身有風
 
 ---
 
-## CC-548 — context.db FTS5 對 CJK 查詢無索引無排序 🔵 active
+## CC-548 — context.db FTS5 對 CJK 查詢無索引無排序（spike）✅ 2026-08-26
 
 **Problem**: [[CC-465]] 修好了注入排序與 prompt/reuse-scan 的 CJK 抽詞，但沒有動
 FTS5 索引層。`context.db` 的 FTS5 表使用 unicode61 tokenizer，對整段中文只會產生
@@ -495,10 +495,22 @@ tokenizer 行為「視為與共用 lib 分離的關注點，允許各自的修
    可省略而直接重建、重建期間的查詢行為）。
 3. Spike 產出 `docs/spikes/CC-548.md` 的 GREEN/AMBER/RED 判定；只有判定為值得做時
    才開實作切片，不因票已存在自動實作。
+- Result log: docs/spikes/CC-548.md — **AMBER，defer**。sqlite trigram 版本下限
+  （3.34.0）對 `docs/platform-support.md` 宣稱的支援平台無硬衝突（該文件未釘選
+  sqlite 最低版本），且 `_ctx_fts5_available()` 既有 probe/fallback idiom 可廉價
+  延伸出三層降級路徑；`_ctx_fts_rebuild()` 本就每次全量 DROP+recreate，切換
+  tokenizer 遷移成本為零。但在本 repo 真實語料（64MB、19806 筆 content_fts）實測：
+  rebuild 慢 ~4.9x（0.87s→4.24s）、索引檔大 +32.6%，查詢期間可能撞見表格不存在
+  的既有 race window 也隨之等比放大（仍為既有缺口，非本票新增）。品質面：5 個真實
+  中文詞抽樣（~68 筆命中）僅 1 筆因 unicode61 把連續中文段落當成單一不可分 token
+  而漏收；`bm25()` 在 unicode61 下已有非退化排序——原票「無 ranking」前提經量測不
+  成立。效益真實但目前偏小、成本非零，故未達開實作切片門檻；不因票已存在自動實作。
 
 **Cross-link**: [[CC-465]]（本票承接其 Requirement 3 殘留）、[[CC-340]]（deferred；
 embeddings/semantic backend——本票是索引層 tokenizer 修正，不是其替代）。
 
+**See**: `docs/spikes/CC-548.md`（AMBER, defer）。
+
 ---
 
 ## CC-468 — dispatch brief 帶 memory 約束：PM 萃取為 constraints 清單（pointer 僅作 provenance）⏸ deferred

diff --git a/docs/spikes/CC-548.md b/docs/spikes/CC-548.md
@@ -0,0 +1,193 @@
+# CC-548 — context.db FTS5 trigram tokenizer for CJK queries (spike result)
+
+**Status**: complete
+**Date**: 2026-08-26
+**Ticket**: BACKLOG.md CC-548
+
+## Investigation scope
+
+1. What sqlite3 version does `tokenize='trigram'` require, and is that floor
+   acceptable against `docs/platform-support.md`'s declared support matrix
+   (including the degraded path when trigram is unavailable)?
+2. What does rebuilding the existing `context.db` FTS5 index actually cost
+   (schema version, whether migration can be skipped in favor of a straight
+   rebuild, query behavior during rebuild)?
+3. Does switching to trigram tokenization deliver a real CJK query-quality
+   improvement worth those costs — GREEN/AMBER/RED, not an automatic green
+   light to implement regardless of outcome.
+
+## Angles
+
+### a1 — sqlite trigram version floor & degraded-path
+
+`tokenize='trigram'` FTS5 support was introduced in sqlite 3.34.0
+(2020-12-01) — a well-known fact not independently re-verified in this
+session against a fetched changelog (no live version-history lookup was
+performed; treat as high-confidence but not machine-checked).
+
+Live smoke test on this machine (the only reliable way to confirm — compile
+flags aren't always visible in `--version` output):
+
+```
+$ sqlite3 --version
+3.37.2 2022-01-06 ...
+$ sqlite3 :memory: "CREATE VIRTUAL TABLE t USING fts5(x, tokenize='trigram');"
+→ succeeds
+```
+
+apt-installed versions by Ubuntu release (general knowledge, **not verified
+live against package changelogs** — medium confidence):
+
+| Release | sqlite3 (apt) | vs 3.34.0 floor |
+|---|---|---|
+| 20.04 focal | ~3.31.1 | **below** — trigram unavailable |
+| 22.04 jammy | ~3.37.2 | above |
+| 24.04 noble | ~3.45.x | above |
+
+`docs/platform-support.md` declares Linux/WSL2 first-class and documents
+`apt install sqlite3` as the install path, but pins **no minimum sqlite3
+version** anywhere — it only requires sqlite3 to exist for FTS5 at all. So
+there is no documented floor to violate; the real exposure is a user on
+Ubuntu 20.04 LTS via WSL2 (still a plausible install target under the
+"WSL2 = Linux" contract), who would fall below the trigram floor while
+22.04/24.04 (the more common current default) clear it comfortably.
+
+Degraded-path cost: `_ctx_fts5_available()` (`runtime/lib/pmctl-context.sh`
+~line 158) already implements the exact idiom needed — a live probe
+(`CREATE VIRTUAL TABLE IF NOT EXISTS ... USING fts5(x)`), cached per-db in
+`_CTX_FTS5_CACHE`, with callers doing `_ctx_fts5_available "$db" || return 0`
+to silently fall back to LIKE-scan when FTS5 itself is absent. A sibling
+`_ctx_fts5_trigram_available()` probing `tokenize='trigram'` specifically,
+feeding a three-tier fallback (trigram → unicode61 FTS5 → LIKE-scan), reuses
+this idiom directly. **Structurally trivial if it were needed.**
+
+### a2 — rebuild cost & migration (measured on real corpus)
+
+Migration machinery required: **none**. `_ctx_fts_rebuild()`
+(`runtime/lib/pmctl-context.sh:623-643`) already does an unconditional
+`DROP TABLE IF EXISTS content_fts` + `CREATE VIRTUAL TABLE ... USING
+fts5(...)` + two `INSERT ... SELECT` on every call, with zero
+incremental/version-aware logic. Both call sites (~line 795 in
+`pmctl_context_index`, ~line 1126 in `pmctl_context_update`) invoke it
+unconditionally already. `index_meta` (schema_version/extractor_version)
+tracks nothing about tokenizer choice — switching the `CREATE VIRTUAL TABLE`
+literal to add `tokenize='trigram'` is a one-line change with no schema
+bump forced.
+
+Measured on this project's real corpus
+(`/home/screenleon/github/pm-dispatch/.pm-dispatch/ctx/context.db`, 64MB,
+1457 files, 4459 symbols, 15347 file_chunks, 19806 content_fts rows), via a
+throwaway `/tmp` copy:
+
+| | unicode61 (current) | trigram |
+|---|---|---|
+| Rebuild wall time | 0.872s | 4.240s (**~4.9x**) |
+| Resulting DB size | 64,053,248 bytes | 84,926,464 bytes (**+32.6%**) |
+| content_fts row count | 19806 | 19806 (identical) |
+
+Query-during-rebuild: `_ctx_fts_rebuild` wraps nothing in `BEGIN`/`COMMIT` —
+only `PRAGMA busy_timeout=5000` is set, so each DROP/CREATE/INSERT runs as
+its own implicit transaction. This is a **pre-existing gap, not introduced
+by this ticket**: a concurrent `pmctl context query` can already observe a
+window where `content_fts` doesn't exist or is half-populated. Switching to
+trigram widens that window roughly 5x on a corpus this size (sub-second →
+up to ~4s) but does not create a new hazard class.
+
+### a3 — real CJK query-quality delta (measured on real corpus)
+
+Method: built a parallel `content_fts_tri` table (`tokenize='trigram'`)
+from the same `symbols`/`file_chunks` rows as the existing unicode61
+`content_fts`, on a throwaway copy of the real corpus. Picked 5 real
+Traditional Chinese multi-character terms confirmed present via LIKE first
+(not assumed): 跨專案分析(8), 既有的(23), 第一輪(12), 沿用既有(18),
+架構設計討論(3) — counts are LIKE-probe hit totals.
+
+- **Recall**: identical hit sets for 4/5 terms. For 跨專案分析, unicode61
+  returned 7 hits vs trigram's 8 — the missing document
+  (`BACKLOG-ARCHIVE.md:6123`) contains the term as a substring of an
+  unbroken run: "...openyida 跨專案分析併入）...". `fts5vocab` confirms
+  unicode61 tokenizes contiguous CJK text into large multi-character runs at
+  punctuation/ASCII boundaries (not per-character or per-word) — "跨專案分
+  析併入" is stored as **one indivisible token**, distinct from "跨專案分
+  析". Since FTS5 matches whole tokens, any query term that is a true
+  substring of a longer unbroken CJK run is silently missed by unicode61.
+  Reproduced from a freshly rebuilt table (not staleness).
+- **Ranking**: both tokenizers produced clearly varying, non-degenerate
+  `bm25()` scores across hits for every term (e.g. 既有的: unicode61 scores
+  ranged -8.2 to -3.6; trigram -7.7 to -5.7). **This contradicts the
+  ticket's Problem-statement assumption that unicode61 "有 no ranking"** —
+  that assumption does not hold on this corpus; ranking works in both.
+  The actual failure mode is silent recall loss on long unbroken CJK runs,
+  not degenerate ranking.
+- **Noise**: the one extra trigram hit was a genuine true positive
+  (contiguous substring), no false-positive noise observed in this sample.
+
+Recall gap is real but small in this 5-term sample (1 missed doc out of
+~68 total hits), because this repo's CJK text is heavily punctuation- and
+list-item-delimited, which keeps most contiguous CJK runs short enough to
+already equal common query terms. The failure mode scales with prose
+density / longer unbroken CJK spans, not with corpus size — it would matter
+more on prose-heavy docs than on BACKLOG-style bulleted entries.
+
+## Findings
+
+- No hard version-floor conflict: `docs/platform-support.md` pins no sqlite
+  minimum; only Ubuntu 20.04 (a still-plausible but less common WSL2 base)
+  would fall short of the 3.34.0 trigram floor. A three-tier fallback
+  (trigram → unicode61 → LIKE) is structurally trivial to add on top of the
+  existing `_ctx_fts5_available()` idiom.
+- Rebuild migration cost is **zero** — the rebuild path is already an
+  unconditional full DROP+recreate today; switching tokenizers is a
+  one-line DDL change.
+- Rebuild **runtime and storage cost are real and measured**: ~4.9x slower
+  (0.87s → 4.24s on this repo's real 64MB corpus) and +32.6% index size,
+  plus a ~5x wider (still sub-5-second) query-during-rebuild race window
+  that already exists today and is not newly introduced.
+- Quality delta is **real but modest on this project's actual corpus**: 1
+  extra recall hit out of ~68 sampled hits, concentrated in long unbroken
+  CJK runs (rare in this repo's punctuation-delimited style). The original
+  ticket's premise that unicode61 produces degenerate/absent ranking is
+  **not supported by measurement** — bm25() ranks non-degenerately under
+  unicode61 today; the real defect is narrower (a token-boundary recall
+  gap) than the Problem statement described.
+
+## Recommendation
+
+**AMBER — defer, do not implement now.**
+
+The benefit is real but narrow and currently small on this project's own
+corpus (a token-boundary recall gap affecting long unbroken CJK spans,
+not a ranking failure as originally assumed). The cost is real and
+non-trivial relative to that benefit: ~5x slower rebuilds, ~33% larger
+index files, and a proportionally wider (if still sub-5-second) window
+where concurrent queries could see a missing or partial `content_fts`
+table. Neither side is a hard blocker (no version-floor conflict per
+platform-support.md; migration cost is zero since rebuild is already
+unconditional), which is why this is AMBER rather than RED — but per the
+ticket's own instruction, a verdict color is **not** an automatic green
+light. Given the measured cost/benefit ratio on the actual repo corpus
+today, opening an implementation slice now is not justified.
+
+## Open risks
+
+- The recall-gap failure mode scales with CJK prose density, which this
+  spike measured on BACKLOG-style bulleted text (low density). A
+  prose-heavy corpus (e.g. long-form docs, chunked file content in
+  `file_chunks`) could show a materially larger gap than the 1/68 sample
+  here — this spike does not measure that case directly.
+- The sqlite 3.34.0 version floor and Ubuntu apt-package version table in
+  Angle a1 were not independently re-verified via a live changelog/package
+  archive fetch in this session; treat as high-confidence general knowledge,
+  not a machine-checked fact.
+- The pre-existing query-during-rebuild race (no `BEGIN`/`COMMIT` wrapping
+  `_ctx_fts_rebuild`) is unrelated to this ticket but was surfaced here;
+  it is not tracked by any ticket today.
+
+## Next tasks
+
+None opened. Per the AMBER verdict, no implementation slice is justified
+at this time. Revisit if either becomes true: (a) usage evidence shows the
+token-boundary recall gap actually causing missed `pmctl context query`
+results on real (non-bulleted) prose content, or (b) the untracked
+query-during-rebuild race from Open risks needs its own fix regardless of
+tokenizer choice.