diff --git a/BACKLOG.md b/BACKLOG.md index db3c04f8..8fe07af6 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -40,7 +40,7 @@ CC-001/CC-002 were consumed by PR #24 fix bundle inline, with no standalone entr | CC-560 | ✅ done | `_gate_scope_reference_index_collect` 每筆 reference 都以 `jq -nc` 建一個 JSON 物件（實測 4.9s×2），與 [[CC-557]] 已修掉的 `_gate_scope_expansion_append` 是同一類寫法；CC-557 未一併處理是因預算餘裕已足，非因不成立 | ops/gate | 2026-08-19 | pr:#511 | P3 | hygiene | | CC-554 | 🔵 active | 永久 regression test 缺少准入門檻：`/ship` 規範「修完每個 finding」但不規範修法形式，reviewer 每提一個邊界就永久長一個阻擋 case，case 又需要 meta-test 保護；QA 規則加六條准入條件＋五條替代路徑，ship.md 加對應例外（明確不設輪數上限，見 [[CC-544]]） | ops/gate | 2026-08-17 | — | P1 | hygiene | | CC-552 | 🔵 active | `test_default_worker_cap` 以 `sleep 0.1` 製造 worker 重疊窗口來驗證併發上限，違反 QA 規則的「不得以 sleep 同步」；主機負載會改變觀測到的重疊數，與 worker-cap 正確性無關（2026-08-17 CC-551 gate round 4 qa-tester，pre-existing） | ops/test | 2026-08-17 | — | P3 | hygiene | -| CC-548 | 🔵 active | context.db FTS5 對 CJK 查詢無索引無排序（[[CC-465]] Requirement 3 殘留）：先 spike 驗 `tokenize='trigram'` 的 sqlite 版本下限與 index rebuild 成本，再決定是否實作 | memory | 2026-08-16 | — | P2 | retrieval | +| CC-548 | ✅ closed 2026-08-26 | **[context.db FTS5 對 CJK 查詢無索引無排序]** Spike 判定 **AMBER — defer，暫不實作**：無 sqlite 版本下限硬衝突，rebuild 遷移成本為零，但實測 trigram rebuild 慢 ~4.9x、索引大 +32.6%，而品質增益在本 repo 實際語料上僅小幅（68 筆抽樣命中中多 1 筆），且原票「unicode61 無 ranking」前提經量測不成立。See `docs/spikes/CC-548.md`. | memory | 2026-08-16 | — | P2 | retrieval | | CC-466 | ⏸ deferred | 記憶卡片生命週期閉環：expires_at 執行 + 關窗式 supersede + usage sidecar 休眠偵測 + doctor→distill 接線；僅在 CC-467 證明 stale/dormant card 已形成實際問題時啟動 | memory | 2026-07-07 | feedback:2026-07-07 | P2 | retrieval | | CC-468 | ⏸ deferred | dispatch brief 帶 memory 約束與信任邊界；完成 CC-465→CC-467 後，僅在 usage evidence 證明有價值時啟動 | ops/memory | 2026-07-07 | — | P2 | retrieval | | CC-011 | 🟢 someday | sync-memory.sh + install 選項：symlink memory 到雲端資料夾實現跨裝置共用 | ux/memory | 2026-05-14 | — | — | — | @@ -473,7 +473,7 @@ pre-existing 缺陷、與 ShellCheck 解析改動無關，且改法本身有風 --- -## CC-548 — context.db FTS5 對 CJK 查詢無索引無排序 🔵 active +## CC-548 — context.db FTS5 對 CJK 查詢無索引無排序（spike）✅ 2026-08-26 **Problem**: [[CC-465]] 修好了注入排序與 prompt/reuse-scan 的 CJK 抽詞，但沒有動 FTS5 索引層。`context.db` 的 FTS5 表使用 unicode61 tokenizer，對整段中文只會產生 @@ -495,10 +495,22 @@ tokenizer 行為「視為與共用 lib 分離的關注點，允許各自的修 可省略而直接重建、重建期間的查詢行為）。 3. Spike 產出 `docs/spikes/CC-548.md` 的 GREEN/AMBER/RED 判定；只有判定為值得做時 才開實作切片，不因票已存在自動實作。 +- Result log: docs/spikes/CC-548.md — **AMBER，defer**。sqlite trigram 版本下限 + （3.34.0）對 `docs/platform-support.md` 宣稱的支援平台無硬衝突（該文件未釘選 + sqlite 最低版本），且 `_ctx_fts5_available()` 既有 probe/fallback idiom 可廉價 + 延伸出三層降級路徑；`_ctx_fts_rebuild()` 本就每次全量 DROP+recreate，切換 + tokenizer 遷移成本為零。但在本 repo 真實語料（64MB、19806 筆 content_fts）實測： + rebuild 慢 ~4.9x（0.87s→4.24s）、索引檔大 +32.6%，查詢期間可能撞見表格不存在 + 的既有 race window 也隨之等比放大（仍為既有缺口，非本票新增）。品質面：5 個真實 + 中文詞抽樣（~68 筆命中）僅 1 筆因 unicode61 把連續中文段落當成單一不可分 token + 而漏收；`bm25()` 在 unicode61 下已有非退化排序——原票「無 ranking」前提經量測不 + 成立。效益真實但目前偏小、成本非零，故未達開實作切片門檻；不因票已存在自動實作。 **Cross-link**: [[CC-465]]（本票承接其 Requirement 3 殘留）、[[CC-340]]（deferred； embeddings/semantic backend——本票是索引層 tokenizer 修正，不是其替代）。 +**See**: `docs/spikes/CC-548.md`（AMBER, defer）。 + --- ## CC-468 — dispatch brief 帶 memory 約束：PM 萃取為 constraints 清單（pointer 僅作 provenance）⏸ deferred diff --git a/docs/spikes/CC-548.md b/docs/spikes/CC-548.md new file mode 100644 index 00000000..3c6595f0 --- /dev/null +++ b/docs/spikes/CC-548.md @@ -0,0 +1,193 @@ +# CC-548 — context.db FTS5 trigram tokenizer for CJK queries (spike result) + +**Status**: complete +**Date**: 2026-08-26 +**Ticket**: BACKLOG.md CC-548 + +## Investigation scope + +1. What sqlite3 version does `tokenize='trigram'` require, and is that floor + acceptable against `docs/platform-support.md`'s declared support matrix + (including the degraded path when trigram is unavailable)? +2. What does rebuilding the existing `context.db` FTS5 index actually cost + (schema version, whether migration can be skipped in favor of a straight + rebuild, query behavior during rebuild)? +3. Does switching to trigram tokenization deliver a real CJK query-quality + improvement worth those costs — GREEN/AMBER/RED, not an automatic green + light to implement regardless of outcome. + +## Angles + +### a1 — sqlite trigram version floor & degraded-path + +`tokenize='trigram'` FTS5 support was introduced in sqlite 3.34.0 +(2020-12-01) — a well-known fact not independently re-verified in this +session against a fetched changelog (no live version-history lookup was +performed; treat as high-confidence but not machine-checked). + +Live smoke test on this machine (the only reliable way to confirm — compile +flags aren't always visible in `--version` output): + +``` +$ sqlite3 --version +3.37.2 2022-01-06 ... +$ sqlite3 :memory: "CREATE VIRTUAL TABLE t USING fts5(x, tokenize='trigram');" +→ succeeds +``` + +apt-installed versions by Ubuntu release (general knowledge, **not verified +live against package changelogs** — medium confidence): + +| Release | sqlite3 (apt) | vs 3.34.0 floor | +|---|---|---| +| 20.04 focal | ~3.31.1 | **below** — trigram unavailable | +| 22.04 jammy | ~3.37.2 | above | +| 24.04 noble | ~3.45.x | above | + +`docs/platform-support.md` declares Linux/WSL2 first-class and documents +`apt install sqlite3` as the install path, but pins **no minimum sqlite3 +version** anywhere — it only requires sqlite3 to exist for FTS5 at all. So +there is no documented floor to violate; the real exposure is a user on +Ubuntu 20.04 LTS via WSL2 (still a plausible install target under the +"WSL2 = Linux" contract), who would fall below the trigram floor while +22.04/24.04 (the more common current default) clear it comfortably. + +Degraded-path cost: `_ctx_fts5_available()` (`runtime/lib/pmctl-context.sh` +~line 158) already implements the exact idiom needed — a live probe +(`CREATE VIRTUAL TABLE IF NOT EXISTS ... USING fts5(x)`), cached per-db in +`_CTX_FTS5_CACHE`, with callers doing `_ctx_fts5_available "$db" || return 0` +to silently fall back to LIKE-scan when FTS5 itself is absent. A sibling +`_ctx_fts5_trigram_available()` probing `tokenize='trigram'` specifically, +feeding a three-tier fallback (trigram → unicode61 FTS5 → LIKE-scan), reuses +this idiom directly. **Structurally trivial if it were needed.** + +### a2 — rebuild cost & migration (measured on real corpus) + +Migration machinery required: **none**. `_ctx_fts_rebuild()` +(`runtime/lib/pmctl-context.sh:623-643`) already does an unconditional +`DROP TABLE IF EXISTS content_fts` + `CREATE VIRTUAL TABLE ... USING +fts5(...)` + two `INSERT ... SELECT` on every call, with zero +incremental/version-aware logic. Both call sites (~line 795 in +`pmctl_context_index`, ~line 1126 in `pmctl_context_update`) invoke it +unconditionally already. `index_meta` (schema_version/extractor_version) +tracks nothing about tokenizer choice — switching the `CREATE VIRTUAL TABLE` +literal to add `tokenize='trigram'` is a one-line change with no schema +bump forced. + +Measured on this project's real corpus +(`/home/screenleon/github/pm-dispatch/.pm-dispatch/ctx/context.db`, 64MB, +1457 files, 4459 symbols, 15347 file_chunks, 19806 content_fts rows), via a +throwaway `/tmp` copy: + +| | unicode61 (current) | trigram | +|---|---|---| +| Rebuild wall time | 0.872s | 4.240s (**~4.9x**) | +| Resulting DB size | 64,053,248 bytes | 84,926,464 bytes (**+32.6%**) | +| content_fts row count | 19806 | 19806 (identical) | + +Query-during-rebuild: `_ctx_fts_rebuild` wraps nothing in `BEGIN`/`COMMIT` — +only `PRAGMA busy_timeout=5000` is set, so each DROP/CREATE/INSERT runs as +its own implicit transaction. This is a **pre-existing gap, not introduced +by this ticket**: a concurrent `pmctl context query` can already observe a +window where `content_fts` doesn't exist or is half-populated. Switching to +trigram widens that window roughly 5x on a corpus this size (sub-second → +up to ~4s) but does not create a new hazard class. + +### a3 — real CJK query-quality delta (measured on real corpus) + +Method: built a parallel `content_fts_tri` table (`tokenize='trigram'`) +from the same `symbols`/`file_chunks` rows as the existing unicode61 +`content_fts`, on a throwaway copy of the real corpus. Picked 5 real +Traditional Chinese multi-character terms confirmed present via LIKE first +(not assumed): 跨專案分析(8), 既有的(23), 第一輪(12), 沿用既有(18), +架構設計討論(3) — counts are LIKE-probe hit totals. + +- **Recall**: identical hit sets for 4/5 terms. For 跨專案分析, unicode61 + returned 7 hits vs trigram's 8 — the missing document + (`BACKLOG-ARCHIVE.md:6123`) contains the term as a substring of an + unbroken run: "...openyida 跨專案分析併入）...". `fts5vocab` confirms + unicode61 tokenizes contiguous CJK text into large multi-character runs at + punctuation/ASCII boundaries (not per-character or per-word) — "跨專案分 + 析併入" is stored as **one indivisible token**, distinct from "跨專案分 + 析". Since FTS5 matches whole tokens, any query term that is a true + substring of a longer unbroken CJK run is silently missed by unicode61. + Reproduced from a freshly rebuilt table (not staleness). +- **Ranking**: both tokenizers produced clearly varying, non-degenerate + `bm25()` scores across hits for every term (e.g. 既有的: unicode61 scores + ranged -8.2 to -3.6; trigram -7.7 to -5.7). **This contradicts the + ticket's Problem-statement assumption that unicode61 "有 no ranking"** — + that assumption does not hold on this corpus; ranking works in both. + The actual failure mode is silent recall loss on long unbroken CJK runs, + not degenerate ranking. +- **Noise**: the one extra trigram hit was a genuine true positive + (contiguous substring), no false-positive noise observed in this sample. + +Recall gap is real but small in this 5-term sample (1 missed doc out of +~68 total hits), because this repo's CJK text is heavily punctuation- and +list-item-delimited, which keeps most contiguous CJK runs short enough to +already equal common query terms. The failure mode scales with prose +density / longer unbroken CJK spans, not with corpus size — it would matter +more on prose-heavy docs than on BACKLOG-style bulleted entries. + +## Findings + +- No hard version-floor conflict: `docs/platform-support.md` pins no sqlite + minimum; only Ubuntu 20.04 (a still-plausible but less common WSL2 base) + would fall short of the 3.34.0 trigram floor. A three-tier fallback + (trigram → unicode61 → LIKE) is structurally trivial to add on top of the + existing `_ctx_fts5_available()` idiom. +- Rebuild migration cost is **zero** — the rebuild path is already an + unconditional full DROP+recreate today; switching tokenizers is a + one-line DDL change. +- Rebuild **runtime and storage cost are real and measured**: ~4.9x slower + (0.87s → 4.24s on this repo's real 64MB corpus) and +32.6% index size, + plus a ~5x wider (still sub-5-second) query-during-rebuild race window + that already exists today and is not newly introduced. +- Quality delta is **real but modest on this project's actual corpus**: 1 + extra recall hit out of ~68 sampled hits, concentrated in long unbroken + CJK runs (rare in this repo's punctuation-delimited style). The original + ticket's premise that unicode61 produces degenerate/absent ranking is + **not supported by measurement** — bm25() ranks non-degenerately under + unicode61 today; the real defect is narrower (a token-boundary recall + gap) than the Problem statement described. + +## Recommendation + +**AMBER — defer, do not implement now.** + +The benefit is real but narrow and currently small on this project's own +corpus (a token-boundary recall gap affecting long unbroken CJK spans, +not a ranking failure as originally assumed). The cost is real and +non-trivial relative to that benefit: ~5x slower rebuilds, ~33% larger +index files, and a proportionally wider (if still sub-5-second) window +where concurrent queries could see a missing or partial `content_fts` +table. Neither side is a hard blocker (no version-floor conflict per +platform-support.md; migration cost is zero since rebuild is already +unconditional), which is why this is AMBER rather than RED — but per the +ticket's own instruction, a verdict color is **not** an automatic green +light. Given the measured cost/benefit ratio on the actual repo corpus +today, opening an implementation slice now is not justified. + +## Open risks + +- The recall-gap failure mode scales with CJK prose density, which this + spike measured on BACKLOG-style bulleted text (low density). A + prose-heavy corpus (e.g. long-form docs, chunked file content in + `file_chunks`) could show a materially larger gap than the 1/68 sample + here — this spike does not measure that case directly. +- The sqlite 3.34.0 version floor and Ubuntu apt-package version table in + Angle a1 were not independently re-verified via a live changelog/package + archive fetch in this session; treat as high-confidence general knowledge, + not a machine-checked fact. +- The pre-existing query-during-rebuild race (no `BEGIN`/`COMMIT` wrapping + `_ctx_fts_rebuild`) is unrelated to this ticket but was surfaced here; + it is not tracked by any ticket today. + +## Next tasks + +None opened. Per the AMBER verdict, no implementation slice is justified +at this time. Revisit if either becomes true: (a) usage evidence shows the +token-boundary recall gap actually causing missed `pmctl context query` +results on real (non-bulleted) prose content, or (b) the untracked +query-during-rebuild race from Open risks needs its own fix regardless of +tokenizer choice.