Skip to content

Fix/FTS5 preflight coverage, store protection, and a discoverable telemetry opt-out - #172

Merged
Yashasvi2229 merged 8 commits into
mainfrom
fix/fts5-preflight-coverage-and-optout-discoverability
Sep 6, 2026
Merged

Fix/FTS5 preflight coverage, store protection, and a discoverable telemetry opt-out#172
Yashasvi2229 merged 8 commits into
mainfrom
fix/fts5-preflight-coverage-and-optout-discoverability

Conversation

@Yashasvi2229

Copy link
Copy Markdown
Collaborator

What

Finishes the four-part report in #110. PR #168 fixed the first part — mex graph now names the problem when Node's bundled SQLite lacks FTS5 — and this picks up what it scoped out, plus a gap that fix left behind.

  • FTS5 preflight now covers every consumer. fix(graph): fail with an actionable message when Node's SQLite lacks FTS5 #168 guarded only the writable graph open. Readers query nodes_fts / source_chunks_fts, and the wiki index has the same dependency via wiki_fts, so mex check, graph scope/query/get, impact and every mex wiki command could still hit SQLite's raw no such module: fts5. The probe moved from graph/db/database.ts to graph/db/sqlite.ts — it describes the SQLite build, not the code graph, and the wiki architecture rule lets the wiki reach that module but not database.ts. database.ts re-exports it, so existing callers are unchanged.
  • .mex/.gitignore is written by store writers, not only by setup. mex graph rebuild/refresh/repair and mex wiki rebuild-index now ensure it exists before creating anything.
  • mex telemetry disable / enable, writing the same ~/.mex/config.json key as mex config set telemetry on|off. The env opt-outs are named in telemetry --help and in status.
  • COMPATIBILITY.md documents the FTS5 requirement, which the preflight's error message already pointed at.

engines stays >=22.5 — see Why.

Why

Closes #110.

Each remaining item was a way for a correct tool to look broken:

  • Coverage. A store built on one Node and read on another fails on read, not on build. Switching versions with a version manager is enough — no copying between machines required. mex wiki rebuild-index run on its own reproduced the original confusing failure verbatim.
  • Untracked databases. Setup writes the ignore rules at step 2, long before it builds anything, but a store writer invoked on its own never passed through setup. The reporter ran mex graph in a checkout that had never run setup, and their next git add -A would have committed a broken database plus its WAL and SHM files.
  • The opt-out. Turning telemetry off was always one command, but the only switch lived under mex config. The reporter typed mex telemetry disable, got unknown command, read telemetry --help, found two commands that only show things, and then guessed at env var names that didn't exist. They said they audited TELEMETRY.md against src/telemetry/index.ts line by line before running anything, because their work touches medical data — that is exactly the user for whom "where is the off switch" must be a ten-second answer.
  • The dangling pointer. fix(graph): fail with an actionable message when Node's SQLite lacks FTS5 #168's message told users to "see COMPATIBILITY.md for which versions are known to work". That document said nothing about FTS5; it repeated the same >=22.5 range that had just failed them.

On not narrowing engines: FTS5 is a compile-time option that Node neither documents nor guarantees, so availability is a property of the build, not the version number — official installers, distro packages and self-compiled Node can differ at the same version. Setting a floor from one field report would lock out working 22.x/23.x installs while still admitting a broken build above the floor. COMPATIBILITY.md instead carries a one-line command to test the Node you actually run, and the two known data points as reports rather than a supported-range claim. More data points would justify a floor; one does not.

Type of change

  • Bug fix
  • New feature
  • Refactor
  • Docs
  • CI/Tooling

How to test

FTS5 preflight (behaviour is unchanged on any Node that has FTS5, which is the common case):

  1. node --no-warnings -e "new (require('node:sqlite').DatabaseSync)(':memory:').exec('CREATE VIRTUAL TABLE t USING fts5(x)')" && echo "FTS5 ok" — the check now documented in COMPATIBILITY.md.
  2. npx vitest run src/graph/__tests__/database-fts5.test.ts src/wiki/index/__tests__/fts5.test.ts — the probe's failure paths are driven through an injected opener, so they are reachable without an FTS5-less Node. Covers the write path, both graph read paths, and a real, intact wiki index that this engine cannot read.

Store protection, in a scratch repo with no .mex/:

  1. git init, add one .ts file, then run mex graph.
  2. git add -A && git status.mex/.gitignore and your source are staged; graph.db and its sidecars are not. On main, all three databases stage.
  3. npx vitest run test/store-ignore-protection.test.ts

Telemetry:

  1. mex telemetry --help — lists disable, DO_NOT_TRACK=1 and MEX_TELEMETRY=0.
  2. mex telemetry disable, then mex telemetry status — the reason is named and explained. DO_NOT_TRACK=1 mex telemetry enable reports that the env var still outranks what was just written.
  3. npx vitest run test/telemetry-optout-cli.test.ts

Checklist

  • Tests pass (npm test)
  • No breaking changes (or documented below)
  • Tested locally with a real project

On "tests pass": measured as branch-vs-main, not as a green board. The full suite was run on this branch and again in a worktree at main, and the failure sets diffed by name: 69 failures here, 71 on main, with one test failing only here and two only on main. All three pass in isolation — that is parallelism flake, not a regression. The ~70 baseline failures are pre-existing on this Windows machine (the documented findConfig trap: TEMP sits under a home directory that is itself a git repo, so tests asserting branch: null see branch: "main"). npm test also does not run the evaluator tests. CI is the arbiter.

Not breaking, and why each near-miss isn't:

  • assertFts5Available moved from graph/db/database.ts to graph/db/sqlite.ts and is re-exported from its old home, so any existing import keeps working. It gained an optional injected-opener parameter; the default is the previous behaviour.
  • WIKI_INDEX_FTS5_UNAVAILABLE is a new wiki diagnostic code — additive, and registered with the coverage test's emitter. The frozen WikiContractReadError vocabulary is untouched: the read session reuses INDEX_UNAVAILABLE, which already means "this index cannot be read right now".
  • Graph reads on an FTS5-less Node now fail at open with a clear message instead of at the first query with a raw SQLite error. Both are failures; only one is legible.

Tested locally with a real project: two private repositories (111 and 64 indexed source files), with the store redirected to a temp path so neither repository's own .mex/ was opened or written. Both produced node counts identical to their existing live stores (1337 and 1163), and read-only and immutable opens both succeeded with FTS queries returning hits. For the wiki, one real scaffold was copied to scratch (markdown only): rebuild-index indexed 31 entities from 19 files, and wiki query — the FTS5 read path guarded here — returned correct results. Temporary stores deleted after measuring.

Code-graph changes

This touches graph infrastructure (db/sqlite.ts, db/database.ts, maintenance.ts) but adds no extractor, resolver, grammar or schema change, so the extractor-specific items below are marked N/A rather than ticked.

  • This PR targets main
  • A linked issue agrees on the bounded extractor/resolver scope — graph: 'no such module: fts5' on Node 23.10.0 (inside documented engines range) #110
  • The change follows the frozen LanguageExtractor or FrameworkResolver interface — N/A, no extractor or resolver added
  • A focused fixture and assertions for the expected node/edge shape are included — N/A, no extraction behaviour changed; node and edge output is byte-identical, verified against two real corpora
  • Any new grammar WASM, extension mapping, extractor, or resolver is registered — N/A, none added
  • No graph identity, reconciliation, schema, or drift-semantics changes are included

The probe added in #168 guarded only the writable open path. Readers
query nodes_fts / source_chunks_fts, so a store built on an FTS5-capable
machine and copied to one without it still failed with SQLite's raw
'no such module: fts5' from check, scope, query, get, and impact.

Running the probe before the store is opened also means a doomed
environment never creates a handle, rather than opening one and closing
it on the way out.
The wiki index's wiki_fts table has the same FTS5 dependency as the graph
but never probed for it, so `mex wiki rebuild-index` still failed with
SQLite's raw 'no such module: fts5'. It now reports
WIKI_INDEX_FTS5_UNAVAILABLE, a new code, rather than borrowing
WIKI_INDEX_REBUILD_REQUIRED, whose remediation would send the user round
a loop rebuilding an index no rebuild can fix.

The probe moves to sqlite.ts, beside openSqlite: FTS5 describes the
SQLite build rather than the code graph, and the wiki architecture rule
lets the wiki reach that module but not database.ts. database.ts
re-exports it, so existing callers are unchanged. It also takes an
injected opener now, which lets the failure paths be tested without
module mocking that could not reach an internal call anyway.

Both store writers also ensure .mex/.gitignore exists before creating
anything, which previously only mex setup did (#110).
Turning telemetry off has always been one command, but the only switch
lived under `mex config set`. Issue #110's reporter typed
`mex telemetry disable`, got 'unknown command', read `telemetry --help`,
found two commands that only show things, and then guessed at env var
names. The env opt-outs were documented solely in the first-run notice,
which scrolls past once and never returns.

Adds `telemetry disable`/`enable` writing the same ~/.mex/config.json key
as `config set`, lists both env opt-outs in the group's help, and has
`status` say what to change rather than only naming a reason code.
Disable and enable report when an env opt-out or a dev checkout outranks
the value just written, so neither ever claims an outcome the next
invocation contradicts.

No new setting and no change to the gate's precedence.
PR #168's error message told users to 'see COMPATIBILITY.md for which
versions are known to work', but that document said nothing about FTS5 —
it repeated the same >=22.5 range that had just failed them.

COMPATIBILITY.md now has an FTS5 section: what it is needed for, that it
is a compile-time property of the Node build rather than the version
number, a one-line command to test the Node you actually run, and the two
known data points as reports rather than a supported-range claim. The
error message points at that section and stops implying a version answer
exists.

engines stays >=22.5. FTS5 availability does not track version order, so
narrowing the range would lock out working builds without excluding
broken ones, on one field report.

Also records what the v0.6.3 fallback costs: the code graph shipped in
0.7.0, so the release we send Node-constrained users to has none of the
feature they would be falling back for (#110).
Asserts the mechanism (the ignore file exists before the store does) and
the consequence the reporter actually hit (`git add -A` cannot stage the
database), plus that an existing hand-written ignore file keeps its own
rules.
The ignore guard sat before the missing-index refusal, so a mistyped
`mex graph refresh` in an unindexed checkout created .mex/.gitignore and
then errored. Rebuild — the mode that creates a store — still gets the
guard before anything is written.
The build path refused early, but the two entry points that open wiki.db
directly — contract status inspection and the read session — let SQLite's
raw 'no such module: fts5' escape. Reachable without copying anything
between machines: build the index on one Node, switch versions with a
version manager, read it on another.

Status reports degraded rather than corrupt, because nothing is wrong
with the store and no rebuild on this Node would improve it. The read
session reuses the existing INDEX_UNAVAILABLE code, which already means
'this index cannot be read right now' and is exactly true here, so the
frozen read-error vocabulary is unchanged and the actionable detail
rides in the message.

The probe-to-diagnostic mapping moves into its own index/fts5 module.
open.ts is the wrong home once a second layer needs it, and query must
not import an opener to ask a question about the engine.

publish.ts needs no guard: it only runs after a rebuild that already
probed, and reads wiki_meta plus integrity_check, neither of which
touches FTS5.
@Yashasvi2229
Yashasvi2229 merged commit 86723dc into main Sep 6, 2026
9 checks passed
@Yashasvi2229
Yashasvi2229 deleted the fix/fts5-preflight-coverage-and-optout-discoverability branch September 6, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

graph: 'no such module: fts5' on Node 23.10.0 (inside documented engines range)

1 participant