diff --git a/CHANGELOG.md b/CHANGELOG.md index 409eb4ce..912f7f20 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,13 @@ All notable changes to this project will be documented in this file. ### Fixed - `mex graph` now fails with an actionable message naming the running Node version when the built-in `node:sqlite` module lacks FTS5 support, instead of surfacing SQLite's raw `no such module: fts5` on the first schema statement that needs it. FTS5 availability is not guaranteed by every Node build/version inside the documented `engines` range (#110). +- The FTS5 preflight now covers every consumer, not only `mex graph`'s writable open: read-only and immutable graph opens (`mex check`, `graph scope`/`query`/`get`, `impact`) and the wiki index, whose `wiki_fts` table has the same dependency. `mex wiki rebuild-index` reports the new `WIKI_INDEX_FTS5_UNAVAILABLE` diagnostic rather than `WIKI_INDEX_REBUILD_REQUIRED`, which would have sent users round a loop rebuilding an index no rebuild can fix (#110). +- The wiki index's two direct read paths — contract status inspection and the read session — also preflight FTS5 now, instead of letting SQLite's raw error escape. Reachable by building the index on one Node and reading it on another (#110). +- COMPATIBILITY.md documents the FTS5 requirement, a one-line command to check the Node you actually run, and that the v0.6.3 fallback predates the code graph. The preflight's error message pointed at a document that said nothing about FTS5 (#110). +- `mex graph rebuild`/`refresh`/`repair` and `mex wiki rebuild-index` now ensure `.mex/.gitignore` exists before creating a store. Only `mex setup` did this, so building a store in a checkout that had never run setup left `graph.db`, `-wal` and `-shm` untracked, ready for the next `git add -A` to commit (#110). + +### Added +- `mex telemetry disable` and `mex telemetry enable`, writing the same `~/.mex/config.json` key as `mex config set telemetry on|off`. `mex telemetry --help` and `mex telemetry status` now name the `DO_NOT_TRACK=1` and `MEX_TELEMETRY=0` env opt-outs and say which one is in effect; previously the only switch lived under `config` and the env vars appeared solely in the first-run notice (#110). ## [0.8.0] - 2026-09-02 diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index be950ac7..9dc067b4 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -2,7 +2,47 @@ ## Runtime requirement -mex 0.8.x requires Node.js 22.5 or newer. The code graph uses the built-in `node:sqlite` module; older Node releases are unsupported. Users who cannot upgrade Node can remain on mex v0.6.3, which supports Node.js 20 or newer. +mex 0.8.x requires Node.js 22.5 or newer. The code graph and the wiki index use +the built-in `node:sqlite` module; older Node releases are unsupported. + +Users who cannot upgrade Node can remain on mex v0.6.3, which supports Node.js +20 or newer. Note what that costs: the code graph shipped in 0.7.0, so v0.6.3 +has no `mex graph`, no `mex impact`, and no code-node grounding. It is a +scaffold-and-drift-checking release, not an older version of the same feature +set. + +### SQLite FTS5 + +**A supported Node version is necessary but not sufficient.** Both databases +need SQLite's FTS5 full-text extension, and `node:sqlite` embeds whatever +SQLite the Node binary was built with. FTS5 is a compile-time option that Node +does not document or guarantee, so whether you have it depends on the *build*, +not the version number alone — official builds, distro packages, and +self-compiled Node can differ at the same version. + +Check the Node you actually run in one command: + +```console +$ node --no-warnings -e "new (require('node:sqlite').DatabaseSync)(':memory:').exec('CREATE VIRTUAL TABLE t USING fts5(x)')" && echo "FTS5 ok" +``` + +Silence plus `FTS5 ok` means you are fine. `no such module: fts5` means that +Node build cannot run the graph or the wiki index; install a different build or +version of Node. mex preflights this itself, so `mex graph` and +`mex wiki rebuild-index` name the problem and your Node version rather than +failing with a bare SQLite error. + +Known data points, which are reports rather than a supported-range claim: + +| Node | Platform | FTS5 | +|---|---|---| +| 23.10.0 | Windows 11 | missing ([#110](https://github.com/mex-memory/mex/issues/110)) | +| 24.11.0 | Windows 11 | present | + +`engines` stays at `>=22.5`: FTS5 does not track version order, so narrowing +the range would lock out working builds without excluding broken ones. If you +hit a build without it, please add it to the table via issue #110 — the sample +is small, and that is the only thing that would justify a floor. This document defines `mex-agent`'s public contract: what's stable, what isn't, and what counts as a breaking change. It is intended for embedders — tools that diff --git a/src/cli.ts b/src/cli.ts index b6fa89a6..4063ddd3 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1040,7 +1040,74 @@ program // ── Telemetry ── const telemetryCmd = program .command("telemetry") - .description("Telemetry transparency commands"); + .description("Telemetry transparency commands, including the opt-out") + .addHelpText( + "after", + "\nOpting out:\n" + + " mex telemetry disable Turn telemetry off for every project (~/.mex/config.json)\n" + + " DO_NOT_TRACK=1 Standard cross-tool opt-out, honoured per invocation\n" + + " MEX_TELEMETRY=0 mex-specific env opt-out, honoured per invocation\n" + + "\nEnvironment variables win over the stored setting. `mex telemetry status`\n" + + "reports which one is in effect.\n", + ); + +/** Explain an opt-out reason in terms of the thing the user would have to change. */ +function describeTelemetryReason(reason: string | undefined): string { + switch (reason) { + case "DO_NOT_TRACK": + return "The DO_NOT_TRACK environment variable is set to 1."; + case "MEX_TELEMETRY": + return "The MEX_TELEMETRY environment variable is set to 0."; + case "dev": + return "This is a mex development checkout; telemetry never runs from one."; + case "config": + return "Stored in ~/.mex/config.json. Re-enable with `mex telemetry enable`."; + default: + return ""; + } +} + +/** + * `telemetry disable` / `enable` write the same `~/.mex/config.json` key as + * `mex config set telemetry off|on`. + * + * The duplication is the point. Issue #110 reported reaching for + * `mex telemetry disable`, getting `unknown command`, and then guessing at env + * var names — because the only switch lived under `config`, which is not where + * anyone looks for it. An alias costs nothing; a user who cannot find the + * opt-out costs trust. + */ +function setTelemetryEnabled(enabled: boolean): void { + try { + setGlobalConfigKey("telemetry", enabled ? "on" : "off"); + } catch (err) { + console.error((err as Error).message); + process.exit(1); + } + console.log(`Telemetry ${enabled ? "enabled" : "disabled"} in ~/.mex/config.json`); + + // Never claim an outcome the next invocation will contradict: an env opt-out + // outranks the stored value, and a dev checkout outranks both. + const active = isEnabled(); + if (active.enabled !== enabled) { + const detail = describeTelemetryReason(active.reason); + console.log( + active.enabled + ? "Telemetry is still on for this project." + : `Telemetry stays off regardless of this setting. ${detail}`.trim(), + ); + } +} + +telemetryCmd + .command("disable") + .description("Turn telemetry off for every project (writes ~/.mex/config.json)") + .action(() => setTelemetryEnabled(false)); + +telemetryCmd + .command("enable") + .description("Turn telemetry back on for every project") + .action(() => setTelemetryEnabled(true)); telemetryCmd .command("inspect") @@ -1073,9 +1140,12 @@ telemetryCmd const result = isEnabled(); if (result.enabled) { console.log("Telemetry: enabled"); - } else { - console.log(`Telemetry: disabled (reason: ${result.reason})`); + console.log("Turn it off with `mex telemetry disable`, DO_NOT_TRACK=1, or MEX_TELEMETRY=0."); + return; } + console.log(`Telemetry: disabled (reason: ${result.reason})`); + const detail = describeTelemetryReason(result.reason); + if (detail) console.log(detail); }); // ── Config ── @@ -1175,6 +1245,7 @@ program console.log(" mex watch Install post-commit hook for auto drift score"); console.log(" mex watch --interval Run heartbeat every 30 minutes (or config value)"); console.log(" mex watch --uninstall Remove the post-commit hook"); + console.log(" mex telemetry disable Turn telemetry off (or DO_NOT_TRACK=1 / MEX_TELEMETRY=0)"); console.log(" mex telemetry inspect Show the exact telemetry payload (without sending)"); console.log(" mex telemetry status Show telemetry enabled/disabled and reason"); console.log(" mex config set Set a global config value (e.g. telemetry off)"); diff --git a/src/graph/__tests__/database-fts5.test.ts b/src/graph/__tests__/database-fts5.test.ts index 56b4f7c1..a406cba4 100644 --- a/src/graph/__tests__/database-fts5.test.ts +++ b/src/graph/__tests__/database-fts5.test.ts @@ -27,21 +27,12 @@ function fakeDb(execImpl: (sql: string) => void): SqliteDatabase { } /** - * Load a fresh `database.js` whose `assertFts5Available` probes against - * `fakeExec` instead of a real `:memory:` connection, by mocking the - * `openSqlite` it imports from `sqlite.js`. `assertFts5Available` no longer - * takes a `SqliteDatabase` parameter (PR #168 review: it must not touch the - * caller's real graph database) — it opens its own throwaway connection - * internally, so exercising the error paths now goes through this module - * mock rather than an injected fake. + * The probe takes an injected opener precisely so its failure paths are + * reachable without an FTS5-less Node build — and without module mocking, which + * cannot reach a call `sqlite.ts` makes to its own `openSqlite`. */ -async function assertFts5AvailableWith(fakeExec: (sql: string) => void) { - vi.resetModules(); - vi.doMock("../db/sqlite.js", () => ({ - openSqlite: () => fakeDb(fakeExec), - })); - const fresh = await import("../db/database.js"); - return fresh.assertFts5Available; +function failingOpener(execImpl: (sql: string) => void) { + return (() => fakeDb(execImpl)) as unknown as Parameters[0]; } describe("assertFts5Available", () => { @@ -52,23 +43,38 @@ describe("assertFts5Available", () => { expect(() => assertFts5Available()).not.toThrow(); }); - it("raises an actionable, Node-version-specific message on the exact SQLite error from issue #110", async () => { - const probe = await assertFts5AvailableWith(() => { + it("raises an actionable, Node-version-specific message on the exact SQLite error from issue #110", () => { + const probe = failingOpener(() => { throw new Error("no such module: fts5"); }); - expect(() => probe()).toThrowError( + expect(() => assertFts5Available(probe)).toThrowError( new RegExp(`Node \\(${process.version.replace(/[.+]/g, "\\$&")}\\).*FTS5.*no such module: fts5`, "s"), ); }); - it("re-throws an unrelated exec failure unchanged, rather than misattributing it to FTS5", async () => { - const probe = await assertFts5AvailableWith(() => { + it("re-throws an unrelated exec failure unchanged, rather than misattributing it to FTS5", () => { + const probe = failingOpener(() => { throw new Error("database is locked"); }); - expect(() => probe()).toThrowError("database is locked"); - expect(() => probe()).not.toThrow(/FTS5/); + expect(() => assertFts5Available(probe)).toThrowError("database is locked"); + expect(() => assertFts5Available(probe)).not.toThrow(/FTS5/); + }); + + it("closes the throwaway probe connection on both the success and failure paths", () => { + let opened = 0; + let closed = 0; + const counting = (execImpl: (sql: string) => void) => (() => { + opened += 1; + return { ...fakeDb(execImpl), close: () => { closed += 1; } }; + }) as unknown as Parameters[0]; + + assertFts5Available(counting(() => {})); + expect(() => assertFts5Available(counting(() => { + throw new Error("no such module: fts5"); + }))).toThrow(/FTS5/); + expect(closed).toBe(opened); }); }); @@ -91,45 +97,39 @@ describe("openGraphDatabase FTS5 preflight", () => { } }); - it("closes the database handle when the FTS5 preflight fails, instead of leaking it open", async () => { - const root = mkdtempSync(join(tmpdir(), "mex-database-fts5-close-")); + it("never opens the store when the preflight fails, on the write path or either read path", async () => { + const root = mkdtempSync(join(tmpdir(), "mex-database-fts5-guarded-")); roots.push(root); + const dbPath = join(root, "graph.db"); + openGraphDatabase(dbPath).close(); vi.resetModules(); vi.doMock("../db/sqlite.js", async () => { const actual = await vi.importActual("../db/sqlite.js"); - let realGraphDb: SqliteDatabase | undefined; + const storeOpens: string[] = []; return { ...actual, + // Only the probe's own :memory: connection is faked; a real store open + // is recorded so the test can prove it never happened. + assertFts5Available: () => actual.assertFts5Available((() => fakeDb(() => { + throw new Error("no such module: fts5"); + })) as unknown as typeof actual.openSqlite), openSqlite: (path: string, options?: { readOnly?: boolean; immutable?: boolean }) => { - if (path === ":memory:") { - // The FTS5 preflight's own throwaway connection: fail it. - return { - prepare: () => { - throw new Error("not used by this test"); - }, - exec: () => { - throw new Error("no such module: fts5"); - }, - pragma: () => {}, - transaction: (fn: () => T) => fn(), - close: () => {}, - open: true, - } satisfies SqliteDatabase; - } - realGraphDb = actual.openSqlite(path, options); - return realGraphDb; + storeOpens.push(path); + return actual.openSqlite(path, options); }, - __getRealGraphDb: () => realGraphDb, + __storeOpens: () => storeOpens, }; }); const fresh = await import("../db/database.js"); - const sqliteMock = (await import("../db/sqlite.js")) as unknown as { - __getRealGraphDb: () => SqliteDatabase | undefined; - }; + const sqliteMock = (await import("../db/sqlite.js")) as unknown as { __storeOpens: () => string[] }; - expect(() => fresh.openGraphDatabase(join(root, "graph.db"))).toThrow(/FTS5/); - expect(sqliteMock.__getRealGraphDb()?.open).toBe(false); + for (const options of [{}, { readOnly: true }, { readOnly: true, immutable: true }]) { + expect(() => fresh.openGraphDatabase(dbPath, options)).toThrow(/FTS5/); + } + // The preflight runs before the store is opened, so there is no handle to + // close — strictly better than opening one and closing it on the way out. + expect(sqliteMock.__storeOpens()).toEqual([]); }); }); diff --git a/src/graph/db/database.ts b/src/graph/db/database.ts index 7c51114e..d275debc 100644 --- a/src/graph/db/database.ts +++ b/src/graph/db/database.ts @@ -25,7 +25,12 @@ import { parseGraphSnapshot, serializeGraphSnapshot, } from "../snapshot.js"; -import { openSqlite, type SqliteDatabase } from "./sqlite.js"; +import { assertFts5Available, openSqlite, type SqliteDatabase } from "./sqlite.js"; + +// The FTS5 preflight lives with the SQLite adapter (it describes the SQLite +// build, not the graph) and is re-exported here for the callers and tests that +// already reach for it through this module. +export { assertFts5Available }; /** The schema version this build writes/expects (matches schema.sql's seed). */ export const DB_SCHEMA_VERSION = 4; @@ -57,42 +62,6 @@ function configureReadOnlyConnection(db: SqliteDatabase): void { db.pragma("query_only = ON"); } -/** - * Probe for FTS5 support and fail fast with an actionable message if it's missing. - * - * `node:sqlite`'s bundled SQLite is not guaranteed to be built with FTS5 on every - * Node build/version, even within the range `package.json`'s `engines` documents - * as supported (issue #110). Without this check, the first FTS5 statement in - * `schema.sql` throws SQLite's raw `no such module: fts5`, which reads like a mex - * bug rather than a Node/SQLite build limitation. Create-and-drop a throwaway - * virtual table rather than querying `pragma_module_list`, since that pragma is - * unavailable on some `node:sqlite` builds too and FTS5 usage is what actually - * needs to work. - * - * FTS5 availability is a property of the SQLite build the running Node binary - * embeds, not of any particular database file, so the probe runs against a - * throwaway `:memory:` connection rather than the caller's real database. - * Probing in place (an earlier version of this function took the caller's - * `SqliteDatabase`) rewrote the on-disk graph on every successful open, which - * broke a read-path non-mutation regression test in CI (PR #168 review). - */ -export function assertFts5Available(): void { - const probe = openSqlite(":memory:"); - try { - probe.exec("CREATE VIRTUAL TABLE __mex_fts5_probe USING fts5(x)"); - } catch (error) { - const msg = error instanceof Error ? error.message : String(error); - if (!/fts5/i.test(msg)) throw error; // a different problem; surface it unchanged - throw new Error( - `Your Node (${process.version}) has SQLite built without FTS5 support, which mex's code graph ` + - "requires. Try a different Node build/version - see COMPATIBILITY.md for which versions are " + - `known to work. Underlying error: ${msg}`, - ); - } finally { - probe.close(); - } -} - /** * Open the graph DB at `dbPath`, creating the file + parent dir and applying the * schema when absent. Idempotent: re-opening an existing DB re-applies PRAGMAs @@ -111,6 +80,13 @@ export function openGraphDatabase( mkdirSync(dir, { recursive: true }); } + // Every graph open needs FTS5: writers apply `schema.sql`'s virtual tables, + // and readers query `nodes_fts` / `source_chunks_fts` (search, scope, impact, + // and the grounding checker). A store built on an FTS5-capable machine and + // copied to one without it fails on read, not on build, so the read-only and + // immutable paths need this preflight just as much as the writable one. + assertFts5Available(); + if (options.readOnly) { return options.immutable ? openImmutableGraphDatabase(dbPath) @@ -120,7 +96,6 @@ export function openGraphDatabase( configureConnection(db); try { - assertFts5Available(); initializeWritableGraphDatabase(db, readFileSync(schemaPath(), "utf-8"), options); return db; } catch (error) { diff --git a/src/graph/db/sqlite.ts b/src/graph/db/sqlite.ts index 8b6750e9..44d39ff5 100644 --- a/src/graph/db/sqlite.ts +++ b/src/graph/db/sqlite.ts @@ -127,3 +127,48 @@ export function openSqlite( ); } } + +/** + * Probe for FTS5 support and fail fast with an actionable message if it's missing. + * + * `node:sqlite`'s bundled SQLite is not guaranteed to be built with FTS5 on every + * Node build/version, even within the range `package.json`'s `engines` documents + * as supported (issue #110). Without this check, the first FTS5 statement — in + * the graph's `schema.sql` or the wiki index's schema — throws SQLite's raw + * `no such module: fts5`, which reads like a mex bug rather than a Node/SQLite + * build limitation. Create-and-drop a throwaway virtual table rather than + * querying `pragma_module_list`, since that pragma is unavailable on some + * `node:sqlite` builds too and FTS5 usage is what actually needs to work. + * + * FTS5 availability is a property of the SQLite build the running Node binary + * embeds, not of any particular database file, so the probe runs against a + * throwaway `:memory:` connection rather than any caller's real database. + * Probing in place (an earlier version of this function took the caller's + * `SqliteDatabase`) rewrote the on-disk graph on every successful open, which + * broke a read-path non-mutation regression test in CI (PR #168 review). + * + * It lives beside {@link openSqlite} rather than in the graph's `database.ts` + * because it describes the SQLite build, not the code graph — and because both + * FTS5 consumers need it, while the wiki may only reach into this module. + * + * @param open Injected opener, for tests that need the probe to fail. Production + * callers always use the default. + */ +export function assertFts5Available(open: typeof openSqlite = openSqlite): void { + const probe = open(":memory:"); + try { + probe.exec("CREATE VIRTUAL TABLE __mex_fts5_probe USING fts5(x)"); + } catch (error) { + const msg = error instanceof Error ? error.message : String(error); + if (!/fts5/i.test(msg)) throw error; // a different problem; surface it unchanged + throw new Error( + `Your Node (${process.version}) has SQLite built without FTS5 support, which mex's code graph ` + + "and wiki index require. FTS5 is a compile-time option, so this is a property of the Node " + + "build rather than the version number - installing a different build or version of Node is " + + "the fix. See the SQLite FTS5 section of COMPATIBILITY.md. " + + `Underlying error: ${msg}`, + ); + } finally { + probe.close(); + } +} diff --git a/src/graph/maintenance.ts b/src/graph/maintenance.ts index 6ff41420..07a6859f 100644 --- a/src/graph/maintenance.ts +++ b/src/graph/maintenance.ts @@ -36,6 +36,7 @@ import { inspectGraphStatus, } from "./status.js"; import { GRAPH_SNAPSHOT_METADATA_KEY } from "./snapshot.js"; +import { tryEnsureSetupIgnoreProtection } from "../setup/ignore.js"; const LOCK_FILE = "graph.db.lock"; const LOCK_GATE_FILE = "graph.db.lock.gate"; @@ -236,6 +237,22 @@ export function acquireGraphMaintenanceLease( "The graph index does not exist. Run `mex graph rebuild` first.", ); } + + // Every graph writer funnels through here, so this is the one place that can + // guarantee a store is ignored by Git before it exists — including the runs + // that never went through `mex setup` (issue #110). Deliberately after the + // missing-index refusal, so a mistyped `refresh` in an unindexed checkout + // leaves nothing behind. + const protection = tryEnsureSetupIgnoreProtection(projectRoot); + if (!protection.ok) { + // Reported, never fatal: the store is still valid, the checkout is merely + // unprotected. Emitted directly rather than through `progress` so it cannot + // turn an already-cancelled signal into a throw before the lease exists. + options.onProgress?.({ + phase: "discover", + message: `Could not ignore local MEX data: ${protection.reason}`, + }); + } const lock = acquireMaintenanceLock(paths, internalOptions); let released = false; let active = false; diff --git a/src/setup/ignore.ts b/src/setup/ignore.ts index 75be3852..49026571 100644 --- a/src/setup/ignore.ts +++ b/src/setup/ignore.ts @@ -116,6 +116,30 @@ export function ensureSetupIgnoreProtection( return result; } +/** + * Best-effort variant for the commands that create a local store outside + * `mex setup` — `mex graph rebuild/refresh/repair` and `mex wiki rebuild-index`. + * + * Issue #110 reported the consequence of not doing this: `mex graph` in a + * checkout that had never run setup left `graph.db`, `-wal` and `-shm` + * untracked, so the next `git add -A` committed a database. Setup writes these + * rules at step 2, long before it builds anything, but a store writer invoked + * on its own never passed through setup. + * + * Never throws. A store build must not fail because a `.gitignore` could not be + * written — the store is still valid, the user is merely unprotected — so the + * caller is handed the reason and decides how loudly to say so. + */ +export function tryEnsureSetupIgnoreProtection( + projectRoot: string, +): { ok: true; result: SetupIgnoreProtectionResult } | { ok: false; reason: string } { + try { + return { ok: true, result: ensureSetupIgnoreProtection({ projectRoot }) }; + } catch (error) { + return { ok: false, reason: error instanceof Error ? error.message : String(error) }; + } +} + /** Render one concise setup status line without coupling the helper to the CLI. */ export function renderSetupIgnoreProtection(result: SetupIgnoreProtectionResult): string { if (!result.changed) { diff --git a/src/wiki/__tests__/diagnostic-coverage.test.ts b/src/wiki/__tests__/diagnostic-coverage.test.ts index fb96bfd4..1dd8f265 100644 --- a/src/wiki/__tests__/diagnostic-coverage.test.ts +++ b/src/wiki/__tests__/diagnostic-coverage.test.ts @@ -57,6 +57,8 @@ import { parseWikiMarkdown } from "../markdown/codec.js"; import type { ParsedEntity } from "../markdown/contract.js"; import { detectRangeOverlaps } from "../index/write.js"; import { openWikiIndex } from "../index/open.js"; +import { fts5UnavailableDiagnostic } from "../index/fts5.js"; +import { assertFts5Available, openSqlite } from "../../graph/db/sqlite.js"; import { rebuildWikiIndex } from "../index/rebuild.js"; import { getEntity } from "../query/get.js"; import { escapedSymlinkDiagnostic } from "../index/discover.js"; @@ -331,6 +333,27 @@ status: promoted return opened.ok ? [] : [opened.diagnostic]; }), + WIKI_INDEX_FTS5_UNAVAILABLE: () => { + // No FTS5-less Node exists to reproduce this on, so the probe's SQLite + // opener is injected. Everything downstream of it — the actionable message + // and the mapping onto this code rather than WIKI_INDEX_REBUILD_REQUIRED — + // is the real production path. + const withoutFts5: typeof openSqlite = () => ({ + prepare: () => { + throw new Error("not reached"); + }, + exec: () => { + throw new Error("no such module: fts5"); + }, + pragma: () => {}, + transaction: (fn) => fn(), + close: () => {}, + open: true, + }); + const emitted = fts5UnavailableDiagnostic("wiki.db", () => assertFts5Available(withoutFts5)); + return emitted ? [emitted] : []; + }, + WIKI_INDEX_REBUILD_REQUIRED: () => inScratch((directory) => { const path = join(directory, "wiki.db"); diff --git a/src/wiki/cli/commands.ts b/src/wiki/cli/commands.ts index 4c4f7f47..2bd83e4d 100644 --- a/src/wiki/cli/commands.ts +++ b/src/wiki/cli/commands.ts @@ -38,6 +38,7 @@ import chalk from "chalk"; import { readFileSync } from "node:fs"; import { resolve } from "node:path"; +import { tryEnsureSetupIgnoreProtection } from "../../setup/ignore.js"; import type { WikiDiagnostic } from "../model/diagnostic.js"; import { inspectDirectWikiSpecMutation } from "./spec-authoring-boundary.js"; import { @@ -312,6 +313,14 @@ export function runGraph(io: CommandIo, flags: CommandFlags): void { } export function runRebuildIndex(io: CommandIo, flags: CommandFlags): void { + // The only command that creates `wiki.db`, and it can run in a checkout that + // never went through `mex setup` — which is how issue #110's reporter ended + // up with an untracked database. Best effort: an unwritable `.gitignore` is + // worth a line of warning, not a failed rebuild. + const projectRoot = io.projectRoot ?? resolve(io.scaffoldRoot, ".."); + const protection = tryEnsureSetupIgnoreProtection(projectRoot); + if (!protection.ok) io.write(chalk.dim(`could not ignore local MEX data: ${protection.reason}`)); + emit(io, wikiRebuildIndex(serviceOptions(io)), flags, (data) => { io.write(`Indexed ${data.entityCount} entities from ${data.fileCount} file(s) into ${data.indexPath}`); for (const swept of data.sweptTempFiles) io.write(chalk.dim(`removed a crashed build's temp index: ${swept}`)); diff --git a/src/wiki/index/__tests__/fts5.test.ts b/src/wiki/index/__tests__/fts5.test.ts new file mode 100644 index 00000000..c4e9ef11 --- /dev/null +++ b/src/wiki/index/__tests__/fts5.test.ts @@ -0,0 +1,99 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fts5UnavailableDiagnostic } from "../fts5.js"; +import { createWikiIndex, openWikiIndex as openWikiIndexReal } from "../open.js"; +import { assertFts5Available, openSqlite } from "../../../graph/db/sqlite.js"; + +/** + * Issue #110: a Node whose bundled SQLite lacks FTS5 cannot read the wiki + * index, and said so with SQLite's raw `no such module: fts5`. No such Node + * exists to test against here, so the probe's opener is injected; everything + * downstream of it is the production path. + */ + +const withoutFts5: typeof openSqlite = () => ({ + prepare: () => { + throw new Error("not reached"); + }, + exec: () => { + throw new Error("no such module: fts5"); + }, + pragma: () => {}, + transaction: (fn) => fn(), + close: () => {}, + open: true, +}); + +describe("fts5UnavailableDiagnostic", () => { + it("stays out of the way when FTS5 works, which is the ordinary case", () => { + expect(fts5UnavailableDiagnostic("wiki.db")).toBeNull(); + }); + + it("reports its own code, not one whose remediation is a rebuild that cannot help", () => { + const emitted = fts5UnavailableDiagnostic("wiki.db", () => assertFts5Available(withoutFts5)); + + expect(emitted?.code).toBe("WIKI_INDEX_FTS5_UNAVAILABLE"); + expect(emitted?.code).not.toBe("WIKI_INDEX_REBUILD_REQUIRED"); + }); + + it("carries the actionable message and the index it was asked about", () => { + const emitted = fts5UnavailableDiagnostic("/tmp/scaffold/wiki.db", () => + assertFts5Available(withoutFts5)); + + expect(emitted?.message).toContain("FTS5"); + expect(emitted?.message).toContain(process.version); + expect(emitted?.file).toBe("/tmp/scaffold/wiki.db"); + }); +}); + +describe("openWikiIndex without FTS5", () => { + const roots: string[] = []; + + afterEach(() => { + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); + vi.doUnmock("../fts5.js"); + vi.resetModules(); + }); + + it("refuses a real, intact index rather than letting SQLite's raw error escape", async () => { + // A genuine index file, built the way a rebuild builds one — the point is + // that nothing is wrong with it. Only the engine reading it is unable. + const root = mkdtempSync(join(tmpdir(), "mex-wiki-fts5-open-")); + roots.push(root); + const path = join(root, "wiki.db"); + createWikiIndex(path).close(); + + vi.resetModules(); + vi.doMock("../fts5.js", async () => { + const actual = await vi.importActual("../fts5.js"); + return { + fts5UnavailableDiagnostic: (file: string) => + actual.fts5UnavailableDiagnostic(file, () => assertFts5Available(withoutFts5)), + }; + }); + const { openWikiIndex } = await import("../open.js"); + + const opened = openWikiIndex(path); + + expect(opened.ok).toBe(false); + if (opened.ok) return; + expect(opened.diagnostic.code).toBe("WIKI_INDEX_FTS5_UNAVAILABLE"); + // The distinction that matters: a rebuild on this Node cannot help, so the + // user must not be sent to `mex wiki rebuild-index`. + expect(opened.diagnostic.code).not.toBe("WIKI_INDEX_REBUILD_REQUIRED"); + expect(opened.diagnostic.message).toContain("FTS5"); + }); + + it("opens that same index normally once the engine can do FTS5", () => { + const root = mkdtempSync(join(tmpdir(), "mex-wiki-fts5-ok-")); + roots.push(root); + const path = join(root, "wiki.db"); + createWikiIndex(path).close(); + + const opened = openWikiIndexReal(path); + expect(opened.ok).toBe(true); + if (opened.ok) opened.index.close(); + }); +}); diff --git a/src/wiki/index/fts5.ts b/src/wiki/index/fts5.ts new file mode 100644 index 00000000..dd27890c --- /dev/null +++ b/src/wiki/index/fts5.ts @@ -0,0 +1,45 @@ +/** + * The one place that asks whether this Node can run the wiki index at all. + * + * The index answers searches through the `wiki_fts` virtual table, and + * `node:sqlite` embeds whatever SQLite the running Node binary was compiled + * with. FTS5 is a compile-time option Node neither documents nor guarantees, so + * a perfectly valid index file can be unreadable on a different Node build of + * the same version — switching versions with a version manager is enough + * (issue #110). + * + * Both the build path and the read paths need this, and they report failures + * differently — a diagnostic here, a typed read error there — so what is shared + * is the probe and the mapping, not the reporting. Reaching for the graph's + * `assertFts5Available` rather than writing a second probe is deliberate: one + * definition of "can this SQLite do FTS5" means one answer. + */ + +import { assertFts5Available } from "../../graph/db/sqlite.js"; +import { diagnostic, type WikiDiagnostic } from "../model/diagnostic.js"; + +/** + * `null` when FTS5 works, otherwise the diagnostic to report. + * + * Deliberately not `WIKI_INDEX_REBUILD_REQUIRED`: nothing is wrong with the + * store, and no rebuild on this Node could improve matters, so pointing the + * user at `mex wiki rebuild-index` would send them round a loop. + * + * @param probe Injected by tests, which have no FTS5-less Node to reproduce + * this on. Production callers take the default. + */ +export function fts5UnavailableDiagnostic( + path: string, + probe: () => void = assertFts5Available, +): WikiDiagnostic | null { + try { + probe(); + return null; + } catch (error) { + return diagnostic( + "WIKI_INDEX_FTS5_UNAVAILABLE", + error instanceof Error ? error.message : String(error), + { file: path }, + ); + } +} diff --git a/src/wiki/index/open.ts b/src/wiki/index/open.ts index d19f9092..cb63e6aa 100644 --- a/src/wiki/index/open.ts +++ b/src/wiki/index/open.ts @@ -17,7 +17,8 @@ */ import { diagnostic, type WikiDiagnostic } from "../model/diagnostic.js"; -import { openSqlite, type SqliteDatabase } from "../../graph/db/sqlite.js"; +import { assertFts5Available, openSqlite, type SqliteDatabase } from "../../graph/db/sqlite.js"; +import { fts5UnavailableDiagnostic } from "./fts5.js"; import { indexExists } from "./dbfile.js"; import { WIKI_META_KEYS, WIKI_SCHEMA_SQL, WIKI_SCHEMA_VERSION } from "./schema.js"; @@ -94,6 +95,9 @@ export function openWikiIndex(path: string, options: OpenIndexOptions = {}): Ope }; } + const fts5 = fts5UnavailableDiagnostic(path); + if (fts5) return { ok: false, diagnostic: fts5 }; + let db: SqliteDatabase; try { const readOnly = options.readOnly !== false; @@ -151,6 +155,9 @@ export function openWikiIndex(path: string, options: OpenIndexOptions = {}): Ope * than seven that could drift apart. */ export function createWikiIndex(path: string): WikiIndexHandle { + // Fail before the file is created, so a Node without FTS5 leaves no partial + // index behind for the next run to trip over. + assertFts5Available(); const db = openSqlite(path); configureConnection(db); db.pragma("journal_mode = WAL"); diff --git a/src/wiki/model/diagnostic.ts b/src/wiki/model/diagnostic.ts index 8dc8e77c..08b98fde 100644 --- a/src/wiki/model/diagnostic.ts +++ b/src/wiki/model/diagnostic.ts @@ -238,6 +238,12 @@ export const WIKI_DIAGNOSTICS = { severity: "error", remediation: "The index was built by a different schema version. Run `mex wiki rebuild-index`.", }, + WIKI_INDEX_FTS5_UNAVAILABLE: { + severity: "error", + remediation: + "The running Node build embeds a SQLite without FTS5, which the wiki index requires. " + + "Rebuilding cannot fix this — use a different Node build/version. See COMPATIBILITY.md.", + }, WIKI_PARSE_ERROR: { severity: "error", remediation: "Fix the malformed Markdown or entity metadata block. Prose is never deleted to resolve this.", diff --git a/src/wiki/query/contract-session.ts b/src/wiki/query/contract-session.ts index af65477a..23a36ac2 100644 --- a/src/wiki/query/contract-session.ts +++ b/src/wiki/query/contract-session.ts @@ -37,6 +37,7 @@ import { } from "../index/corpus-policy.js"; import { readContainedSource } from "../index/source-read.js"; import { WIKI_META_KEYS, WIKI_SCHEMA_VERSION, WIKI_TABLES } from "../index/schema.js"; +import { fts5UnavailableDiagnostic } from "../index/fts5.js"; import { isTeamOwnedReadOnlyPath } from "../model/team-owned-paths.js"; import { estimateTokens } from "./budget.js"; import { healthRank, LIFECYCLE_RANK, MATCH_FIELD_RANK, type MatchField } from "./rank.js"; @@ -444,6 +445,13 @@ export function inspectWikiContractIndex(options: InspectWikiIndexOptions): Cont ), ]); } + // The index answers searches through `wiki_fts`, so a Node whose SQLite + // lacks FTS5 cannot read it even though the file is intact (issue #110). + // Degraded, not corrupt: nothing is wrong with the store, and no rebuild + // on this Node would improve matters. + const fts5 = fts5UnavailableDiagnostic(bound.path); + if (fts5) return status("degraded", observedAt, null, null, null, [fts5]); + db = openSqlite(bound.path, { readOnly: true, immutable: true }); const schemaVersion = readSchemaVersion(db); if (schemaVersion === null) { @@ -568,6 +576,12 @@ export function openWikiContractReadSession(options: InspectWikiIndexOptions): W throw new WikiContractReadError("REVISION_CONFLICT", "Wiki index changed while the read session was opening."); } options.hooks?.beforeSessionImmutableOpen?.(); + // Same FTS5 requirement as the status path, reported through the existing + // INDEX_UNAVAILABLE code: for a reader the index genuinely is unavailable + // on this Node, and the message says why and what to change. + const fts5 = fts5UnavailableDiagnostic(bound.path); + if (fts5) throw new WikiContractReadError("INDEX_UNAVAILABLE", fts5.message); + db = openSqlite(bound.path, { readOnly: true, immutable: true }); if (readSchemaVersion(db) !== WIKI_SCHEMA_VERSION || readMeta(db, WIKI_META_KEYS.indexedRevision) !== initial.indexedRevision diff --git a/test/store-ignore-protection.test.ts b/test/store-ignore-protection.test.ts new file mode 100644 index 00000000..5dcb3f98 --- /dev/null +++ b/test/store-ignore-protection.test.ts @@ -0,0 +1,68 @@ +import { describe, it, expect, afterEach } from "vitest"; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync, readFileSync, existsSync } from "node:fs"; +import { execFileSync } from "node:child_process"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { rebuildGraph } from "../src/graph/maintenance.js"; +import { SETUP_IGNORE_RULES } from "../src/setup/ignore.js"; + +/** + * Issue #110: `mex graph` in a checkout that had never run `mex setup` left + * `graph.db`, `-wal` and `-shm` untracked, so the reporter's next `git add -A` + * would have committed a database. Setup writes `.mex/.gitignore` at step 2, + * but a store writer invoked on its own never passed through setup. + */ + +const roots: string[] = []; + +afterEach(() => { + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +function scratchRepo(): string { + const root = mkdtempSync(join(tmpdir(), "mex-store-ignore-")); + roots.push(root); + execFileSync("git", ["init", "-q", "."], { cwd: root }); + mkdirSync(join(root, "src")); + writeFileSync(join(root, "src", "a.ts"), "export function hello(): number {\n return 1;\n}\n"); + writeFileSync(join(root, "package.json"), JSON.stringify({ name: "probe", version: "0.0.0" })); + return root; +} + +describe("local store protection outside mex setup (#110)", () => { + it("writes .mex/.gitignore before a graph rebuild creates the store", async () => { + const root = scratchRepo(); + + await rebuildGraph(root); + + const ignorePath = join(root, ".mex", ".gitignore"); + expect(existsSync(join(root, ".mex", "graph.db"))).toBe(true); + expect(existsSync(ignorePath)).toBe(true); + const rules = readFileSync(ignorePath, "utf-8").split(/\r?\n/); + for (const rule of SETUP_IGNORE_RULES) expect(rules).toContain(rule); + }); + + it("leaves `git add -A` unable to stage the store, which is the actual complaint", async () => { + const root = scratchRepo(); + + await rebuildGraph(root); + execFileSync("git", ["add", "-A"], { cwd: root }); + const staged = execFileSync("git", ["ls-files", "--cached"], { cwd: root, encoding: "utf-8" }); + + expect(staged).toContain(".mex/.gitignore"); + expect(staged).not.toMatch(/graph\.db/); + }); + + it("does not disturb an existing ignore file's own rules", async () => { + const root = scratchRepo(); + mkdirSync(join(root, ".mex")); + writeFileSync(join(root, ".mex", ".gitignore"), "# hand written\nscratch/\n"); + + await rebuildGraph(root); + + const content = readFileSync(join(root, ".mex", ".gitignore"), "utf-8"); + expect(content).toContain("# hand written"); + expect(content).toContain("scratch/"); + for (const rule of SETUP_IGNORE_RULES) expect(content).toContain(rule); + }); +}); diff --git a/test/telemetry-optout-cli.test.ts b/test/telemetry-optout-cli.test.ts new file mode 100644 index 00000000..ab7c269b --- /dev/null +++ b/test/telemetry-optout-cli.test.ts @@ -0,0 +1,112 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; +import { mkdtempSync, rmSync, readFileSync, existsSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +/** + * Issue #110: the opt-out worked but was undiscoverable. The reporter typed + * `mex telemetry disable`, got `unknown command`, then guessed at env var names. + * These tests pin the three places the answer now appears — the subcommand, the + * group's help text, and `status` — because each one is a thing a user reaches + * for first, and any of them going quiet again reopens the issue. + * + * MEX_HOME is redirected per test (not $HOME: Node's homedir() ignores $HOME on + * Windows), so no test can touch the developer's real ~/.mex/config.json. + */ + +let originalMexHome: string | undefined; +let originalDoNotTrack: string | undefined; +let tempHome: string; + +beforeEach(() => { + originalMexHome = process.env.MEX_HOME; + originalDoNotTrack = process.env.DO_NOT_TRACK; + delete process.env.DO_NOT_TRACK; + tempHome = mkdtempSync(join(tmpdir(), "mex-optout-")); + process.env.MEX_HOME = tempHome; + vi.resetModules(); +}); + +afterEach(() => { + if (originalMexHome === undefined) delete process.env.MEX_HOME; + else process.env.MEX_HOME = originalMexHome; + if (originalDoNotTrack === undefined) delete process.env.DO_NOT_TRACK; + else process.env.DO_NOT_TRACK = originalDoNotTrack; + rmSync(tempHome, { recursive: true, force: true }); + vi.restoreAllMocks(); +}); + +function storedTelemetry(): unknown { + const path = join(tempHome, ".mex", "config.json"); + if (!existsSync(path)) return undefined; + return (JSON.parse(readFileSync(path, "utf-8")) as { telemetry?: unknown }).telemetry; +} + +async function runCli(...args: string[]): Promise { + const lines: string[] = []; + vi.spyOn(console, "log").mockImplementation((...parts: unknown[]) => { + lines.push(parts.join(" ")); + }); + const { program } = await import("../src/cli.js"); + await program.parseAsync(["node", "mex", ...args]); + return lines; +} + +describe("mex telemetry opt-out discoverability (#110)", () => { + it("accepts `telemetry disable`, the command the reporter reached for first", async () => { + const lines = await runCli("telemetry", "disable"); + + expect(storedTelemetry()).toBe("off"); + expect(lines.join("\n")).toContain("Telemetry disabled"); + }); + + it("round-trips back on with `telemetry enable`", async () => { + await runCli("telemetry", "disable"); + expect(storedTelemetry()).toBe("off"); + + await runCli("telemetry", "enable"); + expect(storedTelemetry()).toBe("on"); + }); + + it("writes the same key as `mex config set telemetry off`, rather than a second setting", async () => { + await runCli("config", "set", "telemetry", "off"); + const viaConfig = storedTelemetry(); + + await runCli("telemetry", "enable"); + await runCli("telemetry", "disable"); + + expect(storedTelemetry()).toBe(viaConfig); + }); + + it("does not claim telemetry is off when an env opt-out already outranks the stored value", async () => { + process.env.DO_NOT_TRACK = "1"; + + const lines = (await runCli("telemetry", "enable")).join("\n"); + + // Honest about the outcome: the write happened, and it changes nothing yet. + expect(storedTelemetry()).toBe("on"); + expect(lines).toContain("DO_NOT_TRACK"); + }); + + it("names the env opt-outs in `telemetry --help`, where the reporter looked next", async () => { + const { program } = await import("../src/cli.js"); + const telemetry = program.commands.find((command) => command.name() === "telemetry"); + // `helpInformation()` omits addHelpText hooks, which is exactly where the + // env vars live — render what the user actually sees instead. + let help = ""; + telemetry?.configureOutput({ writeOut: (chunk) => { help += chunk; } }); + telemetry?.outputHelp(); + + expect(telemetry?.commands.map((command) => command.name())).toContain("disable"); + expect(help).toContain("DO_NOT_TRACK=1"); + expect(help).toContain("MEX_TELEMETRY=0"); + }); + + it("tells `status` readers what to change, not just which reason code fired", async () => { + process.env.DO_NOT_TRACK = "1"; + const disabled = (await runCli("telemetry", "status")).join("\n"); + + expect(disabled).toContain("disabled (reason: DO_NOT_TRACK)"); + expect(disabled).toContain("DO_NOT_TRACK environment variable"); + }); +});