Wire Cursor into the session-log collector + backfill (CROW-1095)

[dhilgaertner]

Closes #1095

What & why

The session-log collector (CROW-1056) and historical backfill (CROW-1075) were architected multi-harness; CROW-1089 (#1092) wired Claude + Codex. This wires Cursor end-to-end — live collector and backfill — as the third harness, so cursor-agent transcripts ship to Corveil as session-transcript artifacts the same way.

Cursor's on-disk shape (reverse-engineered)

The blocker recorded in CursorAgent was the missing .sqlite normalizer. Investigation against cursor-agent 2026.08.04 on this machine (488 real chats) established the format:

~/.cursor/chats/<chatId>/<subId>/store.db     # SQLite: blobs + meta
~/.cursor/chats/<chatId>/<subId>/meta.json    # sibling; carries the cwd

• blobs(id TEXT PRIMARY KEY, data BLOB), content-addressed (id == sha256(data)), meta(key TEXT, value TEXT).
• meta['0'] is hex-encoded JSON → latestRootBlobId, agentId (== the <subId> dir), and a blobEncryptionKey.
• The root blob is a protobuf whose repeated field 1 lists the conversation's message-blob ids in order.
• Each message blob is one compact JSON object ({"role":…,"content":…}) — already the NDJSON line.

The blobEncryptionKey is present but today's blobs are plaintext. The extractor is therefore defensive: a message blob that isn't valid UTF-8 JSON is dropped, so if a future Cursor build actually encrypts the blobs the extractor yields nothing rather than shipping ciphertext — the "wrong bytes are worse than none" bar the ticket sets.

How

New .sqlite normalizer + cwd reader (CursorStore, CrowCore, import SQLite3):

• Follows meta['0'].latestRootBlobId → the root blob's ordered field-1 refs → each message blob, emitting one NDJSON line per message. TranscriptNormalizer's .sqlite case concatenates them (max-bytes bounded, whole-line truncation).
• Reads the cwd from the sibling meta.json — Cursor records cwd beside the transcript, not in it, so AgentLogCwdReader (a transcript-head reader, the Codex mechanism) doesn't apply. The collector's cwd filter now dispatches on format: .sqlite → sibling meta.json, else the head.

Live collector: CursorAgent.logSources returns a recursive, cwd-filtered .sqlite directory source over <CursorHome>/chats (fileExtension: "db" uniquely selects store.db, not its -wal/-shm siblings). CursorHome honors $CURSOR_CONFIG_DIR — one source of truth, shared with the MCP-bridge path (mirrors CodexHome).

Backfill: BackfillScanner reconciles Cursor stores too, staying disk-only — the uid is the <subId> dir name and the cwd is the tiny sibling JSON, so no store.db is opened during a scan (only at upload). BackfillService keys the ledger / upload / format (.sqlite) / agentKind off harness.

Attribution that can't misattribute: exact cwd match; a chat with no recoverable cwd is dropped, never guessed.

Docs

session-log-collector.md, session-backfill.md, cli-reference.md, cli.md (regenerated), CLAUDE.md, plus the CLI help text and the backfill web UI: Claude + Codex → + Cursor.

Testing

swift test green across CrowCore (773), CrowCursor (85), and the CrowDaemon logsync + backfill suites. New tests cover the store.db blob extractor (ordering + the encryption/garbage guard + protobuf wire-type skipping), the sibling-meta.json cwd reader, the cwd-filter dispatch (excluding -wal/-shm and unattributable chats), Cursor backfill reconstruction (high-confidence + no-cwd → low), and the harness-scoped mapping. crow + crowd link clean.

Note

Two failing CrowDaemon tests — RPCLanePolicyTests (naming terminal-set/corveil-verify/corveil-reinstall-skill) and WebNotificationCenterTests.bootCatchResetsTheHistory — are pre-existing on main and untouched by this change (none of RPCLanePolicy/RPCHandlers/WebNotification are in this diff; the verbs they name belong to CROW-1085 / CROW-1011).

🤖 Generated with Claude Code

[dgershman]

Code & Security Review

Critical Issues

[Red] Linux CI is red — CursorStore SQLite reads return nil, so CrowCore's new tests fail and a Linux crowd cannot extract Cursor transcripts.

Verified against GitHub Actions run 32748697274 (Build & Test, swift:6.1). CrowCore reported 773 tests, 6 issues; local macOS swift test --package-path Packages/CrowCore --filter CursorStoreTests is green (7/7).

Failing tests:

• CursorStoreTests.extractsOrderedMessagesAndDropsNonJSON — messageLines → nil (CursorStoreTests.swift:125)
• CursorStoreTests.agentIdFromMetaZero — agentId → nil (CursorStoreTests.swift:151)
• CursorStoreTests.normalizerSqliteConcatenatesStoresInOrder / normalizerSqliteTruncatesAtWholeLines
• BackfillScannerTests.reconstructsCursorHighConfidenceSession / cursorChatWithoutCwdIsDroppedToLowConfidence — scan finds no Cursor row

The read path is CursorStore.withDatabase (CursorStore.swift:173-181): sqlite3_open_v2(..., SQLITE_OPEN_READONLY, nil) then return nil on any failure, with no logged sqlite3_errmsg. The fixture creates the DB with read-write sqlite3_open (CursorStoreTests.swift:41), so this is the Linux-readonly (or URL-.path) seam, not "SQLite is missing" — CursorStore.swift compiled and the pure protobuf/JSON tests passed. CrowCore is on the Linux PR allow-list (ADR 0007 / .github/workflows/ci.yml); a Linux crowd (CROW-645) would hit the same nil and upload nothing.

Fix: make the open work on Linux (e.g. path(percentEncoded: false) + withCString, and/or flags that can attach to WAL — SQLITE_OPEN_READONLY cannot create -shm), surface the SQLite error in tests, and keep the suite green on swift:6.1.

Architecture / Existing Patterns

• Existing pathway: CodingAgent.logSources → AgentLogSource directory + cwdFilter (Codex is the reference) → LogSyncCollector.resolveFiles / applyingCwdFilter → TranscriptNormalizer format dispatch → TranscriptUploader; backfill via BackfillScanner.reconstruct* + daemon-injected home (CodexHome / now CursorHome). This is the right extension — CursorStore in CrowCore is justified because the normalizer and scanner live there and CrowCore cannot import CrowCursor.
• [Yellow] Live logSources is looser than backfill, breaking the lockstep CROW-1089 added fileNamePrefix to enforce. Codex selects rollout-*.jsonl on both paths (OpenAICodexAgent.logSources + codexRolloutFiles). Cursor backfill enumerates the exact name store.db (BackfillScanner.swift:301), but CursorAgent.logSources only sets fileExtension: "db" (CursorAgent.swift:335-336) with no fileNamePrefix. A stray *.db under <CursorHome>/chats would be cwd-filtered and fed to CursorStore. Set fileNamePrefix: "store" (or equivalent) so live and backfill stay in lockstep.

Security Review

Strengths:

• Opt-in, gateway-reuse upload path is unchanged (destination/credential still local-only).
• Attribution is exact cwd match; missing cwd is never guessed on the live path (applyingCwdFilter + CursorStore.recordedCwd).
• SQL is parameterized (blobs.id bound); meta['0'] is a literal. Open is read-only so the collector does not take a write lock on a live cursor-agent store.
• Fail-closed on non-UTF-8 / non-JSON blobs (jsonLine) matches the ticket's "wrong bytes are worse than none" bar. Accepted as documented.
• $CURSOR_CONFIG_DIR= empty-is-unset in CursorHome avoids a CWD-relative chats tree (same foot-gun Codex already fixed).

Concerns:

• [Yellow] Quiet-period ignores WAL mtime, so a still-growing Cursor chat can look quiescent. Confirmed against CursorAgent.swift:335 (fileExtension: "db" drops store.db-wal / store.db-shm) and LogSyncCollector.swift:119-127 (newestModification runs only on resolved store.db files). SQLite WAL commits update the -wal file, not necessarily store.db. The server is write-once (409); an early upload freezes a partial transcript. Include sibling -wal/-shm mtimes in the quiet-period calculation (without parsing those files as stores). The PR already knows these siblings exist (CursorStore.swift:170-172).

Code Quality

• [Yellow] docs/harness-transcript-locations.md still marks Cursor "Deferred — needs blob extractor". That file exists so the next harness wiring does not re-discover a path. This PR updates session-log-collector.md / session-backfill.md / CLI help / settings copy, but leaves the field-reference table stale. Flip the Cursor row to Wired (CROW-1095) in the same change.
• concatenateCursorStores loads every message line into memory before applying maxBytes, unlike the incremental NDJSON reader in the same type. Fine for typical chats; worth tightening if stores can be tens of MB. [Green]
• Encryption defense is a UTF-8 / leading-{/[ heuristic, not JSONSerialization. Documented and accepted. [Green]

Summary Table

Color	Meaning	Verdict effect
Red	Must fix	Request changes
Yellow	Should fix	Request changes
Green	Consider	Approve allowed

Recommendation: Request Changes — driven by 1 Red, 3 Yellow, 2 Green findings.

---

🐦‍⬛ Reviewed by Crow via Cursor

[dgershman]

Code Review

Architecture

This is the right shape. The existing pathway is CodingAgent.logSources → collector cwdFilter + TranscriptNormalizer → BackfillScanner reconstruction, as wired for Codex in CROW-1089. Cursor is a globally-pooled store (like Codex, unlike Claude), so it correctly reuses cwdFilter rather than inventing a parallel scanner. CursorHome mirrors CodexHome; CursorStore lives in CrowCore so both the live collector and backfill share one extractor without CrowCore importing CrowCursor. The meta.json cwd split is justified — Cursor does not record cwd in the transcript head, so AgentLogCwdReader cannot apply.

Findings

[Red] Linux CI is red: SQLite3 is unavailable, so the extractor is a stub and 6 tests fail. Packages/CrowCore/Package.swift, Packages/CrowCore/Sources/CrowCore/LogSync/CursorStore.swift

CrowCore is on the Linux allow-list (ADR 0007). CursorStore.messageLines / agentId are wrapped in #if canImport(SQLite3) and return nil otherwise; the test fixture's store.db write is behind the same gate. SQLite3 is a Darwin clang module — canImport(SQLite3) is false in swift:6.1 on Ubuntu, and CrowCore's Package.swift has no .linkedLibrary("sqlite3") / system-library shim (CrowTelemetry uses the same canImport pattern but is not in the Linux lane).

Verified against CI run 32748697274:

• BackfillScannerTests.reconstructsCursorHighConfidenceSession / cursorChatWithoutCwdIsDroppedToLowConfidence — session is nil (store.db was never created)
• CursorStoreTests.extractsOrderedMessagesAndDropsNonJSON / agentIdFromMetaZero / both normalizer sqlite tests — messageLines / normalize return nil

Local macOS: swift test --package-path Packages/CrowCore --filter CursorStoreTests — 7/7 pass.

Fix: make sqlite importable and linked on Linux (system-library / linkedLibrary("sqlite3") + libsqlite3-dev in the CI image if needed) so canImport is true in that lane, then re-run the 6 tests there. Until then the Linux daemon would also silently skip every Cursor transcript.

[Yellow] Quiet-period uses store.db mtime only, so WAL writes look idle and a partial transcript can be frozen write-once. Packages/CrowDaemon/Sources/CrowDaemon/LogSyncCollector.swift, Packages/CrowCursor/Sources/CrowCursor/CursorAgent.swift

CursorAgent.logSources sets fileExtension: "db", which correctly excludes store.db-wal / store.db-shm from the upload set. The collector then uses those same resolved files for quiescence (newestModification(files) vs quietPeriod, skipped only when the session is not terminal). In WAL mode, commits go to the -wal file; the main store.db mtime often stays at creation/last checkpoint.

So a live Cursor session that has been running longer than quietPeriod looks quiescent, the collector uploads a snapshot, and the server 409s every later attempt. Codex does not have this bug: rollout-*.jsonl mtime updates on append.

Fix: include sibling -wal/-shm mtimes in the quiet-period calculation without adding those files to the normalize/upload set.

[Green] reconstructCursor comment says no-cwd chats are dropped; the code emits low-confidence orphans. Packages/CrowCore/Sources/CrowCore/LogSync/BackfillScanner.swift

assemble always returns a session; nil cwd yields .low, matching Claude/Codex orphans and the test cursorChatWithoutCwdIsDroppedToLowConfidence. Live collection does drop (cwd filter). Tighten the comment / PR text so backfill vs live is not confused. Not a logic bug.

[Green] jsonLine is a UTF-8 + {/[ heuristic, not a JSON parse. Documented fail-closed for ciphertext; acceptable. A UTF-8 blob that merely starts with { would still ship. Fine for this threat model.

Tests

• macOS: CursorStoreTests 7/7 pass.
• Linux CI: 6 failures in CursorStoreTests + BackfillScannerTests (see Red).
• Coverage of protobuf skip, encryption/garbage drop, cwd-filter excluding -wal/-shm, and harness-scoped ledger mapping is good — once sqlite actually runs on Linux.

Security

Strengths:

• Opt-in + gateway-only destination/credential (CROW-1070 invariant unchanged).
• SQLITE_OPEN_READONLY; bound id=?; no path from chat content into shell.
• Unattributable chats are not guessed (live drop / backfill low).
• Ciphertext fail-closed rather than uploading opaque blobs.
• blobEncryptionKey stays in sqlite; only message JSON is uploaded.

No new authz, injection, or credential-exposure issue in this diff. Cursor chats can contain secrets — same accepted, opt-in threat model as Claude/Codex.

Summary

Severity	Count	Blocking?
Red	1	Yes
Yellow	1	Should fix
Green	2	No

Verdict: Request changes — Linux CI is red and the WAL quiet-period hole can freeze a partial Cursor transcript write-once.

🐦‍⬛ Reviewed by Crow via Cursor

[dgershman]

Code & Security Review

Critical Issues

None.

Architecture / Existing Patterns

• Existing pathway: CodingAgent.logSources → AgentLogSource directory + cwdFilter (Codex is the reference) → LogSyncCollector.resolveFiles / applyingCwdFilter → TranscriptNormalizer format dispatch → TranscriptUploader; backfill via BackfillScanner.reconstruct* + daemon-injected home (CodexHome / now CursorHome). This is the right extension — CursorStore in CrowCore is justified because the normalizer and scanner live there and CrowCore cannot import CrowCursor. CursorHome correctly mirrors CodexHome (including empty-$CURSOR_CONFIG_DIR is unset).
• [Yellow] Live logSources is looser than backfill, breaking the lockstep CROW-1089 added fileNamePrefix to enforce. Confirmed against Packages/CrowCursor/Sources/CrowCursor/CursorAgent.swift:333-336 (fileExtension: "db" only) vs Packages/CrowCore/Sources/CrowCore/LogSync/BackfillScanner.swift:301 (exact store.db). Codex selects rollout-*.jsonl on both paths (fileNamePrefix: "rollout-"). A stray *.db under <CursorHome>/chats would be cwd-filtered and fed to CursorStore. Set fileNamePrefix: "store" (or equivalent) so live and backfill stay in lockstep.

Security Review

Strengths:

• Opt-in, gateway-reuse upload path is unchanged (destination/credential still local-only).
• Attribution is exact cwd match; missing cwd is never guessed on the live path (applyingCwdFilter + CursorStore.recordedCwd).
• SQL is parameterized (blobs.id bound); meta['0'] is a literal. Open is read-only so the collector does not take a write lock on a live cursor-agent store.
• Fail-closed on non-UTF-8 / non-JSON blobs (jsonLine) matches the ticket's "wrong bytes are worse than none" bar. Accepted as documented.
• $CURSOR_CONFIG_DIR= empty-is-unset in CursorHome avoids a CWD-relative chats tree (same foot-gun Codex already fixed).
• blobEncryptionKey is not uploaded; only plaintext message blobs leave the machine.

Concerns:

• [Yellow] Quiet-period ignores WAL mtime, so a still-growing Cursor chat can look quiescent. Reproduced with a throwaway Python sqlite3 script: after PRAGMA journal_mode=WAL + a committed INSERT, store.db mtime was unchanged and store.db-wal was newer. Confirmed against Packages/CrowCursor/Sources/CrowCursor/CursorAgent.swift:335 (fileExtension: "db" drops store.db-wal / store.db-shm) and Packages/CrowDaemon/Sources/CrowDaemon/LogSyncCollector.swift:119-127 (newestModification runs only on resolved store.db files). Default quiet period is 30 minutes (LogSyncConfig). The server is write-once (409); an early upload freezes a partial transcript and the ledger will skip a later backfill. Include sibling -wal/-shm mtimes in the quiet-period calculation (without parsing those files as stores). The PR already knows these siblings exist (Packages/CrowCore/Sources/CrowCore/LogSync/CursorStore.swift:168-172).

Code Quality

• [Yellow] docs/harness-transcript-locations.md still marks Cursor "Deferred — needs blob extractor". That file exists so the next harness wiring does not re-discover a path, and docs/session-backfill.md still points at it. This PR updates session-log-collector.md / session-backfill.md / CLI help / settings copy, but leaves the field-reference table stale (Cursor row still Deferred · #1095). Flip it to Wired (CROW-1095) in the same change.
• Duplicate ## Scope heading introduced in docs/session-backfill.md:102-104 (the diff added a second heading instead of editing the existing one). [Green]
• concatenateCursorStores loads every message line into memory before applying maxBytes, unlike the incremental NDJSON reader in the same type. Fine for typical chats; worth tightening if stores can be tens of MB. [Green]
• Encryption defense is a UTF-8 / leading-{/[ heuristic, not JSONSerialization. Documented and accepted. [Green]
• Linux CursorStore is a documented no-op (canImport(SQLite3)), matching CrowTelemetry's Darwin-only SQLite pattern. Local macOS CursorStoreTests 7/7, CrowCursor 85/85, LogSyncCollectorTests 20/20. Not re-blocking: the PR explicitly accepts this. [Green]

Summary Table

Color	Meaning	Verdict effect
Red	Must fix	Request changes
Yellow	Should fix	Request changes
Green	Consider	Approve allowed

Recommendation: Request Changes — driven by 0 Red, 3 Yellow, 4 Green findings.

---

🐦‍⬛ Reviewed by Crow via Cursor

[dhilgaertner]

Thanks for the thorough review. Addressed below.

[Red] Linux CI — fixed, but the mechanism is module absence, not a read-only-open seam

CI is green again via #if canImport(SQLite3) gating (commit 860de3b) plus the follow-up here. A note on the root cause, because it changes the fix:

canImport(SQLite3) is false on the swift:6.1 Linux image — SQLite3 is a Darwin-SDK clang module, not part of the Linux Swift toolchain (the same reason CrowTelemetry, which also #if canImport(SQLite3)s, is Darwin-only and excluded from LINUX_PACKAGES). So on Linux the whole SQLite path in CursorStore compiles to the #else return nil no-op; nothing ever calls sqlite3_open_v2.

The read-only-open hypothesis is actually ruled out by the failure itself: the fixture creates the DB with sqlite3_open in rollback (non-WAL) mode, so if the module were present, sqlite3_open_v2(…READONLY) on that fresh, -shm-less file would succeed and the tests would pass. They returned nil instead — which only happens if the fixture's sqlite3_open was compiled out. (If the fixture had failed to create the DB, the failure would surface at its own #expect(sqlite3_open == SQLITE_OK), not at messageLines.)

The fix therefore matches the project's existing boundary:

• The SQLite-backed extraction/normalizer tests are #if canImport(SQLite3)-gated → run on macOS (release.yml), skipped on Linux, exactly as CrowTelemetry's SQLite tests are.
• The fixture now always writes the store.db file (real DB on macOS, placeholder on Linux) + meta.json, so the pure-Foundation backfill-scan / cwd / protobuf / JSON-guard tests keep running on Linux.

On a Linux crowd (CROW-645): the extractor returns nil → the normalizer returns nil → the collector continues, so it uploads nothing (never partial/garbage) — fail-closed, matching the ticket's bar. Making Cursor extraction actually run on Linux needs a CSQLite system-library target + libsqlite3-dev in the Linux lanes; happy to do that as a follow-up if Linux + cursor-agent is a real target, but it's CI-infra scope beyond this feature (and in practice cursor-agent + ~/.cursor + the Crow daemon are the user's Mac).

[Yellow] Live/backfill lockstep — fixed

CursorAgent.logSources now passes fileNamePrefix: "store" alongside fileExtension: "db", so the live path selects exactly store.db, in lockstep with BackfillScanner.cursorStoreFiles's exact-name match. A stray *.db can no longer reach CursorStore.

[Yellow] Quiet-period ignores WAL mtime — fixed

LogSyncCollector.newestModification now also probes each file's -wal/-shm siblings, so a chat still committing to store.db-wal is not read as quiescent and frozen partial under the 409. No-op for Claude/Codex (no such siblings). New test: newestModificationIncludesWalShmSiblings.

[Yellow] harness-transcript-locations.md stale — fixed

Cursor row flipped to Wired (CROW-1095), and the content-filtered attribution mode now notes the sibling-meta.json cwd variant.

[Green] Acknowledged

concatenateCursorStores in-memory accumulation and the UTF-8/opener encryption heuristic are both documented and accepted; a chat is bounded well under the byte cap, so the incremental reader isn't warranted yet.

[dgershman]

Code & Security Review

Critical Issues

None. The prior round's Red (Linux CursorStore tests returning nil) and Yellows (live/backfill filename lockstep, WAL-blind quiet period, stale harness-transcript-locations.md row) are addressed on 87270c0 and the preceding canImport(SQLite3) gate.

Architecture / Existing Patterns

• Existing pathway: CodingAgent.logSources → AgentLogSource directory + cwdFilter (Codex is the reference) → LogSyncCollector.resolveFiles / applyingCwdFilter → TranscriptNormalizer format dispatch → TranscriptUploader; backfill via BackfillScanner.reconstruct* + daemon-injected home (CodexHome / now CursorHome). This is the right extension — CursorStore in CrowCore is justified because the normalizer and scanner live there and CrowCore cannot import CrowCursor. CursorHome correctly mirrors CodexHome (including empty-$CURSOR_CONFIG_DIR is unset).
• Prior Yellows, verified in this HEAD:
  • Packages/CrowCursor/Sources/CrowCursor/CursorAgent.swift now sets fileNamePrefix: "store" alongside fileExtension: "db", matching BackfillScanner.cursorStoreFiles's exact store.db name.
  • Packages/CrowDaemon/Sources/CrowDaemon/LogSyncCollector.swift newestModification probes -wal/-shm siblings (covered by newestModificationIncludesWalShmSiblings).
  • docs/harness-transcript-locations.md Cursor row is Wired (CROW-1095).

Security Review

Strengths:

• Opt-in, gateway-reuse upload path is unchanged (destination/credential still local-only).
• Attribution is exact cwd match; missing cwd is never guessed on the live path (applyingCwdFilter + CursorStore.recordedCwd).
• SQL is parameterized (blobs.id bound); meta['0'] is a literal. Open is read-only so the collector does not take a write lock on a live cursor-agent store.
• Fail-closed on non-UTF-8 / non-JSON blobs (jsonLine) matches the ticket's "wrong bytes are worse than none" bar. Previously accepted.
• $CURSOR_CONFIG_DIR= empty-is-unset in CursorHome avoids a CWD-relative chats tree (same foot-gun Codex already fixed).
• blobEncryptionKey is not uploaded; only plaintext message blobs leave the machine.

Concerns:

• None new. Linux CursorStore remains a documented no-op (#if canImport(SQLite3)), matching CrowTelemetry's Darwin-only SQLite pattern — previously accepted, not re-blocked.

Code Quality

• [Green] docs/session-backfill.md still has a duplicate ## Scope heading (lines 102–104). Harmless; leftover from inserting a second heading instead of editing the existing one.
• [Green] Packages/CrowCore/Sources/CrowCore/LogSync/BackfillScanner.swift reconstructCursor still says a no-cwd chat "is dropped"; assemble always returns a session and nil cwd yields .low (the test is cursorChatWithoutCwdIsDroppedToLowConfidence). Live collection does drop. Tighten the comment so backfill vs live is not confused. Not a logic bug; restated from the prior round, not re-raised as blocking.

Tests

• macOS: CursorStoreTests 7/7, CrowCursor 85/85, BackfillScannerTests 8/8, LogSyncCollectorTests 21/21, BackfillServiceTests 7/7.
• Linux Build & Test was still in flight at review time; the SQLite-backed extraction tests are #if canImport(SQLite3)-gated so the previous 6 failures should not recur. Pure-Foundation scan/cwd/protobuf/JSON-guard tests keep running.

Summary Table

Color	Meaning	Verdict effect
Red	Must fix	Request changes
Yellow	Should fix	Request changes
Green	Consider	Approve allowed

Recommendation: Approve — driven by 0 Red, 0 Yellow, 2 Green findings.

---

🐦‍⬛ Reviewed by Crow via Cursor