Wire OpenCode into the session-log collector + backfill (CROW-1096)

[dhilgaertner]

Closes #1096

Wires OpenCode end-to-end into the session-log collector (CROW-1056) and historical backfill (CROW-1075) — live upload and backfill — the third harness after Claude and Codex (CROW-1089). Follows the pattern of PR #1092 (Codex).

The shape problem — why OpenCode needed a reassembler

Claude and Codex each write one NDJSON transcript per session with the cwd on a head line, so the shared pipeline fits them: AgentLogCwdReader reads the cwd from the head, TranscriptNormalizer concatenates the files.

OpenCode is a multi-file object store. A single session is scattered across:

storage/project/<projectID>.json          # { id, worktree }
storage/session/<projectID>/ses_*.json     # { id, projectID, directory, parentID?, time }
storage/message/<sessionID>/msg_*.json     # one user/assistant message
storage/part/<messageID>/prt_*.json        # the message's content parts

— all pretty-printed single-object JSON, with the cwd on the session anchor's directory field (fallback the project's worktree), not a head line. The head-cwd reader and file-concatenation normalizer don't fit, so a reassembler was needed.

How

• OpenCodeStore (CrowCore) — a pure reader: parses a session anchor (cwd + parentID), enumerates sessions, and reassembles a session's message + part records into ordered NDJSON — session header, then each message in id order, each followed by its parts in id order. OpenCode's msg_/prt_ ids are monotonic ascending by creation time (verified against a live store: a filename sort equals a time.created sort).
• AgentLogFormat.openCodeStore — an internal normalization discriminator. The reassembled artifact is NDJSON, so it uploads stamped as .logDir via the new artifactStamp; the server never sees openCodeStore.
• TranscriptNormalizer reassembles for .openCodeStore; LogSyncCollector's cwd filter is now format-aware — OpenCode parses the whole session anchor for directory and drops child/subagent sessions.
• OpenCodeAgent.logSources returns a recursive, cwd-filtered .openCodeStore source over the session/ tree (path via OpenCodeHome, $XDG_DATA_HOME-aware).
• Backfill: BackfillScanner reconstructs OpenCode sessions (harness .opencode, cwd from the anchor, no git branch → ticket from the worktree name); BackfillService keys ledger / upload / format / agentKind off the harness.

Attribution invariant

Exact cwd match; a session with no readable cwd is dropped, never guessed. Child/subagent sessions (parentID != nil) are excluded — they belong to a parent, mirroring Claude's subagents/ exclusion.

Acceptance

• OpenCode wired end-to-end: live collector and backfill.
• Reassembled transcript order correct and stable; cwd→session attribution cannot misattribute (exact match, no-cwd dropped, children excluded).
• swift test green; new tests cover the object-store reassembler and OpenCode backfill reconstruction.
• Docs updated (session-log-collector, session-backfill, cli-reference, cli, CLAUDE.md).

Testing

swift test green: CrowCore (775), CrowOpenCode (65), and the CrowDaemon LogSync/Backfill suites. New tests cover the object-store reassembler (ordering, byte cap, child exclusion, project-worktree fallback), OpenCodeHome XDG resolution, the .openCodeStore logSources shape, OpenCode backfill reconstruction + harness-scoped ledger keying, and the collector's OpenCode cwd filter. Root swift build (crow + crowd) links clean.

Note

Two CrowDaemon tests fail on the base branch already, unrelated to this change and outside its diff: RPCLanePolicyTests (lane drift for the pre-existing terminal-set/corveil-verify/corveil-reinstall-skill verbs, CROW-1085/CROW-1011) and WebNotificationCenterTests.bootCatchResetsTheHistory (an app.js anchor string). This PR touches no RPC handlers, lanes, or web assets.

🤖 Generated with Claude Code

[dgershman]

Code & Security Review

Critical Issues

• Red — collector reads a JSON object store OpenCode 1.18.x no longer writes. OpenCodeHome.sessionDir() (Packages/CrowOpenCode/Sources/CrowOpenCode/OpenCodeHome.swift:39-42) and OpenCodeAgent.logSources (Packages/CrowOpenCode/Sources/CrowOpenCode/OpenCodeAgent.swift:205-208) point at ~/.local/share/opencode/storage/session/**/ses_*.json. OpenCodeStore.sessionFiles (Packages/CrowCore/Sources/CrowCore/LogSync/OpenCodeStore.swift:123-135) enumerates that tree; if it is missing, the enumerator returns [] and the live collector silently skips (Packages/CrowDaemon/Sources/CrowDaemon/LogSyncCollector.swift:117-121). Docs (docs/session-log-collector.md) present this layout as the wired location.

  Verified (reproduced): OpenCode 1.18.15 (in Crow's documented 1.17.10+/1.18.x window) on this machine has no ~/.local/share/opencode/storage/ directory. Sessions live in ~/.local/share/opencode/opencode.db (SQLite): 34 session rows, 392 message rows, 1829 part rows, with session.directory and session.parent_id as the attribution columns this PR expected on JSON anchors. Current upstream session persistence is SQLite (Database), not Storage.write(["session", …]). A sweep against this install uploads nothing.

  The new tests pass because they stage a synthetic storage/{project,session,message,part} tree that the live harness does not create. Point logSources / backfill at opencode.db (and keep a JSON fallback only if 1.17 still writes it).

Architecture / Existing Patterns

• Existing pathway: CodingAgent.logSources → LogSyncCollector.resolveFiles (cwdFilter) → TranscriptNormalizer.normalize, as extended for Codex (CROW-1089). Extending that pipeline with a format-aware cwd probe is the right shape.
• Red (same defect as above): the on-disk source is the wrong current OpenCode format. AgentLogFormat.sqlite already exists as the Cursor placeholder (Packages/CrowCore/Sources/CrowCore/Agent/AgentLogSource.swift:12-18); 1.18.x OpenCode is a SQLite store (opencode.db, relational session/message/part), not a scattered JSON object store. .openCodeStore + the JSON reassembler invent a parallel reader for a layout that is gone. Schema differs from Cursor's blob store.db, so this still needs an OpenCode-specific SQLite normalizer — but it should sit on the existing .sqlite (or a sibling) path over OpenCodeHome.dataDir()/opencode.db, not storage/session/.

Security Review

Strengths:

• Attribution stays exact-cwd, drop-on-miss — no guessing a worktree from a slug (OpenCodeStore.cwd(ofSessionFile:) returns nil for children and for missing directory).
• Child/subagent sessions (parentID / isChild) are excluded, matching Claude's subagents/ rule.
• Upload still goes only through the workspace's local-only gateway; .openCodeStore is stamped .logDir so an unknown format never hits the server enum (AgentLogFormat.artifactStamp).
• Opt-in remains default-off.

Concerns:

• None beyond the Red: a silent no-op is not an exfil risk, but it also means the security invariant "we upload what this worktree actually ran" is vacuously true (nothing is uploaded).

Code Quality

• Yellow — truncated session 1 can be followed by session 2. TranscriptNormalizer.reassembleOpenCodeStore (Packages/CrowCore/Sources/CrowCore/LogSync/TranscriptNormalizer.swift:117-136) ORs wasTruncated and continues the loop. OpenCodeStore.reassemble leaves leftover budget when a large line will not fit, so the next cwd-matched session's header can be appended after a cut transcript. Codex concatenateNDJSON stops at the cap (break). If the JSON reassembler is kept as a legacy fallback, break after wasTruncated (and add a two-session fixture under a tight cap).
• Tests never open a real opencode.db, so they cannot catch the layout miss. A fixture copied from 1.18.x schema (or a "storage dir absent → zero files" case against the live path) should be part of the fix.
• Green: XDG_DATA_HOME empty-string handling matches the existing XDG_CONFIG_HOME guard. Message/part filename sort is correct for OpenCode's ascending msg_/prt_ ids (sessions themselves are descending ULIDs, but those are not used as the concat order — mtime is).

Summary Table

Color	Meaning	Verdict effect
Red	Must fix	Request changes
Yellow	Should fix	Request changes
Green	Consider	Approve allowed

Recommendation: Request Changes — driven by 1 Red, 1 Yellow, 1 Green findings.

---

🐦‍⬛ Reviewed by Crow via Cursor

[dgershman]

Code & Security Review

Critical Issues

• Red — live collector and backfill read a JSON object store current OpenCode no longer writes. OpenCodeHome.sessionDir() (Packages/CrowOpenCode/Sources/CrowOpenCode/OpenCodeHome.swift:39-42) and OpenCodeAgent.logSources (Packages/CrowOpenCode/Sources/CrowOpenCode/OpenCodeAgent.swift:205-208) point at ~/.local/share/opencode/storage/session/**/ses_*.json. OpenCodeStore.sessionFiles (Packages/CrowCore/Sources/CrowCore/LogSync/OpenCodeStore.swift:123-135) enumerates that tree; a missing directory makes the enumerator return [], and LogSyncCollector silently skips (Packages/CrowDaemon/Sources/CrowDaemon/LogSyncCollector.swift:120-121). Docs present this layout as the wired location (docs/session-log-collector.md:46).

  Verified (reproduced): OpenCode 1.18.15 (inside Crow's documented 1.17.10+ / 1.18.x window in docs/agent-harness-matrix.md) on this machine has no ~/.local/share/opencode/storage/ directory. Sessions live in ~/.local/share/opencode/opencode.db (SQLite): 34 session rows, 392 message rows, 1829 part rows. The attribution columns this PR expected on JSON anchors are session.directory and session.parent_id. A sweep against this install resolves zero files and uploads nothing.

  New tests pass because they stage a synthetic storage/{project,session,message,part} tree the live harness does not create. Point logSources / backfill at opencode.db (keep a JSON fallback only if a still-supported 1.17 build actually writes it — 1.17.3 already used SQLite).

Architecture / Existing Patterns

• Existing pathway: CodingAgent.logSources → LogSyncCollector.resolveFiles (cwdFilter) → TranscriptNormalizer.normalize, as extended for Codex (CROW-1089). Extending that pipeline with a format-aware cwd probe is the right shape; this is not a parallel collector.
• Red (same defect as above): the on-disk source is the wrong current OpenCode format. AgentLogFormat.sqlite already exists as the Cursor placeholder (Packages/CrowCore/Sources/CrowCore/Agent/AgentLogSource.swift:12-18). 1.18.x OpenCode is a SQLite store (opencode.db, relational session / message / part), not a scattered JSON object store. .openCodeStore plus the JSON reassembler invent a reader for a layout that is gone. Schema differs from Cursor's blob store.db, so this still needs an OpenCode-specific SQLite normalizer — but it should sit on the existing .sqlite (or a sibling) path over OpenCodeHome.dataDir()/opencode.db, not storage/session/.

Security Review

Strengths:

• Attribution stays exact-cwd, drop-on-miss — no guessing a worktree from a slug (OpenCodeStore.cwd(ofSessionFile:) returns nil for children and for a missing directory).
• Child/subagent sessions (parentID / isChild) are excluded, matching Claude's subagents/ rule.
• Upload still goes only through the workspace's local-only gateway; .openCodeStore is stamped .logDir so an unknown format never hits the server enum (AgentLogFormat.artifactStamp).
• Opt-in remains default-off.

Concerns:

• None beyond the Red: a silent no-op is not an exfil risk, but the security invariant "we upload what this worktree actually ran" is vacuously true (nothing is uploaded).

Code Quality

• Yellow — after a truncated session, leftover byte budget can start the next session. TranscriptNormalizer.reassembleOpenCodeStore (Packages/CrowCore/Sources/CrowCore/LogSync/TranscriptNormalizer.swift:117-136) ORs wasTruncated and continues the loop. OpenCodeStore.reassemble leaves unused budget when the next line will not fit, so the following cwd-matched session's header can be appended after a cut transcript. Codex concatenateNDJSON stops at the cap (break / break outer, same file ~81-91). If the JSON reassembler is kept as a legacy fallback, break after wasTruncated and add a two-session fixture under a tight cap.
• Tests never open a real opencode.db, so they cannot catch the layout miss. A fixture copied from 1.18.x schema (or a "storage dir absent → zero files" case against the live path) should be part of the fix.
• Green: empty XDG_DATA_HOME= is treated as unset, matching the existing XDG_CONFIG_HOME guard (OpenCodeHome.swift:20-23). Message/part filename sort is correct for OpenCode's ascending msg_ / prt_ ids. Child-session exclusion and artifactStamp → .logDir are well-tested on the synthetic tree.

swift test on this PR's new suites is green: CrowCore OpenCodeStore / artifactStamp (8), CrowOpenCode (65), CrowDaemon LogSync/Backfill OpenCode cases (21). That green is against staged JSON, not against current OpenCode.

Summary Table

Color	Meaning	Verdict effect
Red	Must fix	Request changes
Yellow	Should fix	Request changes
Green	Consider	Approve allowed

Recommendation: Request Changes — driven by 1 Red, 1 Yellow, 3 Green findings.

---

🐦‍⬛ Reviewed by Crow via Cursor

[dhilgaertner]

Thanks — the Red finding was correct and I've reworked the PR around it (pushed in 3fd2eba).

Red — wrong on-disk format. I independently reproduced it: this machine's OpenCode was an ancient 1.1.29 (JSON storage/ tree), which is why the original reader and its synthetic-tree tests looked fine. Checking upstream sst/opencode (packages/core/src/session/sql.ts) confirmed your report — 1.17.10+ (Crow's documented window) keeps every session in ~/.local/share/opencode/opencode.db, relational session/message/part tables, and the pre-1.17 JSON tree is migrated into the DB on upgrade. So the JSON reader was dead on every supported install.

Rewrote OpenCodeStore to read opencode.db:

• Attribution is the session.directory column (absolute cwd) selected inside the DB — the source is now a single .file at opencode.db, cwd-attributed by row, not by dropping files. Child (parent_id) and cwd-less rows are dropped.
• Messages ordered by (time_created, id), parts by id (the DB's own index order); each row's JSON data is re-serialized compact into NDJSON.
• Schema taken from upstream source, not guessed (directoryColumn() stores the absolute path; timestamps are epoch-ms).

Platform. The reader needs the SQLite3 module. The daemon that runs the collector is macOS-only (SQLite3 is an SDK module there — no new dependency, no CI change, no vendored amalgamation). On the Linux CI lane (compiles crowd/crow, runs no daemon) OpenCodeStore is a compiled #else no-op and its behavioral tests are gated to canImport(SQLite3), so they run on macOS.

Yellow — truncation. The SQLite reassembler now breaks the session loop the instant a line won't fit, so a cut transcript can't gain the next session's header. Covered by truncationStopsAndDoesNotStartNextSession (two sessions under a tight cap).

Tests now build a real opencode.db fixture (OpenCodeDBFixture) and exercise enumeration, the cwd probe (child/missing-dir dropped), ordering, multi-session concat, cwd selection, the truncation break, and end-to-end backfill upload. swift test green on macOS across CrowCore/CrowOpenCode/CrowDaemon. Docs updated to SQLite (session-log-collector, session-backfill, cli-reference, harness-transcript-locations).

[dgershman]

Code & Security Review

Critical Issues

None. The previous Red (JSON storage/ tree vs the live opencode.db SQLite store) is fixed: OpenCodeHome.databasePath() (Packages/CrowOpenCode/Sources/CrowOpenCode/OpenCodeHome.swift:32-36) points at <dataDir>/opencode.db, and OpenCodeStore (Packages/CrowCore/Sources/CrowCore/LogSync/OpenCodeStore.swift:234-270) reads session.directory / parent_id / time_created and message/part.data with bound parameters. Verified against a live OpenCode 1.17+ opencode.db on this machine: WAL sidecars present, 34 session / 392 message / 1829 part rows; PRAGMA table_info(session) matches the reader (directory, parent_id, time_created, time_updated). Child rows (parent_id set) are dropped; a missing cwd is dropped, never guessed.

The previous Yellow (truncated session 1 followed by session 2's header) is also fixed: emitSessions (OpenCodeStore.swift:165-174) breaks the session loop, covered by truncationStopsAndDoesNotStartNextSession.

Architecture / Existing Patterns

• Existing pathway: CodingAgent.logSources → LogSyncCollector.resolveFiles → format-aware normalize. Claude/Codex go through TranscriptNormalizer file concatenation; OpenCode correctly bypasses that for a cwd/session-id selector normalize(files:) cannot express (LogSyncCollector.swift:141-143). This is an extension of the existing pipeline, not a parallel collector.

• .openCodeStore as an internal discriminator that stamps .logDir on upload (AgentLogFormat.artifactStamp) is the right sibling to Cursor's unimplemented .sqlite — different schema, different selector. Do not collapse them.

• Yellow — file mtime is the wrong quiescence signal for a shared WAL database. OpenCodeAgent.logSources (Packages/CrowOpenCode/Sources/CrowOpenCode/OpenCodeAgent.swift:205-208) returns a single .file source at opencode.db. resolveFiles (LogSyncCollector.swift:271-274) returns only that path. The quiet-period gate (LogSyncCollector.swift:123-128) then uses newestModification(files) — the main file's contentModificationDate — to decide whether a still-running Crow session is idle enough to upload (write-once 409).

  Verified (reproduced): this install's OpenCode store is WAL (opencode.db-wal / opencode.db-shm). A throwaway WAL SQLite insert did not change the main .db mtime; only the -wal mtime moved. The collector never inspects the WAL sidecar. An in-progress OpenCode session can therefore look idle for longer than quietPeriodMinutes while new messages sit in the WAL; the collector uploads a snapshot, and the server's write-once 409 makes that snapshot permanent. The comments already note the DB is shared across every worktree when omitting agentSessionID (LogSyncCollector.swift:150-152); that same fact makes global file mtime the wrong grain.

  Key quiescence on MAX(time_updated) of the cwd-matched top-level session rows (the analogue of Codex's per-rollout file mtime). Including -wal in the mtime set without a cwd filter would still confuse this worktree with every other OpenCode session on the machine.

Security Review

Strengths:

• Exact-cwd attribution; missing directory and parent_id children dropped — no guessing.
• Read-only sqlite3_open_v2 + SQLITE_TRANSIENT binds; session id is a bound ?, not string-concatenated into SQL.
• Upload still only via the workspace's local-only gateway; .openCodeStore never hits the server enum.
• Opt-in remains default-off.

Concerns:

• None beyond the Yellow: a premature write-once upload is an integrity issue (incomplete transcript locked in), not an exfil issue.

Code Quality

• Tests now build a real SQLite fixture (OpenCodeDBFixture) and cover enumeration, cwd probe, child exclusion, ordering, multi-session concat, truncation, OpenCodeHome XDG (including empty XDG_DATA_HOME=), logSources shape, and backfill ledger keying / upload. swift test --filter OpenCode is green on CrowCore (10), CrowOpenCode, and CrowDaemon (2). Root swift build links crowd.
• Linux #else no-op is documented and matches the macOS-only daemon.
• Green: OpenCodeDBFixture is duplicated in CrowCore and CrowDaemon — a maintainability nit, not a defect.

Summary Table

Color	Meaning	Verdict effect
Red	Must fix	Request changes
Yellow	Should fix	Request changes
Green	Consider	Approve allowed

Recommendation: Request Changes — driven by 0 Red, 1 Yellow, 1 Green findings.

---

🐦‍⬛ Reviewed by Crow via Cursor

[dhilgaertner]

Thanks — addressed the Yellow (pushed in d16d92c).

Yellow — WAL / shared-DB quiescence. You're right that the opencode.db file mtime is the wrong idle signal: a WAL commit lands in the -wal sidecar without bumping the main file, and one DB is shared by every worktree, so the mtime is both stale and machine-global — an in-progress session could get a premature, write-once-permanent snapshot.

Fixed by keying the OpenCode quiescence gate on OpenCodeStore.newestActivity(databaseFiles:cwd:) instead of newestModification. Rather than the session row's time_updated alone, it takes the max of each cwd-matched top-level session's time_updated/time_created and its newest message/part time_created — those tables are append-only, so their timestamps move on every new turn even if the session row isn't rewritten (I didn't want to depend on whether upstream bumps the session row per message). A SQLite read sees WAL-committed rows, and the cwd filter scopes it to this worktree. Claude/Codex/Grok keep file-mtime quiescence.

New tests (all gated to canImport(SQLite3)): latest-write-wins across session/message/part; moves with messages when the session row is stale (the WAL case you described); cwd-scoped so another worktree's churn — and child sessions — don't count; nil when nothing matches or the DB is missing. Fixtures gained part.time_created to match the real schema. swift test green: CrowCore (790), CrowDaemon LogSync/Backfill (29); crow/crowd link.

Green — duplicated OpenCodeDBFixture. Left as-is intentionally: the two test targets are in separate packages (CrowCore, CrowDaemon), so sharing the helper would need either a new test-support module or moving test-fixture code into the production library — both disproportionate for one ~60-line helper. I've kept the two copies identical (both updated this round).

[dgershman]

Code & Security Review

Critical Issues

None. Prior Red (JSON storage/ tree vs live opencode.db) and Yellows (truncation continuing to the next session; opencode.db file mtime as the WAL quiescence signal) are fixed.

Architecture / Existing Patterns

• Existing pathway: CodingAgent.logSources → LogSyncCollector.resolveFiles → format-aware normalize. Claude/Codex/Grok go through TranscriptNormalizer file concatenation; OpenCode correctly bypasses that for a cwd/session-id selector normalize(files:) cannot express (Packages/CrowDaemon/Sources/CrowDaemon/LogSyncCollector.swift). This extends the existing pipeline rather than inventing a parallel collector.
• .openCodeStore as an internal discriminator that stamps .logDir on upload (AgentLogFormat.artifactStamp in Packages/CrowCore/Sources/CrowCore/Agent/AgentLogSource.swift) remains the right sibling to Cursor's unimplemented .sqlite.
• Prior Yellow on shared-WAL quiescence is addressed: OpenCodeStore.newestActivity (Packages/CrowCore/Sources/CrowCore/LogSync/OpenCodeStore.swift) keys idle-ness on cwd-matched session/message/part timestamps instead of the main-file mtime, and the collector actually calls it. Tests cover a stale session row plus a newer message, cwd scoping, and child exclusion.

Security Review

Strengths:

• Exact-cwd attribution; missing directory and parent_id children dropped — no guessing (Packages/CrowCore/Sources/CrowCore/LogSync/OpenCodeStore.swift, Packages/CrowCore/Sources/CrowCore/LogSync/BackfillScanner.swift).
• Read-only sqlite3_open_v2 + SQLITE_TRANSIENT binds; session id is a bound ?. Table-name interpolation in maxTimeCreated is a compile-time "message" / "part" constant, documented as such.
• Upload still only via the workspace's local-only gateway; .openCodeStore never hits the server enum.
• Opt-in remains default-off.

Concerns:

• None blocking. Upstream also has a V2 session_message table; this reader uses message / part, which a prior review verified are populated on a live 1.17+ opencode.db. If OpenCode ever writes only session_message, both reassembly and quiescence would go empty/stale — a version-pin target, not a defect in this change.

Code Quality

• Green — leftover duplicated comments in Packages/CrowCore/Sources/CrowCore/LogSync/BackfillScanner.swift ("Three harnesses" then "Four harnesses") and Packages/CrowDaemon/Sources/CrowDaemon/LogSyncCollector.swift / Packages/CrowDaemon/Sources/CrowDaemon/BackfillService.swift. Harmless merge residue.
• Green — OpenCodeDBFixture duplicated across CrowCore and CrowDaemon test targets. Author declined sharing (separate packages; a test-support module is disproportionate). Restating as Green only.
• Green — readMessages / readParts load the whole session before maxBytes can stop fetching (Packages/CrowCore/Sources/CrowCore/LogSync/OpenCodeStore.swift). concatenateNDJSON reads incrementally against the same cap. Output is still bounded; consider a cursor if real part tables get huge.
• Green — sizeOverride: 0 on OpenCode backfill rows (Packages/CrowCore/Sources/CrowCore/LogSync/BackfillScanner.swift) so the UI does not show the whole-DB size on every session. Conscious tradeoff.
• Green — the GitHub PR description still describes the JSON object store (storage/session/**/ses_*.json, recursive logSources). The code and in-repo docs are SQLite; worth updating the GitHub body so the next reader is not sent to a layout that is gone.

Static analysis: swift test was not re-run in this pass (environment flake); the author reports CrowCore / CrowOpenCode / CrowDaemon green after the quiescence fix, matching the prior review's swift test --filter OpenCode result.

Summary Table

Color	Meaning	Verdict effect
Red	Must fix	Request changes
Yellow	Should fix	Request changes
Green	Consider	Approve allowed

Recommendation: Approve — driven by 0 Red, 0 Yellow, 5 Green findings.

---

🐦‍⬛ Reviewed by Crow via Cursor