feat(analytics): add GitHub Copilot CLI sessions to the analytics report - #449
Open
pigorv wants to merge 19 commits into
Open
feat(analytics): add GitHub Copilot CLI sessions to the analytics report#449pigorv wants to merge 19 commits into
pigorv wants to merge 19 commits into
Conversation
Phase 0 spike over 78 real local Copilot CLI sessions, verified technical analysis, approved spec, and 11-task implementation plan for adding GitHub Copilot CLI sessions to the analytics report. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…nifest reader Discovery manifest reader for GitHub Copilot CLI sessions. workspace.yaml is present in every session directory and carries identity, project path, repo, branch and timestamps, so discovery never opens a multi-megabyte transcript. Honors COPILOT_HOME. Returns null rather than throwing on missing, malformed, or id-less manifests so one bad session directory cannot abort discovery. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
… plugin discoverSessions enumerates ~/.copilot/session-state/<uuid>/, reading only the workspace.yaml manifest so a report run never opens a transcript it will filter out. Skips directories with no events.jsonl (pre-schema sessions with nothing to parse), honors maxAgeDays/cwd/limit, and sorts newest-first. Adds AgentMetadata.analyticsOnly, marking agents CodeMie reads analytics for but never manages. The plugin overrides install/uninstall/run to refuse: inherited, they would npm-install and launch @github/copilot, which is outside this integration's remit. Verified before commit: eslint clean, tsc --noEmit clean, 15/15 unit tests pass. --no-verify used with explicit user approval: two acceptance tests in cost-enricher.test.ts fail on this machine independently of this change (proven by reverting src/ to origin/main), because they assert hardcoded USD totals against live local Claude transcripts that have since grown. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Three tiers: session.shutdown.modelMetrics (authoritative), per-turn assistant.message reconstruction (output tokens and request counts only, marked partial), then no-usage with a reason. Buckets are emitted raw in Copilot's OpenAI-convention field names so the convention conversion has a single seam in readCopilotCli. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…ecomposition Copilot reports inputTokens INCLUSIVE of cacheReadTokens (OpenAI convention, applied even to Anthropic models), while costBreakdown bills input at full rate AND cacheRead separately. readCopilotCli therefore decomposes: freshInput = inputTokens - cacheReadTokens input = freshInput - cacheWriteTokens Verified against a real session: input resolves to 381,719 rather than the raw 14,076,695 a pass-through would bill — a ~36x over-count. reasoningTokens is a subset of outputTokens and is never billed separately. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…totals The cost enricher computes run-level totals through gatherUsageDeduped, not readUsageByModel. Without this branch, per-session numbers are correct while report totals stay $0 — a silent, plausible-looking failure. Copilot is session-local, so no cross-session dedup key is needed. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…h metrics One pass over events.jsonl produces metadata from session.start, raw per-model usage buckets as messages, and metrics: tool counts with success/failure status, file operations from session.shutdown.codeChanges, user prompts, and skill invocations. Mapping skill.invoked into metrics.skillInvocations lets detectSessionSource classify Copilot sessions in the report's Source column the same way it does Claude sessions, rather than defaulting them all to 'Pure chat'. Adds ParsedSession.usageMeta so the enricher can tell 'cost is genuinely zero' from 'cost is unmeasurable'. Named generically rather than after one agent: usagePartial and usageUnavailableReason apply to any agent with incomplete telemetry. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
native-loader resolves session adapters via AgentRegistry.getAgent(name) .getSessionAdapter(), so registration is required for discovery to reach the adapter at all. Updates two registry tests that asserted an exact plugin count of 8. The first now asserts membership by name, so a future plugin change fails with the offending name rather than an opaque length mismatch; the second derives its expected length instead of hardcoding it. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…he ownership gate native-loader.ts:518 tags natively-discovered sessions lacking a CodeMie ownership marker as native-external, and sources/sessions-source.ts filters those out by default. A Copilot session can never carry such a marker, so without this exemption 100% of them are dropped and the feature ships showing nothing. Also corrects two parsing bugs that end-to-end verification exposed, both from assumptions the unit-test fixtures had encoded rather than reality: - tool.execution_complete carries NO tool name and no status string — only toolCallId plus a boolean success. The name and arguments live on tool.execution_start, so the two must be paired by toolCallId. Tool metrics were silently empty before this. - messages now also carry Claude-shaped per-turn records, so synthesizeRawSession derives turns, models, timestamps, cwd and branch correctly. readCopilotCli filters on model+usage and turn counting filters on type='assistant', so the two consumers pick up only what each needs. session.shutdown line totals now merge into the per-file operations instead of appending a duplicate entry for an already-recorded path. Verified against 41 real local sessions: 746 turns, 1454 tool calls (97.7% success), correct per-model attribution, and input priced at 406,071 tokens rather than the ~15.7M a pass-through would have billed. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…s in the report payload
Carries ParsedSession.usageMeta through the cost enricher onto SessionCost and
ReportSessionRecord: premiumRequests (GitHub's real billing unit, which does not
track token volume), usagePartial, and usageUnavailableReason.
All three are optional and spread conditionally, so records for agents that
record full usage are unchanged in shape. Aggregate coverage needs no new
mechanism — the existing AgentCoverage {total, priced, withLog} already carries
the 'N of M unpriced' story.
Verified against real data: 6 sessions report premium requests (3,3,1,1,1,1),
35 carry an unavailable reason.
Generated with AI
Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
… premium requests Adds a GitHub-neutral brand color (#6E7681, the only unsaturated entry so it cannot collide with an existing agent or a PALETTE slot) and introduces an agent display-label mechanism, which the report did not have — raw keys rendered directly, and the Agents-Compare table's text-transform:capitalize would have shown 'Copilot-cli'. Capitalize is now applied only to unmapped keys, so every other agent looks unchanged. labelFor() is wired into the agent chips, the Agents-Compare table and the session modal header; agentLabel() does the same for terminal output. The session modal gains Copilot-only rows: premium requests, a partial-usage note, and the reason a session could not be priced. All render only when present. Note: agent-labels.ts and app.js necessarily hold separate copies of the label map — app.js is a standalone browser bundle that cannot import TypeScript, the same reason AGENT_COLORS lives only there. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…ted limitations Documents two events.jsonl shape assumptions the spec got wrong (tool name lives on execution_start, not _complete; messages must carry Claude-shaped per-turn records for native synthesis) and the accepted D11 limitation where Copilot's un-namespaced skill names do not match the Source detector's prefix rule. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…shutdown rollup
Only the newest Copilot CLI builds put `model` on each assistant.message; 1.0.17
and every 0.0.x omit it, so 38 of 41 local sessions showed 'unknown model' —
including three that were priced and had the model in their own cost breakdown.
Resolution order, most authoritative first:
1. per-turn assistant.message.model
2. the model in effect from the preceding session.model_change, tracked
chronologically so a mid-session switch attributes each turn correctly
3. shutdown modelMetrics when the session used exactly ONE model
Deliberately does not guess when turns are unlabelled and the rollup shows two
or more models — attributing a turn to either would be a fabrication.
Real data: sessions with an attributed model 3 -> 25 of 41, priced-but-unattributed
3 -> 0, distinct models surfaced 3 -> 5 (adds gpt-5-mini, claude-sonnet-4.6).
The remaining 16 carry no model-bearing field anywhere in their transcripts.
Generated with AI
Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…odified file Two defects found reviewing the diff: 1. cacheCreation was not clamped. With inconsistent buckets, input clamped to 0 while the full cacheWriteTokens was still billed — and cache creation prices ABOVE the base input rate, so garbage data could over-charge. Now clamped to the fresh input it was written from, which also makes input non-negative by construction rather than by a second Math.max. The existing test asserted input===0 but never checked cacheCreation, so it passed while over-billing. 2. session.shutdown line totals were attributed to fileOperations[0], which may be a file a tool merely opened. Now prefers an entry that shutdown actually lists in filesModified. Neither changes real-world output: the measured session still reports input 406,071 and cacheCreation 125,660. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Inline review of the 39-file diff. Two defects found and fixed: cache-write tokens were unclamped (over-billing risk on inconsistent buckets, in the one reader whose purpose is not over-billing), and code-churn totals could be credited to a file that was never modified. Verdict records medium rather than high confidence: this was a self-review by the change's author, and the two schema bugs found earlier in this task were both cases where the author's own fixtures encoded a wrong assumption — the exact failure mode a self-review is weakest against. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…anch The 254KB code-review.diff was a snapshot of this very branch's diff, so it got counted inside the diff — 5,820 of the 11,311 lines that pushed the change past the ultrareview size limit and blocked an independent review. It is regenerable at any time with git diff <merge-base>...HEAD, and it is a minority practice: only 4 of 22 task dirs on main commit one, the largest at 55KB. Gitignored so it cannot recur. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Overriding install()/uninstall()/run() to refuse was not enough. codemie update never calls the adapter — updateAgent() reads metadata.npmPackage and runs npm install -g <pkg> --force, so a user with Copilot on PATH would have seen it PRE-CHECKED in the update picker and had their own global install overwritten by an agent CodeMie documents as one it does not manage. Adds AgentRegistry.getManageableAgents() (all agents minus analyticsOnly) and routes the nine management call sites through it: update, install, uninstall, list and first-time. getInstalledAgents() filters too — all four of its consumers (uninstall, list, doctor x2) are management surfaces, and 'installed' is meaningless for an agent CodeMie never installs. getAllAgents() still returns them so the analytics pipeline can resolve session adapters. Copilot's npmPackage is now null (the existing 'null for built-in' idiom, already guarded by every consumer), so even a missed surface fails safe with 'cannot be updated' rather than destructively. Verified: Copilot absent from list/install/uninstall/doctor/update; analytics still discovers all 41 sessions with full model attribution. Found by workflow-backed review; missed by the inline review, which checked that the overrides refused but not whether other paths reach around them. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…dentifiers This repository is public. The committed task artifacts were derived from real local Copilot and Claude transcripts, so they are removed entirely and the identifiers that reached test fixtures are replaced with synthetic values: - session UUIDs -> 11111111-2222-3333-4444-555555555555 - a real session title -> 'example session title' - real branch/repo names -> feat/example, example-org/example-repo - session attribution dropped from test comments Token figures are kept: they carry no identifier on their own and are the exact values that expose the cache-inclusive over-billing bug, so replacing them would weaken the regression guard for no privacy gain. Verified clean: no home paths (only /Users/x/ placeholders), no GitHub logins, no emails, no session UUIDs, no session titles, no references to unrelated private projects. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Five findings from the workflow-backed review:
1. Tier-2 per-turn fallback required data.model, which the CLI builds it exists
for do not write, making it unreachable. Now tracks the model in effect from
session.model_change. NOTE: this recovers nothing on currently-observed data —
all 48 shutdown-less local sessions have no per-turn outputTokens either, so
the review's stated impact ('sessions labelled no-telemetry when tokens are
recoverable') does not hold. It fixes the latent case: a crashed 1.0.17
session, which writes outputTokens but no model and has no shutdown event.
2. Analytics-only sessions were tagged provider 'native', identical to
CodeMie-launched ones, so managed and unmanaged usage became
indistinguishable. Now 'native-unmanaged' — still included by default (the
filter excludes only 'native-external') but honest about origin.
3. File operations were recorded at tool.execution_start, before success was
known, so a failed edit/create counted as a changed file. Now recorded on
successful completion only.
4. labelFor() reached only 3 of 8 agent render sites in the report, and the
session-table search matched the raw key, so a user who searched the label
they could see got no results. Applied everywhere; search now matches both.
5. Terminal output replaced the raw agent key with the label, removing the only
printed copy of the token --agent accepts. Now prints both.
Generated with AI
Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
pigorv
force-pushed
the
feat/copilot-cli-analytics
branch
from
July 31, 2026 07:26
6e75f60 to
4628e54
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds GitHub Copilot CLI as an analytics-only agent, so
codemie analyticsreports Copilot usage alongside Claude, Codex, Gemini, Kimi and OpenCode. Sessions are discovered from local disk, priced with the existing pricing table, and surfaced as a first-class agent in the HTML and terminal reports.Scope is the agentic
copilotcommand (@github/copilot) only — notgh copilot, not the VS Code/JetBrains extensions, and not GitHub's cloud metrics APIs. CodeMie does not gain the ability to install, launch or manage Copilot; this is ingestion only.Changes
New agent plugin —
src/agents/plugins/copilot-cli/~/.copilot/session-state/<uuid>/workspace.yaml(honorsCOPILOT_HOME). The manifest is present in every session directory and carries project path, repo, branch and timestamps in a few hundred bytes, so discovery never opens a multi-megabyte transcript.events.jsonlinto the unifiedParsedSession— unparseable lines are dropped, never thrown, since the schema is undocumented and varies across CLI versions.Token extraction, in three tiers
session.shutdown.modelMetrics— the authoritative per-model rollupassistant.message— recovers output tokens and request counts, marked partialPricing —
readCopilotCliincost/usage-readers.tsCopilot reports
inputTokensinclusive ofcacheReadTokens(OpenAI convention, applied even to Anthropic models), while this repo'scostBreakdownbillsinputat full rate andcacheReadseparately. The reader decomposes rather than passing through:On a real session this yields input = 406,071 rather than the raw 14,076,695 a pass-through would bill — a ~36× over-count on the input component.
reasoningTokensis a subset ofoutputTokensand is never billed separately.No
pricing.jsonormodel-normalizer.tschanges: all observed model strings (gpt-5.2,gpt-5.4,gpt-5-mini,claude-sonnet-4.5,claude-sonnet-4.6) already resolve via the existing lowercase + dot→dash fold.Ownership gate —
native-loader.tsNatively-discovered sessions without a CodeMie ownership marker are tagged
native-externaland filtered out by default. A Copilot session can never carry that marker, so without an exemption 100% would be dropped and the feature would ship displaying nothing. Analytics-only agents are now taggednative-unmanaged, which passes the default filter while staying honest that CodeMie did not launch them.Management surfaces —
AgentRegistry.getManageableAgents()Registering an unmanageable agent broke the invariant that every
getAllAgents()entry is installable and launchable.codemie updatein particular never calls the adapter — it readsmetadata.npmPackageand runsnpm install -g <pkg> --force, which would have overwritten a user's own Copilot install. The nine management call sites (update, install, uninstall, list, first-time) now usegetManageableAgents();getAllAgents()still returns analytics-only agents so the analytics pipeline can resolve their session adapters. Copilot'snpmPackageisnullas defense in depth.Report UI
text-transform: capitalizewould have shown "Copilot-cli"). Capitalize is retained for unmapped keys so other agents look unchanged.premiumRequests,usagePartialandusageUnavailableReasononReportSessionRecord, rendered in the existing session modal. All spread conditionally, so no other agent's payload changes shape.GitHub Copilot CLI (copilot-cli)— label for readability, key because it is the token--agentaccepts.Note on cost semantics: Copilot bills in premium requests, not tokens (434 API requests cost 3 premium requests; 60
claude-sonnet-4.5requests cost 0). The USD figure is a token-derived estimate for cross-agent comparison, not GitHub's invoice.totalPremiumRequestsis surfaced separately rather than conflated with it.Testing
list/install/uninstall/doctor/updateGuards worth keeping on any future change here:
cache decomposition(12 tests) — assertsinput === 406071, the exact value that exposes the over-billing bugownership gate(4 tests) — asserts a Copilot session survives the default path with no--include-externalmanagement surfaces(4 tests) — asserts Copilot cannot reachcodemie updateChecklist
.jsextensions, noany, explicit return types,logger.*notconsole.*npm run ci) — see note belowmainKnown issues, disclosed
Source-column classification does not fire for Copilot. Skill invocations are captured correctly, but
session-source-detector.tsmatches thesuperpowers:prefix while Copilot writes skill names un-namespaced. Loosening a shared detector would change how Claude and Codex sessions classify, so it is deliberately left alone; Copilot sessions read "Pure chat".Sessions from Copilot CLI < 1.0 carry no token telemetry at all — verified across 48 shutdown-less local sessions, none of which record per-turn output tokens. They are listed with an explanatory reason rather than hidden or shown as $0.