Skip to content

feat(analytics): add GitHub Copilot CLI sessions to the analytics report - #449

Open
pigorv wants to merge 19 commits into
codemie-ai:mainfrom
pigorv:feat/copilot-cli-analytics
Open

feat(analytics): add GitHub Copilot CLI sessions to the analytics report#449
pigorv wants to merge 19 commits into
codemie-ai:mainfrom
pigorv:feat/copilot-cli-analytics

Conversation

@pigorv

@pigorv pigorv commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds GitHub Copilot CLI as an analytics-only agent, so codemie analytics reports Copilot usage alongside Claude, Codex, Gemini, Kimi and OpenCode. Sessions are discovered from local disk, priced with the existing pricing table, and surfaced as a first-class agent in the HTML and terminal reports.

Scope is the agentic copilot command (@github/copilot) only — not gh copilot, not the VS Code/JetBrains extensions, and not GitHub's cloud metrics APIs. CodeMie does not gain the ability to install, launch or manage Copilot; this is ingestion only.

Changes

New agent pluginsrc/agents/plugins/copilot-cli/

  • Discovery from ~/.copilot/session-state/<uuid>/workspace.yaml (honors COPILOT_HOME). The manifest is present in every session directory and carries project path, repo, branch and timestamps in a few hundred bytes, so discovery never opens a multi-megabyte transcript.
  • Tolerant JSONL parsing of events.jsonl into the unified ParsedSession — unparseable lines are dropped, never thrown, since the schema is undocumented and varies across CLI versions.
  • Metrics: tool calls with success/failure, file operations, code churn, user prompts, and skill invocations.

Token extraction, in three tiers

  1. session.shutdown.modelMetrics — the authoritative per-model rollup
  2. Per-turn assistant.message — recovers output tokens and request counts, marked partial
  3. Neither — the session is listed unpriced with a reason, not hidden

PricingreadCopilotCli in cost/usage-readers.ts

Copilot reports inputTokens inclusive of cacheReadTokens (OpenAI convention, applied even to Anthropic models), while this repo's costBreakdown bills input at full rate and cacheRead separately. The reader decomposes rather than passing through:

freshInput    = inputTokens − cacheReadTokens
cacheCreation = min(cacheWriteTokens, freshInput)   // cache writes price above base input
input         = freshInput − cacheCreation

On a real session this yields input = 406,071 rather than the raw 14,076,695 a pass-through would bill — a ~36× over-count on the input component. reasoningTokens is a subset of outputTokens and is never billed separately.

No pricing.json or model-normalizer.ts changes: all observed model strings (gpt-5.2, gpt-5.4, gpt-5-mini, claude-sonnet-4.5, claude-sonnet-4.6) already resolve via the existing lowercase + dot→dash fold.

Ownership gatenative-loader.ts

Natively-discovered sessions without a CodeMie ownership marker are tagged native-external and filtered out by default. A Copilot session can never carry that marker, so without an exemption 100% would be dropped and the feature would ship displaying nothing. Analytics-only agents are now tagged native-unmanaged, which passes the default filter while staying honest that CodeMie did not launch them.

Management surfacesAgentRegistry.getManageableAgents()

Registering an unmanageable agent broke the invariant that every getAllAgents() entry is installable and launchable. codemie update in particular never calls the adapter — it reads metadata.npmPackage and runs npm install -g <pkg> --force, which would have overwritten a user's own Copilot install. The nine management call sites (update, install, uninstall, list, first-time) now use getManageableAgents(); getAllAgents() still returns analytics-only agents so the analytics pipeline can resolve their session adapters. Copilot's npmPackage is null as defense in depth.

Report UI

  • Brand colour for Copilot, plus a new agent display-label mechanism (the report had none — raw keys were rendered, and text-transform: capitalize would have shown "Copilot-cli"). Capitalize is retained for unmapped keys so other agents look unchanged.
  • Optional premiumRequests, usagePartial and usageUnavailableReason on ReportSessionRecord, rendered in the existing session modal. All spread conditionally, so no other agent's payload changes shape.
  • Terminal output prints GitHub Copilot CLI (copilot-cli) — label for readability, key because it is the token --agent accepts.

Note on cost semantics: Copilot bills in premium requests, not tokens (434 API requests cost 3 premium requests; 60 claude-sonnet-4.5 requests cost 0). The USD figure is a token-derived estimate for cross-agent comparison, not GitHub's invoice. totalPremiumRequests is surfaced separately rather than conflated with it.

Testing

  • Tests added/updated — 63 new unit tests across discovery, parsing, token extraction, pricing, registry wiring and the ownership gate
  • Manual testing done — verified against 41 real local Copilot sessions: 746 turns, 1454 tool calls (97.7% success), correct per-model attribution, and Copilot absent from list/install/uninstall/doctor/update

Guards worth keeping on any future change here:

  • cache decomposition (12 tests) — asserts input === 406071, the exact value that exposes the over-billing bug
  • ownership gate (4 tests) — asserts a Copilot session survives the default path with no --include-external
  • management surfaces (4 tests) — asserts Copilot cannot reach codemie update

Checklist

  • Code follows project standards — ESM .js extensions, no any, explicit return types, logger.* not console.*
  • Lint clean (zero warnings), typecheck clean, build clean
  • CI is green (npm run ci) — see note below
  • No merge conflicts with main

Known issues, disclosed

Source-column classification does not fire for Copilot. Skill invocations are captured correctly, but session-source-detector.ts matches the superpowers: prefix while Copilot writes skill names un-namespaced. Loosening a shared detector would change how Claude and Codex sessions classify, so it is deliberately left alone; Copilot sessions read "Pure chat".

Sessions from Copilot CLI < 1.0 carry no token telemetry at all — verified across 48 shutdown-less local sessions, none of which record per-turn output tokens. They are listed with an explanatory reason rather than hidden or shown as $0.

pigorv and others added 19 commits July 28, 2026 18:32
Phase 0 spike over 78 real local Copilot CLI sessions, verified technical
analysis, approved spec, and 11-task implementation plan for adding GitHub
Copilot CLI sessions to the analytics report.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…nifest reader

Discovery manifest reader for GitHub Copilot CLI sessions. workspace.yaml is
present in every session directory and carries identity, project path, repo,
branch and timestamps, so discovery never opens a multi-megabyte transcript.

Honors COPILOT_HOME. Returns null rather than throwing on missing, malformed,
or id-less manifests so one bad session directory cannot abort discovery.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
… plugin

discoverSessions enumerates ~/.copilot/session-state/<uuid>/, reading only the
workspace.yaml manifest so a report run never opens a transcript it will filter
out. Skips directories with no events.jsonl (pre-schema sessions with nothing to
parse), honors maxAgeDays/cwd/limit, and sorts newest-first.

Adds AgentMetadata.analyticsOnly, marking agents CodeMie reads analytics for but
never manages. The plugin overrides install/uninstall/run to refuse: inherited,
they would npm-install and launch @github/copilot, which is outside this
integration's remit.

Verified before commit: eslint clean, tsc --noEmit clean, 15/15 unit tests pass.
--no-verify used with explicit user approval: two acceptance tests in
cost-enricher.test.ts fail on this machine independently of this change (proven
by reverting src/ to origin/main), because they assert hardcoded USD totals
against live local Claude transcripts that have since grown.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Three tiers: session.shutdown.modelMetrics (authoritative), per-turn
assistant.message reconstruction (output tokens and request counts only, marked
partial), then no-usage with a reason.

Buckets are emitted raw in Copilot's OpenAI-convention field names so the
convention conversion has a single seam in readCopilotCli.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…ecomposition

Copilot reports inputTokens INCLUSIVE of cacheReadTokens (OpenAI convention,
applied even to Anthropic models), while costBreakdown bills input at full rate
AND cacheRead separately. readCopilotCli therefore decomposes:

  freshInput = inputTokens - cacheReadTokens
  input      = freshInput - cacheWriteTokens

Verified against a real session: input resolves to 381,719 rather than the raw
14,076,695 a pass-through would bill — a ~36x over-count. reasoningTokens is a
subset of outputTokens and is never billed separately.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…totals

The cost enricher computes run-level totals through gatherUsageDeduped, not
readUsageByModel. Without this branch, per-session numbers are correct while
report totals stay $0 — a silent, plausible-looking failure. Copilot is
session-local, so no cross-session dedup key is needed.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…h metrics

One pass over events.jsonl produces metadata from session.start, raw per-model
usage buckets as messages, and metrics: tool counts with success/failure status,
file operations from session.shutdown.codeChanges, user prompts, and skill
invocations.

Mapping skill.invoked into metrics.skillInvocations lets detectSessionSource
classify Copilot sessions in the report's Source column the same way it does
Claude sessions, rather than defaulting them all to 'Pure chat'.

Adds ParsedSession.usageMeta so the enricher can tell 'cost is genuinely zero'
from 'cost is unmeasurable'. Named generically rather than after one agent:
usagePartial and usageUnavailableReason apply to any agent with incomplete
telemetry.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
native-loader resolves session adapters via AgentRegistry.getAgent(name)
.getSessionAdapter(), so registration is required for discovery to reach the
adapter at all.

Updates two registry tests that asserted an exact plugin count of 8. The first
now asserts membership by name, so a future plugin change fails with the
offending name rather than an opaque length mismatch; the second derives its
expected length instead of hardcoding it.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…he ownership gate

native-loader.ts:518 tags natively-discovered sessions lacking a CodeMie
ownership marker as native-external, and sources/sessions-source.ts filters
those out by default. A Copilot session can never carry such a marker, so
without this exemption 100% of them are dropped and the feature ships showing
nothing.

Also corrects two parsing bugs that end-to-end verification exposed, both from
assumptions the unit-test fixtures had encoded rather than reality:

- tool.execution_complete carries NO tool name and no status string — only
  toolCallId plus a boolean success. The name and arguments live on
  tool.execution_start, so the two must be paired by toolCallId. Tool metrics
  were silently empty before this.
- messages now also carry Claude-shaped per-turn records, so synthesizeRawSession
  derives turns, models, timestamps, cwd and branch correctly. readCopilotCli
  filters on model+usage and turn counting filters on type='assistant', so the
  two consumers pick up only what each needs.

session.shutdown line totals now merge into the per-file operations instead of
appending a duplicate entry for an already-recorded path.

Verified against 41 real local sessions: 746 turns, 1454 tool calls (97.7%
success), correct per-model attribution, and input priced at 406,071 tokens
rather than the ~15.7M a pass-through would have billed.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…s in the report payload

Carries ParsedSession.usageMeta through the cost enricher onto SessionCost and
ReportSessionRecord: premiumRequests (GitHub's real billing unit, which does not
track token volume), usagePartial, and usageUnavailableReason.

All three are optional and spread conditionally, so records for agents that
record full usage are unchanged in shape. Aggregate coverage needs no new
mechanism — the existing AgentCoverage {total, priced, withLog} already carries
the 'N of M unpriced' story.

Verified against real data: 6 sessions report premium requests (3,3,1,1,1,1),
35 carry an unavailable reason.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
… premium requests

Adds a GitHub-neutral brand color (#6E7681, the only unsaturated entry so it
cannot collide with an existing agent or a PALETTE slot) and introduces an agent
display-label mechanism, which the report did not have — raw keys rendered
directly, and the Agents-Compare table's text-transform:capitalize would have
shown 'Copilot-cli'. Capitalize is now applied only to unmapped keys, so every
other agent looks unchanged.

labelFor() is wired into the agent chips, the Agents-Compare table and the
session modal header; agentLabel() does the same for terminal output.

The session modal gains Copilot-only rows: premium requests, a partial-usage
note, and the reason a session could not be priced. All render only when present.

Note: agent-labels.ts and app.js necessarily hold separate copies of the label
map — app.js is a standalone browser bundle that cannot import TypeScript, the
same reason AGENT_COLORS lives only there.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…ted limitations

Documents two events.jsonl shape assumptions the spec got wrong (tool name lives
on execution_start, not _complete; messages must carry Claude-shaped per-turn
records for native synthesis) and the accepted D11 limitation where Copilot's
un-namespaced skill names do not match the Source detector's prefix rule.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…shutdown rollup

Only the newest Copilot CLI builds put `model` on each assistant.message; 1.0.17
and every 0.0.x omit it, so 38 of 41 local sessions showed 'unknown model' —
including three that were priced and had the model in their own cost breakdown.

Resolution order, most authoritative first:
  1. per-turn assistant.message.model
  2. the model in effect from the preceding session.model_change, tracked
     chronologically so a mid-session switch attributes each turn correctly
  3. shutdown modelMetrics when the session used exactly ONE model

Deliberately does not guess when turns are unlabelled and the rollup shows two
or more models — attributing a turn to either would be a fabrication.

Real data: sessions with an attributed model 3 -> 25 of 41, priced-but-unattributed
3 -> 0, distinct models surfaced 3 -> 5 (adds gpt-5-mini, claude-sonnet-4.6).
The remaining 16 carry no model-bearing field anywhere in their transcripts.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…odified file

Two defects found reviewing the diff:

1. cacheCreation was not clamped. With inconsistent buckets, input clamped to 0
   while the full cacheWriteTokens was still billed — and cache creation prices
   ABOVE the base input rate, so garbage data could over-charge. Now clamped to
   the fresh input it was written from, which also makes input non-negative by
   construction rather than by a second Math.max. The existing test asserted
   input===0 but never checked cacheCreation, so it passed while over-billing.

2. session.shutdown line totals were attributed to fileOperations[0], which may
   be a file a tool merely opened. Now prefers an entry that shutdown actually
   lists in filesModified.

Neither changes real-world output: the measured session still reports input
406,071 and cacheCreation 125,660.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Inline review of the 39-file diff. Two defects found and fixed: cache-write
tokens were unclamped (over-billing risk on inconsistent buckets, in the one
reader whose purpose is not over-billing), and code-churn totals could be
credited to a file that was never modified.

Verdict records medium rather than high confidence: this was a self-review by
the change's author, and the two schema bugs found earlier in this task were
both cases where the author's own fixtures encoded a wrong assumption — the
exact failure mode a self-review is weakest against.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…anch

The 254KB code-review.diff was a snapshot of this very branch's diff, so it got
counted inside the diff — 5,820 of the 11,311 lines that pushed the change past
the ultrareview size limit and blocked an independent review.

It is regenerable at any time with git diff <merge-base>...HEAD, and it is a
minority practice: only 4 of 22 task dirs on main commit one, the largest at
55KB. Gitignored so it cannot recur.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Overriding install()/uninstall()/run() to refuse was not enough. codemie update
never calls the adapter — updateAgent() reads metadata.npmPackage and runs
npm install -g <pkg> --force, so a user with Copilot on PATH would have seen it
PRE-CHECKED in the update picker and had their own global install overwritten by
an agent CodeMie documents as one it does not manage.

Adds AgentRegistry.getManageableAgents() (all agents minus analyticsOnly) and
routes the nine management call sites through it: update, install, uninstall,
list and first-time. getInstalledAgents() filters too — all four of its consumers
(uninstall, list, doctor x2) are management surfaces, and 'installed' is
meaningless for an agent CodeMie never installs. getAllAgents() still returns
them so the analytics pipeline can resolve session adapters.

Copilot's npmPackage is now null (the existing 'null for built-in' idiom, already
guarded by every consumer), so even a missed surface fails safe with 'cannot be
updated' rather than destructively.

Verified: Copilot absent from list/install/uninstall/doctor/update; analytics
still discovers all 41 sessions with full model attribution.

Found by workflow-backed review; missed by the inline review, which checked that
the overrides refused but not whether other paths reach around them.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…dentifiers

This repository is public. The committed task artifacts were derived from real
local Copilot and Claude transcripts, so they are removed entirely and the
identifiers that reached test fixtures are replaced with synthetic values:

- session UUIDs -> 11111111-2222-3333-4444-555555555555
- a real session title -> 'example session title'
- real branch/repo names -> feat/example, example-org/example-repo
- session attribution dropped from test comments

Token figures are kept: they carry no identifier on their own and are the exact
values that expose the cache-inclusive over-billing bug, so replacing them would
weaken the regression guard for no privacy gain.

Verified clean: no home paths (only /Users/x/ placeholders), no GitHub logins,
no emails, no session UUIDs, no session titles, no references to unrelated
private projects.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Five findings from the workflow-backed review:

1. Tier-2 per-turn fallback required data.model, which the CLI builds it exists
   for do not write, making it unreachable. Now tracks the model in effect from
   session.model_change. NOTE: this recovers nothing on currently-observed data —
   all 48 shutdown-less local sessions have no per-turn outputTokens either, so
   the review's stated impact ('sessions labelled no-telemetry when tokens are
   recoverable') does not hold. It fixes the latent case: a crashed 1.0.17
   session, which writes outputTokens but no model and has no shutdown event.

2. Analytics-only sessions were tagged provider 'native', identical to
   CodeMie-launched ones, so managed and unmanaged usage became
   indistinguishable. Now 'native-unmanaged' — still included by default (the
   filter excludes only 'native-external') but honest about origin.

3. File operations were recorded at tool.execution_start, before success was
   known, so a failed edit/create counted as a changed file. Now recorded on
   successful completion only.

4. labelFor() reached only 3 of 8 agent render sites in the report, and the
   session-table search matched the raw key, so a user who searched the label
   they could see got no results. Applied everywhere; search now matches both.

5. Terminal output replaced the raw agent key with the label, removing the only
   printed copy of the token --agent accepts. Now prints both.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
@pigorv
pigorv force-pushed the feat/copilot-cli-analytics branch from 6e75f60 to 4628e54 Compare July 31, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant