Skip to content

Capture MCP tool calls in PostHog - #128

Open
masnwilliams wants to merge 8 commits into
mainfrom
hypeship/posthog-mcp-analytics
Open

Capture MCP tool calls in PostHog#128
masnwilliams wants to merge 8 commits into
mainfrom
hypeship/posthog-mcp-analytics

Conversation

@masnwilliams

@masnwilliams masnwilliams commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Instruments the MCP server with PostHog MCP analytics (@posthog/mcp, pinned to 0.10.1) so every inbound tools/call, tools/list, and initialize is captured as a $mcp_* event — tool name, latency, error state, client name/version, session.

What that buys us: per-tool call volume, error rate and latency percentiles, which clients connect, and which advertised tools never get called. Today we ship tool changes blind — kernel API traffic tagged X-Source: mcp-server shows endpoints, not tools.

  • src/lib/mcp/analytics.ts — module-scope posthog-node client plus instrument() wiring. identify attributes events to the Clerk user id when the request carries a JWT; API-key requests stay session-scoped rather than getting a made-up identity.
  • src/app/[transport]/route.ts — instrument inside the createMcpHandler callback, mint a session id on the handshake (below), and drain the queue with after() so the flush runs once the response is already on the wire. Capture adds no latency to a tool call.
  • .env.examplePOSTHOG_PROJECT_TOKEN and POSTHOG_HOST. With no token set the whole thing is a no-op, so deploys and local dev without PostHog credentials are unaffected (dev logs a one-time error so the miss isn't silent).

POSTHOG_PROJECT_TOKEN and POSTHOG_HOST are set on the Vercel project (production → prod PostHog project, preview → staging).

Sessions

mcp-handler runs the streamable-HTTP transport statelessly and answers over SSE, so it never issues an Mcp-Session-Id. Left alone that means one PostHog session per HTTP request, and $mcp_client_name / $mcp_client_version — only sent at initialize — missing from every later event.

mintMcpSessionId handles it the way the SDK documents for SSE: on an initialize POST it mints a session token carrying a fresh session id plus the client name and version, injects it on the inbound request (so the handshake event lands in that session too) and echoes it back on the response. Clients replay the header, and any instance decodes the same values out of it — no session store, no sticky routing. The stateless transport ignores an incoming session id (validateSession short-circuits when sessionIdGenerator is undefined), so this can't produce 400/404s. Mcp-Session-Id is added to the CORS allow/expose lists so browser-based clients can round-trip it.

No payloads leave the server

beforeSend drops $mcp_parameters and $mcp_response on every event.

Neither side of a call is safe to capture here. Results are serialized through jsonResponse, so a response reaches the SDK as one JSON string and its key-name redaction can't see inside it — a newly created API key, a TOTP code, or a CDP URL would go out verbatim. Arguments are free-form on many tools: credential field maps (manage_credentials values, manage_auth_connections fields), browser_curl headers and body, computer_action typed text, exec_command commands, execute_playwright_code source. An allow-list of "credential tools" would always be one tool behind, so the payloads go entirely and what remains is call metadata.

If we later want an argument breakdown (e.g. which action is most used), the safe shape is an explicit allow-list of individual parameter keys.

Deliberately left off

instrument() can inject a required context argument into every tool schema to capture what the agent was trying to do ($mcp_intent). It's the most interesting signal, but it changes a public tool surface — a required field plus a 15-25 word instruction on all 17 toolsets — so this ships with context: false. Worth turning on as its own change once the basic numbers are landing.

The SDK is pre-1.0 (0.x), so event names and properties can change in minor releases — hence the exact-version pin.

Verification

  • bunx tsc --noEmit passes (what CI runs).
  • Ran the real route locally against a non-production PostHog project. A handshake plus a tools/list and a tools/call replaying the minted header land as three events under one session id — the exact id inside the token — each carrying client name and version. A control request that doesn't replay the header lands in its own session with a null client, which is what the currently deployed staging server produces for every event.
  • Payload dropping verified the same way: events for a call carrying a canary argument value have neither $mcp_parameters nor $mcp_response present, and no canary string anywhere in the event.
  • bun run build fails locally on this branch and on main alike (page data collection for /authorize without Clerk/Redis env), so that failure is unrelated.

Note

$mcp_server_name currently reads mcp-typescript server on vercel — mcp-handler's default serverInfo, which is also what agents see. Passing serverInfo: { name: "kernel-mcp-server", version } to createMcpHandler would fix both, but it changes the advertised server identity so it's left out of this PR.


Note

Medium Risk
Touches the authenticated MCP request path and CORS for a new header; payload stripping reduces data-leak risk but misconfiguration could still send events externally when a token is set.

Overview
Adds optional PostHog MCP analytics so initialize, tools/list, and tools/call emit $mcp_* events (tool, latency, errors, client, session) without changing tool schemas. Capture is a no-op when POSTHOG_PROJECT_TOKEN is unset; .env.example documents POSTHOG_PROJECT_TOKEN and POSTHOG_HOST.

New src/lib/mcp/analytics.ts wires @posthog/mcp instrument() with Clerk distinctId for JWT requests, beforeSend stripping of parameters and responses, and context: false so tools stay unchanged. The MCP route calls instrumentMcpAnalytics in the handler factory and uses Next after(flushMcpAnalytics) on GET/POST so flushes run after the response.

Because the stateless transport does not issue Mcp-Session-Id, mintMcpSessionId on initialize POST injects and echoes the SDK session header (with CORS allow/expose) so clients can correlate handshake and later calls in one PostHog session.

Reviewed by Cursor Bugbot for commit 1de2e9a. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
mcp Ready Ready Preview Jul 29, 2026 10:47pm
mcp (staging) Ready Ready Preview Jul 29, 2026 10:47pm

@socket-security

socket-security Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedposthog-node@​5.46.17210079100100
Added@​posthog/​mcp@​0.10.17910099100100

View full report

@masnwilliams
masnwilliams marked this pull request as ready for review July 29, 2026 19:11
Comment thread src/lib/mcp/analytics.ts
@masnwilliams
masnwilliams marked this pull request as ready for review July 29, 2026 19:27
Comment thread src/lib/mcp/analytics.ts Outdated
Comment thread src/lib/mcp/analytics.ts Outdated
Comment thread src/lib/mcp/analytics.ts Outdated
Comment thread src/app/[transport]/route.ts

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d9b21ea. Configure here.

Comment thread next-env.d.ts Outdated
@masnwilliams
masnwilliams requested a review from dcruzeneil2 July 29, 2026 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant