External Integration Contracts (Grafana MCP, reporting strategy)

[tallpsmith]

The Why

pmmcp should not exist in isolation. SREs use Grafana, PagerDuty, Jira, Slack, and dozens of other tools. The MCP ecosystem enables Claude to orchestrate across these systems. pmmcp's role is to provide the performance intelligence — the investigation findings, the anomalous metrics, the time ranges of interest — in formats that other MCP servers can consume.

Depends on: #8 (Analysis Tools) and #9 (MCP Prompts)

---

Architecture

pmmcp and mcp-grafana are sibling MCP servers in the user's Claude session. pmmcp never talks to mcp-grafana directly — Claude orchestrates between them.

User's Claude session
├── pmmcp (MCP server) ──→ pmproxy
└── mcp-grafana (MCP server) ──→ Grafana ──→ pmproxy (same instance!)
                                      └── PCP datasource plugin

Hard requirement: Grafana must have a performancecopilot-pcp-app datasource pointing at the same pmproxy instance pmmcp is configured to use. If this shared datasource cannot be confirmed, Grafana visualisation is unavailable and the fallback path activates.

Companion project: grafana/mcp-grafana (~80 tools including dashboard CRUD, Prometheus queries, annotations, deeplinks, and image rendering)

---

Design: Prompt-Driven Grafana Integration

Approach

No new pmmcp tools. All Grafana integration is orchestrated by Claude via enhanced MCP prompts. pmmcp stays in its lane as the performance intelligence provider — it supplies metric names, time ranges, hosts, anomaly data, and statistical context. Claude uses mcp-grafana to create the dashboards.

Preflight: Datasource Discovery & Validation

A prompt-driven workflow (not a tool) run during session initialisation:

1. Claude calls mcp-grafana.list_datasources
2. Finds a datasource of type performancecopilot-pcp-app
3. Calls mcp-grafana.get_datasource to extract its pmproxy URL
4. Compares with pmmcp's configured PMPROXY_URL
5. Match → caches datasource UID in conversation context, logs success, Grafana features enabled
6. No match / no mcp-grafana → logs warning, falls back to HTML reports or text-only output

This runs early (guided by session_init) but the check is idempotent and can be re-triggered if needed.

Future hardening: If prompt-driven validation proves unreliable, fall back to a deterministic pcp_grafana_preflight tool with user-supplied config. See future issue: Deterministic Grafana Preflight Tool.

Dashboard Creation Contract

What pmmcp provides (via existing tool output):

• Metric names (PCP paths, e.g. kernel.cpu.util.user)
• Host names (from pcp_get_hosts, pcp_rank_hosts)
• Time ranges (start/finish timestamps from timeseries tools)
• Anomaly context (z-scores, severity, direction from pcp_detect_anomalies)
• Comparison data (deltas, statistical significance from pcp_compare_windows)
• Correlation data (Pearson coefficients from pcp_correlate_metrics)

What the prompts encode (guidance for Claude):

Convention	Value
Grafana folder	Configurable via PMMCP_GRAFANA_FOLDER, default pmmcp-triage
Dashboard naming	YYYY-MM-DD <short summary> (e.g. 2026-03-10 CPU contention webserver cluster)
Tagging	Always tag pmmcp-generated
Datasource UID	Use the UID discovered during preflight
Deeplink	After creation, call generate_deeplinks with the incident time range, return URL to user
Panel content	Prompts hint at what to visualise; Claude decides actual layout

What pmmcp does NOT do:

• No Grafana JSON generation
• No direct Grafana API calls
• No dashboard templates — Claude decides layout using its judgement and mcp-grafana's update_dashboard / patch_dashboard

Dashboard Lifecycle

Dashboards are semi-persistent. pmmcp creates them in the configured folder with codified names. From there:

• User owns them — rename, edit, export, delete as they see fit
• Enterprise/Cloud users can use Grafana's native PDF export/reporting features
• Dashboard pruning/cleanup is future work (see future issue)

Auto-Trigger Heuristic

Prompts include soft guidance for when Claude should proactively offer visualisation:

"If your investigation has surfaced findings across 3+ metrics or 2+ subsystems, and you haven't already created a visualisation, consider offering one."

User can always initiate or decline. In complex triage cases, Claude may automatically create dashboards to "tell the story in pictures."

---

Design: HTML Fallback (No Grafana)

When: Preflight discovers no mcp-grafana or no shared PCP datasource.

Mechanism: Pure prompt guidance, no new tools. Investigation prompts instruct Claude to:

1. Offer to generate an HTML summary when findings are non-trivial
2. Write a self-contained HTML file (inline CSS, no external dependencies) containing:
   • Investigation summary and timeline
   • Data tables from tool outputs (anomalies, comparisons, rankings)
   • Claude's narrative interpretation
3. Save to configurable output directory (PMMCP_REPORT_DIR, default ~/.pmmcp/reports/)
4. Name consistently: YYYY-MM-DD-<short-summary>.html

No HTML templates in pmmcp code. No charting libraries. Claude uses its native coding ability. If output quality is insufficient, we iterate on prompts, not code.

---

Visualisation Fallback Cascade

Every visualisation-capable prompt follows this cascade:

1. Grafana available → create dashboard in pmmcp-triage/ folder + return deeplink
2. Grafana unavailable → offer HTML report to report_dir
3. User declines both → text/table output as today (existing behaviour)

---

Configuration

Two new settings added to PmproxyConfig:

Setting	Env var	Default	Purpose
grafana_folder	PMMCP_GRAFANA_FOLDER	pmmcp-triage	Grafana folder for generated dashboards
report_dir	PMMCP_REPORT_DIR	~/.pmmcp/reports/	Output directory for HTML fallback reports

No Grafana URL (mcp-grafana owns that), no datasource UID (discovered at runtime), no API keys.

---

Prompt Changes

Prompt	Addition
session_init	Grafana preflight discovery flow
coordinate_investigation	"After synthesis, if findings span multiple subsystems, offer visualisation"
incident_triage	"After triage, create a triage dashboard capturing the incident picture"
fleet_health_check	"After health check, offer a fleet overview dashboard"
compare_periods	"If significant changes found, offer a before/after comparison dashboard"
specialist_investigate	No change — specialists report up, coordinator decides on visualisation

All visualisation-capable prompts include the fallback cascade and auto-trigger heuristic.

---

Scope Boundaries

In scope:

• Prompt-driven Grafana preflight (datasource discovery + validation)
• Prompt guidance for dashboard creation via mcp-grafana (folder, naming, tagging, deeplinks)
• Prompt guidance for HTML fallback reports
• Two new config settings (grafana_folder, report_dir)
• session_init + 4 investigation prompt enhancements
• Documentation updates (README, CLAUDE.md)

Explicitly NOT in scope:

• No new pmmcp tools (zero Python tool code beyond config)
• No Grafana JSON templates
• No PDF generation
• No dashboard pruning/cleanup
• No kitty/inline terminal images
• No Grafana-image-assembled hybrid reports
• No direct pmmcp→mcp-grafana communication
• No charting libraries or new dependencies

---

Acceptance Criteria

• session_init prompt includes Grafana preflight discovery workflow
• Preflight validates shared pmproxy datasource — logs result, caches datasource UID
• coordinate_investigation, incident_triage, fleet_health_check, compare_periods prompts include visualisation guidance with fallback cascade
• Dashboard conventions enforced via prompts: configurable folder, YYYY-MM-DD naming, pmmcp-generated tag, deeplink returned
• HTML fallback works when Grafana unavailable: self-contained file written to report_dir
• Two new config settings: PMMCP_GRAFANA_FOLDER, PMMCP_REPORT_DIR
• No hard dependencies on mcp-grafana — pmmcp works fully standalone
• Documentation updated (README, CLAUDE.md)

---

Future Work (separate issues)

• Dashboard lifecycle management (pruning, retention, keep-tagging) #35 Dashboard lifecycle management (pruning, "keep" tagging, retention)
• Grafana-assembled rich reports (PNG export + narrative stitching) #36 Grafana-assembled rich reports (pull PNGs, Claude stitches narrative + images)
• Kitty/inline terminal chart visualisation #37 Kitty/inline terminal visualisation
• Deterministic Grafana preflight tool (fallback for prompt-driven validation) #38 Deterministic preflight tool (fallback if prompt-driven validation proves unreliable)