feat: specialists should baseline against historical data to distinguish anomalies from normal behaviour

[tallpsmith]

Problem

The 6 specialist sub-agents currently operate in a narrow time window — they discover metrics, fetch recent data, check against static thresholds, and report. They have no sense of what's normal for this host.

A CPU spike to 85% might be a daily batch job that runs like clockwork, or it might be the first time it's ever happened. Without historical context, the specialist can't tell the difference — and the human gets a report full of "findings" that are actually business-as-usual.

What's already available but not wired up

Tool	What it does	Who uses it today
pcp_detect_anomalies	Compares recent window against a historical baseline, flags statistical deviations	incident_triage prompt only
pcp_compare_windows	Before/after statistical comparison of two explicit windows	compare_periods prompt, cross-cutting specialist (mentioned but not mandated)
pcp_quick_investigate	Broad anomaly scan across all namespaces	Cross-cutting specialist only

The domain specialists (cpu, memory, disk, network, process) use none of these. They go straight to pcp_fetch_timeseries with whatever lookback window was requested and judge values against hardcoded thresholds ("idle < 10% = saturated").

Proposal

1. Add a "Baseline" step to the specialist workflow

Currently: Discover → Fetch → Analyse → Report

Proposed: Discover → Baseline → Fetch → Analyse → Report

The new Baseline step should:

1. Fetch the past 7 days of data for key metrics at a coarse interval (e.g., 1hour) using pcp_fetch_timeseries
2. Run pcp_detect_anomalies comparing the investigation window against this 7-day baseline
3. Use the anomaly detection results to classify each finding:
   • Anomaly — this behaviour deviates significantly from the 7-day norm
   • Recurring pattern — this happens regularly (e.g., daily at the same time)
   • Baseline behaviour — this is within normal operating range for this host

This prevents the specialist from crying wolf on a daily batch job or known traffic pattern.

2. Update specialist domain knowledge

Each specialist's heuristics should be augmented with baseline-aware guidance:

• "Before flagging CPU saturation, check if this level is typical for this time of day over the past week"
• "Memory growth is only a leak signal if the trend is new — compare against the 7-day baseline to rule out normal working-set growth"
• "Disk I/O spikes that recur at the same time daily are likely scheduled jobs (backups, log rotation) — note as recurring, not anomalous"

3. Update specialist report structure

Add a classification field to each finding:

- metric: kernel.all.cpu.idle
  value: 8%
  severity: warning
  classification: ANOMALY | RECURRING | BASELINE
  baseline_context: "7-day average at this time of day: 45% idle. This is 5× worse than normal."

4. Cross-cutting specialist should leverage classifications

Update the cross-cutting specialist to:

• Prioritise findings classified as ANOMALY over RECURRING or BASELINE
• Note when multiple subsystems show ANOMALY classification at the same timestamp (correlated anomaly = higher confidence root cause)
• Flag when a subsystem reports BASELINE but another reports ANOMALY — the baseline subsystem is probably not the cause

5. Coordinator synthesis should weight by classification

Update the synthesis phase in coordinate_investigation to:

• Lead with anomalies, not just severity
• Explicitly call out "these findings are normal behaviour for this host" to reduce noise
• Highlight when an apparent anomaly matches a known recurring pattern

Scope

Files to modify

File	Change
src/pmmcp/prompts/specialist.py	Add baseline step to workflow, update domain knowledge per subsystem, add classification to report structure
src/pmmcp/prompts/coordinator.py	Update synthesis guidance to weight by anomaly classification
docs/investigation-flow.md	Update specialist workflow diagram (4 steps → 5 steps)
tests/unit/test_prompts_specialist.py	Verify baseline step and classification appear in prompt output
tests/unit/test_prompts_coordinator.py	Verify synthesis references classification

Not in scope

• Changes to the tools themselves (pcp_detect_anomalies, pcp_compare_windows) — they already do what's needed
• New tool development — this is purely prompt engineering
• Changing the lookback parameter semantics — the baseline window (7 days) is independent of the investigation window

Open questions

• Baseline window: 7 days is a good default for catching daily/weekly patterns. Should this be configurable via the prompt, or is 7 days always right?
• Performance: Adding a 7-day baseline fetch per specialist adds latency. Worth it for accuracy? Could be opt-out (skip_baseline=true) for speed-critical triage.
• Data availability: Not all hosts will have 7 days of history. The specialist should gracefully degrade — "insufficient baseline data, falling back to threshold-only analysis."

Acceptance criteria

• All 6 domain specialists include a baseline step referencing pcp_detect_anomalies
• Specialist report structure includes anomaly classification (ANOMALY / RECURRING / BASELINE)
• Cross-cutting specialist references classification when correlating
• Coordinator synthesis weights findings by classification
• Graceful fallback when historical data is insufficient
• Investigation flow diagram updated
• Unit tests updated