diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index c375704..e9de46d 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,6 +1,13 @@ SYN CROSS --> SYN - SYN --> OUT["Unified Report\n• Root cause narrative\n• Timeline correlation\n• Findings ranked by severity\n• Concrete recommendations"] + SYN --> OUT["Unified Report\n• Root cause narrative\n• Timeline correlation\n• Findings ranked by classification & severity\n (ANOMALY > RECURRING > BASELINE)\n• Concrete recommendations"] style CI fill:#2d6a4f,color:#fff style PAR fill:#40916c,color:#fff @@ -107,10 +107,13 @@ relationships, and interpretation rules from experienced performance engineers. ## Specialist Workflow -Every specialist follows the same 4-step discipline: discover what metrics exist, -fetch the data, analyse against domain heuristics, then report structured findings. -This prevents the common failure mode of querying metrics that don't exist on the -target host. +Domain specialists (CPU, Memory, Disk, Network, Process) follow a 5-step +discipline: discover what metrics exist, establish a 7-day baseline for anomaly +detection, fetch current data, analyse against domain heuristics with baseline +context, then report structured and classified findings. + +The **cross-cutting** specialist does NOT include a Baseline step — it consumes +classifications from the domain specialists rather than baselining independently. ```mermaid sequenceDiagram @@ -126,22 +129,29 @@ sequenceDiagram T-->>S: Available metric names end + rect rgb(225, 235, 225) + Note over S,T: 2. Baseline (domain specialists only) + S->>T: pcp_fetch_timeseries(names=[...], interval="1hour", start="-7days") + T-->>S: 7-day historical data + S->>T: pcp_detect_anomalies(recent vs 7-day baseline) + T-->>S: Anomaly results (z-scores, directions) + Note over S: Note results for step 4
If insufficient data → threshold-only fallback + end + rect rgb(230, 235, 245) - Note over S,T: 2. Fetch + Note over S,T: 3. Fetch S->>T: pcp_fetch_timeseries(names=[...]) - T-->>S: Time-series data - S->>T: pcp_compare_windows(...) [optional] - T-->>S: Statistical comparison + T-->>S: Current investigation window data end rect rgb(245, 235, 225) - Note over S,T: 3. Analyse - Note over S: Apply domain heuristics
Check thresholds & correlations
Identify anomalies + Note over S,T: 4. Analyse + Note over S: Apply domain heuristics
Check thresholds & correlations
Classify: ANOMALY / RECURRING / BASELINE
Assign severity_despite_baseline end rect rgb(240, 230, 230) - Note over S,T: 4. Report - S-->>C: Structured findings
(metric, value, severity,
affected window, recommendation) + Note over S,T: 5. Report + S-->>C: Classified findings
(metric, value, classification,
baseline_context, severity,
severity_despite_baseline) end ``` @@ -155,8 +165,14 @@ coordinator synthesises findings: 2. **Timeline correlation** — the subsystem that changed first is the likely root cause 3. **Unified narrative** — tell the story of what happened, not just list findings -4. **Rank by impact** — order by severity and blast radius -5. **Recommend actions** — concrete next steps, not "investigate further" +4. **Rank by classification, then severity** — ANOMALY findings rank above + RECURRING, which rank above BASELINE (severity is secondary sort within each + tier). What changed is more actionable than what has always been wrong. +5. **Call out normal behaviour** — explicitly identify chronic baseline conditions +6. **Highlight recurring patterns** — flag when an apparent anomaly matches a known + recurring pattern (e.g., daily backup window) +7. **Recommend actions** — concrete next steps, not "investigate further" The output follows a structured format: executive summary → root cause analysis → -findings by severity → recommendations → specialist status. +findings by classification & severity (New Anomalies → Recurring Patterns → +Baseline Behaviour → Normal Operation) → recommendations → specialist status. diff --git a/specs/011-specialist-baselining/checklists/requirements.md b/specs/011-specialist-baselining/checklists/requirements.md new file mode 100644 index 0000000..e4a121c --- /dev/null +++ b/specs/011-specialist-baselining/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Specialist Historical Baselining + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-03-10 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Spec references specific tool names (`pcp_detect_anomalies`, `pcp_fetch_timeseries`) which are domain-specific terminology in this project context, not implementation details — these are the user-facing MCP tool names that the LLM agent invokes. +- The spec references specific file paths in the Scope section, which is acceptable for a purely prompt-engineering feature where the scope is narrow and well-defined. +- All items pass. Spec is ready for `/speckit.plan`. +- Clarification session 2026-03-10: 4 questions asked and resolved (severity_despite_baseline, ranking order, RECURRING detection, cross-cutting scope). diff --git a/specs/011-specialist-baselining/contracts/prompt-contracts.md b/specs/011-specialist-baselining/contracts/prompt-contracts.md new file mode 100644 index 0000000..87f1b48 --- /dev/null +++ b/specs/011-specialist-baselining/contracts/prompt-contracts.md @@ -0,0 +1,56 @@ +# Prompt Contracts: Specialist Historical Baselining + +**Feature**: 011-specialist-baselining | **Date**: 2026-03-10 + +## Contract: No Interface Changes + +This feature makes **zero changes** to function signatures or MCP protocol surface. + +### specialist_investigate (UNCHANGED) + +```python +def specialist_investigate( + subsystem: str, + request: str | None = None, + host: str | None = None, + time_of_interest: str | None = None, + lookback: str | None = None, +) -> list[dict]: +``` + +- Parameters: unchanged +- Return type: `list[dict]` — unchanged +- MCP registration: unchanged + +### coordinate_investigation (UNCHANGED) + +```python +def coordinate_investigation( + request: str, + host: str | None = None, + time_of_interest: str | None = None, + lookback: str | None = None, +) -> list[dict]: +``` + +- Parameters: unchanged +- Return type: `list[dict]` — unchanged +- MCP registration: unchanged + +## Content Contracts (Unit Test Assertions) + +Since no interfaces change, the "contracts" for this feature are content assertions verified in unit tests: + +| Contract | Verified By | +|----------|------------| +| Domain specialists (5) include Baseline step | String assertion: "Baseline" in workflow for cpu, memory, disk, network, process | +| Cross-cutting does NOT include Baseline step | String assertion: "Baseline" NOT in cross-cutting workflow | +| Classification fields in report guidance | String assertion: "classification", "ANOMALY", "RECURRING", "BASELINE" | +| `baseline_context` in report guidance | String assertion: "baseline_context" | +| `severity_despite_baseline` in report guidance | String assertion: "severity_despite_baseline" | +| Coordinator ranks by classification | String assertion: "ANOMALY" ranking guidance in synthesis | +| Graceful degradation | String assertion: "insufficient baseline" or fallback in prompt | + +## Existing Contract Tests (UNCHANGED) + +The existing contract tests in `tests/contract/test_prompts.py` continue to pass unchanged — they verify prompt registration and argument schemas, which are not modified. diff --git a/specs/011-specialist-baselining/data-model.md b/specs/011-specialist-baselining/data-model.md new file mode 100644 index 0000000..b880c01 --- /dev/null +++ b/specs/011-specialist-baselining/data-model.md @@ -0,0 +1,42 @@ +# Data Model: Specialist Historical Baselining + +**Feature**: 011-specialist-baselining | **Date**: 2026-03-10 + +## Overview + +This feature adds no new Pydantic models or database entities. All changes are to prompt text templates that guide LLM behaviour. The "data model" here describes the report structure fields added to prompt output guidance. + +## Report Structure Fields (Prompt-Guided) + +These fields are instructed in the specialist report guidance — the LLM produces them as structured text. They are NOT enforced by code; they are advisory prompt instructions. + +### Finding Classification + +| Field | Type (advisory) | Values | Description | +|-------|-----------------|--------|-------------| +| `classification` | enum string | ANOMALY, RECURRING, BASELINE | Whether the finding is new (anomaly), a known periodic pattern, or normal baseline behaviour | +| `baseline_context` | free text | — | Human-readable comparison to the 7-day baseline (e.g., "CPU idle has been below 15% for the past 7 days") | +| `severity_despite_baseline` | enum string | critical, warning, info, none | Threshold-based severity independent of classification. A BASELINE finding with severity=warning means "your normal is degraded" | + +### Classification Decision Rules (LLM Heuristics) + +| Classification | Condition | +|---------------|-----------| +| ANOMALY | `pcp_detect_anomalies` reports significant z-score AND pattern does not recur at consistent times in the 7-day timeseries | +| RECURRING | 7-day timeseries shows repeated spikes at consistent times of day (batch jobs, log rotation, backups, cron jobs) | +| BASELINE | Current values are within normal range based on 7-day history (low z-score or no anomaly detected) | + +### Coordinator Ranking Order + +| Priority | Classification | Severity Sort | +|----------|---------------|---------------| +| 1 (highest) | ANOMALY | critical → warning → info | +| 2 | RECURRING | critical → warning → info | +| 3 (lowest) | BASELINE | critical → warning → info | + +## Existing Entities (Unchanged) + +- `_SPECIALIST_KNOWLEDGE` dict in `specialist.py` — keys: `prefix`, `display_name`, `domain_knowledge`, `report_guidance`. Structure unchanged; content updated. +- `_specialist_investigate_impl()` signature — unchanged (subsystem, request, host, time_of_interest, lookback) +- `_coordinate_investigation_impl()` signature — unchanged (request, host, time_of_interest, lookback) +- `pcp_detect_anomalies` tool — unchanged (metrics, recent_start, recent_end, baseline_start, baseline_end, z_score_threshold, host, interval) diff --git a/specs/011-specialist-baselining/plan.md b/specs/011-specialist-baselining/plan.md new file mode 100644 index 0000000..41dfd6d --- /dev/null +++ b/specs/011-specialist-baselining/plan.md @@ -0,0 +1,206 @@ +# Implementation Plan: Specialist Historical Baselining + +**Branch**: `011-specialist-baselining` | **Date**: 2026-03-10 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `/specs/011-specialist-baselining/spec.md` + +## Summary + +Add 7-day historical baselining to domain specialist prompts so each finding is classified as ANOMALY, RECURRING, or BASELINE. This eliminates false alarms on known patterns (daily batch jobs, normal working-set growth) by inserting a Baseline step between Discover and Fetch in each specialist's workflow. The coordinator synthesis phase ranks anomalies above baseline/recurring findings. This is purely prompt engineering — no new tools, no new dependencies, no API changes. + +## Technical Context + +**Language/Version**: Python 3.11+ +**Primary Dependencies**: `mcp[cli]` ≥1.2.0 (FastMCP), `pydantic` v2.x — no new dependencies +**Storage**: N/A — prompts are stateless text generators +**Testing**: pytest + pytest-asyncio, unit tests asserting prompt output content +**Target Platform**: MCP server (stdio JSON-RPC) +**Project Type**: Single project +**Performance Goals**: N/A — prompt rendering is synchronous string interpolation +**Constraints**: No changes to tool signatures or MCP protocol surface +**Scale/Scope**: 3 files modified (specialist.py, coordinator.py, investigation-flow.md), ~200 lines of prompt text added + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +| Principle | Status | Notes / Justification | +|-----------|--------|----------------------| +| I. Code Quality — linting enforced, single-responsibility, complexity ≤ 10, peer review, documentation impact reviewed | PASS | Pure text additions to existing prompt modules. No new functions, no complexity increase. Ruff enforced. Docs impact: `docs/investigation-flow.md` (workflow diagram update), `README.md` prompt table (no signature changes, but specialist workflow description needs update). | +| II. Testing Standards — TDD cycle, ≥ 80% unit coverage, contract tests on interface changes | PASS | TDD: write tests asserting baseline/classification presence → implement prompt changes → green. No interface changes (same function signatures). Coverage ≥80% maintained. | +| III. UX Consistency — design system adherence, WCAG 2.1 AA, actionable error messages | N/A | No user-facing UI. Prompt text follows existing specialist output patterns. | +| IV. Performance — latency SLA defined, performance budget in CI, profiling before optimization | N/A | No runtime performance impact — prompt rendering is string interpolation. The additional 7-day fetch is an LLM-executed action, not our code. | +| V. Simplicity — YAGNI posture, no speculative abstractions, complexity justified below | PASS | No new abstractions. Classification is LLM judgement guided by prompt text. No new tools, no periodicity detection, no configuration — YAGNI respected. | + +## Project Structure + +### Documentation (this feature) + +```text +specs/011-specialist-baselining/ +├── plan.md # This file +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output (report structure) +├── quickstart.md # Phase 1 output +└── tasks.md # Phase 2 output (/speckit.tasks) +``` + +### Source Code (files modified) + +```text +src/pmmcp/prompts/ +├── specialist.py # Add Baseline step, classification, domain heuristics +└── coordinator.py # Add classification-based ranking in synthesis + +docs/ +└── investigation-flow.md # Update workflow diagram to 5-step flow + +tests/unit/ +├── test_prompts_specialist.py # New assertions for baseline/classification +└── test_prompts_coordinator.py # New assertions for classification ranking +``` + +**Structure Decision**: Existing single-project layout. All changes are modifications to existing files — no new modules created. + +## Complexity Tracking + +> No violations. All changes are additive text within existing modules. + +--- + +## Design: Changes by File + +### 1. `src/pmmcp/prompts/specialist.py` + +#### 1a. Workflow: Insert Baseline Step (Step 2) + +Current 4-step workflow becomes 5 steps: **Discover → Baseline → Fetch → Analyse → Report** + +The Baseline step (for all 5 domain specialists, NOT cross-cutting) instructs the agent to: + +1. Fetch 7-day historical data at 1hour interval using `pcp_fetch_timeseries` for the discovered metrics +2. Run `pcp_detect_anomalies` comparing the investigation window against the 7-day baseline +3. Note anomaly detection results for use in the Analyse step +4. If `pcp_detect_anomalies` returns insufficient data or errors, note the limitation and proceed with threshold-only analysis + +#### 1b. Report Structure: Add Classification Fields + +Each finding in the specialist report gains three new fields: + +- `classification`: ANOMALY | RECURRING | BASELINE +- `baseline_context`: human-readable comparison (e.g., "CPU idle has been below 15% for the past 7 days") +- `severity_despite_baseline`: threshold-based severity independent of classification (critical/warning/info/none) + +Report guidance instructs the agent to: +- Articulate chronic problems narratively ("this is bad, but based on previous days this is not a new problem") +- Distinguish RECURRING from BASELINE by examining timeseries shape for time-of-day correlation + +#### 1c. Domain Knowledge: Add Baseline-Aware Heuristics + +Each subsystem gets at least one baseline-aware heuristic: + +- **CPU**: Check whether current CPU levels are typical for this time of day over the past week before flagging saturation +- **Memory**: Compare memory growth against 7-day baseline to distinguish leaks from normal working-set growth +- **Disk**: Check whether I/O spikes recur at the same time daily (scheduled jobs like backups, log rotation) +- **Network**: Check whether current packet drop rate is within normal variance for this interface over the past week +- **Process**: Check whether process count and context switch rate match the 7-day pattern before flagging runaway processes + +#### 1d. Cross-Cutting: Classification Consumption (No Baseline Step) + +Cross-cutting domain knowledge updated to: +- Prioritise ANOMALY-classified findings over RECURRING or BASELINE +- Flag correlated anomalies across multiple subsystems at the same timestamp +- Note when one subsystem reports BASELINE while another reports ANOMALY (the anomaly is more likely root cause) + +#### 1e. Graceful Degradation + +The Baseline step includes fallback instructions: +- If `pcp_detect_anomalies` returns insufficient data: fall back to threshold-only analysis +- Note "insufficient baseline data, falling back to threshold-only analysis" in the report +- If baseline data exists but is sparse: attempt anomaly detection but note reduced confidence +- If 0 days of history: skip Baseline step entirely, proceed with threshold-only Fetch → Analyse → Report + +### 2. `src/pmmcp/prompts/coordinator.py` + +#### 2a. Synthesis Phase: Classification-Based Ranking + +Update the synthesis section to: +- Always rank ANOMALY findings above BASELINE/RECURRING regardless of severity +- Within each classification tier, sort by severity (critical → warning → info) +- Explicitly call out findings that are normal behaviour for the host +- Highlight when an apparent anomaly matches a known recurring pattern + +#### 2b. Output Structure Update + +The "Findings by Severity" section becomes "Findings by Classification & Severity": + +``` +## Findings by Classification & Severity + +### New Anomalies (changed from baseline) +1. [CRITICAL] ... +2. [WARNING] ... + +### Recurring Patterns (known periodic behaviour) +1. ... + +### Baseline Behaviour (chronic conditions) +1. [WARNING] ... (your normal is degraded — here's context) + +### Normal Operation +- No anomalies detected for: [subsystems] +``` + +### 3. `docs/investigation-flow.md` + +Update the specialist workflow diagram from 4-step to 5-step: +- Discover → **Baseline** → Fetch → Analyse → Report +- Note that cross-cutting does NOT include the Baseline step +- Update the prompt table if it references specialist workflow steps + +## Test Strategy + +All tests are unit tests asserting prompt output string content — same pattern as existing tests. + +### Specialist Tests (Story 1 + 2 + 5) + +| Test | Asserts | +|------|---------| +| Baseline step present for all 5 domain subsystems | "Baseline" in workflow, `pcp_detect_anomalies` referenced | +| Baseline step absent for cross-cutting | "Baseline" NOT in cross-cutting workflow | +| Classification fields in report structure | "classification", "ANOMALY", "RECURRING", "BASELINE" in report guidance | +| `baseline_context` in report structure | "baseline_context" in report guidance | +| `severity_despite_baseline` in report structure | "severity_despite_baseline" in report guidance | +| Domain heuristics reference baseline | Each subsystem has at least one baseline-aware heuristic | +| Graceful degradation instructions present | "insufficient baseline" or fallback guidance in prompt | +| 7-day window referenced | "7-day" or "7 day" in baseline instructions | +| `pcp_fetch_timeseries` referenced in baseline | Tool name appears in baseline step | + +### Cross-Cutting Tests (Story 3) + +| Test | Asserts | +|------|---------| +| Classification prioritisation in domain knowledge | "ANOMALY" prioritised over "RECURRING"/"BASELINE" | +| Correlated anomaly detection guidance | Cross-subsystem correlation referenced | +| Mixed classification guidance | Guidance for BASELINE + ANOMALY across subsystems | + +### Coordinator Tests (Story 4) + +| Test | Asserts | +|------|---------| +| Classification ranking in synthesis | "ANOMALY" ranked above "BASELINE"/"RECURRING" | +| Baseline behaviour call-out | Normal behaviour explicitly mentioned | +| Recurring pattern highlighting | Pattern matching guidance present | + +## Implementation Order + +Stories are independent enough for parallel implementation in worktrees: + +**Worktree A**: Stories 1 + 2 + 5 (specialist.py changes + specialist tests) +- These are tightly coupled — baseline step, domain heuristics, and degradation are all in the same `_SPECIALIST_KNOWLEDGE` dict and workflow template + +**Worktree B**: Story 3 + 4 (cross-cutting + coordinator changes + tests) +- Cross-cutting domain knowledge update + coordinator synthesis update + their tests + +**Final merge**: Story 3+4 depends on Story 1+2 classification fields existing, but since they're prompt text (not runtime dependencies), the tests can assert independently. Merge worktree A first, then worktree B. + +**Docs** (investigation-flow.md): Can be done in either worktree or main branch after merge. diff --git a/specs/011-specialist-baselining/quickstart.md b/specs/011-specialist-baselining/quickstart.md new file mode 100644 index 0000000..79808d2 --- /dev/null +++ b/specs/011-specialist-baselining/quickstart.md @@ -0,0 +1,44 @@ +# Quickstart: Specialist Historical Baselining + +**Feature**: 011-specialist-baselining | **Date**: 2026-03-10 + +## What Changed + +Domain specialist prompts now include a **Baseline step** that instructs the investigating agent to fetch 7-day historical data and classify each finding as ANOMALY, RECURRING, or BASELINE. The coordinator ranks anomalies above baseline behaviour in its synthesis. + +## Verify It Works + +```bash +# Run all tests (baseline assertions included) +uv run pytest tests/unit/test_prompts_specialist.py tests/unit/test_prompts_coordinator.py -v + +# Full suite with coverage +uv run pytest --cov=pmmcp --cov-report=term-missing +``` + +## Key Files + +| File | Change | +|------|--------| +| `src/pmmcp/prompts/specialist.py` | Baseline step, classification fields, baseline-aware heuristics, graceful degradation | +| `src/pmmcp/prompts/coordinator.py` | Classification-based ranking in synthesis phase | +| `docs/investigation-flow.md` | 5-step workflow diagram | +| `tests/unit/test_prompts_specialist.py` | Baseline/classification presence assertions | +| `tests/unit/test_prompts_coordinator.py` | Classification ranking assertions | + +## How Classification Works + +The specialist prompt instructs the agent to: +1. **Baseline step**: Fetch 7-day history at 1hour interval, run `pcp_detect_anomalies` +2. **Classify each finding**: + - **ANOMALY**: Significant deviation from baseline, not a recurring pattern + - **RECURRING**: Spikes that repeat at the same time of day (batch jobs, backups) + - **BASELINE**: Normal behaviour for this host +3. **Severity despite baseline**: A finding can be BASELINE and still critical (chronic degradation) + +## Impact on Existing Prompts + +- **No signature changes** — `specialist_investigate` and `coordinate_investigation` accept the same parameters +- **No tool changes** — `pcp_detect_anomalies`, `pcp_quick_investigate`, etc. are unchanged +- **No new dependencies** — pure prompt text additions +- **Cross-cutting specialist** does NOT get a Baseline step — it consumes classifications from domain specialists diff --git a/specs/011-specialist-baselining/research.md b/specs/011-specialist-baselining/research.md new file mode 100644 index 0000000..dc5028e --- /dev/null +++ b/specs/011-specialist-baselining/research.md @@ -0,0 +1,88 @@ +# Research: Specialist Historical Baselining + +**Feature**: 011-specialist-baselining | **Date**: 2026-03-10 + +## Research Tasks & Findings + +### R1: How does `pcp_detect_anomalies` work and what does it return? + +**Decision**: Use the existing `pcp_detect_anomalies` tool as-is — no modifications needed. + +**Findings**: +- Signature: `(metrics, recent_start, recent_end, baseline_start, baseline_end, z_score_threshold=2.0, host="", interval="auto")` +- Returns `list[dict]` with: `metric`, `severity` (critical/warning/info/none), `z_score`, `direction` (higher/lower), `recent_mean`, `recent_stddev`, `baseline_mean`, `baseline_stddev` +- Z-score calculation: `(recent_mean - baseline_mean) / baseline_stddev` +- Already handles the statistical comparison between two time windows + +**Rationale**: The tool already accepts arbitrary time windows. The Baseline step just needs to instruct the agent to call it with a 7-day baseline window. + +### R2: How should the Baseline step be structured in the workflow? + +**Decision**: Insert Baseline as step 2 in the specialist workflow (Discover → **Baseline** → Fetch → Analyse → Report). + +**Findings**: +- Current workflow is 4 steps: Discover → Fetch → Analyse → Report +- The Baseline step needs to come after Discover (so we know what metrics exist) but before Fetch (so the live data fetch can be informed by baseline context) +- The Baseline step instructs the agent to: + 1. Fetch 7-day historical data at 1hour interval using `pcp_fetch_timeseries` + 2. Run `pcp_detect_anomalies` comparing investigation window against 7-day baseline + 3. Note anomaly results for use in the Analyse step + +**Alternatives considered**: +- Baseline after Fetch: Rejected — agent benefits from knowing baseline context before interpreting live data +- Baseline merged into Analyse: Rejected — separating concerns keeps each step focused + +### R3: How should classification (ANOMALY/RECURRING/BASELINE) be assigned? + +**Decision**: LLM judgement guided by prompt heuristics — no new tool required. + +**Findings**: +- ANOMALY: `pcp_detect_anomalies` returns a significant z-score and the pattern doesn't recur in the 7-day timeseries +- RECURRING: 7-day timeseries shows repeated spikes at consistent times of day (batch jobs, log rotation, backups) +- BASELINE: Current values are within normal range for this host based on the 7-day history +- The specialist examines timeseries shape for time-of-day correlation + +**Rationale**: Domain knowledge already references daily patterns. LLM is well-suited to pattern recognition in timeseries shapes described textually. A dedicated periodicity tool would be over-engineering for this iteration. + +### R4: How should `severity_despite_baseline` work? + +**Decision**: Independent field that carries threshold-based severity regardless of classification. + +**Findings**: +- A finding can be BASELINE (normal for this host) but still severity=warning/critical (thresholds exceeded) +- This captures the "your normal is sick" scenario — e.g., CPU idle consistently < 10% for a week +- The specialist articulates this narratively: "this is bad, but based on previous days this is not a new problem" + +**Alternatives considered**: +- Merge severity into classification: Rejected — conflates "is this new?" with "is this bad?" +- Separate classification and severity tools: Rejected — over-engineering; the LLM assigns both from the same data + +### R5: Cross-cutting specialist — baseline step or not? + +**Decision**: No baseline step for cross-cutting. It consumes classifications from domain specialists. + +**Findings**: +- Cross-cutting already runs `pcp_quick_investigate` for broad anomaly scanning +- Adding a baseline step would duplicate domain specialists' work +- Cross-cutting's role is correlation, not independent metric analysis +- It receives classification data through domain specialist reports and uses it for prioritisation + +### R6: Graceful degradation approach + +**Decision**: Prompt-level fallback instructions — no code changes needed. + +**Findings**: +- When `pcp_detect_anomalies` returns insufficient data (few or no results), the specialist falls back to threshold-only analysis +- The specialist notes the limitation in its report: "insufficient baseline data, falling back to threshold-only analysis" +- Common causes: new host, recent PCP deployment, archive gaps +- Sparse data (gaps from PCP restarts) still produces results but with reduced confidence — specialist notes this + +### R7: Coordinator classification ranking + +**Decision**: ANOMALY always ranks above BASELINE/RECURRING, regardless of severity. Severity is secondary sort within each tier. + +**Findings**: +- "What changed" is more actionable than "what's chronically wrong" +- Chronic issues (BASELINE with severity warning/critical) are still reported, just after anomalies +- Coordinator synthesis explicitly calls out baseline behaviour to reduce noise +- Highlights when an apparent anomaly matches a known recurring pattern diff --git a/specs/011-specialist-baselining/spec.md b/specs/011-specialist-baselining/spec.md new file mode 100644 index 0000000..d5ea6be --- /dev/null +++ b/specs/011-specialist-baselining/spec.md @@ -0,0 +1,160 @@ +# Feature Specification: Specialist Historical Baselining + +**Feature Branch**: `011-specialist-baselining` +**Created**: 2026-03-10 +**Status**: Draft +**Input**: User description: "Specialists should baseline against historical data to distinguish anomalies from normal behaviour" +**GitHub Issue**: #30 + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Specialist findings include anomaly classification (Priority: P1) + +When a performance investigation runs, each domain specialist (CPU, Memory, Disk, Network, Process) fetches a 7-day historical baseline for its key metrics and classifies every finding as ANOMALY, RECURRING, or BASELINE. This prevents false alarms on known patterns like daily batch jobs or normal working-set growth. + +**Why this priority**: This is the core value proposition — without classification, specialists cry wolf on business-as-usual behaviour, drowning the signal in noise. + +**Independent Test**: Invoke `specialist_investigate(subsystem="cpu")` and verify the prompt output includes instructions to fetch a 7-day baseline, run `pcp_detect_anomalies`, and classify each finding. + +**Acceptance Scenarios**: + +1. **Given** a specialist prompt for any domain subsystem, **When** the prompt is rendered, **Then** it includes a Baseline step between Discover and Fetch that instructs the agent to fetch 7-day historical data and run `pcp_detect_anomalies`. +2. **Given** a specialist prompt for any domain subsystem, **When** the prompt is rendered, **Then** it includes instructions to classify each finding as ANOMALY, RECURRING, or BASELINE with baseline context. +3. **Given** a specialist prompt for any domain subsystem, **When** the prompt is rendered, **Then** the report structure includes `classification`, `baseline_context`, and `severity_despite_baseline` fields. +4. **Given** a specialist prompt for any domain subsystem, **When** the prompt is rendered, **Then** the report guidance instructs the agent to articulate chronic problems narratively (e.g., "this is bad, but based on previous days this is not a new problem — here's what I think is happening"). + +--- + +### User Story 2 - Domain knowledge augmented with baseline-aware guidance (Priority: P1) + +Each specialist's domain knowledge heuristics are updated to reference the baseline before making threshold-based judgements. For example, the CPU specialist checks whether current CPU levels are typical for this time of day before flagging saturation. + +**Why this priority**: Tied with Story 1 — the baseline step is useless if the domain heuristics still say "idle < 10% = saturated" without checking whether that's normal. + +**Independent Test**: Read the domain knowledge for each subsystem and verify at least one heuristic references the baseline or `pcp_detect_anomalies`. + +**Acceptance Scenarios**: + +1. **Given** the CPU specialist domain knowledge, **When** rendered, **Then** it includes guidance to check whether current CPU levels are typical for this time of day over the past week before flagging saturation. +2. **Given** the Memory specialist domain knowledge, **When** rendered, **Then** it includes guidance to compare memory growth against the 7-day baseline to distinguish leaks from normal working-set growth. +3. **Given** the Disk specialist domain knowledge, **When** rendered, **Then** it includes guidance to check whether I/O spikes recur at the same time daily (scheduled jobs like backups, log rotation). +4. **Given** the Network and Process specialist domain knowledge, **When** rendered, **Then** each includes at least one baseline-aware heuristic. + +--- + +### User Story 3 - Cross-cutting specialist prioritises by classification (Priority: P2) + +The cross-cutting specialist uses anomaly classifications from domain specialists to prioritise findings. ANOMALY-classified findings are prioritised over RECURRING or BASELINE. Correlated anomalies across multiple subsystems at the same timestamp receive higher confidence. + +**Why this priority**: Builds on P1 classifications to improve the cross-cutting correlator's signal-to-noise ratio. + +**Independent Test**: Render the cross-cutting specialist prompt and verify it references classification-based prioritisation and correlated anomaly detection. + +**Acceptance Scenarios**: + +1. **Given** the cross-cutting specialist domain knowledge, **When** rendered, **Then** it includes guidance to prioritise ANOMALY-classified findings over RECURRING or BASELINE. +2. **Given** the cross-cutting specialist domain knowledge, **When** rendered, **Then** it includes guidance to flag correlated anomalies across multiple subsystems at the same timestamp. +3. **Given** the cross-cutting specialist domain knowledge, **When** rendered, **Then** it includes guidance to note when one subsystem reports BASELINE while another reports ANOMALY. + +--- + +### User Story 4 - Coordinator synthesis weights findings by classification (Priority: P2) + +The coordinator's synthesis phase leads with anomalies, explicitly calls out baseline behaviour to reduce noise, and highlights when an apparent anomaly matches a known recurring pattern. + +**Why this priority**: Final layer of the classification value chain — ensures the end-user report is noise-reduced. + +**Independent Test**: Render the `coordinate_investigation` prompt and verify the synthesis section references classification weighting. + +**Acceptance Scenarios**: + +1. **Given** the coordinator synthesis prompt, **When** rendered, **Then** it includes guidance to always rank ANOMALY findings above BASELINE/RECURRING regardless of severity, with severity as secondary sort within each tier. +2. **Given** the coordinator synthesis prompt, **When** rendered, **Then** it includes guidance to explicitly call out findings that are normal behaviour for the host. +3. **Given** the coordinator synthesis prompt, **When** rendered, **Then** it includes guidance to highlight when an apparent anomaly matches a known recurring pattern. + +--- + +### User Story 5 - Graceful degradation when baseline data is insufficient (Priority: P2) + +When a host has less than 7 days of historical data (new host, recent PCP deployment, archive gaps), the specialist gracefully falls back to threshold-only analysis and notes the limitation. + +**Why this priority**: Without this, the baseline step would fail or produce misleading results on hosts with limited history. + +**Independent Test**: Verify the specialist prompt includes fallback guidance for insufficient baseline data. + +**Acceptance Scenarios**: + +1. **Given** a specialist prompt, **When** rendered, **Then** it includes instructions to fall back to threshold-only analysis if `pcp_detect_anomalies` returns insufficient data. +2. **Given** a specialist prompt, **When** rendered, **Then** it includes instructions to note "insufficient baseline data, falling back to threshold-only analysis" in the report when degraded. + +--- + +### Edge Cases + +- What happens when the host has exactly 0 days of historical data? Specialist falls back entirely to threshold-based analysis with a clear note. +- What happens when baseline data exists but is sparse (e.g., gaps due to PCP restarts)? Specialist should still attempt anomaly detection but note reduced confidence. +- What happens when the 7-day baseline itself is anomalous (e.g., host was under sustained incident for a week)? The classification should still flag current behaviour relative to the available baseline — the specialist cannot know the baseline is "wrong" but the report should note the baseline window. +- What happens when `pcp_detect_anomalies` returns no anomalies? The specialist should report "no anomalies detected relative to 7-day baseline" — this is a valid finding. + +## Clarifications + +### Session 2026-03-10 + +- Q: Should BASELINE classification distinguish "normal and healthy" from "normal but degraded"? → A: Keep three classifications (ANOMALY / RECURRING / BASELINE) but add a `severity_despite_baseline` field. A finding can be BASELINE and still carry a severity (warning/critical) based on thresholds. This lets the agent articulate "this is your normal, but your normal is sick." +- Q: How should coordinator rank ANOMALY vs BASELINE findings when severity differs? → A: ANOMALY always ranks above BASELINE/RECURRING regardless of severity. "What changed" is more actionable than "what's chronically wrong." Chronic issues are still reported but after new anomalies. +- Q: How should specialists distinguish RECURRING from BASELINE without a periodicity-detection tool? → A: LLM judgement guided by prompt heuristics. The specialist examines the 7-day timeseries shape for time-of-day correlation (repeated spikes at the same hour). Domain knowledge already references daily patterns (batch jobs, log rotation, backups). No new tooling needed. +- Q: Should the cross-cutting specialist get its own baseline step? → A: No. Cross-cutting consumes and correlates classifications from the 5 domain specialists only. It already runs `pcp_quick_investigate` for broad anomaly scanning — adding a baseline step would duplicate the domain specialists' work. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: Each domain specialist prompt (cpu, memory, disk, network, process) MUST include a Baseline step in its workflow between Discover and Fetch. The cross-cutting specialist does NOT get a Baseline step — it consumes and correlates classifications from domain specialists. +- **FR-002**: The Baseline step MUST instruct the agent to fetch 7-day historical data at a coarse interval (e.g., 1hour) using `pcp_fetch_timeseries`. +- **FR-003**: The Baseline step MUST instruct the agent to run `pcp_detect_anomalies` comparing the investigation window against the 7-day baseline. +- **FR-004**: The specialist report structure MUST include a `classification` field with values: ANOMALY, RECURRING, or BASELINE. RECURRING is assigned by LLM judgement when the 7-day timeseries shows repeated spikes at consistent times of day — guided by domain heuristics (batch jobs, log rotation, backups). No dedicated periodicity-detection tool is required. +- **FR-005**: The specialist report structure MUST include a `baseline_context` field providing human-readable comparison to the baseline. +- **FR-005a**: The specialist report structure MUST include a `severity_despite_baseline` field that carries the threshold-based severity (critical/warning/info/none) independently of classification. A BASELINE-classified finding with severity warning or critical indicates a chronic problem — "this is your normal, but your normal is degraded." +- **FR-006**: Each domain specialist's domain knowledge MUST include at least one baseline-aware heuristic relevant to its subsystem. +- **FR-007**: The cross-cutting specialist domain knowledge MUST reference anomaly classification for prioritisation and correlation. +- **FR-008**: The coordinator synthesis phase MUST rank findings with ANOMALY classification above BASELINE/RECURRING regardless of severity. Within each classification tier, findings are sorted by severity (critical → warning → info). Chronic issues (BASELINE/RECURRING with non-zero severity) MUST still be reported, but after anomalies. +- **FR-009**: The specialist prompt MUST include graceful degradation instructions for insufficient baseline data. +- **FR-010**: The investigation flow documentation MUST be updated to reflect the 5-step specialist workflow (Discover → Baseline → Fetch → Analyse → Report). + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: All 5 domain specialist prompts include the Baseline step referencing `pcp_detect_anomalies` in their workflow. +- **SC-002**: All 5 domain specialist prompts include the classification field (ANOMALY / RECURRING / BASELINE), baseline_context, and severity_despite_baseline in their report structure. +- **SC-003**: All 5 domain specialist domain knowledge sections include at least one baseline-aware heuristic. +- **SC-004**: Cross-cutting specialist references classification-based prioritisation in its domain knowledge. +- **SC-005**: Coordinator synthesis phase references classification weighting. +- **SC-006**: Specialist prompts include fallback guidance for insufficient baseline data. +- **SC-007**: Investigation flow diagram shows 5-step workflow (Discover → Baseline → Fetch → Analyse → Report). +- **SC-008**: All unit tests pass, including new tests verifying baseline step and classification presence in prompt output. + +## Assumptions + +- The 7-day baseline window is a fixed default — not user-configurable in this iteration. This covers daily and weekly recurring patterns, which are the most common. Configurability can be added later if needed. +- The `pcp_detect_anomalies` tool already handles the statistical comparison (z-score based) and does not need modification. +- The `pcp_compare_windows` and `pcp_quick_investigate` tools do not need modification — this feature is purely prompt engineering. +- The classification is advisory — the LLM agent interprets the anomaly detection results and assigns the classification. There is no new tool that automatically classifies. +- Performance impact of the additional baseline fetch is acceptable — the 7-day fetch at 1hour interval is a modest data volume. +- This feature does not change the specialist prompt's function signature — no new parameters are added. + +## Scope + +### In scope + +- Modifications to `src/pmmcp/prompts/specialist.py` (workflow, domain knowledge, report structure) +- Modifications to `src/pmmcp/prompts/coordinator.py` (synthesis guidance) +- Updates to `docs/investigation-flow.md` (workflow diagram) +- Unit tests for prompt output verification + +### Not in scope + +- Changes to existing tools (`pcp_detect_anomalies`, `pcp_compare_windows`, `pcp_quick_investigate`) +- New tool development +- Making the baseline window configurable +- Changing the `lookback` parameter semantics diff --git a/specs/011-specialist-baselining/tasks.md b/specs/011-specialist-baselining/tasks.md new file mode 100644 index 0000000..9031ab5 --- /dev/null +++ b/specs/011-specialist-baselining/tasks.md @@ -0,0 +1,290 @@ +# Tasks: Specialist Historical Baselining + +**Input**: Design documents from `/specs/011-specialist-baselining/` +**Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/ + +**Tests**: Per the project constitution (Principle II — Testing Standards, NON-NEGOTIABLE), tests +are mandatory. Unit coverage MUST reach ≥ 80% and contract tests MUST accompany any interface +change. TDD cycle: write failing tests → implement → refactor. + +**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3) +- Include exact file paths in descriptions + +## Path Conventions + +- **Single project**: `src/`, `tests/` at repository root + +--- + +## Phase 1: Setup + +**Purpose**: No new dependencies or project structure changes. This feature is purely prompt text modifications to existing files. + +- [x] T001 Verify existing tests pass and baseline coverage in `tests/unit/test_prompts_specialist.py` and `tests/unit/test_prompts_coordinator.py` by running `uv run pytest tests/unit/test_prompts_specialist.py tests/unit/test_prompts_coordinator.py --cov=pmmcp.prompts -v` + +**Checkpoint**: Green baseline confirmed — all existing prompt tests pass, coverage baseline recorded. + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: No foundational/blocking prerequisites for this feature. All changes are additive text to existing prompt modules. User story phases can begin immediately after Phase 1 confirmation. + +--- + +## Phase 3: User Story 1 — Specialist findings include anomaly classification (Priority: P1) MVP + +**Goal**: Insert a Baseline step (step 2) into the 5 domain specialist workflows and add classification/baseline_context/severity_despite_baseline fields to the report structure. Cross-cutting does NOT get a Baseline step. + +**Independent Test**: Invoke `_specialist_investigate_impl(subsystem="cpu")` and verify the prompt output includes instructions to fetch a 7-day baseline, run `pcp_detect_anomalies`, and classify each finding. + +**Worktree**: A (specialist.py changes — shares worktree with US2 and US5) + +### Tests for User Story 1 *(required per Principle II — Testing Standards)* + +> **NOTE: Write these tests FIRST, ensure they FAIL before implementation (Red-Green-Refactor)** + +- [x] T002 [US1] Write test in `tests/unit/test_prompts_specialist.py` asserting that `_specialist_investigate_impl()` output for each of the 5 domain subsystems (cpu, memory, disk, network, process) contains a "Baseline" step between Discover and Fetch that references `pcp_fetch_timeseries` and `pcp_detect_anomalies` and a "7-day" or "7 day" baseline window +- [x] T003 [US1] Write test in `tests/unit/test_prompts_specialist.py` asserting that `_specialist_investigate_impl(subsystem="crosscutting")` output does NOT contain a "Baseline" step (cross-cutting consumes classifications, it doesn't baseline independently) +- [x] T004 [US1] Write test in `tests/unit/test_prompts_specialist.py` asserting that `_specialist_investigate_impl()` output for each of the 5 domain subsystems includes `classification`, `ANOMALY`, `RECURRING`, `BASELINE`, `baseline_context`, and `severity_despite_baseline` in the report structure guidance +- [x] T005 [US1] Write test in `tests/unit/test_prompts_specialist.py` asserting that `_specialist_investigate_impl()` output for domain subsystems includes narrative guidance for chronic problems (e.g., references to "not a new problem" or "your normal" or similar phrasing that instructs the agent to contextualise baseline-classified findings) +- [x] T006 [US1] Run tests and confirm they FAIL (RED): `uv run pytest tests/unit/test_prompts_specialist.py -v -k "baseline or classification"` — commit failing tests + +### Implementation for User Story 1 + +- [x] T007 [US1] In `src/pmmcp/prompts/specialist.py`, add a Baseline step (step 2) to the specialist workflow template for domain subsystems only (cpu, memory, disk, network, process). The step instructs the agent to: (1) fetch 7-day historical data at 1hour interval using `pcp_fetch_timeseries`, (2) run `pcp_detect_anomalies` comparing investigation window against 7-day baseline, (3) note results for the Analyse step. Cross-cutting workflow remains unchanged (4 steps). +- [x] T008 [US1] In `src/pmmcp/prompts/specialist.py`, update the report structure/guidance for domain subsystems to include three new fields: `classification` (ANOMALY/RECURRING/BASELINE), `baseline_context` (human-readable comparison), `severity_despite_baseline` (threshold severity independent of classification). Add narrative guidance for RECURRING detection (time-of-day correlation in 7-day timeseries) and chronic problem articulation. +- [x] T009 [US1] Run tests and confirm they PASS (GREEN): `uv run pytest tests/unit/test_prompts_specialist.py -v` — commit implementation + +**Checkpoint**: Domain specialist prompts now include Baseline step and classification fields. Cross-cutting unchanged. + +--- + +## Phase 4: User Story 2 — Domain knowledge augmented with baseline-aware guidance (Priority: P1) + +**Goal**: Add at least one baseline-aware heuristic to each domain specialist's domain knowledge section so threshold judgements reference the 7-day baseline before flagging. + +**Independent Test**: Read the domain knowledge for each subsystem and verify at least one heuristic references the baseline or `pcp_detect_anomalies`. + +**Worktree**: A (same worktree as US1 — same file, same data structure) + +### Tests for User Story 2 *(required per Principle II — Testing Standards)* + +- [x] T010 [US2] Write test in `tests/unit/test_prompts_specialist.py` asserting that the `_SPECIALIST_KNOWLEDGE` domain_knowledge for CPU contains guidance to check whether current CPU levels are typical for this time of day over the past week before flagging saturation +- [x] T011 [US2] Write test in `tests/unit/test_prompts_specialist.py` asserting that the `_SPECIALIST_KNOWLEDGE` domain_knowledge for Memory contains guidance to compare memory growth against the 7-day baseline to distinguish leaks from normal working-set growth +- [x] T012 [US2] Write test in `tests/unit/test_prompts_specialist.py` asserting that the `_SPECIALIST_KNOWLEDGE` domain_knowledge for Disk contains guidance to check whether I/O spikes recur at the same time daily (scheduled jobs) +- [x] T013 [US2] Write test in `tests/unit/test_prompts_specialist.py` asserting that the `_SPECIALIST_KNOWLEDGE` domain_knowledge for Network and Process each contain at least one baseline-aware heuristic (referencing "baseline" or "7-day" or "past week" or similar) +- [x] T014 [US2] Run tests and confirm they FAIL (RED): `uv run pytest tests/unit/test_prompts_specialist.py -v -k "baseline_heuristic or domain_knowledge_baseline"` — commit failing tests + +### Implementation for User Story 2 + +- [x] T015 [US2] In `src/pmmcp/prompts/specialist.py`, add baseline-aware heuristic to CPU domain_knowledge: check whether current CPU levels are typical for this time of day over the past week before flagging saturation +- [x] T016 [US2] In `src/pmmcp/prompts/specialist.py`, add baseline-aware heuristic to Memory domain_knowledge: compare memory growth against 7-day baseline to distinguish leaks from normal working-set growth +- [x] T017 [US2] In `src/pmmcp/prompts/specialist.py`, add baseline-aware heuristic to Disk domain_knowledge: check whether I/O spikes recur at the same time daily (scheduled jobs like backups, log rotation) +- [x] T018 [US2] In `src/pmmcp/prompts/specialist.py`, add baseline-aware heuristic to Network domain_knowledge: check whether current packet drop rate is within normal variance over the past week; and Process domain_knowledge: check whether process count and context switch rate match the 7-day pattern before flagging runaway processes +- [x] T019 [US2] Run tests and confirm they PASS (GREEN): `uv run pytest tests/unit/test_prompts_specialist.py -v` — commit implementation + +**Checkpoint**: All 5 domain specialists have baseline-aware heuristics. Worktree A pre-push sanity check. + +--- + +## Phase 5: User Story 5 — Graceful degradation when baseline data is insufficient (Priority: P2) + +**Goal**: Add fallback instructions to the Baseline step so specialists degrade gracefully when historical data is missing or sparse. + +**Independent Test**: Verify the specialist prompt includes fallback guidance for insufficient baseline data. + +**Worktree**: A (same worktree as US1/US2 — modifies the Baseline step text added in US1) + +> **Note**: US5 is prioritised before US3/US4 because it modifies the same Baseline step text from US1, and completing it in the same worktree avoids merge conflicts. + +### Tests for User Story 5 *(required per Principle II — Testing Standards)* + +- [x] T020 [US5] Write test in `tests/unit/test_prompts_specialist.py` asserting that `_specialist_investigate_impl()` output for domain subsystems includes instructions to fall back to threshold-only analysis if `pcp_detect_anomalies` returns insufficient data +- [x] T021 [US5] Write test in `tests/unit/test_prompts_specialist.py` asserting that `_specialist_investigate_impl()` output for domain subsystems includes instructions to note "insufficient baseline" or similar limitation wording in the report when degraded +- [x] T022 [US5] Run tests and confirm they FAIL (RED): `uv run pytest tests/unit/test_prompts_specialist.py -v -k "degradation or fallback or insufficient"` — commit failing tests + +### Implementation for User Story 5 + +- [x] T023 [US5] In `src/pmmcp/prompts/specialist.py`, add graceful degradation instructions to the Baseline step: (1) if `pcp_detect_anomalies` returns insufficient data, fall back to threshold-only analysis; (2) note "insufficient baseline data, falling back to threshold-only analysis" in the report; (3) if data is sparse (gaps from restarts), attempt detection but note reduced confidence; (4) if 0 days of history, skip Baseline entirely +- [x] T024 [US5] Run tests and confirm they PASS (GREEN): `uv run pytest tests/unit/test_prompts_specialist.py -v` — commit implementation +- [x] T025 [US5] Run pre-push sanity for Worktree A: `scripts/pre-push-sanity.sh` — all lint, format, and tests must pass. Push Worktree A branch. + +**Checkpoint**: Worktree A complete (US1+US2+US5). All domain specialist prompt changes done. Merge to feature branch before starting Worktree B. + +--- + +## Phase 6: User Story 3 — Cross-cutting specialist prioritises by classification (Priority: P2) + +**Goal**: Update cross-cutting specialist domain knowledge to prioritise ANOMALY-classified findings and flag correlated anomalies across subsystems. + +**Independent Test**: Render the cross-cutting specialist prompt and verify it references classification-based prioritisation and correlated anomaly detection. + +**Worktree**: B (cross-cutting + coordinator — separate from Worktree A) + +### Tests for User Story 3 *(required per Principle II — Testing Standards)* + +- [x] T026 [P] [US3] Write test in `tests/unit/test_prompts_specialist.py` asserting that `_specialist_investigate_impl(subsystem="crosscutting")` output includes guidance to prioritise ANOMALY-classified findings over RECURRING or BASELINE +- [x] T027 [P] [US3] Write test in `tests/unit/test_prompts_specialist.py` asserting that `_specialist_investigate_impl(subsystem="crosscutting")` output includes guidance to flag correlated anomalies across multiple subsystems at the same timestamp +- [x] T028 [P] [US3] Write test in `tests/unit/test_prompts_specialist.py` asserting that `_specialist_investigate_impl(subsystem="crosscutting")` output includes guidance to note when one subsystem reports BASELINE while another reports ANOMALY +- [x] T029 [US3] Run tests and confirm they FAIL (RED): `uv run pytest tests/unit/test_prompts_specialist.py -v -k "crosscutting_classification or crosscutting_priorit"` — commit failing tests + +### Implementation for User Story 3 + +- [x] T030 [US3] In `src/pmmcp/prompts/specialist.py`, update the crosscutting entry in `_SPECIALIST_KNOWLEDGE` domain_knowledge to: (1) prioritise ANOMALY findings over RECURRING/BASELINE, (2) flag correlated anomalies across subsystems at the same timestamp with higher confidence, (3) note when one subsystem reports BASELINE while another reports ANOMALY (the anomaly is more likely root cause) +- [x] T031 [US3] Run tests and confirm they PASS (GREEN): `uv run pytest tests/unit/test_prompts_specialist.py -v` — commit implementation + +**Checkpoint**: Cross-cutting specialist now consumes and prioritises by classification. + +--- + +## Phase 7: User Story 4 — Coordinator synthesis weights findings by classification (Priority: P2) + +**Goal**: Update coordinator synthesis phase to rank ANOMALY above BASELINE/RECURRING, call out baseline behaviour, and highlight recurring pattern matches. + +**Independent Test**: Render the `coordinate_investigation` prompt and verify the synthesis section references classification weighting. + +**Worktree**: B (same worktree as US3 — different file: coordinator.py) + +### Tests for User Story 4 *(required per Principle II — Testing Standards)* + +- [x] T032 [P] [US4] Write test in `tests/unit/test_prompts_coordinator.py` asserting that `_coordinate_investigation_impl()` output includes guidance to rank ANOMALY findings above BASELINE/RECURRING regardless of severity, with severity as secondary sort within each tier +- [x] T033 [P] [US4] Write test in `tests/unit/test_prompts_coordinator.py` asserting that `_coordinate_investigation_impl()` output includes guidance to explicitly call out findings that are normal behaviour for the host +- [x] T034 [P] [US4] Write test in `tests/unit/test_prompts_coordinator.py` asserting that `_coordinate_investigation_impl()` output includes guidance to highlight when an apparent anomaly matches a known recurring pattern +- [x] T035 [US4] Run tests and confirm they FAIL (RED): `uv run pytest tests/unit/test_prompts_coordinator.py -v -k "classification_ranking or baseline_callout or recurring_pattern"` — commit failing tests + +### Implementation for User Story 4 + +- [x] T036 [US4] In `src/pmmcp/prompts/coordinator.py`, update the synthesis section to: (1) rank ANOMALY above BASELINE/RECURRING regardless of severity, severity as secondary sort within each tier; (2) explicitly call out findings that are normal behaviour; (3) highlight when an apparent anomaly matches a known recurring pattern +- [x] T037 [US4] In `src/pmmcp/prompts/coordinator.py`, update the output structure template from "Findings by Severity" to "Findings by Classification & Severity" with sections: New Anomalies, Recurring Patterns, Baseline Behaviour (chronic conditions), Normal Operation +- [x] T038 [US4] Run tests and confirm they PASS (GREEN): `uv run pytest tests/unit/test_prompts_coordinator.py -v` — commit implementation +- [x] T039 [US4] Run pre-push sanity for Worktree B: `scripts/pre-push-sanity.sh` — all lint, format, and tests must pass. Push Worktree B branch. + +**Checkpoint**: Worktree B complete (US3+US4). Merge to feature branch. + +--- + +## Phase 8: Polish & Cross-Cutting Concerns + +**Purpose**: Documentation updates, final validation, cleanup. + +- [x] T040 [P] Update `docs/investigation-flow.md`: change specialist workflow diagram from 4-step to 5-step (Discover → Baseline → Fetch → Analyse → Report), note cross-cutting does NOT include Baseline step, update the sequence diagram to show the Baseline step with `pcp_fetch_timeseries` and `pcp_detect_anomalies` calls, update the "Specialist Workflow" prose section +- [x] T041 [P] Update `docs/investigation-flow.md`: update the Synthesis Phase section to reference classification-based ranking (ANOMALY > RECURRING > BASELINE) and the new output structure +- [x] T042 [P] Review `README.md` for documentation impact — update the specialist workflow description in the prompt table if it references the 4-step flow, and update tool count / prompt descriptions if affected +- [x] T043 Run full test suite with coverage: `uv run pytest --cov=pmmcp --cov-report=term-missing --cov-fail-under=80` — confirm ≥ 80% coverage maintained +- [x] T044 Run final pre-push sanity: `scripts/pre-push-sanity.sh` — lint, format, all tests green +- [x] T045 Run quickstart.md validation: execute the verification commands from `specs/011-specialist-baselining/quickstart.md` + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Phase 1 (Setup)**: No dependencies — confirm green baseline +- **Phases 3-5 (US1+US2+US5)**: Worktree A — sequential within worktree (same file), can start after Phase 1 +- **Phases 6-7 (US3+US4)**: Worktree B — can start after Worktree A merges (US3 tests assert on classification fields added by US1) +- **Phase 8 (Polish)**: After both worktrees merged + +### Worktree Strategy + +``` +main (or feature branch) + │ + ├── Worktree A: US1 → US2 → US5 (specialist.py — domain specialists) + │ └── merge back after T025 + │ + └── Worktree B: US3 → US4 (specialist.py crosscutting + coordinator.py) + └── merge back after T039 + + → Phase 8: Polish (docs, README, final validation) +``` + +### Within Each User Story + +- Tests MUST be written and FAIL before implementation (Red-Green-Refactor) +- Commit failing tests, then commit passing implementation as separate commits +- Pre-push sanity MUST pass before pushing each worktree + +### Parallel Opportunities + +- **T026, T027, T028**: US3 tests can be written in parallel (different test functions, no dependencies) +- **T032, T033, T034**: US4 tests can be written in parallel (different test functions, no dependencies) +- **T040, T041, T042**: Polish documentation tasks can be done in parallel (different files/sections) +- **Worktrees A and B**: Worktree B can begin setup while Worktree A is in final pre-push, but tests should run after Worktree A merge + +--- + +## Parallel Example: Worktree A + +```bash +# Story 1: Write all failing tests, then implement +# (sequential — tests and impl touch same file sections) + +# Story 2: Write all failing tests, then implement per-subsystem +# T015-T018 modify different subsystem entries in _SPECIALIST_KNOWLEDGE +# but are in the same file — implement sequentially to avoid merge issues + +# Story 5: Write failing tests, implement degradation, pre-push sanity +``` + +## Parallel Example: Worktree B + +```bash +# US3 tests can be written in parallel: +Task: "T026 — cross-cutting classification prioritisation test" +Task: "T027 — correlated anomaly detection test" +Task: "T028 — mixed classification guidance test" + +# US4 tests can be written in parallel: +Task: "T032 — classification ranking test" +Task: "T033 — baseline behaviour call-out test" +Task: "T034 — recurring pattern highlighting test" +``` + +--- + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Complete Phase 1: Confirm green baseline +2. Complete Phase 3: US1 — Baseline step + classification fields +3. **STOP and VALIDATE**: Run specialist tests, verify baseline step renders for all 5 domain subsystems +4. This alone delivers the core value: classification in specialist output + +### Incremental Delivery + +1. US1 → Baseline step and classification fields (core value) +2. US2 → Domain heuristics reference baseline (prevents false threshold alarms) +3. US5 → Graceful degradation (handles edge cases) +4. US3 → Cross-cutting prioritises by classification (correlation layer) +5. US4 → Coordinator ranks by classification (synthesis layer) +6. Polish → Documentation catches up + +### Worktree Strategy (Recommended) + +With parallel execution capacity: + +1. **Worktree A**: US1 → US2 → US5 (all specialist.py domain specialist changes) +2. Merge Worktree A +3. **Worktree B**: US3 → US4 (cross-cutting + coordinator) +4. Merge Worktree B +5. **Main**: Phase 8 polish (docs, README review) + +--- + +## Notes + +- All changes are prompt text modifications — no new Python functions, classes, or modules +- No new dependencies — `pyproject.toml` unchanged +- No interface changes — function signatures for `specialist_investigate` and `coordinate_investigation` are unchanged +- Existing contract tests in `tests/contract/test_prompts.py` should continue to pass without modification +- [P] tasks = different files or test functions, no dependencies +- [Story] label maps task to specific user story for traceability +- Pre-push sanity MUST pass before pushing each worktree: `scripts/pre-push-sanity.sh` diff --git a/src/pmmcp/prompts/coordinator.py b/src/pmmcp/prompts/coordinator.py index b6760ae..69d9161 100644 --- a/src/pmmcp/prompts/coordinator.py +++ b/src/pmmcp/prompts/coordinator.py @@ -95,9 +95,21 @@ def _coordinate_investigation_impl( story of what happened. "At 14:32, memory utilisation crossed 95%, triggering swap \ activity, which caused disk I/O to spike, which manifested as CPU iowait." -4. **Rank by impact**: Order findings by severity and blast radius, not by subsystem. +4. **Rank by classification, then severity**: Group findings by classification tier, \ + not by subsystem. ANOMALY findings rank above RECURRING, which rank above BASELINE — \ + regardless of severity. Within each tier, sort by severity (critical → warning → info). \ + What changed (ANOMALY) is more actionable than what has always been wrong (BASELINE). -5. **Recommend actions**: Concrete next steps — not "investigate further" but \ +5. **Call out normal behaviour**: Explicitly identify findings that are baseline \ + behaviour — chronic conditions that are normal for this host. These still matter \ + (a host whose "normal" is CPU-saturated is sick), but they are not the trigger \ + for the current incident. + +6. **Highlight recurring patterns**: When an apparent anomaly matches a known \ + recurring pattern (RECURRING classification from specialists), call this out — \ + "this spike looks alarming but occurs daily at 2am during the backup window." + +7. **Recommend actions**: Concrete next steps — not "investigate further" but \ "check process X for memory leak" or "increase swap space as immediate mitigation." ## Output Structure @@ -109,10 +121,20 @@ def _coordinate_investigation_impl( ## Root Cause Analysis -## Findings by Severity +## Findings by Classification & Severity + +### New Anomalies 1. [CRITICAL] ... 2. [WARNING] ... -3. [INFO] ... + +### Recurring Patterns +1. [WARNING] ... (occurs daily at