feat(anomaly): control-plane saturation correlated detector (valkey#3927)

[jamby77]

Detector #5 of the board batch — the last one, deliberately sequenced after the COB detector whose REPL_BUFFER_PRESSURE signal it corroborates against. Stacked on #347 (merge that first).

Surfaces the valkey#3927 condition: under CPU saturation the single-threaded event loop starves control-plane traffic — delayed heartbeats and replication events (false node timeouts, replica drops, unnecessary failovers) and admin commands queuing behind the data backlog. Until upstream's trusted-sources QoS lands, this pairs the observable signals.

Detection

The pairing is the whole design — CPU alone never fires, and an RTT blip alone never fires:

sustained CPU ≥90% for 3 consecutive polls
AND at least one of {
  probe-RTT spike (≥4x rolling median, 100ms floor),
  connected_slaves drop since previous poll,
  recent (≤60s) REPLICATION_ROLE / CLUSTER_STATE / CLUSTER_TOPOLOGY /
    RAFT_HEALTH / REPL_BUFFER_PRESSURE anomaly
} → one CRITICAL CONTROL_PLANE_SATURATION correlated group per episode

Episode cooldown: after firing, quiet until the CPU streak fully resets (recovery + re-trigger). The emit uses the ack-after-success contract so a failed emit retries next poll.

Mechanics

• No new MetricType (PRD non-goal): the trigger is one synthetic CRITICAL CPU_UTILIZATION event carrying a structural marker (zScore 0, threshold 90) that isControlPlaneSaturationEvent recognizes — no message sniffing. Z-score CPU anomalies (always non-zero zScore) can't collide.
• Correlator rule placed first so a mixed window containing the marker plus its own side effects (failover, cluster-state events) resolves to CONTROL_PLANE_SATURATION rather than NODE_FAILOVER.
• Probe RTT measured around the poll's existing INFO call (performance.now delta) — zero added commands; server-side used_cpu_* remains the primary signal, RTT is corroboration only.
• FAILOVER_CHURN is included as a corroborator: the enum member is mirrored verbatim from the feat(anomaly): gossip-mode failover-churn detector (valkey#3996) #346 branch, so whichever lands second rebases with a trivial dedupe. A churning shard during sustained saturation is direct starvation evidence.

Tests

13 pure-module cases (streak, RTT baseline/floor/spike, corroboration matrix, ack/cooldown/re-arm, null-sample carry) + 3 correlator cases (marker match, z-score non-match, mixed-window priority) + 5 service-integration cases (replica-drop fire, event corroboration, CPU-only silence, one-per-episode + re-arm, cleanup). Full anomaly-detection suite: 263 green. tsc clean.

---

Note

Medium Risk
New detection logic on the hot poll path can mis-alert on restarts or maintenance if corroboration rules misfire; correlator priority changes how mixed CPU/failover windows are labeled.

Overview
Adds control-plane saturation detection for valkey#3927: sustained ≥90% CPU for three polls only raises a CRITICAL when paired with impact evidence (probe INFO RTT spike, replica drops, or recent replication/cluster/COB-style anomalies). High CPU or RTT alone never fires.

A new control-plane-saturation-detector module owns episode state (streaks, RTT baseline, evidence floors, restart handling). AnomalyService measures probe RTT around getInfoParsed, tracks connected_slaves drops, runs detection after other poll detectors, and emits one synthetic CPU_UTILIZATION event per episode via syntheticPattern: CONTROL_PLANE_SATURATION (no new metric type). CPU z-score config stays threshold-free to avoid double alerts.

The correlator adds a first-priority CONTROL_PLANE_SATURATION rule so mixed windows resolve to saturation over NODE_FAILOVER. AnomalyEvent gains optional syntheticPattern; connection teardown clears saturation maps. Broad unit and service integration tests cover corroboration, cooldown, restart, and cleanup.

^{Reviewed by Cursor Bugbot for commit 3aa05ea. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jamby77]

bugbot run

[jamby77]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0582eb8. Configure here.}

[jamby77]

bugbot run

[jamby77]

bugbot run

[jamby77]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 47c5b88. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d476252. Configure here.}

[KIvanow]

The detector itself is essentially merge-quality - the CPU-streak + impact-evidence pairing is the right design, and the restart-safety (streak/drop/RTT/event resets) and evidenceFloorAt stale-evidence gating are handled carefully and well-tested. Holding it as draft is the right call given the dependencies:

Blocking (sequencing):

1. Stacked on unmerged #347 - can't land until that does.
2. FAILOVER_CHURN enum duplicated with #346. It's added verbatim here, but nothing in this branch actually emits it, so its entry in CONTROL_PLANE_CORROBORATING_METRICS is a dead corroborator until #346 lands. Let #346 own the enum and drop it from this PR to avoid the duplicate-member conflict.

Design concern worth addressing before promoting out of draft:
3. The synthetic-event marker is implicit and fragile. isControlPlaneSaturationEvent recognizes the synthetic event by sniffing CPU_UTILIZATION AND CRITICAL AND zScore===0 AND threshold===90. This is collision-free today only because the CPU config has no absolute threshold - if anyone later adds criticalThreshold: 90 to the CPU config, a poll where value == mean == 90 produces exactly zScore=0, threshold=90 and gets misclassified. Recommend a dedicated marker field (explicit synthetic/anomalyType flag) and/or a single named constructor for the synthetic event, plus a warning comment at the CPU config guarding against adding an absolute threshold.
4. False-positive scenario is untested: a replica removed for maintenance while a primary sits busy-but-healthy at >=90% produces a single connected_slaves drop within the window -> CRITICAL. Rolling replica restarts under load are realistic - worth an explicit test (and maybe a second-signal requirement).

Verdict: draft - not ready. Sequence: land #347 -> land #346 (with the enum) -> rebase this, de-dupe the enum, add the false-positive test.

[jamby77]

Addressed in 69c8b14 (stays draft until #347 → #346 land, per your sequencing):

1. Sequencing — rebased onto feat(anomaly): replication output-buffer pressure detector (valkey#3963) #347's current head (2187a69), which includes its review fixes.
2. Enum de-dup — FAILOVER_CHURN dropped from this branch entirely (enum member, baseline-exclusion guard, and the dead CONTROL_PLANE_CORROBORATING_METRICS entry + its test). feat(anomaly): gossip-mode failover-churn detector (valkey#3996) #346 owns it; a comment marks where the corroborator gets re-added after the post-feat(anomaly): gossip-mode failover-churn detector (valkey#3996) #346 rebase.
3. Synthetic marker — the numeric sniff is gone. AnomalyEvent gains an explicit optional syntheticPattern field (in-memory only, never persisted or inferred), stamped exclusively by a new createControlPlaneSaturationEvent constructor; isControlPlaneSaturationEvent now checks only that marker. Added the exact collision case you described as a regression test (unmarked CPU event with value=90/zScore=0/threshold=90 → not recognized), plus a guard comment at the CPU spike config warning against adding absolute thresholds (they'd double-alert alongside the synthetic event regardless of the marker).
4. Maintenance false positive — implemented the second-signal requirement, graded: a lone single-replica drop (indistinguishable from maintenance) corroborates only alongside an RTT spike or a fresh control-plane anomaly; ≥2 drops in the window — one mass drop or repeated drops, the actual saturation signature — still stand alone. Your scenario (replica removed while the primary sits busy-but-healthy at ≥90%) is now an explicit no-fire test at both the detector and service level.