feat(studio): add native cluster alert collection - #2533
Conversation
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
Large foundation PR that adds native Apache RocketMQ metric collection and alerting to the Dashboard, without requiring Prometheus. The architecture is well-structured with clean separation:
- Collection layer:
CollectorSchedulerwith database lease for multi-replica coordination, pluggableClusterMetricsCollector/BusinessMetricsCollectorinterfaces - Alert domain:
AlertRuleEvaluatorfor condition matching,AlertStateMachinefor PENDING → FIRING → ACKED lifecycle with consecutive-sample threshold - Processing pipeline:
NativeAlertProcessororchestrates evaluation → state machine → persistence → notification outbox - Schema: 10 idempotent SQL migrations with proper upgrade scripts
- Frontend: Three alert surfaces (Business Alerts, Cluster Alerts, Alert Events) wired to the shared domain
The PR is explicitly scoped as a foundation, and the code quality is consistently high across all layers. Test coverage is thorough with dedicated tests for the state machine, evaluator, collectors, and UI components.
Findings
- [Info]
CollectorScheduler.java:56— Synchronous collection across instances could be slow if a broker is unresponsive - [Info]
AlertStateMachine.java:50—firedAttimestamp semantics for multi-sample thresholds (minor) - [Warning]
NativeAlertProcessor.java:52— O(n*m) rule matching per sample batch; consider caching rules by domain
Automated review by github-manager-bot
| return; | ||
| } | ||
| for (InstanceVO instance : instanceRepository.findAll()) { | ||
| collectClusterMetrics(instance); |
There was a problem hiding this comment.
[Info] The collect() method iterates all instances and all collectors synchronously within a single scheduled invocation. If a collector blocks (e.g., slow broker response), it delays subsequent collections. Consider wrapping each instance+collector pair in a CompletableFuture or at least adding a per-collector timeout to prevent one slow instance from starving others.
| int hits = state.consecutiveHits() + 1; | ||
| Instant pendingAt = state.firstPendingAt() == null ? now : state.firstPendingAt(); | ||
| if (hits < required) { | ||
| return new AlertStateUpdate(new AlertRuleState(AlertStateStatus.PENDING, hits, value, pendingAt, |
There was a problem hiding this comment.
[Info] When the state machine transitions from PENDING to FIRING, it creates a new AlertRuleState with firstPendingAt preserved from the pending phase. This is correct for tracking when the alert condition first started, but the firedAt timestamp is set to now rather than the time the threshold was actually crossed. For requiredConsecutiveSamples > 1, the actual threshold crossing happened requiredConsecutiveSamples * interval ago. This is a minor semantic point — the current behavior is reasonable.
| AlertEvaluationResult evaluation = evaluator.evaluate(rule, sample); | ||
| if (!evaluation.matches()) { | ||
| continue; | ||
| } |
There was a problem hiding this comment.
[Warning] The process() method iterates samples × rules with an O(n*m) loop. For each sample, it calls alertService.listRules(sample.domain()) which loads all rules and filters in-memory. If the rule count grows large, consider caching rules by domain or indexing them to avoid repeated full scans per sample.
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
Add native cluster alert collection for RocketMQ Studio — covers cluster health, consumer lag, and DLQ metrics with a complete alerting pipeline (collector → evaluator → state machine → notification outbox). Well-structured with good test coverage, proper lease-based coordination for multi-replica deployments, and clean separation of concerns.
Overall: LGTM — solid feature addition with a few minor optimization suggestions for large-scale deployments.
Findings
- [Info]
CollectorScheduler.java:60— Sequential instance iteration could be parallelized for multi-instance deployments - [Info]
NotificationOutboxService.java:85— Timezone consistency: consider standardizing on Instant/UTC across the alert pipeline - [Info]
AlertSilenceService.java:95—isActive()calls findAll() per evaluation; consider caching for hot paths
Automated review by github-manager-bot
| collectBusinessMetrics(instance); | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
[Info] Sequential instance iteration: collect() iterates all instances sequentially. If one instance has high latency (slow broker response), it delays collection for all subsequent instances. Consider parallelizing with CompletableFuture.supplyAsync() or a bounded thread pool for large multi-instance deployments.
| } | ||
| RmqAlertNotificationOutbox row = new RmqAlertNotificationOutbox(); | ||
| row.setAlertId(alert.getId()); | ||
| row.setChannel(channel); |
There was a problem hiding this comment.
[Info] Timezone consistency: dispatch() uses LocalDateTime.now() for dispatchedAt, while NativeAlertProcessor uses Instant.now() and ZoneOffset.UTC for alert timestamps. Consider standardizing on Instant/UTC across the alert pipeline to avoid timezone-related ordering issues in multi-region deployments.
| private static String trimToNull(String value) { | ||
| return value == null || value.isBlank() ? null : value.trim(); | ||
| } | ||
|
|
There was a problem hiding this comment.
[Info] isActive() calls repository.findAll() on every evaluation. If there are many silences and many rules being evaluated per collection cycle, this could become a hot path. Consider caching active silences with a short TTL (e.g., 10s) or loading them once per collection cycle and passing as a parameter.
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
Re-reviewed the 3 new commits since the previous approval (9366636 broker send-queue usage, 01ceb50 email notification delivery, 52498c6 cloud consumer lag collection). The additions are consistent with the established architecture — new collectors follow the same BusinessMetricsCollector / ClusterMetricsCollector SPI, the NotificationOutboxService provides reliable at-least-once delivery with proper retry/backoff, and the CloudRocketMqBusinessMetricsCollector correctly uses the InstanceProviderRegistry for multi-vendor support.
Findings
- [Info]
CloudRocketMqBusinessMetricsCollector.java— Thecollect()method iterates all consumer groups per instance. For instances with many groups, consider adding a concurrency limit or pagination to avoid long collection cycles. - [Info]
NotificationOutboxService.java— The SMTP delivery usesJavaMailSenderwhich is good. Consider adding a circuit breaker if the mail server is persistently unreachable to avoid queue buildup. - [Info] The email notification channel is a good addition for alert delivery. Ensure the SMTP credentials are configured via environment variables or secrets management, not in application properties.
Overall the incremental changes are well-structured and follow the patterns established in the foundation PR.
Automated review by github-manager-bot
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Re-reviewed the 2 new commits since the previous approval (proxy metrics collector and email notification delivery). The additions are consistent with the established architecture:
- New
ApacheRocketMqProxyMetricsCollector(131 lines): Clean implementation following the same pattern as the existing broker/nameserver collectors. Proper null-safety on cluster/proxy lists, timeout-bounded TCP probes, and correct metric labeling with cluster+proxy identity. NativeAlertRulePolicy: Addedproxy.availability→ CLUSTER domain mapping — consistent with other infrastructure metrics.NotificationOutboxService: Email channel support — minor addition.- Tests: Comprehensive coverage for the new proxy collector (availability up/down, no-proxy, unsupported vendor) and updated policy tests.
No issues found. The proxy collector correctly handles edge cases (null proxy lists, unsupported vendors, probe timeouts) and follows the established collector contract.
Automated review by github-manager-bot
|
This PR has conflicts with the base branch and cannot be merged. Please rebase or merge the base branch into your branch and resolve the conflicts: git fetch origin
git checkout feat/studio-native-alerting-foundation
git rebase origin/rocketmq-studio
# resolve conflicts, then:
git push --force-with-leaseThis is a one-time reminder. Feel free to @mention me for a re-review after conflicts are resolved. Automated notification by github-manager-bot |
b3dd5d8 to
3e8c11b
Compare
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
This PR introduces a comprehensive native alerting system for RocketMQ Studio, including metric collectors (cluster + business), rule evaluation, event lifecycle management, silence handling, and notification delivery (webhook/DingTalk/email). The design doc is thorough and the architecture demonstrates clear separation of concerns.
Given the size of this PR (8154 additions, 119 files), I focused on high-level architectural observations rather than line-by-line review:
Strengths:
- Well-structured package boundaries (cluster/metrics/ for collection, ops/alert/ for evaluation/lifecycle)
- Clean AlertRuleEvaluator with proper null/availability handling
- Multi-replica collection lease prevents duplicate collection in clustered deployments
- Comprehensive test coverage for the evaluator logic
Observations:
- The AlertRuleEvaluator.compare() method uses exact equality (==) for doubles, which can be fragile with floating-point arithmetic. Consider using a small epsilon for equality checks, or document that exact match is intentional.
- The design doc mentions Prometheus rule YAML export as optional interoperability — if this is planned, the rule model should ensure all necessary metadata is preservable for round-trip compatibility.
- With 119 files changed, this PR would benefit from being split into smaller, focused PRs (e.g., collectors, evaluator, notification pipeline) to make review more manageable and reduce merge conflict risk.
Overall, the foundation looks solid. The modular design should make it straightforward to extend with additional metric types and notification channels.
Automated review by github-manager-bot
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
Re-review after new commits. Previous floating-point comparison concern has been addressed — AlertRuleEvaluator.compare() now uses Double.compare() instead of ==, which is the correct approach.
The architecture remains solid:
- CollectorScheduler with lease-based dedup and per-collector error isolation
- AlertStateMachine with proper handling of UNAVAILABLE as opt-in path
- NativeAlertProcessor with domain-scoped rule evaluation and lifecycle-only transitions
- NotificationOutboxService with outbox pattern, exponential backoff retry, and silence support
- Comprehensive test coverage across all components
LGTM — ready to merge.
Automated review by github-manager-bot
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
Re-review after new commit 6d87035 ("fix(alert): complete native alert upgrade path"). The additions strengthen the foundation with two key improvements:
-
Schema auto-migration —
AlertSchemaMigrationnow bootstraps the 5 new tables (rmq_metric_snapshot,rmq_alert_collection_lease,rmq_alert_state,rmq_alert_silence,rmq_alert_notification_outbox) usingCREATE TABLE IF NOT EXISTS. This is safe for re-runs and eliminates manual DDL steps for existing deployments upgrading to the native alerting feature. -
Channel validation (defense in depth) — Notification channels are now validated at both the HTTP layer (
@PatternonAlertRuleRequestDTO) and the domain layer (NativeAlertRulePolicy.validateChannels()). The allowed set (dingtalk|sms|email) is consistent across both layers, and the docs are updated to match.
Test coverage is solid — new tests verify unsupported channel rejection at both layers and the schema migration creates all expected tables.
LGTM — no concerns with the new changes.
Automated review by github-manager-bot
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
This PR modifies 122 file(s) with +8013/-188 lines.
Review Notes
- Files changed: 122
- Lines: +8013 / -188
- Test coverage: ✅ Tests included
- CLA:
⚠️ Unknown
Automated review by github-manager-bot
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
Four new commits since last review address prior feedback cleanly:
- docs: Status updated to implemented with deferred-scope table and E2E verification docs
- server/scripts/native-alert-e2e.sh: Well-structured E2E validation script with proper cleanup trap, env-var validation, and full lifecycle coverage (FIRING → silence → RESOLVED → email/webhook assertions)
- ApacheRocketMqClusterMetricsCollector: Removed redundant
endpointlabel from NameServer availability metrics — simplification is correct since endpoint is already part of instance identity - client.ts: Fixed expired-session redirect from
/to/login— correct UX fix
All changes look good. CLA check not applicable (repo does not use cla-assistant).
Automated review by github-manager-bot
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
This incremental commit adds notification delivery features on top of the previously approved native alerting foundation: reminder/repeat notifications for unacknowledged FIRING alerts, paginated notification delivery listing, manual retry for failed deliveries, test notification endpoint, and DingTalk webhook signing support. The collectionEnabled gate has been removed so native collection is now always active (controlled by the distributed lease).
The changes are well-structured, properly tested, and maintain backward compatibility. The reminder logic in AlertStateMachine correctly handles the FIRING → ACKED lifecycle. Schema migrations add columns with sensible defaults.
Findings
-
[Info]
server/.../NotificationOutboxService.java— This service is growing large (enqueue, dispatch, list, retry, test). Consider splitting intoNotificationDeliveryQueryServiceandNotificationTestServiceas the codebase matures. Not blocking. -
[Info]
server/.../GeneralSettingsUpdateDTO.java— ThedingtalkSigningSecrethas@ToString.Excludebut@Datastill generatesequals/hashCodeincluding the secret. Consider adding@EqualsAndHashCode.Excludeto prevent accidental secret comparison in tests or collections. -
[Info]
server/.../NotificationOutboxService.java:sendWebhook— The return type changed fromResponseEntity<Void>toResponseEntity<Map>. If a webhook endpoint returns non-JSON (e.g., plain text error page), theMapdeserialization will fail. This is caught by the dispatch try-catch and logged, but the error message may be confusing. Consider adding aContent-Typecheck before deserialization. -
[Info]
server/.../CollectorScheduler.java— Removing thecollectionEnabledgate means native collection is now always on. This is a behavioral change from the previous opt-in model. The PR description confirms this is intentional, and the distributed lease prevents duplicate collection across replicas.
Verdict
The code changes are solid and well-tested. The reminder notification pattern is a clean addition to the state machine. Approving.
Automated review by github-manager-bot
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
Re-review after 6 new commits since last approval. Changes add alert rule import/export, semantic fingerprint-based duplicate detection, per-rule notification templates, and pre-rendered outbox content.
Review Notes
New features (all well-implemented):
-
Alert rule import/export (
AlertRuleTransferService): Clean design with version envelope, domain validation, and size limits (200 rules max).@Transactionalensures atomicity. -
Semantic fingerprint (
AlertRuleSemanticFingerprint): SHA-256 hash of evaluation-relevant fields with proper normalization (case, whitespace, threshold formatting). Length-prefixed encoding avoids ambiguity. Database unique index provides safety net against races. -
Notification templates (
AlertNotificationTemplate): Simple placeholder substitution with sensible defaults. Pre-rendered at enqueue time (content frozen even if template changes later). -
Schema migration: Properly adds
notification_template,semantic_fingerprint, andmessage_contentcolumns. NewUNIQUE INDEXon fingerprint enforces database-level duplicate prevention.
Code quality:
- Good test coverage (6 new/updated test files)
- Backward compatibility maintained (fallback template rendering for old outbox rows)
- Defensive validation throughout (version check, domain match, null safety)
- Clean separation of concerns
Minor observations:
AlertService.rejectDuplicateSemanticRule()loads all rules for fingerprint comparison — acceptable for typical rule counts (<1000), but worth monitoring if rule volume grows significantly- Import creates rules sequentially within transaction — correct behavior (rollback on failure), but could be slow for 200 rules; consider batch insert if performance becomes an issue
Verdict
Clean incremental update that addresses prior feedback. Well-tested, backward-compatible, and follows good practices.
Automated review by github-manager-bot
RockteMQ-AI
left a comment
There was a problem hiding this comment.
LGTM — re-approving after rebase. The native alerting foundation is well-structured with proper separation of concerns (CollectorScheduler → NativeAlertProcessor → AlertRuleEvaluator → AlertStateMachine). Good test coverage across all layers.
Automated review by github-manager
ff0e135 to
8c4f83a
Compare
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
Re-reviewed 5 new commits since the previous approval (c02d161 rule testing improvements, 6a2e8b3 native alerting workflow, 8c4f83a route rebase fix, fc47e9c legacy field consistency, bf0efe9 unified business rule metric selection).
The additions are consistent with the established architecture and address the incremental improvements:
- Domain enforcement —
AlertRuleControllerandClusterAlertRuleControllerboth enforce domain server-side (setDomain()), preventing clients from injecting cross-domain rules. Clean. - Import safety —
AlertRuleTransferServicevalidates version, domain match, and caps imports at 200 rules.@Transactionalensures atomicity. - Pagination —
listAlertsPagevalidatespage >= 1,pageSize1–100, andfrom <= to. Proper bounds. - SSRF protection — Webhook delivery uses
UrlHostGuard.check()+NoRedirectClientHttpRequestFactory. DingTalk signing uses HMAC-SHA256 correctly. - Semantic dedup —
semantic_fingerprintunique constraint prevents duplicate rule evaluation conditions. - State machine —
AlertStateMachine.advance()validates inputs and handlesUNAVAILABLEas the sole opt-in path for collection-failure alerts.
Observations (non-blocking)
AlertRuleControllerdual mapping —@RequestMapping({"/api/alert-rules", "/api/business-alert-rules"})with runtime URI dispatch works but is slightly fragile. Consider splitting into two controllers if the routes diverge further.LocalDateTimein API —SystemAlertController.listAlertsPageacceptsLocalDateTimeforfrom/towithout explicit timezone binding. Ensure the frontend and server share the same zone assumption, or considerInstant/OffsetDateTimefor unambiguous semantics.
Verdict
The 5 new commits maintain code quality and address incremental improvements. Architecture remains solid with proper domain separation, input validation, and security controls.
Automated review by RockteMQ-AI
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
Re-review after 10 new commits since the previous approval. The additions strengthen the native alerting system with important refinements:
Key Changes:
-
6a2e8b3— Complete native alerting workflow (1272 additions, 48 files)- New
AlertCorrelationScope— clean instance/resource matching logic for cross-domain alert correlation - New
AlertNotificationSuppressionService— prevents duplicate notifications within configurable windows - New
AlertRuleRuntimeVO— exposes runtime state for monitoring - Updated
NativeAlertProcessor— improved lifecycle transition handling
- New
-
ed625a7— Normalize native ratio thresholdsAlertRuleEvaluatornow handles ratio-based rules consistentlyAlertNotificationTemplateimprovements for variable interpolation
-
UI refinements (
bf0efe9,fc47e9c,884f98b,e713250)- Unified business rule metric selection
- Legacy field consistency
- Notification template variable insertion
- All with corresponding test updates
-
Event formatting and timezone handling (
abe8c31,560cba0)- New
format.tsutilities for consistent date/time display - System alerts page improvements
- New
-
Delivery details (
c02d161)- Enhanced
notificationDeliveries.tsxwith better status display
- Enhanced
Architecture Assessment:
- Clean separation of concerns (collector → evaluator → state machine → notification)
- Proper use of Spring transactions and dependency injection
- Comprehensive test coverage across backend and frontend
- Good use of enums and value objects for domain modeling
No blocking issues found. The incremental commits address prior feedback and add valuable features (notification suppression, correlation scope, runtime monitoring).
LGTM — approving.
Automated review by github-manager-bot
lizhimins
left a comment
There was a problem hiding this comment.
Thanks for this substantial feature — the collection lease, alert state machine, and notification outbox designs are solid and well tested overall. A few project conventions need to be addressed before we can merge:
1. Table schema convention
All new MySQL tables must begin with exactly these three columns, in this fixed order:
id bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT '...'gmt_create datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '...'gmt_modified datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '...'
Currently in server/src/main/resources/db/schema.sql (and the matching DDL in AlertSchemaMigration):
rmq_metric_snapshot:gmt_createis not in position 2 andgmt_modifiedis missingrmq_alert_collection_lease/rmq_alert_state: missinggmt_create;gmt_modifiedsits at the endrmq_alert_silence/rmq_alert_notification_outbox:gmt_create/gmt_modifiedare at the end
Please reorder/add these columns; see rmq_system_alert in the same file as a compliant reference.
2. Test method naming
Test methods must end with Test (e.g. collectsMetricsTest), not start with test. The ~294 new test methods introduced here mostly do not follow this convention (only ~16 do). Please rename the new test methods accordingly.
3. Frontend issues
AlertsPage.test.tsx— the case "disables other alert rule mutations while a bulk action is running" fails deterministically when run in isolation (3/3 failures,pointer-events: noneon the checkbox); the same test passes on the base branch. Please fix.- Minimum font size in the UI is 14px.
web/src/pages/ops/systemAlerts.tsxusesfontSize: 12in 5 places (around lines 603, 609, 615, 633, 656); please raise them to 14.
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
Signed-off-by: liuhy <liuhongyu@apache.org>
3e0af78 to
56c2700
Compare








Closes #2532
Summary
Validation
Notes
This is intentionally a foundation PR. Existing Prometheus integrations remain available; native collection provides the direct RocketMQ path for deployments that do not run Prometheus.