Add state transition latency metrics for Helix participant - #3283
Conversation
Add metrics to track BOOTSTRAP→STANDBY duration, OFFLINE→BOOTSTRAP duration, bootstrap failures, and max time-in-BOOTSTRAP. These fill a gap where no latency instrumentation exists today (Helix's native StateTransitionStatMonitor is denylisted from InGraph due to high cardinality). New metrics: - bootstrapToStandbyDurationMs: histogram of successful bootstrap durations - offlineToBootstrapDurationMs: histogram of offline→bootstrap transition time - bootstrapFailureCount: counter of partitions that failed during bootstrap - maxTimeInBootstrapMs: gauge of the longest currently-bootstrapping partition Motivation: INC-15835 (Helix cascading OOM) showed that we lacked visibility into how long partitions spend in BOOTSTRAP during rebalance storms. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3283 +/- ##
=============================================
- Coverage 64.24% 50.75% -13.50%
+ Complexity 10398 8688 -1710
=============================================
Files 840 938 +98
Lines 71755 80525 +8770
Branches 8611 9692 +1081
=============================================
- Hits 46099 40869 -5230
- Misses 23004 36255 +13251
- Partials 2652 3401 +749 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| } | ||
| logger.info("Before setting partition {} to bootstrap", partitionName); | ||
| localPartitionAndState.put(partitionName, ReplicaState.BOOTSTRAP); | ||
| participantMetrics.recordOfflineToBootstrapDuration(System.currentTimeMillis() - transitionStartMs); |
There was a problem hiding this comment.
If the wait is interrupted, we still record a bootstrap duration. Should we return before these success metrics?
There was a problem hiding this comment.
The existing InterruptedException catch (line 960) intentionally falls through without throw/return — partition still enters BOOTSTRAP state, so recording the duration is correct.
| long maxDuration = 0; | ||
| for (Map.Entry<String, Long> entry : bootstrapStartTimeMs.entrySet()) { | ||
| long duration = now - entry.getValue(); | ||
| if (duration > STALE_BOOTSTRAP_THRESHOLD_MS) { |
There was a problem hiding this comment.
Can this stay read-only? Dropping the timer after 4h can hide the bootstrap time we want to measure.
There was a problem hiding this comment.
Good catch — removed the 4h eviction. Map is bounded by partition count per instance (~1.2k max), and entries are cleaned up on recordBootstrapComplete/recordBootstrapFailure. Fixed in 10dd474.
…er instance (~1.2k max)
3 participants
Summary
Add metrics to track Helix state transition latency in HelixParticipant. Today only transition counts are tracked — there is no duration/latency instrumentation. (Helix's native
StateTransitionStatMonitoris denylisted from InGraph due to per-partition cardinality.)New Metrics
bootstrapToStandbyDurationMsofflineToBootstrapDurationMsbootstrapFailureCountmaxTimeInBootstrapMsMotivation
INC-15835 (Helix cascading OOM in prod-lva1, July 2026) showed we lacked visibility into how long partitions spend in BOOTSTRAP during rebalance storms. These metrics enable:
Changes
HelixParticipantMetrics.java: Added 4 new metrics + helper methods for recording transitionsHelixParticipant.java: InstrumentedonPartitionBecomeBootstrapFromOfflineandonPartitionBecomeStandbyFromBootstrapto record timing and failures