PHOENIX-7962 Fix ReplicationLogGroup mode initialization and handle failover deadlock#2581
Open
tkhurana wants to merge 2 commits into
Open
PHOENIX-7962 Fix ReplicationLogGroup mode initialization and handle failover deadlock#2581tkhurana wants to merge 2 commits into
tkhurana wants to merge 2 commits into
Conversation
…ailover deadlock A ReplicationLogGroup is a writer and must exist only where the local cluster is active for the HA group. Previously it was created unconditionally (lazily on first mutation, and eagerly at RS startup), which let non-active clusters start writers and re-ship, and let the cutover writer deadlock standby failover. Changes: - Role-gated creation: ReplicationLogGroup.get(...) now returns Optional and creates a group only when the local role is active; a non-active role returns empty and is not cached, so the next call after promotion re-checks. Live write path (preBatchMutate) throws on empty (split-brain guard); replay path (preWALRestore) skips re-ship on empty. - Skip eager init on non-active clusters: PhoenixRegionServerEndpoint only eager-inits where the local role is active; others init lazily on promotion. - Failover deadlock fix: a new failoverPending flag (set on entry to the ACTIVE_IN_SYNC_TO_STANDBY cutover gate, cleared on abort) suspends log rotation so no new files land in the peer's shard directory, and turns a SYNC-write failure into an RS abort instead of the illegal cutover->SAF transition. init() seeds the flag on restart-in-cutover. - Consolidated all LOCAL state listeners into subscribeToStateChanges(), each paired with its unsubscribe so close() leaks no watchers; handles cutover suspend, STANDBY/DEGRADED_STANDBY close, and SYNC/SYNC_AND_FORWARD flips. - New constructor overload lets init() reuse the record the role gate read. Tests: 9 new unit tests in ReplicationLogGroupTest and 2 integration tests (testWriterClosesOnDemotionToStandby, testCutoverSuspendsAndResumesRotation).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
ReplicationLogGroupis a writer and should exist only where the local cluster is active for the HA group. Previously it was created unconditionally — lazily on first mutation and eagerly at RS startup — which let non-active clusters start writers and re-ship (split-brain risk), and let a cutover writer deadlock standby failover by continuously rotating files into the peer's shard directory.Changes
ReplicationLogGroup.get(...)now returnsOptionaland creates a group only when the local role is active. A non-active role returns empty and is not cached, so the next call after promotion re-checks. Live write path (preBatchMutate) throws on empty (split-brain guard); replay path (preWALRestore) skips re-ship on empty.PhoenixRegionServerEndpointonly eager-inits where the local role is active; others init lazily on promotion.failoverPendingflag (set on entry to theACTIVE_IN_SYNC_TO_STANDBYcutover gate, cleared on abort) suspends log rotation and turns a SYNC-write failure into an RS abort instead of the illegal cutover→SAF transition.init()seeds the flag on restart-in-cutover.subscribeToStateChanges(), each paired with its unsubscribe soclose()leaks no watchers; handles cutover suspend,STANDBY/DEGRADED_STANDBYclose, and SYNC ↔ SYNC_AND_FORWARD flips.init()reuses the record the role gate already read.Tests
9 new unit tests in
ReplicationLogGroupTestplus 2 integration tests (testWriterClosesOnDemotionToStandby,testCutoverSuspendsAndResumesRotation).Note
Behavioral change: a sync-path write to a non-active cluster now throws rather than silently creating a writer — the intended split-brain guard.