Skip to content

PHOENIX-7962 Fix ReplicationLogGroup mode initialization and handle failover deadlock#2581

Open
tkhurana wants to merge 2 commits into
apache:PHOENIX-7562-feature-newfrom
tkhurana:PHOENIX-7962
Open

PHOENIX-7962 Fix ReplicationLogGroup mode initialization and handle failover deadlock#2581
tkhurana wants to merge 2 commits into
apache:PHOENIX-7562-feature-newfrom
tkhurana:PHOENIX-7962

Conversation

@tkhurana

Copy link
Copy Markdown
Contributor

Summary

A ReplicationLogGroup is a writer and should exist only where the local cluster is active for the HA group. Previously it was created unconditionally — lazily on first mutation and eagerly at RS startup — which let non-active clusters start writers and re-ship (split-brain risk), and let a cutover writer deadlock standby failover by continuously rotating files into the peer's shard directory.

Changes

  • Role-gated creationReplicationLogGroup.get(...) now returns Optional and creates a group only when the local role is active. A non-active role returns empty and is not cached, so the next call after promotion re-checks. Live write path (preBatchMutate) throws on empty (split-brain guard); replay path (preWALRestore) skips re-ship on empty.
  • Skip eager init on non-active clustersPhoenixRegionServerEndpoint only eager-inits where the local role is active; others init lazily on promotion.
  • Failover deadlock fix — a new failoverPending flag (set on entry to the ACTIVE_IN_SYNC_TO_STANDBY cutover gate, cleared on abort) suspends log rotation and turns a SYNC-write failure into an RS abort instead of the illegal cutover→SAF transition. init() seeds the flag on restart-in-cutover.
  • Consolidated state listeners — all LOCAL state subscriptions moved into subscribeToStateChanges(), each paired with its unsubscribe so close() leaks no watchers; handles cutover suspend, STANDBY/DEGRADED_STANDBY close, and SYNC ↔ SYNC_AND_FORWARD flips.
  • Constructor overloadinit() reuses the record the role gate already read.

Tests

9 new unit tests in ReplicationLogGroupTest plus 2 integration tests (testWriterClosesOnDemotionToStandby, testCutoverSuspendsAndResumesRotation).

Note

Behavioral change: a sync-path write to a non-active cluster now throws rather than silently creating a writer — the intended split-brain guard.

…ailover deadlock

A ReplicationLogGroup is a writer and must exist only where the local cluster
is active for the HA group. Previously it was created unconditionally (lazily
on first mutation, and eagerly at RS startup), which let non-active clusters
start writers and re-ship, and let the cutover writer deadlock standby failover.

Changes:
- Role-gated creation: ReplicationLogGroup.get(...) now returns Optional and
  creates a group only when the local role is active; a non-active role returns
  empty and is not cached, so the next call after promotion re-checks. Live
  write path (preBatchMutate) throws on empty (split-brain guard); replay path
  (preWALRestore) skips re-ship on empty.
- Skip eager init on non-active clusters: PhoenixRegionServerEndpoint only
  eager-inits where the local role is active; others init lazily on promotion.
- Failover deadlock fix: a new failoverPending flag (set on entry to the
  ACTIVE_IN_SYNC_TO_STANDBY cutover gate, cleared on abort) suspends log
  rotation so no new files land in the peer's shard directory, and turns a
  SYNC-write failure into an RS abort instead of the illegal cutover->SAF
  transition. init() seeds the flag on restart-in-cutover.
- Consolidated all LOCAL state listeners into subscribeToStateChanges(), each
  paired with its unsubscribe so close() leaks no watchers; handles cutover
  suspend, STANDBY/DEGRADED_STANDBY close, and SYNC/SYNC_AND_FORWARD flips.
- New constructor overload lets init() reuse the record the role gate read.

Tests: 9 new unit tests in ReplicationLogGroupTest and 2 integration tests
(testWriterClosesOnDemotionToStandby, testCutoverSuspendsAndResumesRotation).
@tkhurana
tkhurana requested a review from apurtell July 22, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant