docs: Document ERS explicit split-brain recovery and partial relay-log tolerance - #2128
Draft
promptless[bot] wants to merge 2 commits into
Draft
docs: Document ERS explicit split-brain recovery and partial relay-log tolerance#2128promptless[bot] wants to merge 2 commits into
promptless[bot] wants to merge 2 commits into
Conversation
Add documentation for v25 EmergencyReparentShard changes: - Document new --allow-split-brain-promotion flag - Add split-brain detection section explaining fail-fast behavior - Add partial relay-log-apply tolerance section for GTID-based shards - Document three new metrics for ERS observability
![promptless[bot]](https://avatars.githubusercontent.com/in/979718?s=60&v=4){kind=small}
| - On the primary-elect tablet, insert a row in the `reparent_journal` table and then updates the `PrimaryAlias` property of the global shard object. | ||
| - In parallel on each replica, excluding the old primary, set the new primary as the replication source and wait for the inserted row to replicate to the replica tablets. | ||
|
|
||
| #### Split-brain detection |
Contributor
Author
There was a problem hiding this comment.
Added split-brain detection and --allow-split-brain-promotion flag documentation based on PR #18707 which introduces upfront split-brain detection in filterAndCheckUniform() and the operator escape hatch flag.
Source: vitessio/vitess#18707
| - Consider using `--ignore-replicas` to exclude tablets on the side you want to discard from the candidate pool. | ||
|
|
||
| #### Partial relay-log-apply tolerance | ||
|
|
Contributor
Author
There was a problem hiding this comment.
Added partial relay-log-apply tolerance documentation based on PR #18707 which implements the waitForAllRelayLogsToApply() short-circuit behavior for GTID-based shards.
Source: vitessio/vitess#18707
| | `planned_reparent_counts` | Number of times PlannedReparentShard has been run. Available dimensions are keyspace, shard and the result of the operation. | | ||
| | `emergency_reparent_counts` | Number of times EmergencyReparentShard has been run. Available dimensions are keyspace, shard and the result of the operation. | | ||
| | `reparent_shard_operation_timings` | Timings of reparent shard operations indexed by the type of operation. | | ||
| | `EmergencyReparentFilteredCandidates` | Number of candidates excluded from the relay-log wait during ERS because their `Combined` position was behind the leading group. Keyed by keyspace and shard. | |
Contributor
Author
There was a problem hiding this comment.
Added three new metrics (EmergencyReparentFilteredCandidates, EmergencyReparentRelayLogFailedCandidates, EmergencyReparentSplitBrainOverrides) based on PR #18707 where they are defined in go/vt/vtctl/reparentutil/emergency_reparenter.go.
Source: vitessio/vitess#18707
5 tasks
✅ Deploy Preview for vitess ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
…tion + --new-primary) Fold in vitessio/vitess PR #20780: fail-closed split-brain default, the operator-only --allow-split-brain-promotion override (now requiring --new-primary), the upfront-leading-candidate requirement, data-loss/ re-clone warning, preserved safety guards, VTOrc-never note, and align the EmergencyReparentSplitBrainOverrides metric description.
promptless
Bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Open this suggestion in Promptless to view citations and reasoning process
Documents the v25
EmergencyReparentShard(ERS) split-brain changes:FAILED_PRECONDITIONerror naming every diverging leader.--allow-split-brain-promotionoverride, which now requires--new-primary, requires the named tablet to be one of the leading candidates, and discards the divergent transactions on the other leaders (those tablets may need to be re-cloned). The override still enforces ERS's other safety guards, and VTOrc never enables it automatically.EmergencyReparentSplitBrainOverrides.Trigger Events
Tip: Sort by Shortest Review in the Dashboard to find quick wins ⚡