CASSANDRA-20476 Cluster is unable to recover after shutdown if IPs change - #4979
CASSANDRA-20476 Cluster is unable to recover after shutdown if IPs change#4979beobal wants to merge 28 commits into
Conversation
…ed before allowing startup to proceed
…og processing is suspended
If two nodes restart with new broadcast addresses concurrently and one is a CMS member, the non-member may send a TCM_COMMIT_REQ containing its STARTUP transform to the CMS member's old address. If this is no longer reachable, the sender should time out quickly so it can resend to another CMS member, or try to discover the new CMS address(es).
…CEP-21, pre V9 version
…tructing a CMSLookup
| public CandidateIterator(Collection<InetAddressAndPort> initialContacts, boolean checkLive) | ||
| { | ||
| this.candidates = new ConcurrentLinkedDeque<>(initialContacts); | ||
| this.elements = new HashSet<>(initialContacts); |
There was a problem hiding this comment.
I think this needs to be a concurrent set
| // recalculate CMS placement using endpoint mappings from lookup | ||
| cmsDataPlacement = calculateCMSPlacement(placements, cmsMembership, proposedLookup); | ||
|
|
||
| // We shouldn't need to null out the other lazily initialized CMS fields when we refresh the CMSLookup as the |
There was a problem hiding this comment.
should we assert that they are null?
| logger.debug("Received {} responses. {} required.", responses.size(), requiredResponses); | ||
| if (responses.size() >= requiredResponses) | ||
| { | ||
| isDone = true; |
There was a problem hiding this comment.
markDone() instead to get the logging?
| this(ClusterMetadata.current().metadataIdentifier, MessagingService::instance); | ||
| } | ||
|
|
||
| public SurveyRequestHandler(int metadataId, Supplier<MessageDelivery> messaging) |
| // During first initialization, placements are hardcoded to the local address after the PRE_INITIALIZE_CMS is | ||
| // committed so that the subsequent INITIALIZE_CMS can be. | ||
| // | ||
| // * Some changes to the addresses of the CMS members have been changed and whilst this is known (i.e. has been |
There was a problem hiding this comment.
nit, Some addresses of the CMS members have been changed...
| @Override | ||
| public InetAddressAndPort endpoint(NodeId id) | ||
| { | ||
| if (overrides.containsKey(id)) |
There was a problem hiding this comment.
nit, avoid looking up the key twice:
Pair<InetAddressAndPort, InetAddressAndPort> override = overrides.get(id);
if (override != null)
return override.right;
return lookup.endpoint(id);
...
|
|
||
| Result.Success success = result.success(); | ||
| Directory directory = directorySupplier.get(); | ||
| EndpointLookup lookup = lookupSupplier.get(); |
There was a problem hiding this comment.
is there a possibility that we get "mismatching" directory/lookup here since we call suppliers twice? Maybe the supplier should be something like Supplier<Pair<Directory, EndpointLookup>> to make sure it's backed by the same ClusterMetadata?
| { | ||
| logger.info("Waiting for pending CMS address changes to complete {}", replayed.cmsLookup); | ||
| TimeUnit.MILLISECONDS.sleep(1000); | ||
| replayed = ClusterMetadata.current(); |
There was a problem hiding this comment.
Is there a chance we miss a Startup and get stuck forever here? Should we do an explicit log fetch periodically? Maybe not every second, but once a minute or so, just to make sure?
| } | ||
|
|
||
| logger.info("Proposed endpoint overrides: {}", nextOverrides); | ||
| State state = nextOverrides.isEmpty() ? State.RETIRED : State.ACTIVE; |
There was a problem hiding this comment.
Could/should we retire the lookup once we have a quorum committed? This would allow us to recover if one of the CMS nodes is gone forever
2 participants
Thanks for sending a pull request! Here are some tips if you're new here:
Commit messages should follow the following format:
The Cassandra Jira