Bump the e2e workload node image past the DRA idle deadlock - #421
Open
nicknikolakakis wants to merge 1 commit into
Open
Bump the e2e workload node image past the DRA idle deadlock#421nicknikolakakis wants to merge 1 commit into
nicknikolakakis wants to merge 1 commit into
Conversation
The workload cluster runs kindest/node:v1.34.0, whose kubelet deadlocks once its gRPC connection to a DRA driver has been idle for 30 minutes. The next pod referencing a ResourceClaim then blocks in ContainerCreating with a single Scheduled event, no kubelet error, and NodePrepareResources never reaching the driver. Fixed upstream in v1.34.2 (k/k#133934). CI never sees it, since those clusters are minutes old. A cluster left up overnight hits it on the first ModelDeployment of the morning, and the symptom carries nothing that points at the kubelet. The pin exists for the DRA APIs, GA in 1.34, so any 1.34 image satisfies it. v1.34.8 is the oldest published image that also clears the deadlock, and it carries containerd 2.3.1, past the 2.2.0 that installation.md pins the getting-started cluster away from. Whether Modelplane runs on containerd 2.3.x is untested here; that docs pin is a separate constraint and is left alone, since its cluster runs no DRA driver. The reuse guard now rejects a v1.34.0 or v1.34.1 cluster rather than reusing it. Long-lived dev clusters are the ones that hit this, so bumping only the image would leave them deadlocking. Fixes modelplaneai#419. Signed-off-by: Nick Nikolakakis <nonicked@protonmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #419.
The e2e workload cluster runs
kindest/node:v1.34.0, whose kubelet deadlocks once its connection to a DRA driver has been idle for 30 minutes: the next pod referencing a ResourceClaim sits inContainerCreatingwith oneScheduledevent and nothing else, andNodePrepareResourcesnever reaches the driver. The goroutine dump and the reproduction are in the issue. Fixed upstream by kubernetes/kubernetes#133934, whichCHANGELOG-1.34.mdlists underv1.34.2.The pin's stated reason is the DRA APIs, GA in 1.34, so any 1.34 image satisfies it. The containerd constraint you may remember is a different pin: #315 changed only
docs/content/getting-started/installation.md, where the comment says kind v0.31+ ships containerd 2.2.0.e2e/run.shgot its pin from the e2e PR instead and never carried that constraint. containerd 2.2.0 was also one release: v1.34.3 has 2.2.0, v1.34.8 has 2.3.1, v1.34.11 has 2.3.4, each checked withdocker run --rm --entrypoint containerd <image> --version.So this takes v1.34.8, the oldest published image clearing both the deadlock and 2.2.0. v1.34.11 is there if you would rather track kind's latest. I have not run the full e2e locally, so whether Modelplane is happy on containerd 2.3.x is what the
test-e2elabel would settle. The docs pin is untouched: that cluster runs no DRA driver, so the deadlock doesn't reach it.The reuse guard rejects a v1.34.0 or v1.34.1 cluster now instead of reusing it, since long-lived dev clusters are the population that hits this and bumping only the image would leave them deadlocking.
Touches
e2e/run.shlike #420 does, on different lines.