Skip to content

fix: netty mem leak during failure - #521

Merged
ShubhamChaturvedi7 merged 4 commits into
mainfrom
scchatur/MemLeakFix
Oct 5, 2026
Merged

ShubhamChaturvedi7 merged 4 commits into
mainfrom
scchatur/MemLeakFix

Conversation

@ShubhamChaturvedi7

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:
Drain the stream if a failure has occurred after fetching the payload.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Check any applicable:

  • Were any files moved? Moving files changes their URL, which breaks all hyperlinks to the files.

@ShubhamChaturvedi7
ShubhamChaturvedi7 requested a review from a team as a code owner September 15, 2026 22:10
throw new S3EncryptionClientException("Decryption materials cannot be null. " +
"This may be caused by a misconfigured custom CMM implementation or " +
"a suppressed exception from metadata decoding or CMM invocation due to a network failure.");
// Decryption setup failed in onResponse. AsyncStreamingResponseHandler#onHeaders already

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we are no longer throwing this exception, how are we notifying to custoer that the decryption materials are null and that they may have a misconfigured cmm? removing this exception can be a breaking change, no?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comments explain that - since this is async pipeline, a prev step failed and the Future has exception set. Throwing from here doesn't throw it for the callers, but rather for the pipeline. Which is exactly what was causing the memory leak.

TLDR: The caller still sees the exception - see the test case here

Comment on lines +158 to +163
// prepare() runs once per request attempt (AsyncResponseTransformer#prepare, enforced by
// BaseAsyncClientHandler via IdempotentAsyncResponseHandler keyed on EXECUTION_ATTEMPT).
// Clearing materials is what makes onStream below take the drain path when this attempt's
// onResponse fails, instead of decrypting this attempt's body with the previous attempt's
// materials. onResponse re-resolves materials unconditionally, so this costs no extra CMM call.
materials = null;

@lucasmcdonald3 lucasmcdonald3 Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// prepare() runs once per request attempt (AsyncResponseTransformer#prepare, enforced by
// BaseAsyncClientHandler via IdempotentAsyncResponseHandler keyed on EXECUTION_ATTEMPT).
// Clearing materials is what makes onStream below take the drain path when this attempt's
// onResponse fails, instead of decrypting this attempt's body with the previous attempt's
// materials. onResponse re-resolves materials unconditionally, so this costs no extra CMM call.
materials = null;
shouldDrain = true;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work. We don't need to drain on every call. We only need to drain if materials fail to fetch.

For every prepare call, the SDK also calls onResponse, which either overrides materials or throws. If it overrides, the flow executes normally. If it throws, materials is null then only we should drain.

Essentially we would need to atomically switch shouldDrain based on whether materials is null, which adds unnecessary complication.

A compromise would be to not set materials to null here (like how it was before this change). The flow would then either override it in onResponse or throw. That can cause memory leak in the scenario where the material fetch succeeded on the first attempt but then an exception occurred, SDK retried it but again an exception occurred (so now materials in not null from first attempt, and hence our drain won't fire).

@@ -153,6 +155,12 @@ private class DecryptingResponseTransformer<T> implements AsyncResponseTransform

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
var shouldDrain = false;

Comment on lines 182 to 183
public void onStream(SdkPublisher<ByteBuffer> ciphertextPublisher) {
if (materials == null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public void onStream(SdkPublisher<ByteBuffer> ciphertextPublisher) {
if (materials == null) {
public void onStream(SdkPublisher<ByteBuffer> ciphertextPublisher) {
if (shouldDrain) { /* drain, etc. */ }
if (materials == null) { /* throw existing exception */ }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can never throw from here. See my explanation above. Any error is handled in the exceptionOccurred block so we can only throw once which is done in the prepareMaterialsFromRequest block. Throwing again from here goes into the nettry pipeline.

@lucasmcdonald3 lucasmcdonald3 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR uses "is materials empty" to mean "did an attempt fail", but that 1) is more clearly expressed as a separate check of "an attempt failed, drain the body", 2) collapses into an existing case

@ShubhamChaturvedi7
ShubhamChaturvedi7 merged commit bad1bf8 into main Oct 5, 2026
35 of 36 checks passed
@ShubhamChaturvedi7
ShubhamChaturvedi7 deleted the scchatur/MemLeakFix branch October 5, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants