GH-1143 Add IGNORE QueueNotFoundStrategy option for SQS - #1640
Conversation
maciejwalkowiak
left a comment
There was a problem hiding this comment.
Looks good to me but @tomazfernandes please take a look
|
Thanks @maciejwalkowiak. Let's not block release on this - I'll review it for 4.2.0, since 4.1.0 is already overdue and packed with SQS changes. |
|
@tomazfernandes when do you expect to merge this one? |
|
@BK202503 4.1.0 should be out this week, I'll review and merge this PR after that for 4.2.0. We're working on releasing more often, so 4.2.0 should follow in a timely manner. Thanks for the PR. |
tomazfernandes
left a comment
There was a problem hiding this comment.
Thanks for the PR @BK202503.
The main issue I see is that as is the initialization fails nevertheless, since other components such as the acknowledgement processor are not initialized, which causes the code to throw IllegalArgumentException: acknowledgementExecutor not set.
One other issue is that, even if the source successfully initializes, the polling method would keep being called and the thread would keep spinning.
Perhaps we could instead handle this scenario at the AbstractPollingMessageSource level, and prevent the source from starting at all (isRunning = false).
Let me know your thoughts.
|
@tomazfernandes Thank you for your comment and moving the guard to |
Address review from @tomazfernandes on awspring#1640. Handling the missing queue inside AbstractSqsMessageSource.doStart alone left the container in a half-started state: the acknowledgement processor still initialised and threw IllegalArgumentException: acknowledgementExecutor not set, and the polling thread kept spinning on a source that would only ever return empty batches. Move the guard up to AbstractPollingMessageSource.start(). doStart() now propagates QueueNotFoundException; the base start() catches it, logs a warning, sets running=false, and returns before wiring the ack processor or starting the polling thread. The source-level skipped flag and the short-circuit in doPollForMessages go away with it. Signed-off-by: BK202503 <199436087+BK202503@users.noreply.github.com>
tomazfernandes
left a comment
There was a problem hiding this comment.
Thanks for the update @BK202503, the behavior looks good now.
I've left a suggestion for a path for us to simplify the design further, let me know your thoughts.
| try { | ||
| doStart(); | ||
| } | ||
| catch (QueueNotFoundException qnfe) { |
There was a problem hiding this comment.
We could simplify the design further by inspecting the QueueAttributesResolvingException directly here (wrapped in a CompletionException), and adding a isQueueIgnored() method to it that we'd set in the resolver.
This way we don't need to introduce a new exception type, the catch in the AbstractSqsMessageSource, and we also don't change semantics for the SqsTemplate between IGNORE and FAIL.
What do you think?
There was a problem hiding this comment.
Agreed. Folding the signal into QueueAttributesResolvingException via isQueueIgnored() (set by the resolver on the IGNORE branch) drops the new exception type and keeps SqsTemplate semantics identical between IGNORE and FAIL. Pushed as 7e8a3c5.
Restore the Spring Cloud AWS 2.x behavior where a missing SQS queue at listener startup did not fail the application context. In 3.x today the two strategies are FAIL (throws on missing queue, blocks startup) and CREATE (calls CreateQueue on missing, requires sqs:CreateQueue IAM and can interfere with externally-managed queue configuration); neither fits the common case where a queue may legitimately be absent in some deployments — for example an optional feature queue, or a multi-region rollout that hasn't created the queue yet. Add a third strategy, IGNORE, that logs a warning and skips starting the listener container's message source for that queue. Subsequent polls return empty so the application keeps running cleanly. Wiring: - QueueNotFoundStrategy gets a new IGNORE constant with Javadoc that cross-references the 2.x default behavior. - QueueAttributesResolver.handleException, on QueueDoesNotExistException + IGNORE, logs a warning and fails the future with a new QueueNotFoundException (subtype of QueueAttributesResolvingException so existing catch sites keep working; the dedicated subtype is what the source uses to recognize the IGNORE signal). The CREATE path is unchanged. - QueueAttributesResolver.wrapException passes QueueNotFoundException through unwrapped instead of re-wrapping in QueueAttributesResolvingException, so the source sees the typed signal rather than a generic resolver failure. - AbstractSqsMessageSource.doStart catches CompletionException with QueueNotFoundException as its cause, flips a `skipped` flag, logs a warning, and returns without setting queueAttributes/queueUrl or configuring the conversion context. - AbstractSqsMessageSource.doPollForMessages short-circuits to an empty future when `skipped`, so the polling loop is a no-op for the lifetime of the source. Tests: - QueueAttributesResolverIntegrationTests#shouldIgnoreQueueWhenStrategyIsIgnore mirrors the existing #shouldNotCreateQueue pattern against LocalStack, asserting that resolving a non-existent queue with IGNORE surfaces a QueueNotFoundException whose cause is QueueDoesNotExistException (rather than the generic QueueAttributesResolvingException that FAIL produces). - The full QueueAttributesResolverIntegrationTests class (7 tests) passes locally on LocalStack. Docs: - sqs.adoc table entry for queueNotFoundStrategy gets a sentence on the IGNORE behavior next to the existing FAIL note. The property binding for `spring.cloud.aws.sqs.queue-not-found-strategy` picks up IGNORE automatically because SqsProperties already exposes the enum directly. Closes awspring#1143. Signed-off-by: BK202503 <199436087+BK202503@users.noreply.github.com>
Address review from @tomazfernandes on awspring#1640. Handling the missing queue inside AbstractSqsMessageSource.doStart alone left the container in a half-started state: the acknowledgement processor still initialised and threw IllegalArgumentException: acknowledgementExecutor not set, and the polling thread kept spinning on a source that would only ever return empty batches. Move the guard up to AbstractPollingMessageSource.start(). doStart() now propagates QueueNotFoundException; the base start() catches it, logs a warning, sets running=false, and returns before wiring the ack processor or starting the polling thread. The source-level skipped flag and the short-circuit in doPollForMessages go away with it. Signed-off-by: BK202503 <199436087+BK202503@users.noreply.github.com>
Fold the "queue was ignored" signal into a boolean on QueueAttributesResolvingException instead of a dedicated QueueNotFoundException subtype, per review feedback. QueueAttributesResolver sets the flag on the IGNORE branch; AbstractPollingMessageSource inspects the wrapped QueueAttributesResolvingException at start() and skips the source when the flag is set. Restores original AbstractSqsMessageSource.doStart() and keeps SqsTemplate exception semantics identical between IGNORE and FAIL.
43877ff to
7e8a3c5
Compare
There was a problem hiding this comment.
Thanks for the quick turnaround @BK202503.
Left one remaining design comment in the diff.
A few last points:
- Please update the docs and the
IGNOREjavadocs to reflect the new behavior. - Also, please add
@since 4.2on the newQueueAttributesResolvingExceptionconstructor andisQueueIgnored().
And it would be great if we could add integration tests for these scenarios:
- Container with
IGNOREand a missing queue starts successfully, never polls, and stops normally - Container with
IGNOREand a missing queue starts polling normally after the queue is created and the container is restarted - Container with multiple queues where one is missing consumes normally from the existing queue while the missing one is skipped
FAILwith a missing queue still throws on startup, withisQueueIgnored()returning false
| if (QueueNotFoundStrategy.CREATE.equals(this.queueNotFoundStrategy)) { | ||
| return createQueue(); | ||
| } | ||
| if (QueueNotFoundStrategy.IGNORE.equals(this.queueNotFoundStrategy)) { |
There was a problem hiding this comment.
Let's move the IGNORE logic to wrapException instead, this way we only create the QueueAttributesResolvingException in one place and avoid having the pass through there.
I also think we don't need to log here since we'll be logging in the source. The exception will carry the information in this layer.
Closes #1143.
What
Adds a third option to
QueueNotFoundStrategy—IGNORE— that logs a warning and skips starting the listener container's message source when its queue does not exist, so application startup is not blocked. Subsequent polls for that source return empty.This restores the default behavior of Spring Cloud AWS 2.x (
spring-cloud-starter-aws-messaging) for users that rely on it during the 2.x → 3.x migration, as flagged by several commenters on #1143.Why
Today the two strategies are:
FAIL— throws on a missing queue and blocks application startup. Wrong for deployments where the queue is legitimately optional.CREATE— callsCreateQueueon a missing queue. Requiressqs:CreateQueueIAM and can interfere with externally-managed queue configuration / cross-team queue ownership.Neither fits the case where a queue may legitimately be absent in some deployments (optional feature queues, multi-region rollout where the queue hasn't been created yet, etc.) and the listener should simply skip rather than crash the app.
@tomazfernandes wrote on the issue: "I don't have a solution in mind, but happy to take a look at a PR with a proposal." — this is that proposal.
How
The main concern from the issue was "graciously handle the error thrown by the
QueueAttributesResolverand interrupting container startup". The patch routes that through a typed signal rather than reusing the generic resolver exception:QueueNotFoundStrategygets a newIGNOREconstant with Javadoc that cross-references the 2.x default behavior.QueueAttributesResolver.handleException, onQueueDoesNotExistException+IGNORE, logs a warning and fails the future with a newQueueNotFoundException(subtype ofQueueAttributesResolvingException, so existing catch sites keep working; the dedicated subtype is what lets the source distinguish "ignore this queue" from a generic resolver failure). TheCREATEpath is unchanged.QueueAttributesResolver.wrapExceptionpassesQueueNotFoundExceptionthrough unwrapped instead of re-wrapping inQueueAttributesResolvingException, so the source sees the typed signal.AbstractSqsMessageSource.doStartcatchesCompletionExceptionwithQueueNotFoundExceptionas its cause, flips askippedflag, logs a warning, and returns without settingqueueAttributes/queueUrlor configuring the conversion context.AbstractSqsMessageSource.doPollForMessagesshort-circuits to an empty future whenskipped, so the polling loop is a no-op for the lifetime of the source.The
@RetryableTopic-style orFAILpaths are untouched — only the newIGNOREconstant exercises the new branches. The property binding forspring.cloud.aws.sqs.queue-not-found-strategy=IGNOREpicks up automatically becauseSqsPropertiesalready exposes the enum directly.Test plan
QueueAttributesResolverIntegrationTests#shouldIgnoreQueueWhenStrategyIsIgnoremirrors the existing#shouldNotCreateQueuepattern against LocalStack, asserting that resolving a non-existent queue withIGNOREsurfaces aQueueNotFoundExceptionwhose cause isQueueDoesNotExistException(rather than the genericQueueAttributesResolvingExceptionthatFAILproduces).QueueAttributesResolverIntegrationTestsclass (7 tests) passes locally on LocalStack — the existingFAIL/CREATEpaths still produce their original exception types becausewrapExceptiononly passes throughQueueNotFoundExceptionspecifically../mvnw -B spotless:applyclean.Docs
docs/src/main/asciidoc/sqs.adoc— the table entry forqueueNotFoundStrategygets a sentence on theIGNOREbehavior next to the existingFAILnote.AI agent disclosure
I used an AI coding agent (Claude Code) to navigate the existing strategy / resolver / source code paths and draft this patch. I have read every line, verified the dispatch through the
CompletionExceptionunwrap indoStart, traced the property-binding to confirm noSqsPropertieschange is needed, and confirmed the new IGNORE test passes against LocalStack alongside the existing 6 resolver tests. I will be the one responding to review.