Skip to content

Prime reranking auth before parallel batches - #2531

Closed
erichare wants to merge 1 commit into
mainfrom
fix/prime-reranking-authorizer-cache
Closed

Prime reranking auth before parallel batches#2531
erichare wants to merge 1 commit into
mainfrom
fix/prime-reranking-authorizer-cache

Conversation

@erichare

@erichare erichare commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What this PR does:

Runs reranking batch 0 to completion before constructing and subscribing to the remaining batches. A successful first batch primes the upstream API Gateway authorizer cache; the tail batches then retain their existing concurrent, fail-fast execution.

This reduces the authorization cache-miss fan-out of one multi-batch findAndRerank request from as many as one miss per batch to one miss per request. Concurrent top-level requests can still race each other, so this is intentionally paired with the Lambda timeout and resilience changes in riptano/cloud-helm-charts#1095.

The tradeoff is one reranking-batch round trip on the critical path before the concurrent tail starts.

Which issue(s) this PR fixes:

Companion to riptano/cloud-helm-charts#1095.

Validation:

  • JAVA_HOME=/Library/Java/JavaVirtualMachines/graalvm-21.jdk/Contents/Home ./mvnw -B -ntp -Dstyle.color=always clean test (3,424 tests)
  • Focused reranking provider and gateway tests (10 tests)
  • Added coverage for first-batch sequencing, concurrent tail execution, first-batch failure, single-batch requests, empty passages, result indexes, and model usage aggregation

Checklist

  • Changes manually tested in a deployed environment
  • Automated Tests added/updated
  • Documentation not required (no API or configuration change)
  • CLA Signed: DataStax CLA

@erichare
erichare requested a review from amorton August 6, 2026 14:31
@erichare
erichare marked this pull request as ready for review August 6, 2026 14:31
@erichare
erichare requested a review from a team as a code owner August 6, 2026 14:31
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Unit Test Coverage Report

Overall Project 53% 🍏
Files changed 100% 🍏

File Coverage
RerankingProvider.java 59.34% 🍏

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Integration Test Coverage Report (dse69-it)

Overall Project 71.39% -0.08% 🍏
Files changed 0%

File Coverage
RerankingProvider.java 12.09% -18.9%

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Integration Test Coverage Report (hcd-it)

Overall Project 72.71% -0.08% 🍏
Files changed 0%

File Coverage
RerankingProvider.java 12.09% -18.9%

@amorton amorton left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-1 blocking until we know more , at the surface level this does not feel like the best idea, we should fix the upstream problem

@erichare erichare closed this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants