fix(es): keep exact metadata lookup below max_clause_count for 300+ tables - #620
Open
Coder-xiaosuo wants to merge 1 commit into
Open
Conversation
…ables
findInElasticsearch serialized the metadata filter into a Lucene
query_string. For an IN over 300+ table names only the first value kept
the field prefix; the rest became bare words re-scattered across all
searchable fields by index.query.default_field, exceeding
indices.query.bool.max_clause_count (1024) and failing every shard with
TooManyClauses ("all shards failed").
Replace the query_string path with a native bool/term/terms DSL query:
string equality now matches on metadata.<key>.keyword and each multi-value
IN collapses into a single terms clause, so clause count no longer grows
with the number of table names. Also fixes the sibling getTableDocuments /
getColumnDocumentsByTableName callers sharing getDocumentsOnlyByFilter.
Closes spring-ai-alibaba#608 (issue 5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #608 (Question 5) — hasTableDocuments fails with search_phase_execution_exception: all shards failed when a datasource has 300+ tables.
What this PR does / why we need it
MetadataDocumentRetriever.findInElasticsearch serialized the metadata filter into a Lucene query_string . For an IN over 300+ table names only the first value kept the metadata.name: field prefix; the rest became bare words re-scattered across all searchable fields by index.query.default_field , pushing the query past indices.query.bool.max_clause_count (default 1024). Every shard then failed with TooManyClauses , surfaced as search_phase_execution_exception: all shards failed .
The fix stops using query_string for exact metadata retrieval and builds a native bool / term / terms DSL query instead:
Sibling callers sharing the same getDocumentsOnlyByFilter path ( SchemaServiceImpl.getTableDocuments / getColumnDocumentsByTableName ) are fixed as well.
How to verify
New unit test MetadataDocumentRetrieverQueryTest (no live ES): asserts that 320 table names translate to 2 term + 1 terms ( metadata.name.keyword ) leaves and that no query_string is emitted; guards for column lookups, NIN , and EQ-only filters.
Local checks passed: spring-javaformat validate OK, Checkstyle 0 violations, all 5 new tests green plus related vector-store tests.
PR Type
Bug fix
Feature
Refactoring
Documentation