Skip to content

Support isNull and isNotNull in PgVectorFilterExpressionConverter - #6904

Open
draviteja wants to merge 1 commit into
spring-projects:mainfrom
draviteja:pgvector-isnull-support
Open

Support isNull and isNotNull in PgVectorFilterExpressionConverter#6904
draviteja wants to merge 1 commit into
spring-projects:mainfrom
draviteja:pgvector-isnull-support

Conversation

@draviteja

Copy link
Copy Markdown

isNull() and isNotNull() from FilterExpressionBuilder don't work with
PgVector. They throw when the search runs.

The cause is in PgVectorFilterExpressionConverter.doExpression, which starts
with:

Assert.state(expression.right() != null, "expression should have a right operand");

ISNULL and ISNOTNULL have no right operand, so this always fails. The
awkward bit is that it only shows up at query time. The expression itself
builds fine, so any test that stops at building it will pass.

This change handles the two types before that assert. The JSONPath it
generates:

ISNULL     (!exists($."k") || $."k" == null)
ISNOTNULL  (exists($."k") && $."k" != null)

A key counts as null if it's missing, or if it's there but set to JSON null.
That's the same rule SimpleVectorStoreFilterExpressionEvaluator uses.

The Elasticsearch converter does it differently and only checks whether the
field exists. That gives a different answer for a key that exists holding
null, which can happen in JSONB. I went with the SimpleVectorStore
behaviour, but happy to switch if you'd rather match Elasticsearch.

The brackets are needed. JSONPath gives && higher precedence than ||, so
without them isNull(k) && year >= 2020 would also match a document that has
no k and year 2019.

One thing this doesn't fix: NOT(x IS NULL) still fails with "Unknown
expression type: ISNULL" from FilterHelper.negate. That's in core and
affects every store, not just PgVector. I have a fix for it if you want it as
a separate PR.

Tests: 7 unit tests in PgVectorFilterExpressionConverterTests and 4 cases in
PgVectorStoreIT. I couldn't run the IT locally (no Docker, no OpenAI key),
so I ran the generated SQL against a local Postgres to check the behaviour
instead.

Signed-off-by: Raviteja Daggupati <raviteja5255@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants