Skip to content

fix(rig-postgres): filtered and thresholded searches generate valid SQL - #2377

Open
jackthepunished wants to merge 5 commits into
0xPlaygrounds:mainfrom
jackthepunished:fix/postgres-search-sql
Open

fix(rig-postgres): filtered and thresholded searches generate valid SQL#2377
jackthepunished wants to merge 5 commits into
0xPlaygrounds:mainfrom
jackthepunished:fix/postgres-search-sql

Conversation

@jackthepunished

@jackthepunished jackthepunished commented Aug 18, 2026

Copy link
Copy Markdown

Fixes #2376

rig-postgres produced SQL that Postgres rejects for any VectorSearchRequest that carries a member filter, a single-condition filter, or a threshold; and the threshold compared in the wrong direction. Found by auditing the vector-store crates against rig-sqlite, which gets all of this right.

# bug severity
1 PgSearchFilter::member renders id is in ($3, $4) — not a Postgres operator high (hard error)
2 threshold rendered as distance > $N in the inner WHERE, where distance is only a select-list alias → column "distance" does not exist high (hard error)
3 > on a pgvector distance while threshold is documented as a minimum similarity → would keep the worst rows high
4 "WHERE" + condition with no separator → WHEREprice >= $3 for any single-condition filter high (hard error)

Generated SQL on origin/main, as printed by the new tests when run there:

WHEREprice >= $3
WHEREdistance > $3
WHERE(distance > $3) AND ((kind = $4) AND (id is in ($5, $6)))

and on this branch:

WHERE (price >= $3)
WHERE (1 - (embedding <=> $1) >= $3)
WHERE ((kind = $3) AND (id IN ($4, $5))) AND (-(embedding <-> $1) >= $6)

Fix

  • member renders IN.
  • The threshold is applied as a minimum similarity on a per-operator expression that repeats the distance operator rather than naming the alias: 1 - (embedding <=> $1) for cosine and jaccard, the negated distance for inner product / L2 / L1 / hamming (pgvector's <#> is already the negative inner product). This is the approach rig-sqlite takes. It is composed after the $ renumbering pass so the $1 inside the expression survives, and its bind value is appended after the filter's.
  • WHERE is separated from its first condition.

Non-breaking: returned scores are still raw pgvector distances in ascending order; only requests that previously errored change behavior.

Tests

  • 4 unit tests fail on origin/main and pass here (threshold_is_a_minimum_similarity_on_the_repeated_expression, threshold_placeholder_follows_the_filter_placeholders, single_condition_filter_is_separated_from_where, plus the existing every_parameterised_operator_uses_dollar_placeholders, whose assertion pinned is in). They call the private search_query on a store built with a lazy pool and rig_core::test_utils::MockEmbeddingModel, so no database is needed. similarity_expression_inverts_each_distance_operator covers all six operators.

Verified locally: cargo fmt --all -- --check, cargo clippy -p rig-postgres --all-targets --all-features, cargo test -p rig-postgres.

The threshold predicate was rendered as `distance > $N` inside the inner
select, where `distance` is only a select-list alias, so Postgres rejected
every thresholded search. It also compared a pgvector distance with `>`,
which would have kept the least similar rows. Compare a per-operator
similarity expression against the threshold instead, and separate the
`WHERE` keyword from the first condition, which a single-condition filter
used to abut.
Copilot AI lite review requested due to automatic review settings August 18, 2026 06:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(rig-postgres): member filters, single-condition filters and thresholds generate invalid SQL

2 participants