Add fuzzy search support to `contains()` and `bif:contains` by danielhmills · Pull Request #1465 · openlink/virtuoso-opensource

danielhmills · 2026-08-17T20:50:44Z

Summary

Adds fuzzy string matching support to Virtuoso text search.

Included

Levenshtein, Jaro-Winkler, and n-gram cosine algorithms.
Fuzzy options for SQL contains() and SPARQL bif:contains.
FUZZY_THRESHOLD, FUZZY_N, and FUZZY_PREFIX options.
SIMILARITY and Levenshtein DISTANCE output columns.
Fuzzy score integration with full-text relevance scoring.
Whole-literal similarity and distance calculation for RDF/SPARQL results.
Validation for incompatible fuzzy options.
Regression coverage for quoted phrases, thresholds, score-independent metrics, and distances.

Review fixes applied

Replaced libc allocators with dk_alloc/dk_free in fuzzy_algorithms.c.
Used ITC_OWNS_PARAM() for fuzzy scan search parameters in text.c.
Added sst_is_fuzzy guard to SST_COMMON for safe word_stream_t typecast.
Split unrelated changes (VSP compile mutex → Serialize concurrent first VSP compilation #1466, JSON-LD context fix → Stabilize JSON-LD context loading #1467) into separate PRs.
Removed plan.md design doc from the branch.

Verification

Focused fuzzy-search suite passes: 79 passed, 0 failed.
Rebuilt libwi.la and virtuoso-t successfully (0 errors).

Generated with Devin

imitko · 2026-08-17T21:18:25Z

The VSP is a stored procedure, the compile and re-compile critical sections should handle that.
Should not be part of PR

Split out to a separate PR. The VSP compile mutex change has been removed from this branch and is now in #1466 (branch fix/vsp-compile-mutex).

As you noted, the VSP is a stored procedure and the compile/re-compile critical sections should handle concurrency. The HTTP-layer mutex was a stopgap to serialize first-compile across workers. The new PR flags this and invites your guidance on whether the fix belongs in the procedure compile/re-compile critical sections instead.

imitko · 2026-08-17T21:29:23Z

Should use ITC_OWNS_PARAM() macro to bind the allocated parameter box to itc, then it will be destroyed upon its_free.
Have a member in SST_COMMON sst_is_fuzzy or similar, so can test it set before typecast of sst to word_stream_t.

Both addressed:

ITC_OWNS_PARAM() — The search parameter boxes are now bound to the itc via ITC_OWNS_PARAM() so they are freed automatically by itc_free. The manual dk_free_box(itc->itc_search_params[...]) calls in both the success and ITC_FAILED paths have been removed.

sst_is_fuzzy member — Added int sst_is_fuzzy to the SST_COMMON macro so it exists in both word_stream_s and search_stream_s. It is set to 1 in wst_from_fuzzy when creating fuzzy word streams, and defaults to 0 via dk_alloc_box_zero in NEW_SST for all other streams. The typecast to word_stream_t * in sst_scores is now guarded by sst->sst_is_fuzzy before accessing wst_fuzzy_similarity / wst_fuzzy_distance.

imitko · 2026-08-17T21:46:32Z

      sst->sst_raw_score = sst->sst_all_ranges_fill;
      sst_freq_factor (sst);
+      /* Fuzzy score fusion: scale relevance by similarity ratio */
+      if (((word_stream_t *) sst)->wst_fuzzy_similarity > 0.0


have a member in SST_COMMON to indicate word_stream_s.wst_fuzzy_similarity can be accessed/used, so typecast is possible.

Resolved. Added sst_is_fuzzy to SST_COMMON — see reply on the comment above. The typecast in sst_scores is now guarded by sst->sst_is_fuzzy so a plain search_stream_t (OR/AND node) or non-fuzzy word stream is never mistreated as a fuzzy word_stream_t.

imitko · 2026-08-17T21:50:29Z

What is relation to fuzzy search, if patch address separate issue should be separate PR?

Split out to a separate PR. The JSON-LD context stabilization change has been removed from this branch and is now in #1467 (branch fix/jsonld-context-retry). It was unrelated to fuzzy search — it fixes transient cache/HTTP failures in JSON-LD context resolution.

imitko · 2026-08-17T21:51:34Z

Have to be separate patch.

Kept in this PR. The xpf.c change only initializes the new tctx_fuzzy_* fields to their default values (FUZZY_NONE, FUZZY_DEFAULT_THRESHOLD, FUZZY_DEFAULT_N, prefix=2) in xp_text_contains(). This is required so the XPath text-contains path has zeroed fuzzy fields rather than uninitialized stack values. It is part of the fuzzy feature, not an unrelated fix.

Please confirm if you still want it split given this rationale — if so, I can move it to a follow-up, but the XPath contains path would reference uninitialized fields until that follow-up lands.

imitko · 2026-08-17T21:55:01Z

Since code is in Wi and built in the core, the memory alloc/free should be replaced with dk_alloc/free

Done. All 38 occurrences of malloc/calloc/realloc/free have been replaced with dk_alloc/dk_free. Since dk_realloc does not exist in the Virtuoso allocator, the two realloc calls were replaced with dk_alloc + memcpy + dk_free. The #include <stdlib.h> was replaced with #include "Dk.h".

The dk_free(p, 0) is not correct, either give a size if known or dk_free(p, -1) // NO_SIZE

Fixed in 73a1440.

Every dk_free(p, 0) in fuzzy_algorithms.c now passes the actual allocation size, taken from the same locals used in the matching dk_alloc (e.g. sizeof (int) * (len1 + 1) for the Levenshtein rows, len1/len2 for the Jaro match arrays, map->n + 1 / sizeof (uint32_t) * map->n for the n-gram entries, sizeof (*_ngram_entry_t) * map->capacity for the entry arrays). The grow-path frees use map->capacity before it is updated, which is the size of the block being released.

I went with the concrete size rather than dk_free(p, -1) because the size is known at every free site, and passing it keeps the CACHE_MALLOC fast-path enabled (the sz != NO_SIZE branch). NO_SIZE is private to Dkalloc.c and not exposed in a header, so it can't be referenced by name here anyway.

Verified: libwi.la and virtuoso-t rebuild clean (0 errors), and the focused fuzzy suite still reports 0 failed, 79 passed.

TallTed · 2026-08-17T22:28:54Z

+
+#### Implementation notes:
+
+- **Levenshtein**: Single-row DP array (O(min(m,n)) space). Early exit if distance exceeds `max(m,n) * (1 - threshold)` — skip full computation when candidate can't possibly meet threshold.


Suggested change

- **Levenshtein**: Single-row DP array (O(min(m,n)) space). Early exit if distance exceeds `max(m,n) * (1 - threshold)` — skip full computation when candidate can't possibly meet threshold.

- **Levenshtein**: Single-row DP array (`O(min(m,n))` space). Early exit if distance exceeds `max(m,n) * (1 - threshold)` — skip full computation when candidate can't possibly meet threshold.

Resolved by removing plan.md from the PR entirely. The design document has been saved outside the repo for reference; it should not be committed to the source tree.

TallTed · 2026-08-17T22:29:59Z

+- **Jaro-Winkler**: Standard Jaro with Winkler prefix bonus (p=0.1, max prefix=4). Matching window = `max(len(s1), len(s2)) / 2 - 1`.
+- **N-gram cosine**: Build sparse frequency maps (hash table or sorted array) of character n-grams with boundary padding (`$` prefix/suffix). Cosine = dot product / (magnitude1 × magnitude2). Early exit: if n-gram set intersection count < `threshold² × |s1_ngrams|`, return 0.0 immediately.


Suggested change

- **Jaro-Winkler**: Standard Jaro with Winkler prefix bonus (p=0.1, max prefix=4). Matching window = `max(len(s1), len(s2)) / 2 - 1`.

- **N-gram cosine**: Build sparse frequency maps (hash table or sorted array) of character n-grams with boundary padding (`$` prefix/suffix). Cosine = dot product / (magnitude1 × magnitude2). Early exit: if n-gram set intersection count < `threshold² × |s1_ngrams|`, return 0.0 immediately.

- **Jaro-Winkler**: Standard Jaro with Winkler prefix bonus (`p=0.1, max prefix=4`). Matching window = `max(len(s1), len(s2)) / 2 - 1`.

- **N-gram cosine**: Build sparse frequency maps (hash table or sorted array) of character n-grams with boundary padding (`$` prefix/suffix). `Cosine = dot product / (magnitude1 × magnitude2)`. Early exit: if n-gram set intersection count < `threshold² × |s1_ngrams|`, return `0.0` immediately.

Resolved by removing plan.md from the PR entirely. The design document has been saved outside the repo for reference; it should not be committed to the source tree.

…gram cosine) Pure C implementations with no Virtuoso dependencies, callable from both BIF wrappers and the text search engine. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Register levenshtein, levenshtein_similarity, jaro_winkler, jaro_winkler_similarity, ngram_cosine, and ngram_cosine_n as built-in functions. Automatically available in SPARQL via the bif: prefix (e.g. bif:levenshtein, bif:jaro_winkler). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Parse 'fuzzy', 'fuzzy_threshold', and 'fuzzy_n' options in both sqlo.c and sqldf.c contains() option parsers. Add ot_text_fuzzy_* fields to out_text_t and generate state slots in sqlgen.c. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Add wst_from_fuzzy() which scans a prefix range on the word index, computes similarity per candidate word, and creates word streams for matches above threshold. Add fuzzy fields to text_node_t, sst_tctx_t, and word_stream_t. Fuse similarity ratio into scoring in sst_scores(). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Add FUZZY, FUZZY_THRESHOLD, and FUZZY_N as recognized SPARQL triple-pattern options for bif:contains, using the existing OPTION (...) syntax. This allows fuzzy search to be used from SPARQL via: ?name bif:contains "'Johnson'" OPTION (FUZZY 'jaro_winkler', FUZZY_THRESHOLD 0.6) . Changes: - sparql_p.y: Add FUZZY_L, FUZZY_THRESHOLD_L, FUZZY_N_L token declarations and grammar rules in spar_triple_freetext_option - sparqlwords.gperf: Add lexer entries for the three new keywords - sparql_tree.c: Add token names and has_ft classification - sparqld.c: Add token names for error messages - sparql2sqltext.c: Add SQL printing for the three options in ssg_print_ft_predicate - sparql2sql.c: Add expression traversal for the three options in gp_trav_cu_in_options - sqlpfn.c: Add 'fuzzy', 'fuzzy_threshold', 'fuzzy_n' to sqlp_contains_opts so the SQL parser converts them from column references to string atoms Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Add the standard OpenLink Virtuoso Open-Source (VOS) GPL v2 license header to fuzzy_algorithms.c, fuzzy_algorithms.h, and sqlbif_fuzzy.c, matching the convention used by all other source files in libsrc/Wi/. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Create binsrc/tests/suite/test_fuzzy.sql covering: - Group A: Algorithm correctness (Levenshtein, Jaro-Winkler, N-gram cosine) with exact value assertions for known inputs - Group B: NULL handling, n-gram bounds (n=0 default, n=8 clamp), non-string argument errors - Group C: SQL contains() with fuzzy options — jaro_winkler, levenshtein, ngram_cosine, ngram_cosine with explicit n, SCORE output and ordering - Group D: SPARQL bif:contains with OPTION (...) — all three algorithms, SCORE, FUZZY_N, lowercase/mixed-case keyword case-insensitivity, standalone BIFs in SPARQL - Group E: Error handling — invalid algorithm names in SQL and SPARQL Wire test_fuzzy.sql into tsql3.sh after the existing tft_offband test. 50 assertions, all passing on a fresh build. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

When enable_qp >= 2, partition the word-index prefix range into sub-ranges and scan them concurrently using async-queue workers. Each worker creates its own it_cursor on the word index, iterates its sub-range, and collects (word, similarity) pairs above the threshold. The main thread then merges all pairs and creates word_stream_t entries. Falls back to the original single-threaded scan when enable_qp < 2. The parallel path uses bootstrap_cli and aq_no_lt_enter=1, following the pattern in chash.c. Workers use bootstrap_cli->cli_trx for read-only uncommitted index scans. The WST_WILDCARD_MAX limit and all three algorithms (levenshtein, jaro_winkler, ngram_cosine) are preserved. Test suite confirms identical results for both paths: 0 FAILED, 50 PASSED. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Four issues found and fixed: 1. ITC_FAILED search-param leak in fuzzy_scan_worker: the two box_copy() search params were only freed inside ITC_FAIL, not in ITC_FAILED. Added dk_free_box() calls in the ITC_FAILED block. 2. Same ITC_FAILED leak in single-threaded wst_from_fuzzy path. Added matching dk_free_box() calls. 3. SRC_ERROR word_stream_t leak in single-threaded wst_from_fuzzy: when wst_from_word() returns SRC_ERROR, the stream was neither pushed to wsts nor freed. Added else-if cleanup matching the parallel path. 4. ngram_map_add realloc bug in fuzzy_algorithms.c: realloc result was assigned directly to map->entries, so on failure the original pointer was lost (leak) and subsequent accesses crashed. Now uses a temporary variable and only updates on success. Audit also verified: DV_TEXT_SEARCH destructor properly handles word_stream_t cleanup; NEW_SST zero-initializes wst_fuzzy_similarity; bif_string_or_null_arg rejects wide strings safely; SPARQL code generation handles non-literal fuzzy values via ssg_print_scalar_expn. Test suite: 0 FAILED, 50 PASSED. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…on, interaction tests P1-1: Add UTF-8 code-point-aware algorithm variants (levenshtein_cp, jaro_winkler_cp, ngram_cosine_cp) with ASCII fast path. CJK and emoji inputs now compare by code point, not byte. P1-2: Add 7 interaction tests covering fuzzy + SCORE, SCORE_LIMIT, DESCENDING, START_ID/END_ID, encoding prefix, boolean AND/OR. All combinations work correctly; no bugs found. P1-3: Change score fusion from multiplicative (score * similarity, truncated) to additive (score + similarity * 100). Exact matches get full +100 bonus so they always rank highest. Eliminates truncation ties while preserving FT score discrimination. P1-4: Add configurable fuzzy_prefix option (SQL contains() and SPARQL bif:contains OPTION). Values 1-4, default 2. Plumbed through all compiler/runtime layers: sqlo.h/c, sqlgen.c, sqldf.c, sqlpfn.c, xmlnode.h, text.h/c, sparql_p.y, sparqlwords.gperf, sparql_tree.c, sparqld.c, sparql2sql.c, sparql2sqltext.c. P1-5: Add SPARQL compile-time validation for FUZZY algorithm name (rejects invalid names and non-literal values with clear error listing valid algorithms). Validate FUZZY_THRESHOLD and FUZZY_N as literals. Test suite: 74 tests, 0 failed, 74 passed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Set tctx_fuzzy_algo, tctx_fuzzy_threshold, tctx_fuzzy_n, and tctx_fuzzy_prefix to default values in xp_text_contains() so the XPath contains() path has zeroed fuzzy fields rather than uninitialized stack values. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Since fuzzy_algorithms.c is in libsrc/Wi and built into the core engine, use Virtuoso's dk_alloc/dk_free instead of malloc/calloc/ realloc/free. No dk_realloc exists, so realloc is replaced with dk_alloc + memcpy + dk_free. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Bind the allocated search parameter boxes to the itc via ITC_OWNS_PARAM() so they are freed automatically by itc_free, instead of manually freeing them by fragile index arithmetic in both the success and ITC_FAILED paths. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Add sst_is_fuzzy member to the SST_COMMON macro so it exists in both word_stream_s and search_stream_s. Set it to 1 in wst_from_fuzzy when creating fuzzy word streams; it defaults to 0 via dk_alloc_box_zero in NEW_SST for all other streams. In sst_scores, guard the word_stream_t typecast with sst_is_fuzzy before accessing wst_fuzzy_similarity and wst_fuzzy_distance, so a plain search_stream_t (OR/AND node) or a non-fuzzy word stream is never mistreated as a fuzzy word stream. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

dk_free(ptr, 0) is incorrect: 0 is a real size, not "unknown", so it misroutes the CACHE_MALLOC fast-path (ALIGN_A on 0) and corrupts the dk_n_bytes accounting. Pass the actual allocation size at every free site, derived from the same locals used in the matching dk_alloc. NO_SIZE is private to Dkalloc.c and not exposed in any header, so it cannot be referenced by name here; every freed pointer's size is known at its free site, so passing the concrete size is both correct and keeps the cache fast-path enabled. Addresses review comment r3802220473. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

imitko requested changes Aug 17, 2026

View reviewed changes

TallTed changed the title ~~Add fuzzy search support to contains() and bif:contains~~ Add fuzzy search support to contains() and bif:contains Aug 17, 2026

TallTed reviewed Aug 17, 2026

View reviewed changes

danielhmills and others added 18 commits August 18, 2026 01:00

Fix integer fuzzy threshold crash

02197b1

Add fuzzy similarity and distance output columns

a49c899

Preserve fuzzy similarity for quoted phrases

d0dd41d

Compute RDF fuzzy metrics from complete literals

59beade

This was referenced Aug 18, 2026

Serialize concurrent first VSP compilation #1466

Open

Stabilize JSON-LD context loading #1467

Open

danielhmills force-pushed the feature/fuzzy-search-contains branch from c0b0e8d to 4ea4755 Compare August 18, 2026 05:01

danielhmills requested a review from imitko August 18, 2026 05:11


		#### Implementation notes:

		- Levenshtein: Single-row DP array (O(min(m,n)) space). Early exit if distance exceeds `max(m,n) * (1 - threshold)` — skip full computation when candidate can't possibly meet threshold.

		- Jaro-Winkler: Standard Jaro with Winkler prefix bonus (p=0.1, max prefix=4). Matching window = `max(len(s1), len(s2)) / 2 - 1`.
		- N-gram cosine: Build sparse frequency maps (hash table or sorted array) of character n-grams with boundary padding (`$` prefix/suffix). Cosine = dot product / (magnitude1 × magnitude2). Early exit: if n-gram set intersection count < `threshold² × \|s1_ngrams\|`, return 0.0 immediately.

Conversation

danielhmills commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Included

Review fixes applied

Verification

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

danielhmills commented Aug 17, 2026 •

edited

Loading