feat(embeddings): add cross-database vector search - #1592
Open
kkopanidis wants to merge 28 commits into
Open
Conversation
Require object-form Vector fields and validate dimensions, similarity, and index methods. Extract testable Mongo/Postgres mappings and capability helpers.
Load hidden source hashes, suppress embeddings-owned write-back events, and deduplicate queue work so one source change yields one provider call.
…aries Vector and semantic search now require a subject, scope, or verified admin operator on authorization-enabled schemas. Embeddings jobs use a module-identity-scoped read/write context, and provider/config/queue paths fail closed on SSRF, secrets, and malformed payloads.
…arity scores Replace unbounded authorized-ID materialization with ANN prefilters plus bounded candidate checks, and document a higher-is-better score contract with strict shared filter and limit validation.
Prevent silent index mismatches, expose queryability, and apply declared vector indexes so search fails closed until indexes are ready.
Add an embeddings-owned BackfillRun schema and a pure transition/pagination helper so operational backfills can resume, cancel, and report progress without scanning in the gRPC thread.
Persist a queued BackfillRun and return after enqueueing a controller job, then scan bounded pages through continuation jobs with gated execution, accurate counters, and unlabeled queue metrics.
Replace unreleased JSON-string proto responses with typed config, status, backfill, and search messages so operators can manage embeddings through gRPC, Admin, and Hermes MCP without exposing config or backfill on client routes.
Teach TYPE.Vector as a finite number array in Zod, OpenAPI, GraphQL, and MCP so clients never see a phantom Vector type, and keep managed vector/hash/select:false fields out of CMS create/update bodies.
…s gaps Material EmbeddingConfig changes now invalidate hashes, recreate indexes when needed, and schedule an explicit backfill instead of reusing stale vectors. Backfill start/cancel/resume is idempotent, enqueue failures cannot leave queued orphans, and drain polling fails closed on timeout with sanitized diagnostics.
Package embeddings as a BullMQ service bundle with bake/CI wiring while keeping it out of standalone v1. Production still requires GRPC_KEY and capability/index readiness before activation.
Dedupe BullMQ jobs only while waiting/active so completed/failed ids cannot block later updates. Increment backfill counts atomically, provision vector indexes on first upsert, and bound updateMany mutation id collection.
Prevent schema-owner gRPC callers from bypassing hidden or sensitive source rejection, and add defense-in-depth bounds for job reads, backfill filters, Mongo index drops, and client search limits.
Keep concurrent backfill counters on the Database $inc contract. Production tsc can then accept findByIdAndUpdate without casts.
Require a non-empty GRPC_KEY in profile enablement examples, distinguish Helm workload install.embeddings.enabled from convict enabled, and keep embeddings-only image rebuilds off standalone v1.
… counters Evaluate createVectorIndex canModify against the requested vector field so embeddings can index its own extension fields, persist failed provisioning as disabled, and keep page/state saves from overwriting atomic processed/failed counts.
Authorize ExtensionOnly deletes from the live index field, and provision a versioned replacement before retiring the previous index so pending builds cannot strand search.
Treat live indexes as ready only when field, dimensions, similarity, and method match, so a failed recreate cannot enable or search the old index.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
7 tasks
Include pgvector in the database bundle install, emit docker target JSON even when GitHub sets GITHUB_OUTPUT, and clear branch-local CodeFactor unused and complexity findings.
6 tasks
Operator settings now own provider models and dimensions, while endpoint SSRF checks derive the HTTPS hostname internally and production GRPC_KEY stays a deployment requirement.
Reject configs unless the provider and model exist in settings. Persist catalogue dimensions and require enabled, extendable schemas.
Cover Core GET/PATCH redaction, default-model selection, derived dimensions, extension collisions, and provider invocation against the operator catalogue shape the Admin UI consumes.
Flatten default-model selection and drop control-flow try/catch without changing catalogue migration.
…r methods Treat empty proto method and missing Mongo indexingMethod as hnsw so live indexes are reused instead of recreated as _vN. Persist migrated catalogue-only provider settings on module config lifecycle so a later unrelated Admin PATCH cannot wipe them.
Admin, middleware, Client, and Database-owned system schemas can be extendable in the CMS list; they still must not be embedding sources.
Share source-hash naming, redaction detection, and catalogue strictness so the latest migration and schema-policy work matches surrounding style.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
getVectorCapabilitiesprobes MongoDB Search withlistSearchIndexes/createSearchIndex(Atlas Vector Search and self-hosted Search). Missing commands or probe errors keep vector storage true and indexing/search false. Postgres probespgvectorviaSELECT 'vector'::regtypeafter a best-effortCREATE EXTENSION IF NOT EXISTS vector; other SQL dialects store JSON only.enableddefault false) owns OpenAI-compatible generation, cursorBackfillRuns, Admin/embeddings/*plus Hermes MCP tools, and textsemanticSearch. Settings own the provider model catalogue; embedding configs persist catalogue dimensions. Config, backfill, capabilities, and status stay operator-only; clientPOST /embeddings/searchtakes user/scope from router auth and does not accept raw vectors oradminOperator.GRPC_KEYis deployment-owned, not Settings; configs require enabled and extendable schemas (module-owned without CMS block may be eligible) and deny Database/core/router/authorization internals even when extendable; schema-owner upserts cannot bypass source allowlists; CMS omits Vector/*SourceHash/select:falsewrites; search and enablement wait for a matching queryable vector index.--profile embeddings(non-emptyGRPC_KEY). Standalone v1 does not include it. Helminstall.embeddings.enabledonly deploys the process; convictenabledis a separate Core switch. This change does not publishdocker.io/conduitplatform/embeddings.Test plan
nvm usethenpnpm exec turbo run build --filter=@conduitplatform/embeddings... --filter=@conduitplatform/database... --filter=@conduitplatform/hermes... --filter=@conduitplatform/grpc-sdk... --filter=@conduitplatform/module-tools... --filter=@conduitplatform/corepnpm --filter @conduitplatform/embeddings test(153)pnpm --filter @conduitplatform/database test --testPathIgnorePatterns=integration(109)pnpm --filter @conduitplatform/hermes test(9)GRPC_KEY=ci-nonempty-grpc-key docker compose --profile mongodb --profile embeddings config --servicesincludesembeddings;docker compose --profile mongodb config --servicesdoes notCHANGED_FILES=modules/embeddings/src/index.ts node scripts/resolve-docker-targets.mjsselectsembeddingsand excludesconduit-standaloneenabledor pullingdocker.io/conduitplatform/embeddings:${IMAGE_TAG}