Skip to content

Add local Docker vector databases and prefilled adapter settings - #1

Open
CompleteDotTech wants to merge 51 commits into
mainfrom
feature/local-vector-databases
Open

CompleteDotTech wants to merge 51 commits into
mainfrom
feature/local-vector-databases

Conversation

@CompleteDotTech

@CompleteDotTech CompleteDotTech commented Sep 3, 2026

Copy link
Copy Markdown

Summary

Adds a complete local Docker path for the open-source vector databases supported by the Vector DB setup UI:

  • Qdrant
  • PostgreSQL with pgvector
  • Weaviate
  • Milvus (standalone with etcd and MinIO)

The change also makes the existing adapter forms usable for the local deployment by pre-filling the connection values returned by each adapter schema.

Implementation

  • Adds persistent Compose services, volumes, health checks, and a pgvector initialization script.
  • Uses internal Compose DNS names for backend/worker connections.
  • Adds local HTTP handling to the Weaviate adapter while preserving the existing Weaviate Cloud path.
  • Pins the compatible weaviate-client version and updates the affected lockfiles.
  • Adds schema regression coverage for the local defaults.
  • Documents service ports, volumes, credentials, and local-vs-external connection settings.
  • Pinecone remains cloud-managed and is not presented as a local container.

The local Postgres schema defaults are development defaults from the Train Compose environment. Deployments with different essentials.env values should replace those fields before submitting the adapter.

Live Train deployment

Deployed at https://etl.home.complete.tech:

  • Onboarding route: https://etl.home.complete.tech/mock_org/onboard (HTTP 200).
  • All 13 application containers are running with zero restarts.
  • Qdrant, pgvector/Postgres, Weaviate, and the Milvus/etcd/MinIO stack are running in six separate containers.
  • The backend exposes local defaults for all four adapters, including the Postgres form fields that were previously blank.
  • Worker-side connection tests passed for all four adapters; each test created, queried, and removed a temporary collection/table.
  • The public health endpoint correctly returns HTTP 401 without authentication; no credentialed extraction workflow is claimed here.

The deployed application images were built from an isolated merge of the latest Train checkout with this PR, so concurrent uncommitted host files and persistent vector volumes were preserved.

Validation

  • SDK schema and Weaviate adapter tests: 4 passed.
  • JSON, whitespace, Python, shell, Compose, and lockfile validation completed.
  • Live backend schema defaults verified.
  • Live worker adapter tests passed for Qdrant, Postgres/pgvector, Weaviate, and Milvus.

@CompleteDotTech CompleteDotTech changed the title Provision local open-source vector databases Add local Docker vector databases and prefilled adapter settings Sep 3, 2026
CompleteDotTech and others added 20 commits September 7, 2026 02:54
Record the PostgreSQL WAL recovery failure, retained-container readiness
checks, and guardrails against WAL reset, volume removal, or recreation.
Keep the public recovery record free of deployment-specific host and container
identifiers.
…0907-3lwkgkey

Document disk exhaustion recovery without destructive database reset
Record the PostgreSQL WAL recovery failure, retained-container readiness
checks, and guardrails against WAL reset, volume removal, or recreation.
Keep the public recovery record free of deployment-specific host and container
identifiers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant