Add local Docker vector databases and prefilled adapter settings

Caddyfile.snippet

@@ -0,0 +1,26 @@
+(etl_home_access_log) {
+    log {
+        level INFO
+        format json
+        output file /var/log/caddy/etl-home-access.jsonl {
+            roll_size 50MiB
+            roll_keep 14
+            roll_keep_for 336h
+        }
+    }
+}
+
+etl.home.complete.tech:80 {
+    redir https://etl.home.complete.tech{uri} permanent
+}
+
+etl.home.complete.tech:443 {
+    import etl_home_access_log
+    tls /certs/fullchain.pem /certs/privkey.pem
+
+    reverse_proxy 172.30.88.1:13110 {
+        header_up X-Forwarded-For {remote_host}
+        header_up X-Forwarded-Proto https
+        header_up X-Forwarded-Host {host}
+    }
+}

DOMAIN_CONTEXT.md

@@ -0,0 +1,77 @@
+# Domain Context: etl.home.complete.tech
+
+Local home: `C:\Users\timot\Documents\projects\domains\etl.home.complete.tech`
+
+Public URL: `https://etl.home.complete.tech/`
+
+Source: `https://github.com/Zipstack/unstract`
+
+Service host: `completetrain@train.home.complete.tech`
+
+Remote root: `/home/completetrain/etl.home.complete.tech`
+
+Runtime: rootless Podman under `completetrain`, fronted by the shared rootful
+Caddy container `home-complete-tech-domain-proxy`.
+
+## Deployment shape
+
+- Upstream source is pinned to the release tag `v0.187.2`.
+- Unstract is run with the upstream Docker Compose files plus
+  `docker/compose.train.yaml`.
+- The Unstract Traefik service is published only on host port `13110`; all
+  database, broker, storage, and worker ports remain internal to
+  `unstract-network`.
+- The runner and Unstract Traefik use the rootless Podman API socket at
+  `/run/user/1000/podman/podman.sock`, mounted at `/var/run/docker.sock` in
+  their containers. This preserves Unstract's Docker-compatible container
+  spawning path without exposing the rootful host socket.
+- The frontend is mounted with `docker/frontend-nginx.conf` and listens on
+  container port `8080`, because the published frontend image runs Nginx as a
+  non-root user under rootless Podman.
+- Shared Caddy proxies `etl.home.complete.tech` to `172.30.88.1:13110` and
+  uses the existing wildcard certificate mounted at `/certs`.
+- LAN DNS maps `etl.home.complete.tech` to `192.168.1.146` on
+  `router.complete.tech`.
+
+## Persistent data
+
+The Compose named volumes are owned by the rootless project and must not be
+removed during upgrades:
+
+- `unstract-etl-home-complete-tech_postgres_data`
+- `unstract-etl-home-complete-tech_redis_data`
+- `unstract-etl-home-complete-tech_minio_data`
+- `unstract-etl-home-complete-tech_qdrant_data`
+- `unstract-etl-home-complete-tech_prompt_studio_data`
+- `unstract-etl-home-complete-tech_rabbitmq_data`
+- `unstract-etl-home-complete-tech_flipt_data`
+- `/home/completetrain/etl.home.complete.tech/docker/workflow_data`
+
+The backend and platform-service `ENCRYPTION_KEY` values are generated once
+and must be backed up securely. Losing or changing that key makes encrypted
+adapter credentials inaccessible.
+
+## Operational commands
+
+```sh
+cd /home/completetrain/etl.home.complete.tech
+export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
+export VERSION=v0.187.2
+docker compose -f docker/docker-compose.yaml -f docker/compose.train.yaml ps
+docker compose -f docker/docker-compose.yaml -f docker/compose.train.yaml logs --tail=100 backend
+```
+
+The rootless Podman user socket and the shared rootful Caddy proxy are separate
+ownership domains. Inspect both before changing either one.
+
+## Verification
+
+```sh
+curl -fsSI https://etl.home.complete.tech/
+curl -ksS -o /dev/null -w '%{http_code}\n' https://etl.home.complete.tech/api/v1/health
+ssh router.complete.tech "nslookup etl.home.complete.tech 127.0.0.1"
+```
+
+The unauthenticated health request should return `401`; an authenticated health
+request should return `200`. The public root should return the Unstract
+frontend, with API and WebSocket paths routed by Traefik to the backend.

README.md

@@ -89,6 +89,10 @@ That's it!
 - Login with username: `unstract` password: `unstract`
 - Start extracting data!
 
+The Compose quickstart also provisions local Qdrant, pgvector/PostgreSQL,
+Weaviate, and Milvus services. See [local vector database connection
+settings](docker/README.md#local-vector-databases) for the adapter values.
+
 ## 📦 Other Deployment Options
 
 ### Docker Compose

docker/README.md

@@ -30,6 +30,63 @@ VERSION=dev docker compose -f docker-compose.yaml --profile optional up -d
 
 Now access frontend at http://frontend.unstract.localhost
 
+## Local vector databases
+
+The default development Compose stack starts the open-source vector database
+backends supported by Unstract:
+
+- Qdrant
+- PostgreSQL with the `pgvector` extension
+- Weaviate
+- Milvus Standalone (with private etcd and MinIO dependencies)
+
+Pinecone is supported as a hosted provider and is intentionally not included
+in the local stack because it is not a self-hosted open-source service.
+
+The services use persistent named volumes and are bound to loopback on the host.
+Qdrant and Weaviate use anonymous access because this is a local development
+stack; do not expose these ports beyond the local machine without adding
+authentication and TLS. The Unstract workers connect over the Compose network,
+so use the internal addresses below when creating an adapter in the UI:
+
+| Adapter | UI fields | Address from Unstract containers | Host address | Local credentials |
+|---------|-----------|----------------------------------|--------------|-------------------|
+| Qdrant | URL, API Key | `http://qdrant:6333` | `http://localhost:6333` | Leave API Key empty |
+| Postgres | Database, Host, Port, User, Password, Enable SSL | Host `postgres-vector`, port `5432` | Host `localhost`, port `5433` | Values from `docker/essentials.env`; SSL is enabled by default |
+| Weaviate | URL, API Key | `http://weaviate:8080` | `http://localhost:8084` | Leave API Key empty |
+| Milvus | URI, Token | `http://milvus:19530` | `http://localhost:19530` | Leave Token empty |
+
+When the adapter schema is served by the local development stack, these
+internal values are pre-filled in the form, including the sample PostgreSQL
+credentials from `docker/essentials.env` and SSL enabled for the local
+connection. The bundled `postgres-vector` service generates a self-signed
+certificate and starts PostgreSQL with TLS enabled. The adapter uses
+`sslmode=require` without certificate verification for this local-only
+certificate. If `essentials.env` has customized the PostgreSQL credentials,
+replace the pre-filled values with the customized values before testing the
+connection. Keep the container DNS names (`qdrant`, `postgres-vector`,
+`weaviate`, and `milvus`) when the adapter is used by Unstract workers; use the
+host addresses only for clients running outside the Compose network.
+
+For the Postgres adapter, use the `POSTGRES_USER`, `POSTGRES_PASSWORD`, and
+`POSTGRES_DB` values in `docker/essentials.env`; the vector database has its own
+container and volume even though it reuses the platform's local development
+credentials. The initialization script enables `vector` and creates the
+`POSTGRES_SCHEMA` schema on first boot.
+
+The default `run-platform.sh` flow creates `docker/.env` and
+`docker/essentials.env` before starting Compose. To start only the vector
+services during development, run:
+
+```bash
+cd docker
+VERSION=dev docker compose -f docker-compose.yaml up -d \
+  qdrant postgres-vector weaviate milvus
+```
+
+Do not use `docker compose down -v` unless deleting local vector data is
+intentional.
+
 ## Overriding a service's config
 
 By making use of the [merge compose files](https://docs.docker.com/compose/how-tos/multiple-compose-files/merge/) feature its possible to override some configuration that's used by the services.

docker/compose.train.healthchecks.yaml

@@ -0,0 +1,98 @@
+# Train health coverage overlay.
+#
+# Apply this file after docker/compose.train.yaml:
+#
+#   docker compose -f docker/docker-compose.yaml \
+#     -f docker/compose.train.yaml \
+#     -f docker/compose.train.healthchecks.yaml up -d
+#
+# The overlay only adds a read-only probe mount and a bounded application
+# healthcheck.  It does not replace ports, volumes, images, environment, or
+# container names from the owner Compose files.  Worker and runner checks are
+# staged separately in docker/compose.train.worker-healthchecks.yaml.
+
+x-unstract-service-probe: &unstract_service_probe
+  - ${UNSTRACT_HEALTHCHECK_SOURCE:-./healthchecks/unstract-services.sh}:/usr/local/bin/unstract-services.sh:ro
+
+x-unstract-healthcheck: &unstract_healthcheck
+  interval: 30s
+  timeout: 10s
+  retries: 3
+
+services:
+  db:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 30s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "db"]
+
+  redis:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 15s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "redis"]
+
+  minio:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 60s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "minio"]
+
+  reverse-proxy:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 60s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "proxy"]
+
+  qdrant:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 60s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "vector-db"]
+
+  rabbitmq:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 60s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "rabbitmq"]
+
+  weaviate:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 120s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "weaviate"]
+
+  x2text-service:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 120s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "x2text-service"]
+
+  platform-service:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 120s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "platform-service"]
+
+  backend:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 180s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "backend"]
+
+  frontend:
+    volumes: *unstract_service_probe
+    healthcheck:
+      <<: *unstract_healthcheck
+      start_period: 60s
+      test: ["CMD", "/usr/local/bin/unstract-services.sh", "frontend"]

docker/compose.train.worker-healthchecks.yaml

@@ -0,0 +1,97 @@
+# Worker and runner health coverage overlay for the dirty Train checkout.
+#
+# Apply this after docker/compose.train.yaml and before the core service
+# overlay.  It contains the same checks committed in docker-compose.yaml, so a
+# coordinator can stage the health configuration without checking out over
+# Train's local Compose and embedding changes.
+
+x-worker-healthcheck: &worker_healthcheck
+  interval: 30s
+  timeout: 5s
+  retries: 3
+  start_period: 30s
+
+services:
+  runner:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:5002/v1/api/health"]
+
+  worker-log-history-scheduler-v2:
+    environment:
+      LOG_HISTORY_SCHEDULER_HEALTH_PORT: "8092"
+      LOG_HISTORY_SCHEDULER_HEALTH_STALE_SECONDS: ${LOG_HISTORY_SCHEDULER_HEALTH_STALE_SECONDS:-120}
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8092/health"]
+
+  worker-pg-orchestrator-api:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8090/health"]
+
+  worker-pg-orchestrator-general:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8090/health"]
+
+  worker-pg-fileproc:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8090/health"]
+
+  worker-pg-callback:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8090/health"]
+
+  worker-pg-scheduler:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8090/health"]
+
+  worker-pg-metrics:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8090/health"]
+
+  worker-log-stream-consumer:
+    environment:
+      LOG_STREAM_CONSUMER_HEALTH_PORT: "8091"
+      LOG_STREAM_CONSUMER_HEALTH_STALE_SECONDS: ${LOG_STREAM_CONSUMER_HEALTH_STALE_SECONDS:-15}
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8091/health"]
+
+  worker-pg-executor:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8090/health"]
+
+  worker-pg-ide-callback:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8090/health"]
+
+  worker-pg-notification:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8090/health"]
+
+  worker-pg-reaper:
+    healthcheck:
+      <<: *worker_healthcheck
+      test: ["CMD", "/usr/bin/curl", "--fail", "--silent", "--show-error",
+             "--max-time", "3", "http://127.0.0.1:8086/health"]