fix: align oss readiness updates

.github/workflows/ci.yml

@@ -38,6 +38,25 @@ jobs:
       - name: Biome check
         run: pnpm exec biome check .
 
+  repo-check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+          cache-dependency-glob: api/uv.lock
+
+      - name: Install API dependencies
+        run: uv sync --dev
+        working-directory: api
+
+      - uses: ./.github/actions/setup-node-pnpm
+
+      - name: Run root check
+        run: pnpm check
+
   sdk-typecheck:
     runs-on: ubuntu-latest
     steps:

AGENTS.md

@@ -26,7 +26,7 @@ The repo follows aggressive package-style splits. If a single file grows past ~3
 - `api/bigrag/app_factory/` (lifespan, exception_handlers, routers)
 - `api/bigrag/mcp/` (tools, unscoped, scoped, cli)
 - `api/bigrag/services/{embedding,retrieval,webhook,vector_store,storage,url_security,access_log,event_bus,queue_conversion,queue_embedding,chat,runtime_setting_specs}/` packages
-- `sdks/python/src/bigrag/resources/admin/` and `sdks/typescript/src/resources/admin/` (settings, backups, realtime, users, api_keys, access, audit, connectors, embedding_presets, mcp_servers)
+- `sdks/python/src/bigrag/resources/admin/` and `sdks/typescript/src/resources/admin/` (settings, users, api_keys, access, audit, connectors, embedding_presets, mcp_servers, vector_storage)
 
 When adding new code, prefer the smallest meaningful module instead of dropping it into the nearest catch-all.
 
@@ -94,7 +94,7 @@ Use lint, typecheck, build, compile, and runtime smoke checks for current verifi
 - Tenant scoping: route handlers call `enforce_collection_pin(user, collection_name)` from `routers/__init__.py` to honor pinned API keys. Connector accounts/sources carry an optional `tenant_id` column.
 - Background workers: Dramatiq actors live in `services/jobs/`. The ingestion queue (`services/queue.py`) drains via `bigrag-worker`.
 - MCP server: `api/bigrag/mcp/` package (`tools`, `unscoped`, `scoped`, `cli`); entry point `bigrag-mcp = "bigrag.mcp:cli"`.
-- SDK uses resource namespaces: `client.collections.list()`, `client.documents.upload()`, `client.admin.users.list()` etc. The `admin` resource is itself a package — sub-resources live in `resources/admin/{settings,backups,realtime,users,api_keys,access,audit,connectors,embedding_presets,mcp_servers}.py`.
+- SDK uses resource namespaces: `client.collections.list()`, `client.documents.upload()`, `client.admin.users.list()` etc. The `admin` resource is itself a package — sub-resources live in `resources/admin/{settings,users,api_keys,access,audit,connectors,embedding_presets,mcp_servers,vector_storage}.py`.
 - SDK reliability: both Python and TypeScript SDKs send `Idempotency-Key: <uuid4>` on POST/PUT/PATCH/DELETE and only retry mutating calls when an idempotency key is present. Both expose typed error subclasses (`BadRequestError`, `ConflictError`, `PayloadTooLargeError`, `UnprocessableEntityError`, `BadGatewayError`, `ServiceUnavailableError`, etc.).
 
 ## Code conventions

CONTRIBUTING.md

@@ -83,18 +83,11 @@ bigrag/
 ### Verifying Changes
 
 ```bash
-# Website build check
-pnpm --filter @bigrag/docs build
-
-# SDK and app build checks
-pnpm --filter @bigrag/client build
-pnpm --filter @bigrag/app build
-
-# Lint everything
-pnpm lint          # TypeScript (Biome)
-cd api && uv run ruff check . && uv run ruff format --check .  # Python
+pnpm check
 ```
 
+`pnpm check` runs Biome, Ruff, workspace typechecks, app / SDK / docs builds, and a Python backend compile pass. Use targeted package commands while iterating, then run the root check before opening a PR.
+
 ### Commit Messages
 
 Use [Conventional Commits](https://www.conventionalcommits.org/):
@@ -124,7 +117,7 @@ When cutting a coordinated platform release, keep the API package, SDK packages,
 
 ### PR Requirements
 
-- All CI checks must pass (lint, biome, sdk-typecheck, website-build, app-build)
+- All CI checks must pass, including the root `pnpm check` job
 - At least one maintainer approval
 - No merge conflicts with `main`
 

README.md

@@ -51,20 +51,35 @@ This starts the bigRAG API, worker, admin UI, Postgres, and Redis. Open **[local
 > [!IMPORTANT]
 > Configure Turbopuffer from onboarding before ingesting or querying collections.
 
-Once Turbopuffer is configured, you can drive everything over HTTP:
+Once Turbopuffer is configured, create the first admin and mint an API key for HTTP clients:
 
 ```bash
+export BASE="http://localhost:4000"
+
+curl -X POST "$BASE/v1/auth/setup" \
+  -H "Content-Type: application/json" \
+  -c cookies.txt \
+  -d '{"email": "admin@example.com", "password": "a-strong-password", "display_name": "Admin"}'
+
+export BIGRAG_API_KEY=$(curl -s -X POST "$BASE/v1/admin/api-keys" \
+  -b cookies.txt \
+  -H "Content-Type: application/json" \
+  -d '{"name": "local-dev", "scopes": ["*:*"]}' | jq -r .key)
+
 # Create a collection
-curl -X POST http://localhost:4000/v1/collections \
+curl -X POST "$BASE/v1/collections" \
+  -H "Authorization: Bearer $BIGRAG_API_KEY" \
   -H "Content-Type: application/json" \
   -d '{"name": "docs", "embedding_api_key": "sk-..."}'
 
 # Upload a document
-curl -X POST http://localhost:4000/v1/collections/docs/documents \
+curl -X POST "$BASE/v1/collections/docs/documents" \
+  -H "Authorization: Bearer $BIGRAG_API_KEY" \
   -F "file=@paper.pdf"
 
 # Query
-curl -X POST http://localhost:4000/v1/collections/docs/query \
+curl -X POST "$BASE/v1/collections/docs/query" \
+  -H "Authorization: Bearer $BIGRAG_API_KEY" \
   -H "Content-Type: application/json" \
   -d '{"query": "What are the main findings?"}'
 ```
@@ -78,11 +93,11 @@ curl -X POST http://localhost:4000/v1/collections/docs/query \
 ### Docker Images
 
 ```bash
-docker pull yoginth/bigrag-api:2026.4.30
-docker pull yoginth/bigrag-ui:2026.4.30
+docker pull yoginth/bigrag-api:latest
+docker pull yoginth/bigrag-ui:latest
 ```
 
-Release artifacts use CalVer (`YYYY.M.D`). Docker also publishes `latest`; the Python and TypeScript SDKs publish dated PyPI and npm releases.
+Release artifacts use CalVer (`YYYY.M.D`). Docker publishes `latest` for quick starts; pin a dated tag from the release you deploy in production.
 
 ## Architecture
 
@@ -148,8 +163,6 @@ graph TD
 | `DELETE` | `/v1/collections/{name}` | Delete collection |
 | `GET` | `/v1/collections/{name}/stats` | Collection stats |
 | `POST` | `/v1/collections/{name}/truncate` | Delete all documents, keep the collection |
-| `POST` | `/v1/collections/{name}/realtime-token` | Create a short-lived collection realtime token |
-| `WS` | `/v1/realtime` | Subscribe to realtime snapshots and collection events |
 | **Documents** | | |
 | `POST` | `/v1/collections/{name}/documents` | Upload document |
 | `GET` | `/v1/collections/{name}/documents` | List documents |
@@ -321,7 +334,7 @@ Bootstrap settings use the `BIGRAG_` prefix as environment variables, or configu
 |----------|-------------|---------|
 | `BIGRAG_DATABASE_URL` | Postgres URL (`postgres:5432` inside docker-compose, `localhost:5432` for bare-metal dev) | `postgres://bigrag:bigrag@localhost:5432/bigrag?sslmode=disable` |
 | `BIGRAG_DB_POOL_MIN` | Min Postgres pool size | `5` |
-| `BIGRAG_DB_POOL_MAX` | Max Postgres pool size | `10` |
+| `BIGRAG_DB_POOL_MAX` | Max Postgres pool size | `20` |
 | `BIGRAG_MIGRATION_TIMEOUT_SECONDS` | Startup migration check timeout (`0` disables the timeout) | `60` |
 | `BIGRAG_REDIS_URL` | Redis URL | `redis://localhost:6379/0` |
 

STYLEGUIDE.md

@@ -2569,22 +2569,20 @@ if (lastError.name === "TimeoutError" || lastError.name === "AbortError") {
 }
 ```
 
-### Realtime Streaming
+### Paginated Iteration
 
-Use async generators for WebSocket realtime subscriptions:
+Use async generators for SDK pagination helpers:
 
 ```typescript
-async *streamEvents(name: string): AsyncGenerator<ProgressEvent> {
-  const connection = new BigRAGRealtimeConnection(this._client);
-  try {
-    for await (const message of connection.subscribe<ProgressEvent>("collection.events", {
-      collection: name,
-    })) {
-      if (message.type === "event") yield message.payload;
+async *listAllDocuments(collection: string): AsyncGenerator<Document> {
+  let cursor: string | undefined;
+  do {
+    const page = await this.list(collection, { cursor });
+    for (const document of page.documents) {
+      yield document;
     }
-  } finally {
-    await connection.close();
-  }
+    cursor = page.next_cursor ?? undefined;
+  } while (cursor);
 }
 ```
 

api/bigrag/logging.py

@@ -8,6 +8,7 @@
 from bigrag.logging_redaction import (
     is_sensitive_log_key,
     redact_secrets,
+    safe_url_value,
     truncate_log_value,
 )
 from bigrag.logging_rendering import console_renderer
@@ -29,6 +30,7 @@
     "current_worker_label",
     "get_logger",
     "is_sensitive_log_key",
+    "safe_url_value",
     "truncate_log_value",
 ]
 

api/bigrag/logging_redaction.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import re
+from urllib.parse import SplitResult, parse_qsl, urlencode, urlsplit, urlunsplit
 
 _SENSITIVE_KEYS = frozenset(
     {
@@ -87,6 +88,34 @@ def truncate_log_value(value: str) -> str:
     return f"{value[:_MAX_LOG_VALUE_LENGTH]}..."
 
 
+def safe_url_value(value: str) -> str:
+    try:
+        parts = urlsplit(value)
+    except ValueError:
+        return truncate_log_value(value)
+    if not parts.scheme or not parts.netloc:
+        return truncate_log_value(value)
+    query = urlencode([(key, "[REDACTED]") for key, _ in parse_qsl(parts.query, True)])
+    fragment = "[REDACTED]" if parts.fragment else ""
+    return truncate_log_value(
+        urlunsplit((parts.scheme, _safe_url_netloc(parts), parts.path, query, fragment))
+    )
+
+
+def _safe_url_netloc(parts: SplitResult) -> str:
+    if not parts.netloc or (parts.username is None and parts.password is None):
+        return parts.netloc
+    hostname = parts.hostname or ""
+    if ":" in hostname and not hostname.startswith("["):
+        hostname = f"[{hostname}]"
+    try:
+        port = parts.port
+    except ValueError:
+        port = None
+    suffix = f":{port}" if port is not None else ""
+    return f"[REDACTED]@{hostname}{suffix}"
+
+
 def log_field_value(value: object) -> str:
     return truncate_log_value(escape_control_characters(str(value)))
 
@@ -114,5 +143,7 @@ def redact_log_value(value: object) -> object:
     if isinstance(value, tuple):
         return tuple(redact_log_value(item) for item in value)
     if isinstance(value, str):
+        if "://" in value:
+            return safe_url_value(value)
         return truncate_log_value(value)
     return value

api/bigrag/middleware/idempotency.py

@@ -16,7 +16,16 @@
 _MAX_REQUEST_BODY_BYTES = 8 * 1024 * 1024
 _SENSITIVE_RESPONSE_HEADERS = frozenset({"content-length", "set-cookie"})
 _IN_FLIGHT_SENTINEL = "__in_flight__"
+_COMPLETED_UNREPLAYABLE = "__completed_unreplayable__"
 _IN_FLIGHT_TTL_SECONDS = 60
+_UNCACHEABLE_RESPONSE_ROUTES: tuple[tuple[str, str], ...] = (
+    ("POST", "/v1/admin/api-keys"),
+    ("POST", "/v1/admin/api-keys/"),
+    ("POST", "/v1/admin/mcp-servers"),
+    ("POST", "/v1/admin/mcp-servers/"),
+    ("POST", "/v1/admin/webhooks"),
+    ("POST", "/v1/admin/webhooks/"),
+)
 
 
 def _cache_key(principal: str, idem_key: str, method: str, path: str) -> str:
@@ -55,6 +64,13 @@ def _should_skip_body(scope) -> bool:
     return content_length is not None and content_length > _MAX_CACHED_BODY_BYTES
 
 
+def _should_skip_response_cache(method: str, path: str) -> bool:
+    for route_method, route_prefix in _UNCACHEABLE_RESPONSE_ROUTES:
+        if method == route_method and path.startswith(route_prefix):
+            return True
+    return False
+
+
 async def _read_request_body(receive) -> tuple[bytes, bool, list[dict]]:
     chunks: list[bytes] = []
     messages: list[dict] = []
@@ -123,6 +139,21 @@ async def _send_in_flight(send) -> None:
     await send({"type": "http.response.body", "body": body})
 
 
+async def _send_unreplayable(send) -> None:
+    body = orjson.dumps({"detail": "Idempotency-Key already completed for a one-time response"})
+    await send(
+        {
+            "type": "http.response.start",
+            "status": 409,
+            "headers": [
+                (b"content-type", b"application/json"),
+                (b"content-length", str(len(body)).encode("ascii")),
+            ],
+        }
+    )
+    await send({"type": "http.response.body", "body": body})
+
+
 class IdempotencyMiddleware:
     def __init__(self, app, ttl_seconds: int = _DEFAULT_TTL_SECONDS) -> None:
         self.app = app
@@ -151,6 +182,7 @@ async def __call__(self, scope, receive, send):
 
         method = scope["method"]
         path = scope.get("path", "")
+        skip_response_cache = _should_skip_response_cache(method, path)
         principal = principal_id(scope)
         cache_key = _cache_key(principal, idem_key, method, path)
         request_body, can_fingerprint, messages = await _read_request_body(receive)
@@ -164,6 +196,9 @@ async def __call__(self, scope, receive, send):
             if cached.get("status") == _IN_FLIGHT_SENTINEL:
                 await _send_in_flight(send)
                 return
+            if cached.get("status") == _COMPLETED_UNREPLAYABLE:
+                await _send_unreplayable(send)
+                return
             if cached.get("request_hash") != fingerprint:
                 await _send_conflict(send)
                 return
@@ -186,6 +221,9 @@ async def __call__(self, scope, receive, send):
                 await _send_in_flight(send)
                 return
             if existing:
+                if existing.get("status") == _COMPLETED_UNREPLAYABLE:
+                    await _send_unreplayable(send)
+                    return
                 if existing.get("request_hash") != fingerprint:
                     await _send_conflict(send)
                     return
@@ -220,7 +258,7 @@ async def send_wrapper(message):
         cached_response = False
         try:
             await self.app(scope, _replay_receive(messages), send_wrapper)
-            if 200 <= status_code < 300 and cacheable_body:
+            if 200 <= status_code < 300 and cacheable_body and not skip_response_cache:
                 body = b"".join(body_chunks)
                 await redis_cache.set(
                     cache_key,
@@ -237,6 +275,16 @@ async def send_wrapper(message):
                     ttl=self.ttl_seconds,
                 )
                 cached_response = True
+            elif 200 <= status_code < 300 and skip_response_cache:
+                await redis_cache.set(
+                    cache_key,
+                    {
+                        "request_hash": fingerprint,
+                        "status": _COMPLETED_UNREPLAYABLE,
+                    },
+                    ttl=self.ttl_seconds,
+                )
+                cached_response = True
         finally:
             if not cached_response:
                 await redis_cache.delete(cache_key)