diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 44d8de2f..03ff1421 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -38,6 +38,25 @@ jobs: - name: Biome check run: pnpm exec biome check . + repo-check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: astral-sh/setup-uv@v5 + with: + enable-cache: true + cache-dependency-glob: api/uv.lock + + - name: Install API dependencies + run: uv sync --dev + working-directory: api + + - uses: ./.github/actions/setup-node-pnpm + + - name: Run root check + run: pnpm check + sdk-typecheck: runs-on: ubuntu-latest steps: diff --git a/AGENTS.md b/AGENTS.md index a3a1db56..0eefdf6c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ The repo follows aggressive package-style splits. If a single file grows past ~3 - `api/bigrag/app_factory/` (lifespan, exception_handlers, routers) - `api/bigrag/mcp/` (tools, unscoped, scoped, cli) - `api/bigrag/services/{embedding,retrieval,webhook,vector_store,storage,url_security,access_log,event_bus,queue_conversion,queue_embedding,chat,runtime_setting_specs}/` packages -- `sdks/python/src/bigrag/resources/admin/` and `sdks/typescript/src/resources/admin/` (settings, backups, realtime, users, api_keys, access, audit, connectors, embedding_presets, mcp_servers) +- `sdks/python/src/bigrag/resources/admin/` and `sdks/typescript/src/resources/admin/` (settings, users, api_keys, access, audit, connectors, embedding_presets, mcp_servers, vector_storage) When adding new code, prefer the smallest meaningful module instead of dropping it into the nearest catch-all. @@ -94,7 +94,7 @@ Use lint, typecheck, build, compile, and runtime smoke checks for current verifi - Tenant scoping: route handlers call `enforce_collection_pin(user, collection_name)` from `routers/__init__.py` to honor pinned API keys. Connector accounts/sources carry an optional `tenant_id` column. - Background workers: Dramatiq actors live in `services/jobs/`. The ingestion queue (`services/queue.py`) drains via `bigrag-worker`. - MCP server: `api/bigrag/mcp/` package (`tools`, `unscoped`, `scoped`, `cli`); entry point `bigrag-mcp = "bigrag.mcp:cli"`. -- SDK uses resource namespaces: `client.collections.list()`, `client.documents.upload()`, `client.admin.users.list()` etc. The `admin` resource is itself a package — sub-resources live in `resources/admin/{settings,backups,realtime,users,api_keys,access,audit,connectors,embedding_presets,mcp_servers}.py`. +- SDK uses resource namespaces: `client.collections.list()`, `client.documents.upload()`, `client.admin.users.list()` etc. The `admin` resource is itself a package — sub-resources live in `resources/admin/{settings,users,api_keys,access,audit,connectors,embedding_presets,mcp_servers,vector_storage}.py`. - SDK reliability: both Python and TypeScript SDKs send `Idempotency-Key: ` on POST/PUT/PATCH/DELETE and only retry mutating calls when an idempotency key is present. Both expose typed error subclasses (`BadRequestError`, `ConflictError`, `PayloadTooLargeError`, `UnprocessableEntityError`, `BadGatewayError`, `ServiceUnavailableError`, etc.). ## Code conventions diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8e44a3bb..8262ce8e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -83,18 +83,11 @@ bigrag/ ### Verifying Changes ```bash -# Website build check -pnpm --filter @bigrag/docs build - -# SDK and app build checks -pnpm --filter @bigrag/client build -pnpm --filter @bigrag/app build - -# Lint everything -pnpm lint # TypeScript (Biome) -cd api && uv run ruff check . && uv run ruff format --check . # Python +pnpm check ``` +`pnpm check` runs Biome, Ruff, workspace typechecks, app / SDK / docs builds, and a Python backend compile pass. Use targeted package commands while iterating, then run the root check before opening a PR. + ### Commit Messages Use [Conventional Commits](https://www.conventionalcommits.org/): @@ -124,7 +117,7 @@ When cutting a coordinated platform release, keep the API package, SDK packages, ### PR Requirements -- All CI checks must pass (lint, biome, sdk-typecheck, website-build, app-build) +- All CI checks must pass, including the root `pnpm check` job - At least one maintainer approval - No merge conflicts with `main` diff --git a/README.md b/README.md index ab68136d..059e126c 100644 --- a/README.md +++ b/README.md @@ -51,20 +51,35 @@ This starts the bigRAG API, worker, admin UI, Postgres, and Redis. Open **[local > [!IMPORTANT] > Configure Turbopuffer from onboarding before ingesting or querying collections. -Once Turbopuffer is configured, you can drive everything over HTTP: +Once Turbopuffer is configured, create the first admin and mint an API key for HTTP clients: ```bash +export BASE="http://localhost:4000" + +curl -X POST "$BASE/v1/auth/setup" \ + -H "Content-Type: application/json" \ + -c cookies.txt \ + -d '{"email": "admin@example.com", "password": "a-strong-password", "display_name": "Admin"}' + +export BIGRAG_API_KEY=$(curl -s -X POST "$BASE/v1/admin/api-keys" \ + -b cookies.txt \ + -H "Content-Type: application/json" \ + -d '{"name": "local-dev", "scopes": ["*:*"]}' | jq -r .key) + # Create a collection -curl -X POST http://localhost:4000/v1/collections \ +curl -X POST "$BASE/v1/collections" \ + -H "Authorization: Bearer $BIGRAG_API_KEY" \ -H "Content-Type: application/json" \ -d '{"name": "docs", "embedding_api_key": "sk-..."}' # Upload a document -curl -X POST http://localhost:4000/v1/collections/docs/documents \ +curl -X POST "$BASE/v1/collections/docs/documents" \ + -H "Authorization: Bearer $BIGRAG_API_KEY" \ -F "file=@paper.pdf" # Query -curl -X POST http://localhost:4000/v1/collections/docs/query \ +curl -X POST "$BASE/v1/collections/docs/query" \ + -H "Authorization: Bearer $BIGRAG_API_KEY" \ -H "Content-Type: application/json" \ -d '{"query": "What are the main findings?"}' ``` @@ -78,11 +93,11 @@ curl -X POST http://localhost:4000/v1/collections/docs/query \ ### Docker Images ```bash -docker pull yoginth/bigrag-api:2026.4.30 -docker pull yoginth/bigrag-ui:2026.4.30 +docker pull yoginth/bigrag-api:latest +docker pull yoginth/bigrag-ui:latest ``` -Release artifacts use CalVer (`YYYY.M.D`). Docker also publishes `latest`; the Python and TypeScript SDKs publish dated PyPI and npm releases. +Release artifacts use CalVer (`YYYY.M.D`). Docker publishes `latest` for quick starts; pin a dated tag from the release you deploy in production. ## Architecture @@ -148,8 +163,6 @@ graph TD | `DELETE` | `/v1/collections/{name}` | Delete collection | | `GET` | `/v1/collections/{name}/stats` | Collection stats | | `POST` | `/v1/collections/{name}/truncate` | Delete all documents, keep the collection | -| `POST` | `/v1/collections/{name}/realtime-token` | Create a short-lived collection realtime token | -| `WS` | `/v1/realtime` | Subscribe to realtime snapshots and collection events | | **Documents** | | | | `POST` | `/v1/collections/{name}/documents` | Upload document | | `GET` | `/v1/collections/{name}/documents` | List documents | @@ -321,7 +334,7 @@ Bootstrap settings use the `BIGRAG_` prefix as environment variables, or configu |----------|-------------|---------| | `BIGRAG_DATABASE_URL` | Postgres URL (`postgres:5432` inside docker-compose, `localhost:5432` for bare-metal dev) | `postgres://bigrag:bigrag@localhost:5432/bigrag?sslmode=disable` | | `BIGRAG_DB_POOL_MIN` | Min Postgres pool size | `5` | -| `BIGRAG_DB_POOL_MAX` | Max Postgres pool size | `10` | +| `BIGRAG_DB_POOL_MAX` | Max Postgres pool size | `20` | | `BIGRAG_MIGRATION_TIMEOUT_SECONDS` | Startup migration check timeout (`0` disables the timeout) | `60` | | `BIGRAG_REDIS_URL` | Redis URL | `redis://localhost:6379/0` | diff --git a/STYLEGUIDE.md b/STYLEGUIDE.md index 31de4792..4ef6ceae 100644 --- a/STYLEGUIDE.md +++ b/STYLEGUIDE.md @@ -2569,22 +2569,20 @@ if (lastError.name === "TimeoutError" || lastError.name === "AbortError") { } ``` -### Realtime Streaming +### Paginated Iteration -Use async generators for WebSocket realtime subscriptions: +Use async generators for SDK pagination helpers: ```typescript -async *streamEvents(name: string): AsyncGenerator { - const connection = new BigRAGRealtimeConnection(this._client); - try { - for await (const message of connection.subscribe("collection.events", { - collection: name, - })) { - if (message.type === "event") yield message.payload; +async *listAllDocuments(collection: string): AsyncGenerator { + let cursor: string | undefined; + do { + const page = await this.list(collection, { cursor }); + for (const document of page.documents) { + yield document; } - } finally { - await connection.close(); - } + cursor = page.next_cursor ?? undefined; + } while (cursor); } ``` diff --git a/api/bigrag/logging.py b/api/bigrag/logging.py index 9789ec86..5aca8312 100644 --- a/api/bigrag/logging.py +++ b/api/bigrag/logging.py @@ -8,6 +8,7 @@ from bigrag.logging_redaction import ( is_sensitive_log_key, redact_secrets, + safe_url_value, truncate_log_value, ) from bigrag.logging_rendering import console_renderer @@ -29,6 +30,7 @@ "current_worker_label", "get_logger", "is_sensitive_log_key", + "safe_url_value", "truncate_log_value", ] diff --git a/api/bigrag/logging_redaction.py b/api/bigrag/logging_redaction.py index 2eb4786c..a80ca805 100644 --- a/api/bigrag/logging_redaction.py +++ b/api/bigrag/logging_redaction.py @@ -1,6 +1,7 @@ from __future__ import annotations import re +from urllib.parse import SplitResult, parse_qsl, urlencode, urlsplit, urlunsplit _SENSITIVE_KEYS = frozenset( { @@ -87,6 +88,34 @@ def truncate_log_value(value: str) -> str: return f"{value[:_MAX_LOG_VALUE_LENGTH]}..." +def safe_url_value(value: str) -> str: + try: + parts = urlsplit(value) + except ValueError: + return truncate_log_value(value) + if not parts.scheme or not parts.netloc: + return truncate_log_value(value) + query = urlencode([(key, "[REDACTED]") for key, _ in parse_qsl(parts.query, True)]) + fragment = "[REDACTED]" if parts.fragment else "" + return truncate_log_value( + urlunsplit((parts.scheme, _safe_url_netloc(parts), parts.path, query, fragment)) + ) + + +def _safe_url_netloc(parts: SplitResult) -> str: + if not parts.netloc or (parts.username is None and parts.password is None): + return parts.netloc + hostname = parts.hostname or "" + if ":" in hostname and not hostname.startswith("["): + hostname = f"[{hostname}]" + try: + port = parts.port + except ValueError: + port = None + suffix = f":{port}" if port is not None else "" + return f"[REDACTED]@{hostname}{suffix}" + + def log_field_value(value: object) -> str: return truncate_log_value(escape_control_characters(str(value))) @@ -114,5 +143,7 @@ def redact_log_value(value: object) -> object: if isinstance(value, tuple): return tuple(redact_log_value(item) for item in value) if isinstance(value, str): + if "://" in value: + return safe_url_value(value) return truncate_log_value(value) return value diff --git a/api/bigrag/middleware/idempotency.py b/api/bigrag/middleware/idempotency.py index 3a5c74d7..eea2f3aa 100644 --- a/api/bigrag/middleware/idempotency.py +++ b/api/bigrag/middleware/idempotency.py @@ -16,7 +16,16 @@ _MAX_REQUEST_BODY_BYTES = 8 * 1024 * 1024 _SENSITIVE_RESPONSE_HEADERS = frozenset({"content-length", "set-cookie"}) _IN_FLIGHT_SENTINEL = "__in_flight__" +_COMPLETED_UNREPLAYABLE = "__completed_unreplayable__" _IN_FLIGHT_TTL_SECONDS = 60 +_UNCACHEABLE_RESPONSE_ROUTES: tuple[tuple[str, str], ...] = ( + ("POST", "/v1/admin/api-keys"), + ("POST", "/v1/admin/api-keys/"), + ("POST", "/v1/admin/mcp-servers"), + ("POST", "/v1/admin/mcp-servers/"), + ("POST", "/v1/admin/webhooks"), + ("POST", "/v1/admin/webhooks/"), +) def _cache_key(principal: str, idem_key: str, method: str, path: str) -> str: @@ -55,6 +64,13 @@ def _should_skip_body(scope) -> bool: return content_length is not None and content_length > _MAX_CACHED_BODY_BYTES +def _should_skip_response_cache(method: str, path: str) -> bool: + for route_method, route_prefix in _UNCACHEABLE_RESPONSE_ROUTES: + if method == route_method and path.startswith(route_prefix): + return True + return False + + async def _read_request_body(receive) -> tuple[bytes, bool, list[dict]]: chunks: list[bytes] = [] messages: list[dict] = [] @@ -123,6 +139,21 @@ async def _send_in_flight(send) -> None: await send({"type": "http.response.body", "body": body}) +async def _send_unreplayable(send) -> None: + body = orjson.dumps({"detail": "Idempotency-Key already completed for a one-time response"}) + await send( + { + "type": "http.response.start", + "status": 409, + "headers": [ + (b"content-type", b"application/json"), + (b"content-length", str(len(body)).encode("ascii")), + ], + } + ) + await send({"type": "http.response.body", "body": body}) + + class IdempotencyMiddleware: def __init__(self, app, ttl_seconds: int = _DEFAULT_TTL_SECONDS) -> None: self.app = app @@ -151,6 +182,7 @@ async def __call__(self, scope, receive, send): method = scope["method"] path = scope.get("path", "") + skip_response_cache = _should_skip_response_cache(method, path) principal = principal_id(scope) cache_key = _cache_key(principal, idem_key, method, path) request_body, can_fingerprint, messages = await _read_request_body(receive) @@ -164,6 +196,9 @@ async def __call__(self, scope, receive, send): if cached.get("status") == _IN_FLIGHT_SENTINEL: await _send_in_flight(send) return + if cached.get("status") == _COMPLETED_UNREPLAYABLE: + await _send_unreplayable(send) + return if cached.get("request_hash") != fingerprint: await _send_conflict(send) return @@ -186,6 +221,9 @@ async def __call__(self, scope, receive, send): await _send_in_flight(send) return if existing: + if existing.get("status") == _COMPLETED_UNREPLAYABLE: + await _send_unreplayable(send) + return if existing.get("request_hash") != fingerprint: await _send_conflict(send) return @@ -220,7 +258,7 @@ async def send_wrapper(message): cached_response = False try: await self.app(scope, _replay_receive(messages), send_wrapper) - if 200 <= status_code < 300 and cacheable_body: + if 200 <= status_code < 300 and cacheable_body and not skip_response_cache: body = b"".join(body_chunks) await redis_cache.set( cache_key, @@ -237,6 +275,16 @@ async def send_wrapper(message): ttl=self.ttl_seconds, ) cached_response = True + elif 200 <= status_code < 300 and skip_response_cache: + await redis_cache.set( + cache_key, + { + "request_hash": fingerprint, + "status": _COMPLETED_UNREPLAYABLE, + }, + ttl=self.ttl_seconds, + ) + cached_response = True finally: if not cached_response: await redis_cache.delete(cache_key) diff --git a/api/bigrag/middleware/request_logging.py b/api/bigrag/middleware/request_logging.py index cf671774..39d58c6c 100644 --- a/api/bigrag/middleware/request_logging.py +++ b/api/bigrag/middleware/request_logging.py @@ -2,7 +2,7 @@ import time import uuid -from urllib.parse import SplitResult, parse_qsl, urlencode, urlsplit, urlunsplit +from urllib.parse import parse_qsl import structlog from starlette.datastructures import MutableHeaders @@ -12,6 +12,7 @@ REQUEST_ID_HEADER, get_logger, is_sensitive_log_key, + safe_url_value, truncate_log_value, ) @@ -31,34 +32,7 @@ def _request_id_from_scope(scope: Scope) -> str: def _safe_url(value: str) -> str: - try: - parts = urlsplit(value) - except ValueError: - return truncate_log_value(value) - query = urlencode( - [ - (key, "[REDACTED]" if is_sensitive_log_key(key) else truncate_log_value(param_value)) - for key, param_value in parse_qsl(parts.query, keep_blank_values=True) - ] - ) - fragment = "[REDACTED]" if parts.fragment else "" - return truncate_log_value( - urlunsplit((parts.scheme, _safe_netloc(parts), parts.path, query, fragment)) - ) - - -def _safe_netloc(parts: SplitResult) -> str: - if not parts.netloc or (parts.username is None and parts.password is None): - return parts.netloc - hostname = parts.hostname or "" - if ":" in hostname and not hostname.startswith("["): - hostname = f"[{hostname}]" - try: - port = parts.port - except ValueError: - port = None - suffix = f":{port}" if port is not None else "" - return f"[REDACTED]@{hostname}{suffix}" + return safe_url_value(value) def _safe_value(key: str, value: str) -> str: diff --git a/api/bigrag/models/auth.py b/api/bigrag/models/auth.py index 3c338bed..ebd184ba 100644 --- a/api/bigrag/models/auth.py +++ b/api/bigrag/models/auth.py @@ -138,9 +138,9 @@ class CreateApiKeyRequest(BaseModel): scopes: list[str] | None = Field( default=None, description=( - "List of 'resource:action' strings. Omit for a " - "full-access key. Examples: ['collection:read', " - "'document:upload'] — or ['*:*'] for unrestricted." + "List of 'resource:action' strings. Omit or pass [] to " + "store ['*:*'] for a full-access key. Examples: " + "['collection:read', 'document:upload']." ), ) collection: str | None = Field( diff --git a/api/bigrag/models/collection.py b/api/bigrag/models/collection.py index cee4226f..5ab6c185 100644 --- a/api/bigrag/models/collection.py +++ b/api/bigrag/models/collection.py @@ -43,7 +43,7 @@ class CreateCollectionRequest(BaseModel): reranking_api_key: str | None = None multimodal_enabled: bool = False multimodal_enrichment_enabled: bool = False - default_top_k: int = Field(default=10, ge=1, le=1000) + default_top_k: int = Field(default=10, ge=1, le=200) default_min_score: float | None = None default_search_mode: str = Field(default="semantic", pattern=r"^(semantic|keyword|hybrid)$") @@ -65,7 +65,7 @@ class UpdateCollectionRequest(BaseModel): reranking_api_key: str | None = None multimodal_enabled: bool | None = None multimodal_enrichment_enabled: bool | None = None - default_top_k: int | None = Field(default=None, ge=1, le=1000) + default_top_k: int | None = Field(default=None, ge=1, le=200) default_min_score: float | None = None default_search_mode: str | None = Field(default=None, pattern=r"^(semantic|keyword|hybrid)$") chunk_strategy: str | None = Field(default=None, pattern=r"^(paragraph|recursive)$") diff --git a/api/bigrag/routers/admin_api_keys.py b/api/bigrag/routers/admin_api_keys.py index 3d9d3d81..7f6be8dd 100644 --- a/api/bigrag/routers/admin_api_keys.py +++ b/api/bigrag/routers/admin_api_keys.py @@ -59,6 +59,10 @@ def _validate_scopes(scopes: list[str] | None) -> None: validate_scope_string(s) +def _stored_scopes(scopes: list[str] | None) -> list[str]: + return scopes or ["*:*"] + + @router.get("", response_model=ApiKeyListResponse) async def list_api_keys( limit: int = Query(default=50, ge=1, le=200), @@ -105,8 +109,8 @@ async def create_api_key( collection = await validate_collection_name(session, body.collection) permissions: dict = {} - if body.scopes: - permissions["scopes"] = body.scopes + scopes = _stored_scopes(body.scopes) + permissions["scopes"] = scopes if collection: permissions["collection"] = collection plaintext, prefix, key_hash = generate_api_key() @@ -132,7 +136,7 @@ async def create_api_key( resource_id=str(key.id), metadata={ "name": body.name, - "scopes": body.scopes or [], + "scopes": scopes, "collection": collection, }, ) @@ -173,10 +177,7 @@ async def update_api_key( raise HTTPException( status_code=400, detail=safe_error_detail(e, "Invalid scopes.") ) from e - if body.scopes: - existing["scopes"] = body.scopes - else: - existing.pop("scopes", None) + existing["scopes"] = _stored_scopes(body.scopes) fields.append("scopes") if body.collection is not None: collection = await validate_collection_name(session, body.collection) diff --git a/api/bigrag/routers/auth.py b/api/bigrag/routers/auth.py index 9a0771db..c1cd5566 100644 --- a/api/bigrag/routers/auth.py +++ b/api/bigrag/routers/auth.py @@ -1,6 +1,7 @@ from __future__ import annotations import asyncio +import hashlib import uuid import sqlalchemy as sa @@ -34,7 +35,7 @@ lock_setup, set_session_cookie, ) -from bigrag.services import audit +from bigrag.services import audit, redis_cache from bigrag.services.auth import ( DUMMY_PASSWORD_HASH, hash_password, @@ -42,10 +43,49 @@ needs_rehash, verify_password, ) +from bigrag.services.client_ip import client_ip logger = get_logger("bigrag.routers.auth") router = APIRouter(prefix="/v1/auth", tags=["auth"]) +_AUTH_THROTTLE_WINDOW_SECONDS = 15 * 60 +_AUTH_THROTTLE_MAX_FAILURES = 10 + + +def _auth_throttle_key(request: Request, action: str, identity: str) -> str: + source = client_ip(request) or "unknown" + digest = hashlib.sha256(f"{action}|{source}|{identity}".encode()).hexdigest() + return f"bigrag:auth:throttle:{digest}" + + +async def _raise_if_auth_limited(request: Request, action: str, identity: str) -> None: + redis = redis_cache.get_redis() + if redis is None: + return + raw = await redis.get(_auth_throttle_key(request, action, identity)) + try: + attempts = int(raw or 0) + except (TypeError, ValueError): + attempts = 0 + if attempts >= _AUTH_THROTTLE_MAX_FAILURES: + raise HTTPException(status_code=429, detail="Too many attempts. Try again later.") + + +async def _record_auth_failure(request: Request, action: str, identity: str) -> None: + redis = redis_cache.get_redis() + if redis is None: + return + key = _auth_throttle_key(request, action, identity) + attempts = await redis.incr(key) + if attempts == 1: + await redis.expire(key, _AUTH_THROTTLE_WINDOW_SECONDS) + + +async def _clear_auth_failures(request: Request, action: str, identity: str) -> None: + redis = redis_cache.get_redis() + if redis is None: + return + await redis.delete(_auth_throttle_key(request, action, identity)) @router.get("/setup-status", response_model=SetupStatusResponse) @@ -63,14 +103,17 @@ async def setup( response: Response, session: AsyncSession = Depends(get_session), ) -> SessionResponse: + email = body.email.lower() + await _raise_if_auth_limited(request, "setup", email) await lock_setup(session) existing = await session.scalar(sa.select(sa.func.count()).select_from(User)) if existing > 0: + await _record_auth_failure(request, "setup", email) raise HTTPException(status_code=409, detail="Setup has already been completed") user = User( id=uuid7(), - email=body.email.lower(), + email=email, password_hash=await asyncio.to_thread(hash_password, body.password), display_name=body.display_name, role="admin", @@ -83,7 +126,8 @@ async def setup( await session.refresh(user) await set_session_cookie(response, token) - logger.info("first admin created", email=body.email) + await _clear_auth_failures(request, "setup", email) + logger.info("first admin created", email=email) audit.record( request, user={"id": str(user.id), "email": user.email}, @@ -103,6 +147,7 @@ async def login( session: AsyncSession = Depends(get_session), ) -> SessionResponse: email = body.email.lower() + await _raise_if_auth_limited(request, "login", email) user = await session.scalar(sa.select(User).where(User.email == email)) if user is None: await asyncio.to_thread(verify_password, body.password, DUMMY_PASSWORD_HASH) @@ -110,6 +155,7 @@ async def login( else: password_ok = await asyncio.to_thread(verify_password, body.password, user.password_hash) if user is None or not password_ok: + await _record_auth_failure(request, "login", email) audit.record( request, user={"id": None, "email": email}, @@ -128,6 +174,7 @@ async def login( await session.commit() await session.refresh(user) await set_session_cookie(response, token) + await _clear_auth_failures(request, "login", email) audit.record( request, user={"id": str(user.id), "email": user.email}, diff --git a/api/bigrag/routers/documents/crud.py b/api/bigrag/routers/documents/crud.py index aa8270e4..6c9beccc 100644 --- a/api/bigrag/routers/documents/crud.py +++ b/api/bigrag/routers/documents/crud.py @@ -24,6 +24,7 @@ from bigrag.services import audit, collection_cache from bigrag.services.document_progress import document_progress, publish_queued_progress from bigrag.services.documents import ( + check_document_tenant, content_hash_match, document_response, get_document_payload, @@ -168,6 +169,7 @@ async def delete_document( ) if doc is None: raise HTTPException(status_code=404, detail="Document not found") + check_document_tenant(user, doc, collection) await ingestion_queue.cancel_documents([document_id]) diff --git a/api/bigrag/routers/documents/elements.py b/api/bigrag/routers/documents/elements.py index fde42cbc..f9826dc3 100644 --- a/api/bigrag/routers/documents/elements.py +++ b/api/bigrag/routers/documents/elements.py @@ -27,23 +27,26 @@ async def get_document_elements( enforce_collection_pin(user, collection_name) collection = await get_collection_or_404(collection_name) doc_id = uuid_or_404(document_id, "Document") - exists = await session.scalar( - sa.select(Document.id) + doc = await session.scalar( + sa.select(Document) .where(Document.id == doc_id) .where(Document.collection_id == collection["id"]) ) - if exists is None: + if doc is None: raise HTTPException(status_code=404, detail="Document not found") + check_document_tenant(user, doc, collection) total = await session.scalar( sa.select(sa.func.count()) .select_from(DocumentElement) .where(DocumentElement.document_id == doc_id) + .where(DocumentElement.collection_id == collection["id"]) ) rows = ( await session.scalars( sa.select(DocumentElement) .where(DocumentElement.document_id == doc_id) + .where(DocumentElement.collection_id == collection["id"]) .order_by(DocumentElement.element_index.asc()) .limit(limit) .offset(offset) diff --git a/api/bigrag/routers/documents/listing.py b/api/bigrag/routers/documents/listing.py index 1dcaa496..62a91c25 100644 --- a/api/bigrag/routers/documents/listing.py +++ b/api/bigrag/routers/documents/listing.py @@ -37,4 +37,5 @@ async def list_documents( offset=offset, cursor=cursor, include_total=include_total, + user=user, ) diff --git a/api/bigrag/routers/query.py b/api/bigrag/routers/query.py index 290ae680..9ad86f50 100644 --- a/api/bigrag/routers/query.py +++ b/api/bigrag/routers/query.py @@ -174,6 +174,7 @@ async def multi_collection_query( embedding_models = {} reranking_configs = {} + filters_by_collection = {} resolve_semaphore = asyncio.Semaphore(_FANOUT_LIMIT) async def _resolve(col_name: str): @@ -183,10 +184,13 @@ async def _resolve(col_name: str): resolved_collections = await asyncio.gather( *[_resolve(col_name) for col_name in body.collections] ) - multi_filters = body.filters for col_name, collection in zip(body.collections, resolved_collections, strict=True): enforce_collection_pin(principal, col_name) - multi_filters = enforce_tenant_filters(collection, multi_filters, principal) + filters_by_collection[col_name] = enforce_tenant_filters( + collection, + body.filters, + principal, + ) embedding_models[col_name] = resolve_embedding_model( collection, error_label=f"Collection '{col_name}'", @@ -202,7 +206,8 @@ async def _resolve(col_name: str): query=body.query, embedding_models=embedding_models, top_k=body.top_k, - filters=multi_filters, + filters=body.filters, + filters_by_collection=filters_by_collection, min_score=body.min_score, search_mode=body.search_mode, reranking_configs=reranking_configs, diff --git a/api/bigrag/services/access_log/payload.py b/api/bigrag/services/access_log/payload.py index 2fbc8ae4..2c248913 100644 --- a/api/bigrag/services/access_log/payload.py +++ b/api/bigrag/services/access_log/payload.py @@ -4,6 +4,8 @@ from collections.abc import Mapping from typing import Any +from bigrag.logging import safe_url_value + _MAX_METADATA_DEPTH = 4 _MAX_METADATA_ITEMS = 32 _MAX_STRING_LEN = 300 @@ -26,6 +28,8 @@ def _safe_scalar(value: Any) -> Any: if value is None or isinstance(value, bool | int | float): return value if isinstance(value, str): + if "://" in value: + return safe_url_value(value) if len(value) <= _MAX_STRING_LEN: return value return f"{value[:_MAX_STRING_LEN]}..." diff --git a/api/bigrag/services/audit.py b/api/bigrag/services/audit.py index ef596b48..ffdba7a7 100644 --- a/api/bigrag/services/audit.py +++ b/api/bigrag/services/audit.py @@ -13,6 +13,7 @@ from bigrag.ids import uuid7 from bigrag.logging import get_logger from bigrag.models.auth import AuditLogEntry, AuditLogListResponse +from bigrag.services.access_log.payload import _safe_metadata from bigrag.services.pagination import paginate logger = get_logger("bigrag.audit") @@ -256,7 +257,7 @@ def record( action=action, resource_type=resource_type, resource_id=resource_id, - metadata=metadata or {}, + metadata=_safe_metadata(metadata or {}), ip=ip, user_agent=user_agent, ) diff --git a/api/bigrag/services/collection_scope.py b/api/bigrag/services/collection_scope.py index 1f893f1a..b868c916 100644 --- a/api/bigrag/services/collection_scope.py +++ b/api/bigrag/services/collection_scope.py @@ -35,9 +35,7 @@ def _extract_collection_name(path: str) -> str | None: def assert_collection_matches_pin(pinned: str, target: str) -> None: if target != pinned: - raise ForbiddenError( - f"This API key is pinned to collection {pinned!r}; request targeted {target!r}." - ) + raise ForbiddenError("This API key cannot access the requested collection.") async def enforce_collection_scope(request: Request, pinned: str) -> None: @@ -46,10 +44,7 @@ async def enforce_collection_scope(request: Request, pinned: str) -> None: stripped = path.rstrip("/") if (method, stripped) in _FORBIDDEN_FOR_SCOPED_SET: - raise ForbiddenError( - f"This API key is pinned to collection {pinned!r} and cannot " - "use cross-collection endpoints." - ) + raise ForbiddenError("This API key cannot use cross-collection endpoints.") target = _extract_collection_name(path) if target is not None: @@ -60,7 +55,4 @@ async def enforce_collection_scope(request: Request, pinned: str) -> None: len(parts) == 2 and parts[1] == "collections" ) if is_collection_root and method in _FORBIDDEN_METHODS_ON_PINNED_COLLECTION: - raise ForbiddenError( - f"This API key is pinned to collection {pinned!r}; reconfiguring or " - "deleting collections is not allowed." - ) + raise ForbiddenError("This API key cannot reconfigure or delete collections.") diff --git a/api/bigrag/services/documents/__init__.py b/api/bigrag/services/documents/__init__.py index 0f168710..f6e7cfe9 100644 --- a/api/bigrag/services/documents/__init__.py +++ b/api/bigrag/services/documents/__init__.py @@ -14,13 +14,18 @@ list_documents_payload, ) from bigrag.services.documents.serialize import document_response -from bigrag.services.documents.tenant import check_document_tenant, prepare_document_metadata +from bigrag.services.documents.tenant import ( + check_document_tenant, + document_tenant_metadata_filter, + prepare_document_metadata, +) __all__ = [ "SUPPORTED_EXTENSIONS", "UploadBudget", "check_document_tenant", "content_hash_match", + "document_tenant_metadata_filter", "document_response", "get_document_payload", "get_document_with_collection", diff --git a/api/bigrag/services/documents/queries.py b/api/bigrag/services/documents/queries.py index f43049ec..86da7e1a 100644 --- a/api/bigrag/services/documents/queries.py +++ b/api/bigrag/services/documents/queries.py @@ -11,7 +11,7 @@ from bigrag.services.collection_cache import get_or_404 as get_collection_or_404 from bigrag.services.document_progress import document_progress, document_progress_map from bigrag.services.documents.serialize import document_response -from bigrag.services.documents.tenant import check_document_tenant +from bigrag.services.documents.tenant import check_document_tenant, document_tenant_metadata_filter from bigrag.services.pagination import paginate from bigrag.services.tenant_enforcement import tenant_field @@ -37,6 +37,7 @@ async def list_documents_payload( offset: int, cursor: str | None, include_total: bool, + user: dict, ) -> DocumentListResponse: collection = await get_collection_or_404(collection_name) sort_column = _DOCUMENT_SORT_COLUMNS.get(sort) @@ -72,6 +73,10 @@ async def list_documents_payload( if status: stmt = stmt.where(Document.status == status) count_stmt = count_stmt.where(Document.status == status) + tenant_filter = document_tenant_metadata_filter(user, collection) + if tenant_filter is not None: + stmt = stmt.where(Document.meta.contains(tenant_filter)) + count_stmt = count_stmt.where(Document.meta.contains(tenant_filter)) if cursor and sort != "created_at": raise HTTPException( diff --git a/api/bigrag/services/documents/tenant.py b/api/bigrag/services/documents/tenant.py index a85d4638..dcc05987 100644 --- a/api/bigrag/services/documents/tenant.py +++ b/api/bigrag/services/documents/tenant.py @@ -1,10 +1,14 @@ from __future__ import annotations from bigrag.db.models import Document +from bigrag.exceptions import ValidationError from bigrag.services import metadata_schema from bigrag.services.tenant_enforcement import ( enforce_document_tenant_access, + is_admin_org_global, + principal_tenant_id, require_tenant_metadata, + tenant_field, ) @@ -16,3 +20,18 @@ def prepare_document_metadata(collection: dict, metadata: dict) -> dict: def check_document_tenant(user: dict, doc: Document, collection: dict) -> None: enforce_document_tenant_access(user, collection, doc.meta) + + +def document_tenant_metadata_filter(user: dict, collection: dict) -> dict | None: + field = tenant_field(collection) + if not field: + return None + tenant = principal_tenant_id(user) + if tenant is not None: + return {field: tenant} + if is_admin_org_global(user): + return None + raise ValidationError( + f"This API key is not scoped to a tenant; access to collection " + f"{collection.get('name')!r} (tenant_field {field!r}) is denied" + ) diff --git a/api/bigrag/services/ingestion_job.py b/api/bigrag/services/ingestion_job.py index 6fcd7788..abe4c128 100644 --- a/api/bigrag/services/ingestion_job.py +++ b/api/bigrag/services/ingestion_job.py @@ -25,6 +25,7 @@ class IngestionJob: document_epoch: int = 0 attempt: int = 0 max_attempts: int = 3 + admission_released: bool = False job_id: str = field(default_factory=lambda: uuid.uuid4().hex[:8]) @property @@ -51,6 +52,7 @@ def serialize(self) -> bytes: "tenant_field": self.tenant_field, "attempt": self.attempt, "max_attempts": self.max_attempts, + "admission_released": self.admission_released, "job_id": self.job_id, } ) diff --git a/api/bigrag/services/jobs/actors.py b/api/bigrag/services/jobs/actors.py index 4e002e9f..91d1c8bc 100644 --- a/api/bigrag/services/jobs/actors.py +++ b/api/bigrag/services/jobs/actors.py @@ -102,6 +102,10 @@ async def _process_ingestion_job(payload: str) -> None: job = IngestionJob.deserialize(payload.encode()) logger.debug("ingestion actor received job", job=job.job_id, doc=job.document_id) if await is_active(): + await queue.ingestion_queue.defer_admitted_job(job) + enqueue_ingestion_job(job, delay_seconds=10) + return + if not await queue.ingestion_queue.restore_deferred_admission(job): enqueue_ingestion_job(job, delay_seconds=10) return await queue.ingestion_queue.process_leased_job(current_worker_label(), job) diff --git a/api/bigrag/services/queue.py b/api/bigrag/services/queue.py index b8126316..201eee79 100644 --- a/api/bigrag/services/queue.py +++ b/api/bigrag/services/queue.py @@ -92,6 +92,21 @@ async def _admit_job(self) -> None: async def release_job(self) -> None: await queue_state.release_inflight(self._redis) + async def defer_admitted_job(self, job: IngestionJob) -> None: + if not job.admission_released: + await self.release_job() + job.admission_released = True + + async def restore_deferred_admission(self, job: IngestionJob) -> bool: + if not job.admission_released: + return True + try: + await self._admit_job() + except QueueFullError: + return False + job.admission_released = False + return True + async def enqueue(self, job: IngestionJob) -> None: from bigrag.services.jobs.actors import enqueue_ingestion_job from bigrag.services.maintenance import MaintenanceActiveError, ensure_writes_allowed @@ -149,6 +164,11 @@ async def stats(self) -> dict: ) stats = await queue_state.queue_stats(self._redis) + if stats.get("stale_processing"): + recovered = await self._recover_stuck_jobs() + if recovered: + stats = await queue_state.queue_stats(self._redis) + stats["recovered_stale_processing"] = recovered stats["pending"] = 0 stats["retrying"] = 0 try: diff --git a/api/bigrag/services/queue_embedding/embed_batches.py b/api/bigrag/services/queue_embedding/embed_batches.py index c9da8d0b..d17c74bd 100644 --- a/api/bigrag/services/queue_embedding/embed_batches.py +++ b/api/bigrag/services/queue_embedding/embed_batches.py @@ -15,7 +15,6 @@ logger = get_logger("bigrag.queue") BATCH_BACKOFF_BASE = 2 -EMBED_CONCURRENCY = 8 async def embed_all_batches( @@ -90,7 +89,9 @@ async def _embed_batch( continue raise - embed_sem = asyncio.Semaphore(EMBED_CONCURRENCY) + from bigrag.services.runtime_settings import get_value + + embed_sem = asyncio.Semaphore(int(await get_value("embedding_concurrency"))) async def _embed_batch_bounded(bn, bs, be, bc): async with embed_sem: diff --git a/api/bigrag/services/queue_recovery.py b/api/bigrag/services/queue_recovery.py index 7780d511..802c9851 100644 --- a/api/bigrag/services/queue_recovery.py +++ b/api/bigrag/services/queue_recovery.py @@ -18,23 +18,28 @@ async def recover_stuck_jobs(redis) -> int: jobs = await queue_state.recover_stuck_jobs(redis) if jobs: - await mark_recovered_jobs_pending(jobs) - for job in jobs: + recovered_jobs = await mark_recovered_jobs_pending(jobs) + for job in recovered_jobs: enqueue_ingestion_job(job) - await redis.hincrby(queue_state.STATS_KEY, "queued", len(jobs)) - logger.info("queue requeued stuck jobs", recovered=len(jobs)) - return len(jobs) + await redis.hincrby(queue_state.STATS_KEY, "queued", len(recovered_jobs)) + logger.info("queue requeued stuck jobs", recovered=len(recovered_jobs)) + return len(recovered_jobs) + return 0 -async def mark_recovered_jobs_pending(jobs: list[IngestionJob]) -> None: +async def mark_recovered_jobs_pending(jobs: list[IngestionJob]) -> list[IngestionJob]: ids = [uuid.UUID(job.document_id) for job in jobs] async with session_factory()() as session: - await session.execute( + result = await session.scalars( sa.update(Document) .where(Document.id.in_(ids)) + .where(Document.status.in_(("pending", "processing"))) .values( status="pending", error_message="Recovered stale processing lease; requeued.", ) + .returning(Document.id) ) + recovered_ids = {str(document_id) for document_id in result.all()} await session.commit() + return [job for job in jobs if job.document_id in recovered_ids] diff --git a/api/bigrag/services/retrieval/fusion.py b/api/bigrag/services/retrieval/fusion.py index 5a1f7b38..71ab0103 100644 --- a/api/bigrag/services/retrieval/fusion.py +++ b/api/bigrag/services/retrieval/fusion.py @@ -32,6 +32,7 @@ def reciprocal_rank_fusion( ) -> list[dict]: scores: dict[str, float] = {} items: dict[str, dict] = {} + max_score = len(ranked_lists) / (k + 1) if ranked_lists else 1.0 for ranked_list in ranked_lists: for rank, item in enumerate(ranked_list): @@ -44,7 +45,7 @@ def reciprocal_rank_fusion( result = [] for item_id in sorted_ids: item = items[item_id].copy() - item["score"] = round(scores[item_id], 6) + item["score"] = round(scores[item_id] / max_score, 6) result.append(item) return result diff --git a/api/bigrag/services/retrieval/multi.py b/api/bigrag/services/retrieval/multi.py index 06b69a49..6936ff39 100644 --- a/api/bigrag/services/retrieval/multi.py +++ b/api/bigrag/services/retrieval/multi.py @@ -14,6 +14,7 @@ async def retrieve_multi( embedding_models: dict[str, EmbeddingModel], top_k: int = 10, filters: dict | None = None, + filters_by_collection: dict[str, dict | None] | None = None, min_score: float | None = None, search_mode: str = "semantic", reranking_configs: dict[str, dict] | None = None, @@ -28,12 +29,17 @@ async def retrieve_multi( async def search_one(col_name: str) -> list[dict]: async with semaphore: col_reranking = (reranking_configs or {}).get(col_name) + col_filters = ( + filters_by_collection.get(col_name, filters) + if filters_by_collection is not None + else filters + ) outcome = await retrieve( collection_name=col_name, query=query, embedding_model=embedding_models[col_name], top_k=top_k, - filters=filters, + filters=col_filters, min_score=min_score, search_mode=search_mode, reranking_config=col_reranking, diff --git a/app/package.json b/app/package.json index 6ee1f6ab..7d8cda73 100644 --- a/app/package.json +++ b/app/package.json @@ -30,6 +30,7 @@ "remark-gfm": "4.0.1", "sonner": "^2.0.7", "tailwind-merge": "^3.6.0", + "ts-pattern": "5.9.0", "zustand": "^5.0.13" }, "devDependencies": { diff --git a/app/src/features/api-keys/api-key-form-state.ts b/app/src/features/api-keys/api-key-form-state.ts index e40b99e6..bbdfdfaf 100644 --- a/app/src/features/api-keys/api-key-form-state.ts +++ b/app/src/features/api-keys/api-key-form-state.ts @@ -1,3 +1,5 @@ +import { match } from "ts-pattern"; + export const API_KEY_UNSCOPED = "__all__"; export type ApiKeyFormValues = { @@ -43,14 +45,18 @@ export const apiKeyBodyFromValues = (values: ApiKeyFormValues) => ({ }); const scopesFromAccessLevel = ({ accessLevel, scopesText }: ApiKeyFormValues) => { - if (accessLevel === "full") return null; - if (accessLevel === "read") { - return ["collection:read", "document:read", "query:read", "chat:read"]; - } - if (accessLevel === "write") { - return ["collection:read", "document:read", "document:upload", "query:read", "chat:write"]; - } - return parseScopes(scopesText); + return match(accessLevel) + .with("full", () => ["*:*"]) + .with("read", () => ["collection:read", "document:read", "query:read", "chat:read"]) + .with("write", () => [ + "collection:read", + "document:read", + "document:upload", + "query:read", + "chat:write", + ]) + .with("custom", () => parseScopes(scopesText)) + .exhaustive(); }; const parseScopes = (value: string) => @@ -60,11 +66,13 @@ const parseScopes = (value: string) => .filter(Boolean); const expirationFromPreset = ({ customExpiresAt, expiresPreset }: ApiKeyFormValues) => { - if (expiresPreset === "never") return null; - if (expiresPreset === "custom") - return customExpiresAt ? new Date(customExpiresAt).toISOString() : null; - const days = Number.parseInt(expiresPreset, 10); - const expires = new Date(); - expires.setDate(expires.getDate() + days); - return expires.toISOString(); + return match(expiresPreset) + .with("never", () => null) + .with("custom", () => (customExpiresAt ? new Date(customExpiresAt).toISOString() : null)) + .otherwise((preset) => { + const days = Number.parseInt(preset, 10); + const expires = new Date(); + expires.setDate(expires.getDate() + days); + return expires.toISOString(); + }); }; diff --git a/app/src/features/chat/use-chat-page-controller.ts b/app/src/features/chat/use-chat-page-controller.ts index 1fbe9af8..38892ad0 100644 --- a/app/src/features/chat/use-chat-page-controller.ts +++ b/app/src/features/chat/use-chat-page-controller.ts @@ -9,14 +9,21 @@ import { useChatQuestionSuggestions, useGenerateChatQuestions } from "@/hooks/us import { useCollections } from "@/hooks/use-collections"; import { usePreferences, useUpdatePreferences } from "@/hooks/use-preferences"; -const useSelectFirstCollection = ( +const useSelectAvailableCollection = ( collections: { name: string }[], current: string, - selectFirstCollection: (collections: readonly { name: string }[]) => void, + selectCollection: (collection: string) => void, ) => { useEffect(() => { - if (!current) selectFirstCollection(collections); - }, [collections, current, selectFirstCollection]); + const first = collections[0]?.name ?? ""; + if (!current && first) { + selectCollection(first); + return; + } + if (current && !collections.some((item) => item.name === current)) { + selectCollection(first); + } + }, [collections, current, selectCollection]); }; export const useChatPageController = () => { @@ -32,7 +39,6 @@ export const useChatPageController = () => { isStreaming, messages, selectCollection, - selectFirstCollection, setMessages, setStreaming, updateMessage, @@ -44,14 +50,13 @@ export const useChatPageController = () => { isStreaming: state.isStreaming, messages: state.messages, selectCollection: state.selectCollection, - selectFirstCollection: state.selectFirstCollection, setMessages: state.setMessages, setStreaming: state.setStreaming, updateMessage: state.updateMessage, })), ); - useSelectFirstCollection(collections, collection, selectFirstCollection); + useSelectAvailableCollection(collections, collection, selectCollection); const state: ChatState = useMemo(() => { const chat = prefsQuery.data?.data.chat ?? {}; diff --git a/app/src/features/collections/collection-settings/retrieval-defaults-card.tsx b/app/src/features/collections/collection-settings/retrieval-defaults-card.tsx index 3a042acd..8fae9c08 100644 --- a/app/src/features/collections/collection-settings/retrieval-defaults-card.tsx +++ b/app/src/features/collections/collection-settings/retrieval-defaults-card.tsx @@ -48,7 +48,7 @@ export const RetrievalDefaultsCard = ({ label="Default top K" type="number" min={1} - max={100} + max={200} value={topK} onChange={(e) => onTopKChange(Number(e.target.value))} /> diff --git a/app/src/features/collections/document-detail-route.tsx b/app/src/features/collections/document-detail-route.tsx index 2b623ae9..0b18357a 100644 --- a/app/src/features/collections/document-detail-route.tsx +++ b/app/src/features/collections/document-detail-route.tsx @@ -2,6 +2,7 @@ import { getRouteApi, Link, useNavigate } from "@tanstack/react-router"; import { ArrowLeft, Trash2 } from "lucide-react"; import { useEffect, useState } from "react"; import { toast } from "sonner"; +import { match, P } from "ts-pattern"; import { Badge } from "@/components/ui/badge"; import { Button } from "@/components/ui/button"; import { Card, CardContent } from "@/components/ui/card"; @@ -9,6 +10,7 @@ import { ConfirmDialog } from "@/components/ui/confirm-dialog"; import { Empty } from "@/components/ui/empty"; import { Page } from "@/components/ui/page"; import { ProgressBar } from "@/components/ui/progress-bar"; +import { QueryError } from "@/components/ui/query-error"; import { Spinner } from "@/components/ui/spinner"; import { decodeCollectionName } from "@/features/collections/use-collection-name"; import { useChunks, useDeleteDocument, useDocument } from "@/hooks/use-documents"; @@ -93,13 +95,33 @@ export const DocumentDetail = () => { const navigate = useNavigate(); const [deleteOpen, setDeleteOpen] = useState(false); - const { data: doc, dataUpdatedAt, isFetching, isPending } = useDocument(name, docId); - const { data: chunks, refetch: refetchChunks } = useChunks(name, docId); + const document = useDocument(name, docId); + const chunksQuery = useChunks(name, docId); const remove = useDeleteDocument(name); + const doc = document.data; + const chunks = chunksQuery.data; + const dataUpdatedAt = document.dataUpdatedAt; - useRefreshChunksWhenReady(doc?.status, refetchChunks); + useRefreshChunksWhenReady(doc?.status, chunksQuery.refetch); - if (isPending || !doc) { + const detailState = match(document) + .with({ isError: true }, () => "error" as const) + .with({ isPending: true }, () => "loading" as const) + .with({ data: P.nullish }, () => "loading" as const) + .otherwise(() => "ready" as const); + + if (detailState === "error") { + return ( + document.refetch()} + title="Document could not load" + /> + ); + } + + if (detailState === "loading" || !doc) { return (
@@ -150,7 +172,7 @@ export const DocumentDetail = () => {

{progress.message}

- {isFetching && doc.status !== "ready" && doc.status !== "failed" ? ( + {document.isFetching && doc.status !== "ready" && doc.status !== "failed" ? ( ) : ( {progressPct}% @@ -188,7 +210,13 @@ export const DocumentDetail = () => { )} - {chunks ? ( + {chunksQuery.isError ? ( + chunksQuery.refetch()} + title="Chunks could not load" + /> + ) : chunks ? ( chunks.chunks.length === 0 ? ( ) : ( diff --git a/app/src/features/collections/s3/s3-connector-panel.tsx b/app/src/features/collections/s3/s3-connector-panel.tsx index 3efa0c4e..95558d87 100644 --- a/app/src/features/collections/s3/s3-connector-panel.tsx +++ b/app/src/features/collections/s3/s3-connector-panel.tsx @@ -63,18 +63,27 @@ export const S3ConnectorPanel = ({ collection }: { collection: string }) => { onDelete={async (sourceId) => { await deleteSource.mutateAsync(sourceId); }} + onRetrySources={() => sources.refetch()} onSync={(sourceId) => syncSource.mutate(sourceId)} onToggleSchedule={(sourceId, enabled) => updateSource.mutate({ body: { schedule_enabled: enabled }, sourceId }) } sources={sources.data?.sources ?? []} + sourcesError={sources.error} + sourcesIsError={sources.isError} sourcesPending={sources.isPending} syncPending={syncSource.isPending} updatePending={updateSource.isPending} workerOffline={workerOffline} /> {addSourceOpen && ( diff --git a/app/src/features/collections/s3/sources-panel.tsx b/app/src/features/collections/s3/sources-panel.tsx index 6c616413..d62685fe 100644 --- a/app/src/features/collections/s3/sources-panel.tsx +++ b/app/src/features/collections/s3/sources-panel.tsx @@ -3,6 +3,7 @@ import { useState } from "react"; import { Button } from "@/components/ui/button"; import { ConfirmDialog } from "@/components/ui/confirm-dialog"; import { Empty } from "@/components/ui/empty"; +import { QueryError } from "@/components/ui/query-error"; import { Spinner } from "@/components/ui/spinner"; import { Tooltip } from "@/components/ui/tooltip"; import { SourceRow } from "@/features/collections/s3/source-row"; @@ -16,9 +17,12 @@ export const SourcesPanel = ({ onAddSource, onChangeInterval, onDelete, + onRetrySources, onSync, onToggleSchedule, sources, + sourcesError, + sourcesIsError, sourcesPending, syncPending, updatePending, @@ -31,9 +35,12 @@ export const SourcesPanel = ({ onAddSource: () => void; onChangeInterval: (sourceId: string, hours: number) => void; onDelete: (sourceId: string) => Promise; + onRetrySources: () => void; onSync: (sourceId: string) => void; onToggleSchedule: (sourceId: string, enabled: boolean) => void; sources: S3Source[]; + sourcesError: unknown; + sourcesIsError: boolean; sourcesPending: boolean; syncPending: boolean; updatePending: boolean; @@ -66,7 +73,14 @@ export const SourcesPanel = ({ Scheduled syncs wait until bigrag-worker is online. )} - {sources.length ? ( + {sourcesIsError ? ( + + ) : sources.length ? (
    {sources.map((source) => ( void; }) => (

    Sync monitor

    - {isPending ? ( + {isError ? ( + + ) : isPending ? (
    diff --git a/app/src/features/connectors/connectors-page.tsx b/app/src/features/connectors/connectors-page.tsx index a982d674..d2859daf 100644 --- a/app/src/features/connectors/connectors-page.tsx +++ b/app/src/features/connectors/connectors-page.tsx @@ -6,6 +6,7 @@ import { Button } from "@/components/ui/button"; import { ConfirmDialog } from "@/components/ui/confirm-dialog"; import { Empty } from "@/components/ui/empty"; import { Page } from "@/components/ui/page"; +import { QueryError } from "@/components/ui/query-error"; import { Spinner } from "@/components/ui/spinner"; import { Tooltip } from "@/components/ui/tooltip"; import { @@ -71,7 +72,21 @@ export const ConnectorsPage = () => { - {sources.isPending ? ( + {sources.isError ? ( + sources.refetch()} + title="Connector sources could not load" + /> + ) : syncJobs.isError ? ( + syncJobs.refetch()} + title="Connector sync jobs could not load" + /> + ) : sources.isPending ? (
    diff --git a/app/src/features/settings/instance-settings-helpers.ts b/app/src/features/settings/instance-settings-helpers.ts index 474459e0..1812f79f 100644 --- a/app/src/features/settings/instance-settings-helpers.ts +++ b/app/src/features/settings/instance-settings-helpers.ts @@ -1,3 +1,4 @@ +import { match } from "ts-pattern"; import type { InstanceSettingSpec, InstanceSettingValue } from "@/types/bigrag"; export type DraftValue = boolean | string; @@ -98,7 +99,26 @@ export const valuesForSubmit = ( const value = draft[spec.key]; if (spec.kind === "secret" && !value) continue; if (spec.kind !== "secret" && value === draftValue(spec, settingValues[spec.key])) continue; - values[spec.key] = value; + values[spec.key] = valueForSubmit(spec, value); } return values; }; + +const valueForSubmit = (spec: InstanceSettingSpec, value: DraftValue): unknown => + match(spec.kind) + .with("int", () => (value === "" ? null : Number.parseInt(String(value), 10))) + .with("float", () => (value === "" ? null : Number.parseFloat(String(value)))) + .with("int_list", () => + String(value) + .split(/[\n,]/) + .map((item) => item.trim()) + .filter(Boolean) + .map((item) => Number.parseInt(item, 10)), + ) + .with("string_list", () => + String(value) + .split(/[\n,]/) + .map((item) => item.trim()) + .filter(Boolean), + ) + .otherwise(() => value); diff --git a/app/src/features/settings/instance-settings/instance-settings-tab.tsx b/app/src/features/settings/instance-settings/instance-settings-tab.tsx index 2fdaf101..58dc5a89 100644 --- a/app/src/features/settings/instance-settings/instance-settings-tab.tsx +++ b/app/src/features/settings/instance-settings/instance-settings-tab.tsx @@ -1,6 +1,7 @@ import { useStore } from "@tanstack/react-form"; import { useEffect, useState } from "react"; import { ConfirmDialog } from "@/components/ui/confirm-dialog"; +import { QueryError } from "@/components/ui/query-error"; import { useInstanceSettingsForm } from "@/features/settings/instance-settings/instance-settings-form"; import { RuntimeSettingsPanel } from "@/features/settings/instance-settings/runtime-settings-panel"; import { @@ -40,7 +41,7 @@ export const InstanceSettingsTab = ({ stacked = false, }: InstanceSettingsTabProps) => { const targetGroups = useTargetGroups(group, groups); - const { data, isPending } = useInstanceSettings(); + const { data, error, isError, isPending, refetch } = useInstanceSettings(); const save = useUpdateInstanceSettings(); const purgeEmbeddingCache = usePurgeEmbeddingCache(); const form = useInstanceSettingsForm(); @@ -62,6 +63,12 @@ export const InstanceSettingsTab = ({ ); } + if (isError) { + return ( + refetch()} title="Runtime settings could not load" /> + ); + } + return (
    {targetGroups.map((targetGroup, index) => { diff --git a/bigrag.toml b/bigrag.toml index ca09fcb2..28160ca1 100644 --- a/bigrag.toml +++ b/bigrag.toml @@ -18,7 +18,7 @@ # database_url = "postgres://bigrag:bigrag@localhost:5432/bigrag?sslmode=disable" # db_pool_min = 5 -# db_pool_max = 10 +# db_pool_max = 20 # Connection budget: each API/worker process opens its own pool of up to # db_pool_max connections. Keep (api_workers * api_replicas + worker_processes) # * db_pool_max under Postgres max_connections (default 200), leaving headroom diff --git a/package.json b/package.json index 3d4a6a1a..ba3247e6 100644 --- a/package.json +++ b/package.json @@ -12,7 +12,8 @@ "typecheck": "pnpm -r typecheck", "lint": "biome check .", "lint:fix": "biome check --write .", - "lint:api": "cd api && uv run ruff check . && uv run ruff format --check ." + "lint:api": "cd api && uv run ruff check . && uv run ruff format --check .", + "check": "pnpm lint && pnpm lint:api && pnpm typecheck && pnpm build:app && pnpm build:sdk && pnpm build:website && cd api && uv run python -m compileall -q bigrag" }, "devDependencies": { "@biomejs/biome": "^2.4.15" diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 15fd1581..9ace79b3 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -72,6 +72,9 @@ importers: tailwind-merge: specifier: ^3.6.0 version: 3.6.0 + ts-pattern: + specifier: 5.9.0 + version: 5.9.0 zustand: specifier: ^5.0.13 version: 5.0.13(@types/react@19.2.14)(react@19.2.6)(use-sync-external-store@1.6.0(react@19.2.6)) @@ -2986,6 +2989,9 @@ packages: resolution: {integrity: sha512-q5W7tVM71e2xjHZTlgfTDoPF/SmqKG5hddq9SzR49CH2hayqRKJtQ4mtRlSxKaJlR/+9rEM+mnBHf7I2/BQcpQ==} engines: {node: '>=6.10'} + ts-pattern@5.9.0: + resolution: {integrity: sha512-6s5V71mX8qBUmlgbrfL33xDUwO0fq48rxAu2LBE11WBeGdpCPOsXksQbZJHvHwhrd3QjUusd3mAOM5Gg0mFBLg==} + tslib@2.8.1: resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==} @@ -6209,6 +6215,8 @@ snapshots: ts-dedent@2.2.0: {} + ts-pattern@5.9.0: {} + tslib@2.8.1: {} tsx@4.22.1: diff --git a/website/content/docs/api-reference/api-keys.mdx b/website/content/docs/api-reference/api-keys.mdx index 2d3e9450..72b72206 100644 --- a/website/content/docs/api-reference/api-keys.mdx +++ b/website/content/docs/api-reference/api-keys.mdx @@ -61,7 +61,7 @@ POST /v1/admin/api-keys | Field | Type | Required | Notes | |-------|------|----------|-------| | `name` | string | yes | Human label | -| `scopes` | string[] | no | Omit / empty = full access. See [scopes](/docs/api-reference/authentication#scopes) | +| `scopes` | string[] | no | Omit / empty stores `["*:*"]` for full access. See [scopes](/docs/api-reference/authentication#scopes) | | `expires_at` | string (RFC3339) | no | `null` = never | | `collection` | string \| null | no | Pin this key to one collection | @@ -99,7 +99,7 @@ PATCH /v1/admin/api-keys/{key_id} { "name": "ingest-prod", "active": false, "scopes": ["query:read"] } ``` -Any combination of `name`, `active`, `expires_at`, `scopes`, and `collection` is permitted. Deactivating (`active: false`) is the non-destructive way to freeze a leaked key while you investigate; the audit log still references it. +Any combination of `name`, `active`, `expires_at`, `scopes`, and `collection` is permitted. Updating `scopes` to `[]` stores `["*:*"]` for full access. Deactivating (`active: false`) is the non-destructive way to freeze a leaked key while you investigate; the audit log still references it. **Response** `200`: updated key (without `key`). diff --git a/website/content/docs/api-reference/authentication.mdx b/website/content/docs/api-reference/authentication.mdx index 0b7619b0..79871b4c 100644 --- a/website/content/docs/api-reference/authentication.mdx +++ b/website/content/docs/api-reference/authentication.mdx @@ -87,7 +87,7 @@ curl http://localhost:4000/v1/collections \ |-------|-------| | `name` | Human-readable label | | `prefix` | First 12 chars; safe to display | -| `scopes` | `["resource:action", …]` — omit / empty = full access | +| `scopes` | `["resource:action", …]`; omit / empty stores `["*:*"]` for full access | | `expires_at` | RFC3339; `null` = never | | `active` | Disable without deleting | | `last_used_at` | Updated on each successful auth | @@ -112,7 +112,7 @@ Scopes use `resource:action` with `*` as a wildcard. | `vector:write` | Direct raw vector upsert | | `vector:delete` | Direct raw vector delete | | `audit:read` | Read the API-key-capable usage rollups at `/v1/usage` and `/v1/status/usage` | -| `*:*` | Full access (equivalent to no `scopes`) | +| `*:*` | Full access | A key missing a required scope returns `403` with `{"detail": "API key missing required scope: "}`. diff --git a/website/content/docs/api-reference/collections.mdx b/website/content/docs/api-reference/collections.mdx index 3769c977..e86bfd6a 100644 --- a/website/content/docs/api-reference/collections.mdx +++ b/website/content/docs/api-reference/collections.mdx @@ -102,9 +102,9 @@ All endpoints require [authentication](/docs/api-reference/authentication) — a | `chunk_size` | integer | no | `512` | 64–10,000 | | `chunk_overlap` | integer | no | `50` | 0–5,000, must be `< chunk_size` | | `chunk_strategy` | string | no | `"paragraph"` | `paragraph` (split on blank lines) or `recursive` (hierarchical) | -| `tenant_field` | string | no | — | Metadata key required on uploads, raw vector upserts, query filters, and chat filters | +| `tenant_field` | string | no | — | Metadata key used for tenant isolation on uploads, raw vector upserts, queries, chat, document listing, and document reads | | `metadata_schema` | object | no | — | JSON Schema validated on every upload / upsert | -| `default_top_k` | integer | no | `10` | 1–1,000 | +| `default_top_k` | integer | no | `10` | 1–200 | | `default_min_score` | float | no | `null` | Minimum similarity score | | `default_search_mode` | string | no | `"semantic"` | `semantic`, `keyword`, `hybrid` | | `reranking_enabled` | boolean | no | `false` | Re-rank results with Cohere cross-encoder | @@ -146,7 +146,7 @@ Only the fields below are mutable. Embedding provider / model / dimension, chunk | `reranking_api_key` | string | — | | `multimodal_enabled` | boolean | Disabling it also disables enrichment for future ingestions | | `multimodal_enrichment_enabled` | boolean | Enabling it also enables `multimodal_enabled` | -| `default_top_k` | integer | 1–1,000 | +| `default_top_k` | integer | 1–200 | | `default_min_score` | float | — | | `default_search_mode` | string | `semantic`, `keyword`, `hybrid` | diff --git a/website/content/docs/api-reference/health.mdx b/website/content/docs/api-reference/health.mdx index ef015dd1..51f17091 100644 --- a/website/content/docs/api-reference/health.mdx +++ b/website/content/docs/api-reference/health.mdx @@ -14,7 +14,7 @@ import { Accordion, Accordions } from "fumadocs-ui/components/accordion"; No authentication required. Returns `200` if the server is running. ```json -{ "status": "ok", "version": "2026.4.30" } +{ "status": "ok", "version": "2026.x.y" } ``` @@ -28,7 +28,7 @@ No authentication required. Tests connectivity to Postgres, Turbopuffer, Redis, ```json { "status": "ok", - "version": "2026.4.30", + "version": "2026.x.y", "postgres": true, "vector_store": true, "redis": true, @@ -42,7 +42,7 @@ No authentication required. Tests connectivity to Postgres, Turbopuffer, Redis, ```json { "status": "degraded", - "version": "2026.4.30", + "version": "2026.x.y", "postgres": true, "vector_store": true, "redis": false, diff --git a/website/content/docs/api-reference/query.mdx b/website/content/docs/api-reference/query.mdx index 3985c7cc..0cc011b2 100644 --- a/website/content/docs/api-reference/query.mdx +++ b/website/content/docs/api-reference/query.mdx @@ -32,7 +32,7 @@ Run a search against one collection's Turbopuffer namespace. Semantic and hybrid | Field | Type | Required | Default | Constraints | |-------|------|----------|---------|-------------| | `query` | string | yes | — | Natural language or keyword query | -| `top_k` | integer | no | Collection default | 1–1,000 | +| `top_k` | integer | no | Collection default | 1–200 | | `filters` | object | no | — | Metadata filters — see [operators](/docs/concepts/search#filters) | | `min_score` | float | no | Collection default | Drop results below this score | | `search_mode` | string | no | Collection default | `semantic`, `keyword`, or `hybrid` | @@ -127,7 +127,7 @@ Search across multiple collections and merge the results by score. |-------|------|----------|-------------| | `query` | string | yes | Search query | | `collections` | string[] | yes | Collection names to search | -| `top_k` | integer | no | Max results per collection | +| `top_k` | integer | no | Max results per collection, 1–200 | | `filters` | object | no | Metadata filters | | `min_score` | float | no | Minimum score | | `search_mode` | string | no | `semantic`, `keyword`, `hybrid` | diff --git a/website/content/docs/concepts/collections.mdx b/website/content/docs/concepts/collections.mdx index 4ff0a43f..116a224b 100644 --- a/website/content/docs/concepts/collections.mdx +++ b/website/content/docs/concepts/collections.mdx @@ -50,7 +50,7 @@ curl -X POST http://localhost:4000/v1/collections \ | `dimension` | integer | no | Server default | Embedding vector dimension | | `chunk_size` | integer | no | `512` | 64–10,000 characters per chunk | | `chunk_overlap` | integer | no | `50` | 0–5,000, must be less than `chunk_size` | -| `default_top_k` | integer | no | `10` | Default number of results (1–1,000) | +| `default_top_k` | integer | no | `10` | Default number of results (1–200) | | `default_min_score` | float | no | `null` | Default minimum similarity score | | `default_search_mode` | string | no | `"semantic"` | `semantic`, `keyword`, or `hybrid` | | `reranking_enabled` | boolean | no | `false` | Enable reranking for this collection | @@ -91,7 +91,7 @@ curl -X POST http://localhost:4000/v1/collections \ }' ``` -When querying, if `top_k`, `min_score`, or `search_mode` are omitted, the collection's defaults are used. +When querying, if `top_k`, `min_score`, or `search_mode` are omitted, the collection's defaults are used. Hybrid scores are normalized after reciprocal rank fusion, so a `default_min_score` such as `0.3` can be used consistently across semantic and hybrid search. ## Reranking @@ -149,7 +149,7 @@ curl -X PUT http://localhost:4000/v1/collections/research_papers \ | `reranking_api_key` | string | Cohere API key | | `multimodal_enabled` | boolean | Enable/disable element extraction for future ingestions | | `multimodal_enrichment_enabled` | boolean | Enable/disable asynchronous LLM element summaries | -| `default_top_k` | integer | Default number of results (1–1,000) | +| `default_top_k` | integer | Default number of results (1–200) | | `default_min_score` | float | Default minimum similarity score | | `default_search_mode` | string | `semantic`, `keyword`, or `hybrid` | diff --git a/website/content/docs/concepts/search.mdx b/website/content/docs/concepts/search.mdx index b888334f..1f3830d5 100644 --- a/website/content/docs/concepts/search.mdx +++ b/website/content/docs/concepts/search.mdx @@ -55,7 +55,7 @@ Each bigRAG collection maps to one Turbopuffer namespace. Document chunks are wr | Field | Type | Default | Description | |-------|------|---------|-------------| | `query` | string | — | Required query text | -| `top_k` | integer | Collection default | 1–1,000 | +| `top_k` | integer | Collection default | 1–200 | | `filters` | object | — | Metadata filters — see below | | `min_score` | float | Collection default | Drop hits below this score | | `search_mode` | string | Collection default | `semantic`, `keyword`, `hybrid` | @@ -90,9 +90,9 @@ Pass a plain value for exact match, or use operators for more control: } ``` -Multiple filters are combined with AND. When a collection was created with `tenant_field`, bigRAG configures that field for backend filtering and requires it in every query and chat filter. Missing tenant filters return `400`. +Multiple filters are combined with AND. When a collection was created with `tenant_field`, bigRAG configures that field for backend filtering. Tenant-scoped API keys have their tenant filter applied automatically on query and chat requests. Session admins can pass explicit tenant filters. Unscoped API keys without a tenant cannot access tenant-guarded collections and receive `400`. -Keyword search uses Turbopuffer BM25 over the chunk `text` field. Hybrid search runs Turbopuffer ANN and BM25 queries, then merges the two result sets with reciprocal rank fusion before optional reranking. +Keyword search uses Turbopuffer BM25 over the chunk `text` field. Hybrid search runs Turbopuffer ANN and BM25 queries, then merges the two result sets with normalized reciprocal rank fusion before optional reranking. ## Reranking diff --git a/website/content/docs/concepts/security.mdx b/website/content/docs/concepts/security.mdx index e25f5a8a..5bd495cf 100644 --- a/website/content/docs/concepts/security.mdx +++ b/website/content/docs/concepts/security.mdx @@ -52,7 +52,7 @@ If you need to automate any of these, proxy through a trusted service that holds ## Scopes on API keys -Scopes are `resource:action` pairs. A key without `scopes` has full access; a key **with** `scopes` is restricted to the union of those patterns. `*` is a wildcard. +Scopes are `resource:action` pairs. Full-access keys are stored with `["*:*"]`; legacy keys without `scopes` are also treated as full access. Any narrower scope list is restricted to the union of those patterns. `*` is a wildcard. ``` collection:read # GET /v1/collections, GET /v1/collections/{name}, stats @@ -67,14 +67,14 @@ chat:write # POST /v1/chat, POST /v1/chat/question-suggestions vector:write # POST /v1/collections/{name}/vectors/upsert vector:delete # POST /v1/collections/{name}/vectors/delete audit:read # GET /v1/usage -*:* # equivalent to no scopes +*:* # full access ``` Missing-scope calls return `403` with `{"detail": "API key missing required scope: "}`. Collection-pinned keys apply that pin consistently, including the global document-read helpers under `/v1/documents/{id}`. -Collections created with `tenant_field` require that field in upload metadata, raw vector metadata, query filters, and chat filters. Missing tenant constraints return `400` before retrieval. +Collections created with `tenant_field` require that field in upload metadata and raw vector metadata. Tenant-scoped API keys get tenant filters applied automatically on query, chat, document listing, and document reads. Session admins can use explicit tenant filters. Unscoped API keys without a tenant are denied with `400` before tenant-guarded retrieval. ## Audit log diff --git a/website/content/docs/cookbook/multi-tenant-saas.mdx b/website/content/docs/cookbook/multi-tenant-saas.mdx index 02b38387..3fbfa4ba 100644 --- a/website/content/docs/cookbook/multi-tenant-saas.mdx +++ b/website/content/docs/cookbook/multi-tenant-saas.mdx @@ -51,7 +51,7 @@ Collections-per-tenant is simple but creates operational overhead as tenant coun ## Pattern B — shared collection with tenant filters -One collection, tenant isolation via a required metadata filter. Setting `tenant_field` tells bigRAG to configure that field for backend filtering and reject uploads, raw vector upserts, queries, and chat calls that omit the tenant field. +One collection, tenant isolation via a required metadata field. Setting `tenant_field` tells bigRAG to configure that field for backend filtering, reject writes that omit the tenant field, and apply the tenant filter automatically for tenant-scoped API keys. ```python await client.collections.create({ @@ -73,7 +73,7 @@ results = await client.queries.query("shared_tenants", { For tenant-guarded collections, the tenant filter must identify one tenant. `{"tenant_id": {"$eq": "acme"}}` and `{"tenant_id": {"$in": ["acme"]}}` are valid; multi-value `$in` filters are rejected because they cross tenant boundaries. -You still need to pass the tenant filter on every tenant-scoped query and chat request. bigRAG rejects missing filters, but your application still owns mapping the authenticated tenant to the exact allowed tenant value. +For session admins, pass the tenant filter on query and chat requests when you want a single tenant. For tenant-scoped API keys, bigRAG injects the tenant filter server-side, so your application owns mapping each authenticated tenant to the key or backend principal that carries the exact allowed tenant value. ## Per-tenant usage + billing diff --git a/website/content/docs/deployment/docker.mdx b/website/content/docs/deployment/docker.mdx index 98486d07..7d2f96d6 100644 --- a/website/content/docs/deployment/docker.mdx +++ b/website/content/docs/deployment/docker.mdx @@ -13,16 +13,16 @@ import { Callout } from "fumadocs-ui/components/callout"; ### Pull the images ```bash -docker pull yoginth/bigrag-api:2026.4.30 -docker pull yoginth/bigrag-ui:2026.4.30 +docker pull yoginth/bigrag-api:latest +docker pull yoginth/bigrag-ui:latest ``` ### Start the stack ```bash -BIGRAG_API_IMAGE=yoginth/bigrag-api:2026.4.30 \ -BIGRAG_UI_IMAGE=yoginth/bigrag-ui:2026.4.30 \ +BIGRAG_API_IMAGE=yoginth/bigrag-api:latest \ +BIGRAG_UI_IMAGE=yoginth/bigrag-ui:latest \ docker compose up -d --no-build ``` @@ -47,7 +47,7 @@ The default `docker-compose.yml` runs the API, worker, admin UI, Postgres, and R ```yaml services: bigrag-api: - image: yoginth/bigrag-api:2026.4.30 + image: yoginth/bigrag-api:latest ports: - "4000:4000" volumes: @@ -77,7 +77,7 @@ services: retries: 3 bigrag-worker: - image: yoginth/bigrag-api:2026.4.30 + image: yoginth/bigrag-api:latest command: ["bigrag-worker", "--processes", "${BIGRAG_WORKER_PROCESSES:-5}", "--threads", "${BIGRAG_WORKER_THREADS:-8}"] volumes: - bigrag_data:/data @@ -113,7 +113,7 @@ services: start_period: 60s bigrag-ui: - image: yoginth/bigrag-ui:2026.4.30 + image: yoginth/bigrag-ui:latest ports: - "3000:3000" environment: diff --git a/website/content/docs/deployment/production.mdx b/website/content/docs/deployment/production.mdx index 68843238..b7e69500 100644 --- a/website/content/docs/deployment/production.mdx +++ b/website/content/docs/deployment/production.mdx @@ -42,7 +42,7 @@ When the guard trips, it logs every violation before exiting so you can fix the ```yaml services: bigrag-api: - image: yoginth/bigrag-api:2026.4.30 + image: yoginth/bigrag-api: ports: - "4000:4000" volumes: @@ -80,7 +80,7 @@ services: retries: 3 bigrag-worker: - image: yoginth/bigrag-api:2026.4.30 + image: yoginth/bigrag-api: command: ["bigrag-worker", "--processes", "1", "--threads", "8"] volumes: - bigrag_data:/data @@ -116,7 +116,7 @@ services: start_period: 60s bigrag-ui: - image: yoginth/bigrag-ui:2026.4.30 + image: yoginth/bigrag-ui: ports: - "3000:3000" environment: diff --git a/website/content/docs/development/testing.mdx b/website/content/docs/development/testing.mdx index 23470ad2..d0250781 100644 --- a/website/content/docs/development/testing.mdx +++ b/website/content/docs/development/testing.mdx @@ -5,6 +5,12 @@ description: Current verification workflow for bigRAG changes. bigRAG no longer carries package-level unit, integration, end-to-end, or coverage suites. Do not add those runners back to feature work unless the project deliberately reintroduces them. +Use the root check before reporting a source change complete: + +```bash +pnpm check +``` + Use the smallest verification set that matches the changed surface: | Surface | Verification | diff --git a/website/content/docs/getting-started/installation.mdx b/website/content/docs/getting-started/installation.mdx index 2e16d824..5a05ad72 100644 --- a/website/content/docs/getting-started/installation.mdx +++ b/website/content/docs/getting-started/installation.mdx @@ -37,11 +37,11 @@ Turbopuffer is configured from the admin UI after setup and provides the managed Pre-built images are published to Docker Hub: ```bash -docker pull yoginth/bigrag-api:2026.4.30 -docker pull yoginth/bigrag-ui:2026.4.30 +docker pull yoginth/bigrag-api:latest +docker pull yoginth/bigrag-ui:latest ``` -Release artifacts use CalVer (`YYYY.M.D`). Docker also publishes `latest` for convenience, but production deployments should pin the dated tag. +Release artifacts use CalVer (`YYYY.M.D`). Docker publishes `latest` for convenience, but production deployments should pin the dated tag from the release they deploy. @@ -101,7 +101,7 @@ If the admin UI shows `bigrag-worker is offline`, check the terminal for the col ```bash curl http://localhost:4000/health -# → {"status": "ok", "version": "2026.4.30"} +# → {"status": "ok", "version": "2026.x.y"} ```