diff --git a/README.md b/README.md index d0ff245..517242e 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ Launch requirements: - Return each chat turn as one complete JSON response; do not stream partial output. - Preserve conversation context across follow-up questions. - Let administrators choose the pages where the chat widget appears and configure its color scheme, screen position, and welcome message. -- Select OpenAI or Groq for chat generation through one server environment variable while keeping all provider credentials and model settings in environment variables. +- Select OpenAI or Groq for chat generation through encrypted installation configuration stored by the backend. - Support both native and optional Docker server deployment, with ParadeDB BM25 + pgvector as the normal production retrieval mode only after its package and readiness gates pass. - Provide an admin Test Chat submenu for exercising the widget UI and backend API integration. - Power the website first while keeping the backend reusable for a future mobile app. @@ -73,7 +73,7 @@ Future-facing requirements: - [`server/RANKING_AND_CITATION_CONTRACT.md`](server/RANKING_AND_CITATION_CONTRACT.md): versioned relevance-first ranking, review aggregation, promotion disclosures, citations, uncertainty, deduplication, and evaluation gates. - [`server/CONVERSATION_CONTEXT_CONTRACT.md`](server/CONVERSATION_CONTEXT_CONTRACT.md): exact visitor ownership, bounded history, turn audit records, PostgreSQL checkpoints, history access, deletion, anonymization, and retention. - [`server/SEMANTIC_SEARCH_ARCHITECTURE_AND_FLOW_GUIDE.md`](server/SEMANTIC_SEARCH_ARCHITECTURE_AND_FLOW_GUIDE.md): Ask Sunny indexing, retrieval, chat, caching, security, and semantic-search flows. -- [`server/DATABASE_SCHEMA.md`](server/DATABASE_SCHEMA.md): PostgreSQL schema for content, embeddings, conversations, user data, analytics, admin sessions, and migrations. +- [`server/DATABASE_SCHEMA.md`](server/DATABASE_SCHEMA.md): PostgreSQL schema for content, embeddings, conversations, user data, analytics, API keys, and migrations. - [`server/REST_API_CONTRACT.md`](server/REST_API_CONTRACT.md): backend REST endpoints called by WordPress, future mobile clients, and server admins. ### Plugin @@ -112,7 +112,7 @@ WordPress remains responsible for collecting site content, configuring and rende - Ask Sunny is single-tenant. Do not use a multi-tenant `sites` and `site_domains` model as the main architecture. - Browser JavaScript calls WordPress REST only. Browser code never receives AI-provider, embedding-provider, or backend API keys. - The backend uses LangGraph for orchestration and short-term workflow state. Application tables store durable conversation, message, tool-call, profile, and usage records. -- The backend uses a provider-neutral abstraction for Responses API calls. `AI_PROVIDER=openai|groq` selects a registered adapter at runtime; provider keys, base URLs, models, and embedding settings remain environment configuration, while database tables store no provider discriminator or provider-specific conversation state. Chat responses are not streamed. +- The backend uses a provider-neutral abstraction for Responses API calls. One singleton global database record selects a registered adapter and supplies its encrypted API key and chat model for every installation; non-secret endpoints, timeouts, and independent embedding settings remain environment configuration. Conversation tables store no provider-specific response or conversation state. Chat responses are not streamed. - Native or Dockerized ParadeDB uses `pg_search` for BM25 and pgvector for dense similarity. Hybrid retrieval is the verified production default and fuses both candidate sets only after applying the stored data-source allowlist and structured filters. Installation and upgrades begin with hybrid disabled; it must not be enabled until the installed package/image matches the running PostgreSQL major version, execution OS, and architecture and all extension, index, and smoke checks pass. - WordPress and Directorist remain the content source of truth for launch. Backend content tables are an indexed search/read model. - Backend content storage is separated by source kind. Directorist listings use a dedicated `listings` table with inline normalized state and vector data; reviews and optional WordPress content use their own content and embedding tables. diff --git a/plugin/ARCHITECTURE.md b/plugin/ARCHITECTURE.md index 902e0ef..0568c33 100644 --- a/plugin/ARCHITECTURE.md +++ b/plugin/ARCHITECTURE.md @@ -247,11 +247,11 @@ The Data Sources submenu should include: - Index-status counts update with search and tab-specific filters but are calculated before applying the selected index-status value, so the administrator can see how many records exist in every status without clearing the filter. - An explicit **Delete indexed data** action on each item and a destructive **Delete all indexed data** action for each source tab, both protected by confirmation and `manage_options`. - Diagnostics. -- Recent usage summary fetched from backend. +- Diagnostics summary fetched from backend. -Backend diagnostics and usage use `GET /installation/diagnostics` and -`GET /installation/usage` with the installation credential. The plugin must not call `/admin/*`, -store an admin key/session, or expose the installation key to browser code. +Backend diagnostics use `GET /system/diagnostics` with the website credential. The backend exposes +no website usage route. The plugin must not call `/admin/*`, store an admin key, or expose the +website key to browser code. The Test Chat submenu should render the production widget component in an isolated admin preview and send messages through an admin-only WordPress REST route. It displays connection/provider/hybrid-search diagnostics, request correlation ID, latency, answer, citations, recommendations, and sanitized errors. Test conversations use the backend `admin_test` channel and must not bypass the same response validation or source allowlist used by public chat. diff --git a/plugin/REST_API_CONTRACT.md b/plugin/REST_API_CONTRACT.md index f915aca..9256780 100644 --- a/plugin/REST_API_CONTRACT.md +++ b/plugin/REST_API_CONTRACT.md @@ -305,7 +305,7 @@ Explicitly deletes all indexed records for a source after admin confirmation. Th ### `POST /provision` -Calls backend `/auth/provision-installation` using a server-side provisioning key. +Calls backend `/auth/provision` using a server-side provisioning key. Response: @@ -387,9 +387,8 @@ Returns local indexing status. ### `GET /diagnostics` Checks WordPress-side state and the backend's installation-scoped diagnostics. WordPress calls -backend `GET /installation/diagnostics` with the installation credential; it never calls a backend -`/admin/*` route or stores a backend administrator credential. Recent usage is fetched separately -from backend `GET /installation/usage` and reduced to the same safe aggregates for the dashboard. +backend `GET /system/diagnostics` with the website credential; it never calls a backend `/admin/*` +route or stores a backend administrator credential. The backend exposes no website usage route. ```json { @@ -398,9 +397,9 @@ from backend `GET /installation/usage` and reduced to the same safe aggregates f "backend": { "ok": true, "database": "ok", - "ai_provider": "groq", - "ai_provider_configured": true, - "embedding_provider": "openai", + "chat_ai_router": "groq", + "chat_ai_configured": true, + "embedding_ai_router": "openai", "hybrid_search": "enabled", "paradedb": "ok", "allowed_data_sources_version": 6, diff --git a/server/ARCHITECTURE.md b/server/ARCHITECTURE.md index 91db308..4a825e6 100644 --- a/server/ARCHITECTURE.md +++ b/server/ARCHITECTURE.md @@ -21,8 +21,8 @@ The server is responsible for: - Language: JavaScript, following the backend service's Bun/Hono runtime pattern. - HTTP framework: Hono. - Agent framework: LangGraph.js. -- Model API: provider-neutral generation interface with adapters registered by name and selected at runtime from `AI_PROVIDER`. -- Embeddings: independently configured embedding provider; OpenAI is the launch default. +- Model API: router-neutral generation interface selected by database-backed chat router/model options. +- Embeddings: independently selected OpenAI or Gemini router/model options with fixed dimensions. - Database: ParadeDB's PostgreSQL distribution with `pg_search` and pgvector. - Search: hybrid BM25 keyword matching plus dense vector similarity. - Cache: Redis optional. @@ -38,27 +38,14 @@ LOG_LEVEL=info REQUEST_BODY_LIMIT=2mb ASK_SUNNY_INSTALLATION_PROVISIONING_KEY=replace-with-long-random-secret -ASK_SUNNY_ADMIN_EMAIL=admin@example.com +ASK_SUNNY_ADMIN_USERNAME=admin ASK_SUNNY_ADMIN_PASSWORD=replace-with-strong-password -ASK_SUNNY_ADMIN_SESSION_TTL_SECONDS=86400 DATABASE_URL=postgres://ask_sunny:strong-password@127.0.0.1:5432/ask_sunny PG_POOL_MAX=10 -AI_PROVIDER=openai AI_REQUEST_TIMEOUT_MS=45000 -OPENAI_API_KEY=replace-with-openai-api-key -OPENAI_BASE_URL=https://api.openai.com/v1 -OPENAI_CHAT_MODEL=replace-with-supported-openai-model - -GROQ_API_KEY=replace-with-groq-api-key -GROQ_BASE_URL=https://api.groq.com/openai/v1 -GROQ_CHAT_MODEL=replace-with-supported-groq-model - -EMBEDDING_PROVIDER=openai -OPENAI_EMBEDDINGS_URL=https://api.openai.com/v1/embeddings -EMBEDDING_MODEL=text-embedding-3-small EMBEDDING_DIMENSIONS=1536 EMBEDDING_REQUEST_TIMEOUT_MS=15000 EMBEDDING_MAX_RETRIES=2 @@ -92,7 +79,20 @@ MAX_TOOL_ITERATIONS=6 DEFAULT_TIMEZONE=UTC ``` -`AI_PROVIDER` is the only switch for chat generation. A provider registry resolves that value to an adapter implementing the provider-neutral generation interface; orchestration, persistence, routes, and domain services must not branch on provider names. The selected adapter's API key, base URL, and model must be valid at startup; credentials for an inactive provider may be omitted. Embeddings are configured independently because generation and embedding providers do not have identical capabilities. Changing `AI_PROVIDER` does not change stored vector dimensions or trigger re-embedding, and provider identity is not stored in application database tables. +The global `options` key/value rows store the selected generation provider, encrypted API key, and chat +model globally. A provider registry resolves that database record to an adapter implementing the +provider-neutral generation interface; orchestration, routes, and domain services must not branch +on provider names. Missing or incomplete stored provider configuration +fails the related request before work begins and never falls back to process environment settings. +Non-secret provider base URLs and the shared request timeout remain deployment controls. Embeddings +remain independently configured because generation and embedding providers do not have identical +capabilities. + +The option repository is the single persistence boundary for the `options` table. It exposes only +generic insert, get, update, delete, `getByKeys`, and `updateMany` operations. Provider key names, +mapping, validation, and summary projection remain in the application layer; the repository has no +provider-specific helpers. `updateMany` accepts only option items and always assigns row +`updated_at` values from the database clock. Embedding requests use independent timeout and retry controls. `EMBEDDING_REQUEST_TIMEOUT_MS` defaults to 15000 and accepts 1000 through 60000. `EMBEDDING_MAX_RETRIES` defaults to 2 and accepts @@ -102,7 +102,7 @@ milliseconds, and may be raised by a valid `Retry-After` value up to that same c Because chat is returned as one complete response, the WordPress proxy timeout must be greater than `AI_REQUEST_TIMEOUT_MS`; a 60-second WordPress timeout provides application overhead around the 45-second provider timeout. -Model names are deployment configuration, not hardcoded constants. Verify the selected provider's current model, Responses API, structured-output, and tool-use support before production launch. The provider adapter must not send parameters unsupported by the active provider. +Model names are global database configuration, not hardcoded constants. Verify the selected provider's current model, Responses API, structured-output, and tool-use support before production launch. The provider adapter must not send parameters unsupported by the active provider. The example connection URLs target native services. Docker Compose overrides their hosts with Compose service names such as `paradedb` and `redis`; application code and all other configuration remain identical. @@ -181,7 +181,11 @@ Use the configured provider's Responses API for: - Complete structured response generation for the widget. - Multi-turn continuity through server-side conversation context. -The launch adapter registry includes `openai` and `groq`. `AI_PROVIDER=openai` resolves the OpenAI adapter and `AI_PROVIDER=groq` resolves the Groq adapter. Adding a future provider requires registering another implementation, not editing orchestration or persistence code. Each adapter owns request construction, supported parameters, structured-output validation, tool-call normalization, usage normalization, timeout handling, and error mapping. +The launch adapter registry includes `openai` and `groq`; the global `options` key/value store +provider type resolves the matching adapter. Adding a future provider requires registering another +implementation, not editing orchestration or conversation persistence code. Each adapter owns +request construction, supported parameters, structured-output validation, tool-call normalization, +usage normalization, timeout handling, and error mapping. Do not depend on provider-hosted conversation state. The application loads and persists provider-neutral conversation history and LangGraph state, then supplies the required context on every turn. Database tables do not store the active provider or provider-specific conversation IDs. This keeps provider switching deterministic and avoids coupling to provider-specific response-storage features. diff --git a/server/DATABASE_SCHEMA.md b/server/DATABASE_SCHEMA.md index 8a6f7df..d0a3c17 100644 --- a/server/DATABASE_SCHEMA.md +++ b/server/DATABASE_SCHEMA.md @@ -19,28 +19,9 @@ These extension statements may run only after the deployment compatibility gate. ## Core Configuration ```sql -CREATE TABLE installation_config ( - id BOOLEAN PRIMARY KEY DEFAULT true CHECK (id = true), - installation_name TEXT NOT NULL DEFAULT 'WordPress Site', - primary_domain TEXT NOT NULL, - wordpress_site_url TEXT NOT NULL, - timezone TEXT NOT NULL DEFAULT 'UTC', - allowed_data_source_keys TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], - allowed_data_sources_version BIGINT NOT NULL DEFAULT 0, - allowed_data_sources_updated_at TIMESTAMPTZ NULL, - settings JSONB NOT NULL DEFAULT '{}'::jsonb, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - -CREATE INDEX installation_allowed_data_sources_gin_idx -ON installation_config USING GIN (allowed_data_source_keys); - -CREATE TABLE installation_domains ( - domain TEXT PRIMARY KEY, - wordpress_site_url TEXT NOT NULL UNIQUE, - installation_name TEXT NOT NULL, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), +CREATE TABLE options ( + key TEXT PRIMARY KEY, + value JSONB NOT NULL, updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); @@ -48,18 +29,41 @@ CREATE TABLE api_keys ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), key_prefix TEXT NOT NULL UNIQUE, key_hash TEXT NOT NULL UNIQUE, - key_type TEXT NOT NULL CHECK (key_type IN ('wordpress_installation', 'admin', 'mobile_service')), + owner_id TEXT NULL, + key_type TEXT NOT NULL CHECK (key_type IN ('website', 'admin')), status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active', 'revoked')), metadata JSONB NOT NULL DEFAULT '{}'::jsonb, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), last_used_at TIMESTAMPTZ NULL, revoked_at TIMESTAMPTZ NULL ); + +CREATE UNIQUE INDEX api_keys_active_owner_id_uidx +ON api_keys (owner_id) +WHERE key_type = 'website' AND status = 'active'; ``` -Allowlist replacement uses one conditional statement that matches -`allowed_data_sources_version = expected_version`, writes the complete canonical array, increments -the version by exactly one, and sets `allowed_data_sources_updated_at` from the database clock. A +`options` is a global key/value store with no synthetic identifier. Each setting occupies one row. +AI credential keys are `openai_api_key`, `groq_api_key`, and `gemini_api_key`; each present value is +AES-256-GCM ciphertext protected by the server encryption key. Service selections use +`chat_ai_router`, `chat_ai_model`, `embedding_ai_router`, and `embedding_ai_model`. Retrieval uses +`allowed_data_source_keys`, `allowed_data_sources_version`, and, after the first allowlist update, +`allowed_data_sources_updated_at`. AI configuration applies to every installation and must never be +copied into `api_keys.metadata`. Credential option values and masked fragments are never returned. + +Single and bulk option writes assign the row `updated_at` value with the database clock. Callers do +not provide this persistence timestamp. + +The global-configuration cutover clears all existing rows from `api_keys`, the former admin +authentication tables, and the legacy `installation_domains` registry after the provider has been copied to +the configuration store. This deliberately invalidates every previously issued installation/admin +credential. Follow-up cleanup migrations copy the retrieval allowlist into the configuration store, drop +`installation_config`, and drop the unused `installation_domains` table. + +Allowlist replacement uses one conditional statement that matches the JSON number stored under +`allowed_data_sources_version` to `expected_version`, writes the complete canonical array under +`allowed_data_source_keys`, increments the version by exactly one, and stores the database clock +under `allowed_data_sources_updated_at`. A zero-row update is `409 retrieval_config_conflict`; it must not retry against a newer version or partially alter the array. The initial version is `0`, including before the first allowlist sync. @@ -76,17 +80,25 @@ For WordPress installation credentials, `key_prefix` is the unique digest of the complete high-entropy API key. The digest is used only after the prefix selects a candidate row and is compared in constant time. Plaintext keys are never persisted. -The `metadata` object for a WordPress installation key contains its fixed `scopes`, a -canonical `domain`, `wordpress_site_url`, `rotation_id`, and either `rotated_from_key_ids` on the -newly issued key or `revocation_reason` plus `replaced_by_key_id` on keys revoked by rotation. -Provisioning/rotation atomically upserts the domain registry row, ensures the singleton -`installation_config` row exists for shared settings, inserts the new key, and revokes every -previously active `wordpress_installation` key for the same canonical domain only. Credentials for -other registered domains remain active. A failed transaction must leave the prior credential active. +The request's trimmed `provisioning_id` is persisted in the WordPress installation key's dedicated +`owner_id` column. A provisioning identity accepts any Unicode string after trimming, is 5 through +255 characters, and is compared exactly after that normalization. The `metadata` object contains +only the fixed `scopes` and a `revocation_reason` after disconnect. The partial unique index on +`owner_id` enforces at most one active WordPress installation key per identity under concurrency. + +Provisioning creates `website` credentials. The only other supported persistent key type is +`admin`; the obsolete `mobile_service` type is not supported, and migration removes any rows that +used it. + +Provisioning never rotates or revokes an existing key. When an active row already owns the requested +identity, the transaction returns `409 provisioning_id_already_provisioned` and creates no row. The +existing authenticated key must call disconnect first; disconnect revokes only that key. A later +provisioning request for the same identity may then create a new key while preserving the revoked +row as audit history. ## Data Source Metadata -The backend stores the identity and retrieval context of data sources represented by received content. It does not reproduce the WordPress settings UI or indexing-filter configuration. WordPress computes the allowed keys from its local settings and synchronizes them into `installation_config.allowed_data_source_keys`; the backend enforces that persisted list for RAG. +The backend stores the identity and retrieval context of data sources represented by received content. It does not reproduce the WordPress settings UI or indexing-filter configuration. WordPress computes the allowed keys from its local settings and synchronizes them into the `options` row keyed by `allowed_data_source_keys`; the backend enforces that persisted list for RAG. ```sql CREATE TABLE data_sources ( @@ -319,7 +331,8 @@ does not store placeholder hashes. All `data_sources` refresh plus matching cont transaction. Review writes resolve both the classified parent source and the composite parent listing before inserting, returning `409 parent_listing_missing` with no orphan content row when absent. -Registering or refreshing `data_sources` never modifies `installation_config.allowed_data_source_keys`. +Registering or refreshing `data_sources` never modifies the `options` value keyed by +`allowed_data_source_keys`. The same `source_id` remains unique only within its concrete `data_source_id`; it may exist under a different source key without collision. @@ -502,30 +515,15 @@ CREATE INDEX usage_events_type_idx ON usage_events (event_type); -- allowlist version, safe limits, source-kind count, candidate counts, and branch latencies. Query -- text, filters, result identities/content, raw scores, vectors, SQL, and provider identity are forbidden. -CREATE TABLE admin_users ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - email TEXT NOT NULL UNIQUE, - password_hash TEXT NOT NULL, - display_name TEXT NOT NULL DEFAULT '', - status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active', 'disabled')), - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - -CREATE TABLE admin_sessions ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - admin_user_id UUID NOT NULL REFERENCES admin_users(id) ON DELETE CASCADE, - session_hash TEXT NOT NULL UNIQUE, - expires_at TIMESTAMPTZ NOT NULL, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - CREATE TABLE schema_migrations ( version TEXT PRIMARY KEY, applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); ``` +Admin authentication uses `api_keys.key_type=admin`. The admin username is stored as `owner_id`, +fixed admin scopes are stored in metadata, and no separate admin-user or session table exists. + ## LangGraph Checkpoints Use the official LangGraph checkpoint storage package or a local PostgreSQL checkpoint table during implementation. Keep checkpoint rows separate from `conversation_messages`; checkpoints are replay/recovery state, while conversation tables are product/audit state. diff --git a/server/GROUNDED_CHAT_CONTRACT.md b/server/GROUNDED_CHAT_CONTRACT.md index b888740..82962c7 100644 --- a/server/GROUNDED_CHAT_CONTRACT.md +++ b/server/GROUNDED_CHAT_CONTRACT.md @@ -59,9 +59,16 @@ no unsupported factual claim, and returns empty grounded arrays. ## 4. Provider-Neutral Generation Boundary -The provider registry resolves `AI_PROVIDER` once during startup. Only `openai` and `groq` are -registered. Orchestration, tools, HTTP routes, and persistence receive the selected adapter through -the common boundary and never branch on its name. +The router registry resolves `chat_ai_router`, `chat_ai_model`, and that router's encrypted +credential from `options` for every turn. `openai`, `groq`, and `gemini` are registered. Selection +and credentials are application-wide, never installation-specific, and never come from provider, +model, or key environment variables. Orchestration, tools, HTTP routes, and conversation +persistence receive the selected adapter through the common boundary and never branch on its name. + +Provider resolution occurs after request/authentication validation but before a conversation turn, +retrieval, tool, or upstream provider call is created. Missing, incomplete, or undecryptable stored +chat configuration returns `503 chat_ai_not_configured` without falling back to another router, a +local adapter, or environment configuration. The internal request contains only: @@ -91,16 +98,16 @@ public errors or durable records. ### OpenAI Responses adapter -The OpenAI adapter uses only `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `OPENAI_CHAT_MODEL`, and -`AI_REQUEST_TIMEOUT_MS`. It calls the configured base URL's `/responses` endpoint, supplies the +The OpenAI adapter uses only the stored installation API key and chat model plus the non-secret +`OPENAI_BASE_URL` and `AI_REQUEST_TIMEOUT_MS` endpoint controls. It calls the configured base URL's `/responses` endpoint, supplies the complete server-owned input, uses strict function schemas and a strict JSON-schema text format, and sets `store: false`. It never sends or persists `previous_response_id`, a provider conversation ID, or a provider response ID. ### Groq Responses adapter -The Groq adapter uses only `GROQ_API_KEY`, `GROQ_BASE_URL`, `GROQ_CHAT_MODEL`, and -`AI_REQUEST_TIMEOUT_MS`. It calls the configured base URL's `/responses` endpoint and supplies the +The Groq adapter uses only the stored installation API key and chat model plus the non-secret +`GROQ_BASE_URL` and `AI_REQUEST_TIMEOUT_MS` endpoint controls. It calls the configured base URL's `/responses` endpoint and supplies the same complete server-owned history. It omits unsupported state and request parameters, including `store`, `previous_response_id`, `conversation`, `truncation`, `include`, `prompt`, `prompt_cache_key`, and `safety_identifier`. diff --git a/server/HYBRID_SEARCH_PLAN.md b/server/HYBRID_SEARCH_PLAN.md index 01a38b5..be37180 100644 --- a/server/HYBRID_SEARCH_PLAN.md +++ b/server/HYBRID_SEARCH_PLAN.md @@ -115,7 +115,7 @@ The same allowed keys and structured predicates must constrain both BM25 and vec For each normalized query: -1. Load `installation_config.allowed_data_source_keys`; an empty list fails closed. +1. Load the `options` value keyed by `allowed_data_source_keys`; an empty list fails closed. 2. Intersect model-selected keys with the persisted list. 3. Extract validated structured constraints and build kind-specific predicates. 4. Generate or load the query embedding. diff --git a/server/OPERATIONS_ADMIN_CONTRACT.md b/server/OPERATIONS_ADMIN_CONTRACT.md index 5da7899..f41a47a 100644 --- a/server/OPERATIONS_ADMIN_CONTRACT.md +++ b/server/OPERATIONS_ADMIN_CONTRACT.md @@ -2,44 +2,39 @@ ## 1. Scope -This contract is normative for SV-US-012. It fixes admin authentication, diagnostics, privacy-safe -usage aggregation, reindex coordination, operational metrics, and alert guidance. It does not make +This contract is normative for SV-US-012. It fixes admin authentication, diagnostics, reindex +coordination, operational metrics, and alert guidance. It does not make the backend an editorial source of truth: WordPress still re-sends content for every reindex. ## 2. Admin Authentication -Admin routes accept either: +Admin routes accept either an active `api_keys.key_type=admin` bearer key with the required +`admin:read` or `admin:write` scope, or an active `api_keys.key_type=website` bearer key with its +fixed `operations:read` scope. -- an active `api_keys.key_type=admin` bearer key whose server-owned metadata contains the required - `admin:read` or `admin:write` scope; or -- an unexpired opaque admin session created by `POST /admin/sessions` and sent as a bearer token. +The launch provisioning route creates `website` keys with server-defined installation scopes; +those keys may access the administrative routes described by this contract. Malformed, unknown, +hash-mismatched, and revoked credentials return the generic +`401 authentication_error`. Only successful authorization updates key last-use. -The launch provisioning route continues to create only `wordpress_installation` keys with the -server-defined installation scopes. A valid installation key on an admin route returns the same generic -`403 forbidden` as any authenticated wrong-scope key and does not fall back to session parsing. -Malformed, unknown, hash-mismatched, expired, disabled-user, and revoked credentials return the -generic `401 authentication_error`. Only successful authorization updates key/session last-use. +`POST /auth/admin` is the admin-key provisioning boundary. Its strict body contains `username` and +`password`; it validates against `ASK_SUNNY_ADMIN_USERNAME` and `ASK_SUNNY_ADMIN_PASSWORD` using +constant-time comparisons, then creates an `api_keys.key_type=admin` credential owned by the +normalized username with fixed `admin:read` and `admin:write` scopes. The plaintext API key is +returned once, and only its prefix and SHA-256 digest are persisted. Login failures never reveal +which field differed and never log credentials. There are no admin-user or admin-session tables. -`POST /admin/sessions` is the bootstrap login boundary. Its strict body contains `email` and -`password`; it validates against `ASK_SUNNY_ADMIN_EMAIL` and `ASK_SUNNY_ADMIN_PASSWORD` using -constant-time comparisons, upserts the configured admin user with a one-way password hash, and -returns a new opaque token once with `expires_at`. Only the token digest is stored. Login failures -never reveal which field differed and never log credentials. Sessions expire after -`ASK_SUNNY_ADMIN_SESSION_TTL_SECONDS`; creating a session deletes expired sessions for that user. +For admin keys, read routes require `admin:read` and mutations require `admin:write`. Website keys +use `operations:read` for both read and mutation routes; authorization still validates the stored +key type and server-owned fixed scope. -The session token format is `ask_admin_session_<43 base64url characters>`. Admin API-key creation -and rotation remain an operator-controlled secret-management operation outside the public HTTP API. - -Read routes require `admin:read`; reindex creation requires `admin:write`. - -WordPress installation operations are a separate boundary. Active `wordpress_installation` keys -receive `operations:read`; this scope authorizes only `GET /installation/diagnostics` and -`GET /installation/usage`. The migration adds the scope idempotently to existing active installation -credential metadata without rotating credentials. It does not authorize any `/admin/*` route. +Active `website` keys receive `operations:read`; this scope authorizes the website diagnostics +projection, AI configuration, safe option inspection, and reindex coordination. Diagnostics remain +projected by key type, so website keys do not receive admin-only deployment details. ## 3. Diagnostics -`GET /admin/diagnostics` returns safe current operational state: +`GET /system/diagnostics` with an `admin:read` key returns safe current operational state: - deployment mode and service version; - database, Redis, and pool total/idle/waiting state; @@ -56,34 +51,9 @@ credential metadata without rotating credentials. It does not authorize any `/ad Diagnostics are read-only and bounded. Dependency probe failures return the same schema with safe `error`/`unavailable` states and do not expose SQL or exception text. -## 4. Usage - -`GET /admin/usage` requires RFC3339 `from` and `to`, with `from < to`, a maximum inclusive range of -92 days, and optional `event_type` from the stored server event taxonomy. Unknown parameters or -event types return `400 validation_error`. - -The response contains `from`, `to`, optional filter, totals, and UTC daily buckets. Totals include: - -- chat turns, indexing/mutation events, and retrieval events; -- successes, errors, average and p95 latency; -- input/output tokens and retrieval result count; -- vector, BM25, and fused candidate counts; -- vector-only fallback count and error counts by stable `error_code`. - -Daily buckets contain only date, event counts, errors, average latency, tokens, retrieval count, and -fallback count. Aggregation reads the existing safe `usage_events` columns/metadata. It never -selects or returns conversation message content, tool arguments/results, visitor identities, -source/result identities, query/filter text, raw metadata, provider name/model, or provider state. -Provider identity remains runtime diagnostic/ephemeral metric context and is not added to usage rows. - -The installation usage route applies the same validation and aggregation but returns only the safe -projection for the authenticated WordPress installation. It never returns identities, messages, -queries, filters, source/result identities, raw metadata, provider bodies, or administrative session -state. - -## 4.1 Installation Diagnostics Projection +## 4. Website Diagnostics Projection -`GET /installation/diagnostics` reuses the operational probes but explicitly projects only service +`GET /system/diagnostics` with a website key reuses the operational probes but explicitly projects only service version and dependency status; selected AI and embedding configuration; ParadeDB, vector, BM25, and requested/effective hybrid state; retrieval-configuration version/update time; content counts; and the latest safe indexing time/outcome. It excludes credentials, URLs, pool internals, package paths, @@ -101,8 +71,9 @@ Keys must be a non-empty subset of currently stored data-source descriptors and a retained source. The route inserts one record and returns `202` with `ok`, `job_id`, `status=awaiting_wordpress`, requested keys, and `created_at`. -`GET /admin/reindex/:job_id` requires `admin:read` and returns that same bounded record or a generic -`404 reindex_job_not_found`. `GET /admin/diagnostics` exposes the latest record. No backend worker +`GET /admin/reindex/:job_id` requires `admin:read` for admin keys or `operations:read` for website +keys and returns that same bounded record or a generic +`404 reindex_job_not_found`. The admin projection of `GET /system/diagnostics` exposes the latest record. No backend worker claims to rebuild WordPress content. The administrator/plugin uses the record as coordination, causes WordPress to re-send eligible source-of-truth payloads through normal idempotent content routes, and verifies indexing/usage state. Completion mutation is deferred until a WordPress-owned @@ -110,7 +81,7 @@ reporting contract exists; launch status therefore remains honestly `awaiting_wo ## 6. Metrics And Correlation -The runtime exposes safe metrics through diagnostics/usage and structured logs: +The runtime exposes safe metrics through diagnostics and structured logs: - request correlation, route, status, latency, and stable error code; - job ID/correlation and requested source count; diff --git a/server/REST_API_CONTRACT.md b/server/REST_API_CONTRACT.md index 486c9e5..185b6da 100644 --- a/server/REST_API_CONTRACT.md +++ b/server/REST_API_CONTRACT.md @@ -41,25 +41,36 @@ Returns server health. "status": "enabled", "reason": null }, - "ai_provider": "openai", + "chat_ai_router": "openai", + "chat_ai_model": "gpt-5.4-mini", + "embedding_ai_router": "openai", + "embedding_ai_model": "text-embedding-3-small", + "ai_routers": { + "openai": true, + "groq": false, + "gemini": false + }, "redis": "disabled" } ``` `hybrid_search.status` may report `disabled` or `degraded` when package compatibility is unproven, `pg_search`, a required BM25 index, or smoke verification is unavailable. Health must not report BM25 as enabled merely because the environment flag is set. A requested-but-ineffective hybrid configuration reports `requested: true`, `effective: false`, and a stable reason code. -### `POST /auth/provision-installation` +`ai_routers` contains every supported router key. A value is `true` only when that router has an +encrypted credential row in `options`; it does not imply that an upstream request was made during +the health check or that either AI service currently selects that router. + +### `POST /auth/provision` -Creates or rotates the WordPress installation API key. Uses the provisioning secret, not an existing installation key. +Creates a WordPress installation API key for one unprovisioned identity. Uses the provisioning +secret, not an existing installation key, and never rotates an active credential. Request: ```json { "provisioning_key": "long-shared-secret", - "domain": "example.com", - "wordpress_site_url": "https://example.com", - "installation_name": "Example WordPress Site" + "provisioning_id": "wordpress-production" } ``` @@ -77,20 +88,13 @@ Response: "conversations:read", "operations:read" ], - "rotated_previous_key": false, - "installation": { - "domain": "example.com", - "timezone": "UTC" - } + "provisioning_id": "wordpress-production" } ``` -`domain` is a lower-case hostname without a scheme, port, path, query, or fragment. -`wordpress_site_url` is an absolute HTTPS URL whose hostname exactly matches `domain`; subdirectory -installations may retain a path, while query and fragment components are rejected. The backend trims -the installation name, canonicalizes the identity, and registers that domain for the singleton -backend. Later requests may register additional canonical domains served by the same backend; each -domain receives and rotates its own credential without revoking credentials for other domains. +`provisioning_id` accepts any string after trimming surrounding whitespace, must contain 5 through +255 characters, and is stored exactly after trimming. The request rejects every field other than +`provisioning_key` and `provisioning_id`. The key format is `ask_live_<16-lowercase-hex-key-id>_<43-character-base64url-secret>`. The unique `key_prefix` is the format through the key-id segment and may be logged for credential identification; @@ -99,16 +103,14 @@ of the complete high-entropy API key. A provisioned WordPress installation key r listed server-defined scopes; the caller cannot add scopes in the request. The migration that introduces `operations:read` adds it idempotently to every active -`wordpress_installation` credential's stored scope metadata. It does not rotate or reveal the +`website` credential's stored scope metadata. It does not rotate or reveal the credential, change its status, or grant access to `/admin/*` routes. -Provisioning is also the rotation operation for a canonical domain. The first successful request for -a domain returns `rotated_previous_key: false`. A later successful request for the same canonical -domain creates a new key, immediately revokes every prior active `wordpress_installation` key for -that domain in the same database transaction, returns `rotated_previous_key: true`, and writes -cross-referenced rotation metadata on the new and revoked rows. If any part of the transaction -fails, the previous key remains active and no new key is issued. The plaintext key is returned only -in this response and cannot be recovered. +If an active key already owns the normalized provisioning identity, provisioning returns +`409 provisioning_id_already_provisioned`, creates no key, and leaves the existing credential +unchanged. The existing key must successfully call `POST /auth/disconnect` before that +identity can be provisioned again. Disconnect never reveals or rotates a key. The plaintext key is +returned only in its successful provisioning response and cannot be recovered. An invalid provisioning secret returns the same `401 authentication_error` used for invalid bearer credentials and performs no installation or key write. Provisioning-secret comparison is @@ -119,6 +121,43 @@ same `401 authentication_error`. An authenticated key missing a route's required `403 forbidden` without naming the missing scope. Only a fully authorized request updates `last_used_at`. +### `POST /auth/admin` + +Validates a strict `username` and `password` request against `ASK_SUNNY_ADMIN_USERNAME` and +`ASK_SUNNY_ADMIN_PASSWORD`, then creates a persistent `admin` API key with `admin:read` and +`admin:write` scopes. The response returns the plaintext API key once, using the same key format as +website provisioning; only its prefix and SHA-256 digest are stored. Invalid credentials return the +generic `401 authentication_error`. No admin user or session record is created. + +### AI configuration routes + +All AI configuration routes use the `/system/ai-config` prefix. Admin keys require `admin:read` for +reads or `admin:write` for mutations; website keys require their fixed `operations:read` scope. + +- `GET /system/ai-config/routers` returns the hardcoded OpenAI, Groq, and Gemini router catalog with + compatible text and embedding model IDs. +- `POST /system/ai-config/routers` accepts exactly `router_type` and `api_key`, validates the key + against the selected router before encrypting and atomically inserting or rotating it, and never + returns key material. +- `DELETE /system/ai-config/routers/{router_type}` idempotently removes that credential and + atomically clears every chat or embedding selection that references it. It never selects a + fallback router. +- `PUT /system/ai-config/chat` accepts exactly `router_type` and `model`, requires a connected + router and catalogued text model, and atomically replaces `chat_ai_router` and `chat_ai_model`. +- `PUT /system/ai-config/embedding` applies the equivalent embedding selection and reports + `reindex_required=true` when indexed content exists and the selection changed. + +`GET /system/options` requires `admin:read` for admin keys or `operations:read` for website keys and +returns explicitly classified safe option rows plus +per-router configured booleans. Encrypted credentials, plaintext, masked fragments, and unknown +option keys are never returned. The retired `POST /system/provider` route is absent. + +### `POST /auth/disconnect` + +Requires any active `website` or `admin` API key. It revokes only the presented key and records the +generic disconnect reason. After disconnect succeeds, the key receives `401 authentication_error` +on every protected route. A disconnected website key's `provisioning_id` may be provisioned again. + ## Retrieval Configuration Routes ### `PUT /retrieval/allowed-data-sources` @@ -206,8 +245,8 @@ Response: ``` Before the first sync, the response contains an empty list, version `0`, and `updated_at: null`. -This route is the narrow installation-facing diagnostic surface; it does not weaken the separate -admin-authentication requirement for `GET /admin/diagnostics`. +This route is the narrow retrieval-configuration surface; it does not weaken the separate admin-key +requirement for the administrative projection of `GET /system/diagnostics`. The list uses concrete `data_source_key` classifications rather than broad `source_kind` values. For example, `directorist:events` and `directorist:events:reviews` can be allowed independently even though both are Directorist data. @@ -437,7 +476,7 @@ Tombstones every active record for a key in its source-kind table. WordPress cal explicit admin **Delete all indexed data** action or an equivalent deliberate maintenance operation. Disabling an optional WordPress source must not call this route. A missing key is an idempotent success with zero items. The operation updates only the resolved content table in one transaction and -never inserts, deletes, or updates `installation_config.allowed_data_source_keys`. +never inserts, deletes, or updates the `options` value keyed by `allowed_data_source_keys`. Request: @@ -500,7 +539,10 @@ The chat caller does not provide `allowed_data_source_keys`. The backend loads i `channel` accepts `web`, `mobile`, or `admin_test`. WordPress sends `web` for the public widget and `admin_test` only from its capability-protected Test Chat route. Channel is product context, not an AI-provider selector. -The chat caller also cannot choose the AI provider or model. The server uses `AI_PROVIDER` and the selected provider's environment configuration for the entire turn. +The chat caller cannot override the AI router or model. The server resolves the application-wide +database-backed chat selection and connected router credential for the entire turn. Missing or +incomplete configuration returns `503 chat_ai_not_configured` before a conversation turn, +retrieval, tool, or upstream router call is created. SV-US-008 adds no public retrieval endpoint. `search_content` and `get_content_detail` are server-owned application/tool boundaries used by the later chat workflow. Their validated filter @@ -591,18 +633,19 @@ Missing, mismatched, deleted, and malformed conversation IDs share: } ``` -## Installation Operations Routes +## System Operations Routes -These routes require an active WordPress installation credential with `operations:read`. They are -safe, read-only projections for the WordPress plugin and do not accept an admin key/session in place -of installation authentication. Installation credentials remain forbidden from every `/admin/*` -route. +System routes accept only active API keys. Website credentials use their bounded operational +projection; admin credentials receive the administrative projection when their scopes authorize it. +Website credentials remain forbidden from every `/admin/*` route. -### `GET /installation/diagnostics` +### `GET /system/diagnostics` -Returns the bounded operational state needed by WordPress without credentials, URLs, visitor or +Requires `operations:read` for a `website` key or `admin:read` for an `admin` key. Website requests +return the bounded operational state needed by WordPress without credentials, URLs, visitor or conversation data, query/content text, raw errors, pool details, package-install coordinates, or -other admin-only deployment data. +other admin-only deployment data. Admin requests return the full safe administrative diagnostics +projection documented below. ```json { @@ -631,26 +674,10 @@ other admin-only deployment data. Dependency probe failures preserve this shape with safe `unavailable` or `degraded` values and a stable reason. The projection is read-only and bounded to the provisioned installation's data. -### `GET /installation/usage` - -Uses the same `from`, `to`, and optional `event_type` validation as `GET /admin/usage`, including the -92-day maximum range. It returns only the installation's totals and UTC daily buckets from the safe -usage projection documented in the operations contract. - -```json -{ - "from": "2026-07-01T00:00:00Z", - "to": "2026-07-20T00:00:00Z", - "event_type": null, - "totals": {"events": 150, "successes": 147, "errors": 3, "average_latency_ms": 420, "p95_latency_ms": 900}, - "daily": [] -} -``` - ## Admin Routes -Admin routes require an admin API key or admin session. -Exact session, scope, diagnostics, usage, reindex tracking, privacy, and failure behavior is +Admin routes require an admin API key. +Exact scope, diagnostics, reindex tracking, privacy, and failure behavior is normative in [`OPERATIONS_ADMIN_CONTRACT.md`](OPERATIONS_ADMIN_CONTRACT.md). ### `POST /admin/reindex` @@ -676,30 +703,7 @@ Response: } ``` -### `GET /admin/usage` - -Returns usage and latency metrics. - -Query parameters: - -- `from` -- `to` -- `event_type` - -Response: - -```json -{ - "totals": { - "chat_turns": 120, - "indexing_events": 30, - "errors": 2 - }, - "daily": [] -} -``` - -### `GET /admin/diagnostics` +### Admin projection from `GET /system/diagnostics` Returns operational state. @@ -780,7 +784,7 @@ Returns operational state. - Deleted content requires only `data_source_key` and `source_id`. - WordPress applies indexing filters before sending content and synchronizes source allowance separately. Every backend candidate query, vector search, detail lookup used by RAG, and model tool call must constrain results to the stored allowlist. - Chat routes must never accept raw SQL, arbitrary tool names, or model overrides from clients. -- Chat routes must reject or ignore caller-supplied `ai_provider`, provider API keys, base URLs, and model names; only environment configuration is authoritative. +- Chat routes must reject caller-supplied provider overrides; only the `options` key/value settings for provider type, encrypted API key, and chat model are authoritative. - Hybrid retrieval must constrain both BM25 and vector candidates to persisted allowed data-source keys and active records before fusion. ### Content And Metadata Safety Limits diff --git a/server/SEMANTIC_SEARCH_ARCHITECTURE_AND_FLOW_GUIDE.md b/server/SEMANTIC_SEARCH_ARCHITECTURE_AND_FLOW_GUIDE.md index 9ce05c3..62e60e3 100644 --- a/server/SEMANTIC_SEARCH_ARCHITECTURE_AND_FLOW_GUIDE.md +++ b/server/SEMANTIC_SEARCH_ARCHITECTURE_AND_FLOW_GUIDE.md @@ -50,8 +50,8 @@ This guide defines the semantic indexing, retrieval, chat, and failure flows. De - API: Hono. - Orchestration: LangGraph.js. - Database: PostgreSQL with ParadeDB `pg_search`, pgvector, and `pgcrypto`. -- Chat generation: provider-neutral adapter selected by `AI_PROVIDER=openai|groq`. -- Embeddings: independently selected with `EMBEDDING_PROVIDER` and embedding environment settings. +- Chat generation: router-neutral adapter selected by `chat_ai_router` and `chat_ai_model` options. +- Embeddings: independently selected by `embedding_ai_router` and `embedding_ai_model` options. - Deployment: native services or optional Docker Compose. - Response transport: one complete JSON response; no partial token streaming. @@ -61,15 +61,7 @@ Relevant environment contract: DATABASE_URL=postgres://ask_sunny:strong-password@127.0.0.1:5432/ask_sunny PG_POOL_MAX=10 -AI_PROVIDER=openai -OPENAI_API_KEY=replace-with-openai-api-key -OPENAI_CHAT_MODEL=replace-with-supported-openai-model -GROQ_API_KEY=replace-with-groq-api-key -GROQ_CHAT_MODEL=replace-with-supported-groq-model - -EMBEDDING_PROVIDER=openai -OPENAI_EMBEDDINGS_URL=https://api.openai.com/v1/embeddings -EMBEDDING_MODEL=text-embedding-3-small +AI_REQUEST_TIMEOUT_MS=45000 EMBEDDING_DIMENSIONS=1536 HYBRID_SEARCH_ENABLED=false @@ -91,7 +83,9 @@ MAX_METADATA_NESTING_DEPTH=4 The hybrid flag begins `false` for installation or upgrade. It changes to `true` only after the `pg_search` package is proven compatible with the running PostgreSQL major version, execution OS, and architecture and all verification checks pass. The exact gate is defined in [`HYBRID_SEARCH_PLAN.md`](HYBRID_SEARCH_PLAN.md). -`AI_PROVIDER` affects generation only. Provider identity and provider-specific conversation identifiers are not stored in application tables. Changing generation provider does not change embedding dimensions or reindex content. +Stored installation provider configuration affects generation only. Provider-specific conversation +identifiers are not stored in conversation tables. Changing generation provider does not change +embedding dimensions or reindex content. ## 4. High-Level Architecture @@ -129,7 +123,10 @@ The source-kind repositories remain separate: | `directorist_review` | `directorist_reviews` | `directorist_review_embeddings` | | `wordpress_post` | `wordpress_content` | `wordpress_content_embeddings` | -`data_sources` stores source labels and retrieval context. `installation_config.allowed_data_source_keys` stores the authoritative retrieval allowlist. Disabling an optional source removes its key from that list but does not delete indexed rows. An explicit delete operation tombstones content. +`data_sources` stores source labels and retrieval context. The `options` value keyed by +`allowed_data_source_keys` stores the authoritative retrieval allowlist. Disabling an optional +source removes its key from that list but does not delete indexed rows. An explicit delete operation +tombstones content. Every retrieval begins by loading the stored allowlist. Model-selected or request-derived keys are intersected with it; neither a chat caller nor a model can expand it. A missing or empty allowlist fails closed with no candidates. @@ -367,7 +364,7 @@ Primary semantic-search routes are: - `POST /content/delete`: tombstone one source record. - `POST /content/delete-by-data-source`: explicit administrative tombstone operation, not a disable-source action. - `POST /chat`: execute one complete grounded chat turn. -- `GET /admin/diagnostics`: report search capability and indexing state. +- `GET /system/diagnostics`: report search capability and indexing state. `POST /chat` never accepts an AI provider, model, or caller-supplied source allowlist. It accepts conversational context only; server environment and persisted installation configuration control generation and retrieval. @@ -470,7 +467,7 @@ Diagnostics must report requested/effective hybrid mode, PostgreSQL version, `pg - Review evidence remains linked to its parent listing. - Citations contain valid direct URLs and claims trace to retrieved evidence. - Multi-turn context and clarification behavior work without provider-hosted conversation state. -- Switching `AI_PROVIDER` needs no database migration and does not change retrieval. +- Switching the stored installation provider does not change retrieval or embedding dimensions. - Native and Docker deployments both pass the `pg_search` package compatibility gate. - A mismatch or missing extension keeps hybrid disabled and vector-only diagnostics honest. - Backup, migration, restore, reindex, and cache invalidation procedures are rehearsed. diff --git a/server/USER_STORIES.md b/server/USER_STORIES.md index f5afb2c..ae9681b 100644 --- a/server/USER_STORIES.md +++ b/server/USER_STORIES.md @@ -348,15 +348,15 @@ checkpoint, history route, deletion, anonymization, and retention rules are defi 4. **Given** a successful turn, **when** `POST /chat` returns, **then** one non-streaming JSON payload contains the conversation ID, message ID, answer, recommendations, and optional follow-up questions. 5. **Given** a retrieval, model, schema, or timeout failure, **when** the turn ends, **then** the server returns a stable friendly error/fallback, persists the failure, and does not present unsupported claims. 6. **Given** caller-supplied SQL, model overrides, raw tool names, or allowed-source settings, **when** validation runs, **then** those values cannot alter server policy or execution. -7. **Given** `AI_PROVIDER=openai`, **when** a turn runs, **then** the OpenAI adapter uses only OpenAI environment configuration and returns the common internal response shape. -8. **Given** `AI_PROVIDER=groq`, **when** a turn runs, **then** the Groq adapter uses only Groq environment configuration, excludes unsupported provider parameters, supplies server-owned conversation history, and returns the same internal response shape. -9. **Given** an invalid provider value or missing selected-provider configuration, **when** the service starts, **then** startup fails without exposing any API key. +7. **Given** a stored OpenAI installation provider, **when** a turn runs, **then** the OpenAI adapter uses only that database-backed configuration and returns the common internal response shape. +8. **Given** a stored Groq installation provider, **when** a turn runs, **then** the Groq adapter uses only that database-backed configuration, excludes unsupported provider parameters, supplies server-owned conversation history, and returns the same internal response shape. +9. **Given** missing or invalid stored provider configuration, **when** chat is requested, **then** the request fails early without exposing any API key. 10. **Given** any conversation, message, tool-call, or usage record, **when** it is persisted, **then** no AI-provider discriminator or provider-specific conversation state is written to the database. **Tasks** -- [ ] Add `AI_PROVIDER=openai|groq` as the single chat-generation switch. -- [ ] Add environment-only OpenAI and Groq keys, base URLs, models, and shared provider timeout. +- [ ] Add database-backed OpenAI and Groq generation selection and credentials. +- [ ] Keep non-secret provider base URLs and the shared provider timeout in deployment configuration. - [ ] Keep embedding provider, model, URL, and dimensions independently configured. - [ ] Implement a common generation-provider interface with OpenAI and Groq Responses adapters. - [ ] Resolve adapters through a runtime registry so orchestration, routes, domain services, and persistence contain no provider-name branches. @@ -380,24 +380,23 @@ checkpoint, history route, deletion, anonymization, and retention rules are defi **User story** -> As a **server operator**, I want diagnostics, usage reporting, and controlled reindex coordination, so that I can detect failures and support the WordPress integration. +> As a **server operator**, I want diagnostics and controlled reindex coordination, so that I can detect failures and support the WordPress integration. **Acceptance criteria** 1. **Given** an authorized diagnostics request, **when** it runs, **then** it reports native-service or Docker dependency health, ParadeDB extensions and BM25 indexes, hybrid mode, runtime generation/embedding configuration, allowlist version, source counts, and latest indexing state. -2. **Given** an authorized usage query with a date range, **when** it runs, **then** it returns chat, indexing, latency, token, BM25/vector/fused retrieval, fallback, and error aggregates without exposing private message content or persisting provider identity. -3. **Given** a reindex coordination request, **when** it is accepted, **then** it receives a tracked status while WordPress remains responsible for re-sending source-of-truth content. -4. **Given** a wrong-scope installation key, **when** an admin-only endpoint is called, **then** access is denied. -5. **Given** an operational failure, **when** thresholds are exceeded, **then** logs and metrics provide enough correlation to diagnose the affected request or job. +2. **Given** a reindex coordination request, **when** it is accepted, **then** it receives a tracked status while WordPress remains responsible for re-sending source-of-truth content. +3. **Given** an active website key with its fixed `operations:read` scope, **when** an administrative endpoint is called, **then** it is authorized while diagnostics remain projected for the website key type. +4. **Given** an operational failure, **when** thresholds are exceeded, **then** logs and metrics provide enough correlation to diagnose the affected request or job. **Tasks** -- [ ] Implement `GET /admin/diagnostics` and `GET /admin/usage`. +- [ ] Implement the admin projection of `GET /system/diagnostics`. - [ ] Implement `POST /admin/reindex` as a tracked coordination boundary. -- [ ] Add admin key/session scope enforcement. +- [ ] Add admin API-key scope enforcement. - [ ] Instrument chat, BM25/vector/fused retrieval, indexing, database-pool, selected-provider, embedding-provider, and error metrics. - [ ] Add health, latency, error-rate, rate-limit, and stale-index alert guidance. -- [ ] Add diagnostics, usage, authorization, and privacy tests. +- [ ] Add diagnostics, authorization, and privacy tests. **Dependencies:** SV-US-007, SV-US-011 **Priority:** Must have @@ -447,20 +446,20 @@ checkpoint, history route, deletion, anonymization, and retention rules are defi **User story** -> As a **WordPress administrator**, I want installation-scoped diagnostics and usage telemetry, so that I can operate the integration without receiving a backend administrator credential. +> As a **WordPress administrator**, I want website-scoped diagnostics, so that I can operate the integration without receiving a backend administrator credential. **Acceptance criteria** -1. **Given** an active WordPress installation key with `operations:read`, **when** installation diagnostics or usage is requested, **then** only the safe operational projection required by the plugin is returned. +1. **Given** an active website key with `operations:read`, **when** system diagnostics is requested, **then** only the safe operational projection required by the plugin is returned. 2. **Given** an existing active WordPress installation key, **when** the scope migration runs, **then** `operations:read` is added idempotently without changing its secret, status, or other scopes. -3. **Given** an installation key, **when** an `/admin/*` route is requested, **then** it remains forbidden and cannot gain administrative session or write authority. -4. **Given** diagnostics or usage data, **when** it is projected for WordPress, **then** credentials, visitor data, messages, queries, content-record identities, raw errors, and admin-only deployment details are absent while bounded counts may remain grouped by data-source key. -5. **Given** a degraded dependency or bounded usage query, **when** the route responds, **then** it preserves the documented stable shape, validation, and correlation behavior. +3. **Given** a website key, **when** an `/admin/*` route is requested, **then** it remains forbidden and cannot gain administrative write authority. +4. **Given** diagnostics data, **when** it is projected for WordPress, **then** credentials, visitor data, messages, queries, content-record identities, raw errors, and admin-only deployment details are absent while bounded counts may remain grouped by data-source key. +5. **Given** a degraded dependency, **when** the route responds, **then** it preserves the documented stable shape and correlation behavior. **Tasks** - [ ] Add `operations:read` to new WordPress installation credentials and migrate active existing credential metadata idempotently. -- [ ] Add `GET /installation/diagnostics` and `GET /installation/usage` behind installation authentication. +- [ ] Add the website projection of `GET /system/diagnostics` behind API-key authentication. - [ ] Reuse the operations service through explicit safe installation projections rather than exposing `/admin/*` responses directly. - [ ] Document request, response, authorization, validation, privacy, and degraded-state contracts. - [ ] Add migration, provisioning, authorization, scoping, privacy, validation, route, and OpenAPI tests. @@ -487,6 +486,40 @@ checkpoint, history route, deletion, anonymization, and retention rules are defi **Dependencies:** SV-US-011 **Priority:** Must have +### SV-US-016 — Configure multiple AI routers and immutable identities + +**Normative contracts:** [`GROUNDED_CHAT_CONTRACT.md`](GROUNDED_CHAT_CONTRACT.md), [`REST_API_CONTRACT.md`](REST_API_CONTRACT.md), [`ARCHITECTURE.md`](ARCHITECTURE.md) + +**User story** + +> As a **server operator**, I want multiple connected AI routers with independent chat and embedding selections, so that each service can use an explicitly configured router/model while credentials remain global, encrypted, and revocable. + +**Acceptance criteria** + +1. **Given** configured OpenAI, Groq, or Gemini routers, **when** chat or embedding work resolves its selection, **then** it uses the encrypted router credential and service-specific router/model values in `options`, never installation metadata or provider/model/key environment configuration. +2. **Given** a valid provisioning secret and a trimmed `provisioning_id` of at least five characters, **when** provisioning succeeds, **then** a new `website` key is returned, its identity is stored in `api_keys.owner_id`, and only its hash and fixed scopes are otherwise persisted. +3. **Given** an active key for a `provisioning_id`, **when** the same identity is provisioned again, **then** the server returns `409 provisioning_id_already_provisioned`, creates no credential, and does not revoke or rotate the existing key. +4. **Given** the active key disconnects, **when** the same `provisioning_id` is provisioned later, **then** a new key may be created while the disconnected key remains revoked. +5. **Given** missing, incomplete, disconnected, or undecryptable chat or embedding configuration, **when** the related service is requested, **then** it returns `503 chat_ai_not_configured` or `503 embedding_ai_not_configured` before conversation, retrieval, indexing, mutation, tool, or router work begins. +6. **Given** an existing database with installation-scoped provider metadata, **when** the forward migration runs, **then** the latest valid provider is copied to global app configuration before provider metadata is removed from every installation key. +7. **Given** independently configured chat and embedding selections, **when** one selection changes, **then** the other remains unchanged; embedding dimensions remain fixed at 1536 and a changed embedding selection reports whether reindexing is required. +8. **Given** legacy authentication, site-identity, and installation-configuration data, **when** the global-configuration cutover migrations run, **then** credentials are cleared, the retrieval allowlist moves into the application configuration store, and the unused `installation_config` and `installation_domains` tables are removed. +9. **Given** the application configuration table, **when** settings are persisted, **then** it is named `options`, has no synthetic ID, and stores each setting as a distinct `key` and JSON `value` row. +10. **Given** application code needs to manage settings, **when** it accesses persistence, **then** the option repository exposes only generic `insert`, `get`, `update`, `delete`, `getByKeys`, and `updateMany` operations, contains no provider-specific helpers, and `updateMany` accepts only the option items. +11. **Given** valid configured admin username and password values, **when** `POST /auth/admin` succeeds, **then** it returns a one-time plaintext `admin` API key with fixed admin scopes and persists no admin user or session row. +12. **Given** the revised route contract, **when** clients provision, disconnect, inspect diagnostics, or manage AI configuration, **then** they use `/auth/provision`, `/auth/disconnect`, `/system/diagnostics`, and the `/system/ai-config/*` routes, while `/system/provider` and every former usage route are absent. +13. **Given** any active website or admin API key, **when** it calls `POST /auth/disconnect`, **then** only that presented key is revoked. +14. **Given** an authorized admin or website key, **when** supported routers are requested, **then** the API returns the hardcoded stable compatible router/model catalog without consulting a provider or database. +15. **Given** an authorized admin or website key and a supported router credential, **when** the router is connected or rotated, **then** the key is validated upstream before its encrypted value is atomically stored; invalid or unavailable validation performs no write. +16. **Given** a connected router and supported model, **when** chat or embedding selection is updated, **then** only the corresponding service selection is replaced and becomes effective without restarting the API. +17. **Given** a connected router used by either service, **when** that router is disconnected, **then** its credential and every dependent router/model selection are deleted atomically without selecting a fallback router. +18. **Given** an authorized admin or website key, **when** system options are requested, **then** only explicitly classified safe rows and per-router configured booleans are returned; plaintext, ciphertext, masked fragments, and unknown option keys are absent. +19. **Given** any health request, **when** router state is reported, **then** `ai_routers` includes every supported router as a boolean indicating whether its encrypted credential exists, without validating or exposing the credential. +20. **Given** an active `website` key with `operations:read`, **when** it calls an `/admin/*`, `/system/ai-config/*`, or `/system/options` route, **then** it is authorized by key type while admin keys continue to require their admin read/write scopes. + +**Dependencies:** SV-US-014, SV-US-015 +**Priority:** Must have + ## Recommended Story Order 1. SV-US-001 → SV-US-004: service, database, authentication, and retrieval policy. @@ -495,6 +528,7 @@ checkpoint, history route, deletion, anonymization, and retention rules are defi 4. SV-US-010 and SV-US-011: durable conversation and grounded chat. 5. SV-US-012 → SV-US-014: operations, security, resilience, release, and WordPress-safe telemetry. 6. SV-US-015: complete conversation message restoration. +7. SV-US-016: database-backed generation provider authority. ## Related Specifications diff --git a/shared/SETUP_AND_OPERATIONS.md b/shared/SETUP_AND_OPERATIONS.md index 3a7eae7..64b8656 100644 --- a/shared/SETUP_AND_OPERATIONS.md +++ b/shared/SETUP_AND_OPERATIONS.md @@ -78,20 +78,7 @@ Use Docker volumes for ParadeDB data and any Redis persistence. Add health check Keep all server configuration in `.env`, the native service-manager environment, or an equivalent Docker secret mechanism. Important controls include: ```dotenv -AI_PROVIDER=openai AI_REQUEST_TIMEOUT_MS=45000 - -OPENAI_API_KEY= -OPENAI_BASE_URL=https://api.openai.com/v1 -OPENAI_CHAT_MODEL= - -GROQ_API_KEY= -GROQ_BASE_URL=https://api.groq.com/openai/v1 -GROQ_CHAT_MODEL= - -EMBEDDING_PROVIDER=openai -OPENAI_EMBEDDINGS_URL=https://api.openai.com/v1/embeddings -EMBEDDING_MODEL=text-embedding-3-small EMBEDDING_DIMENSIONS=1536 HYBRID_SEARCH_ENABLED=false @@ -119,7 +106,13 @@ CONVERSATION_RETENTION_DAYS=90 CONVERSATION_DELETED_GRACE_DAYS=30 ``` -`AI_PROVIDER=openai|groq` is the single generation-provider switch. The runtime provider registry resolves the selected adapter without changing orchestration or persistence code. The selected adapter's key, base URL, and model must validate at startup. Credentials for the inactive generation provider may be omitted. Embeddings remain independently configured so changing the chat provider never silently changes vector dimensions or forces a reindex. Provider identity is not persisted in application tables. +The runtime router registries resolve connected OpenAI, Groq, and Gemini credentials plus the +independent chat and embedding router/model selections from `options` on each related request. +Missing or invalid stored configuration fails before related processing. Provider endpoints, +provider/model names, and API keys are not environment authority; only generic operational +timeouts, retries, fixed embedding dimensions, and the credential-encryption master key remain. +Changing embedding selection keeps dimensions at 1536, excludes mismatched stored vectors, and +requires WordPress content resend when `reindex_required` is reported. `HYBRID_SEARCH_ENABLED=false` is the required safe value during installation and upgrade. Hybrid is the expected production mode only after the compatibility, extension, migration, index, direct-query, and application gates below pass; then set it to `true` deliberately. @@ -146,7 +139,7 @@ Environment safety rules: - Never print provider, embedding, provisioning, database, or installation secrets. - Use long random provisioning and installation keys. - Keep the generated backend installation key only in WordPress server-side options. -- Set secure cookies for admin sessions behind HTTPS. +- Store issued admin API keys only in an approved secret manager. ## Deployment Flow @@ -324,7 +317,7 @@ Recovery sequence: - Check WordPress logs and backend logs using the correlation ID. - Verify the backend installation key and `/health`. -- Verify `AI_PROVIDER` and that provider's API key, base URL, model, and supported request parameters. +- Verify the global app provider type, configured status, model, endpoint, and supported request parameters without printing its key. - Verify the embedding-provider configuration separately. ### Results Are Irrelevant @@ -348,18 +341,27 @@ Recovery sequence: - Verify backend logs for `authentication_error`. - Rotate the provisioning key only after updating both sides. -### Emergency Installation Credential Rotation +### Emergency Installation Credential Replacement + +The global-configuration cutover invalidates all existing installation and admin credentials. After +that migration, provision each required installation identity again and create a new admin API key. +AI configuration and retrieval settings are both preserved in `app_config`. + +1. Verify the stored `provisioning_id` from a trusted administrative workflow. +2. Call `POST /auth/disconnect` with the existing key. That key is immediately revoked. +3. Send one provisioning request with the same identity and capture the returned installation key without logging it. +4. Store the new key in the WordPress server-side option before making further backend calls. +5. Run an authenticated diagnostic with the new key and confirm the old key receives the generic `401 authentication_error`. +6. Record the non-secret `key_prefix`, replacement time, identity, and operator. Never record the full key. -1. Verify the canonical WordPress domain and site URL from a trusted administrative session. -2. Send one provisioning request from trusted server-side code and capture the returned installation key without logging it. -3. Store the new key in the WordPress server-side option before making further backend calls. A successful response means every previous installation key was revoked atomically. -4. Run an authenticated diagnostic with the new key and confirm the old key now receives the generic `401 authentication_error`. -5. Record the non-secret `key_prefix`, rotation time, and operator in the incident record. Never record the full key. -6. If the request fails or its response is lost, retry provisioning; the last successful response is the only active key and must replace any earlier captured value. +Provisioning an identity that still has an active key returns +`409 provisioning_id_already_provisioned`; it never rotates that key. If a successful provisioning +response is lost, disconnect the newly created key through an authorized recovery workflow before +trying the same identity again. If the provisioning secret itself may be exposed, replace it in the backend secret store, restart the -service, update the trusted WordPress-side provisioning workflow, and only then rotate the -installation credential. Do not place either secret in command history, tickets, or logs. +service and update the trusted WordPress-side provisioning workflow. Do not place either secret in +command history, tickets, or logs. ## Production Readiness Checklist @@ -372,7 +374,7 @@ secret-free final report follow - ParadeDB, `pg_search`, and pgvector are installed and compatible; any missing or mismatched evidence keeps hybrid disabled. - Required migrations, BM25 indexes, `ANALYZE`, direct BM25 smoke queries, and application checks pass before hybrid search is enabled. - Backend `/health` and WordPress diagnostics pass. -- `AI_PROVIDER` selects a configured, verified OpenAI or Groq adapter. +- Connected router credentials and independent chat/embedding selections exist in `options`. - Initial reindex completes. - Every Directorist directory type has a required listing source, and reviews are controlled by one global optional Listing Reviews setting. - Global reviews and optional WordPress sources honor enabled state and filters. @@ -383,7 +385,7 @@ secret-free final report follow - Per-item indexing status and failures are visible in WordPress admin. - Chat works for anonymous and logged-in visitors and returns one complete response with citations and recommendations. - Widget page targeting, color scheme, position, and welcome message match the saved configuration. -- OpenAI, Groq, embedding-provider, and backend installation keys are absent from browser source. +- OpenAI, Groq, Gemini, and backend installation keys are absent from browser source. - Featured recommendations and configured promotion disclosures are labeled. - Backup and restore rehearsals pass. - Error logs and alerts are monitored.