diff --git a/.github/screenshots/vision-chat-sidecar-picker.png b/.github/screenshots/vision-chat-sidecar-picker.png new file mode 100644 index 0000000000..53a8a4ae1d Binary files /dev/null and b/.github/screenshots/vision-chat-sidecar-picker.png differ diff --git a/README.md b/README.md index 22d6956c6c..51d6f12791 100644 --- a/README.md +++ b/README.md @@ -146,8 +146,9 @@ see the [installation docs](https://opencodex.me/getting-started/installation/). [sub-agent guide](https://opencodex.me/guides/sub-agent-surface/). - **Log in once, skip the API key** — OAuth for xAI, Anthropic, and Kimi; or forward `codex login`, paste a key, or use `${ENV_VAR}` references. -- **Web search & vision sidecars** — non-OpenAI models get real web search and image understanding - through a sidecar over your ChatGPT login. +- **Web search & vision sidecars** — routed models get real web search through an OpenAI/Anthropic + sidecar, while image understanding can also use an explicit provider-backed Chat vision sidecar. + Chat vision targets use `provider/model`; see the [sidecar guide](https://opencodex.me/guides/sidecars/). - **See what's happening** — the dashboard shows providers, OAuth status, model selection, and a live request log with cache token counts. - **Clean exit, zero residue** — `ocx stop` restores Codex to its original configuration. diff --git a/docs-site/src/content/docs/fr/getting-started/how-it-works.mdx b/docs-site/src/content/docs/fr/getting-started/how-it-works.mdx index ad012d22ca..6dc30901af 100644 --- a/docs-site/src/content/docs/fr/getting-started/how-it-works.mdx +++ b/docs-site/src/content/docs/fr/getting-started/how-it-works.mdx @@ -71,8 +71,9 @@ le modèle et l’effort de raisonnement à utiliser. Les requêtes v2 conserven de transfert direct refuse de poursuivre si les informations d’identification requises sont indisponibles. 4. **Service auxiliaire de vision (facultatif)** — si le modèle routé figure dans `provider.noVisionModels` et que - la requête contient une image, opencodex fait décrire chaque image par le service auxiliaire de vision ChatGPT - configuré, puis la remplace par du texte afin qu’un modèle textuel puisse tout de même la traiter. + la requête contient une image, opencodex utilise le backend de vision configuré — OpenAI via la connexion + ChatGPT, Anthropic via OAuth ou le backend Chat réservé à la vision avec une cible explicite `provider/model` — + puis remplace chaque image par du texte afin qu’un modèle textuel puisse tout de même la traiter. Voir [Services auxiliaires](/fr/guides/sidecars/). 5. **Transfert direct rapide** — si l’adaptateur transfère directement les Responses (`openai-responses` ou diff --git a/docs-site/src/content/docs/fr/getting-started/installation.md b/docs-site/src/content/docs/fr/getting-started/installation.md index 5a9ae40163..47ea5f0c47 100644 --- a/docs-site/src/content/docs/fr/getting-started/installation.md +++ b/docs-site/src/content/docs/fr/getting-started/installation.md @@ -5,8 +5,8 @@ description: Installez le proxy opencodex (ocx), ses prérequis et vérifiez qu' opencodex installe deux commandes équivalentes, `ocx` et `opencodex`. Toutes deux lancent le même petit serveur HTTP local, fondé sur Bun. Les requêtes de modèles sont envoyées au fournisseur choisi par le -routage. Les services auxiliaires facultatifs de vision et de recherche web peuvent également utiliser votre connexion -ChatGPT lorsqu’un modèle routé en a besoin. +routage. La vision peut utiliser OpenAI via votre connexion ChatGPT, Anthropic via OAuth ou le backend Chat +réservé à la vision avec une cible explicite `provider/model` ; la recherche web utilise les backends dédiés. ## Prérequis diff --git a/docs-site/src/content/docs/fr/guides/claude-code.md b/docs-site/src/content/docs/fr/guides/claude-code.md index 8a5ccf79b2..0b91a104e9 100644 --- a/docs-site/src/content/docs/fr/guides/claude-code.md +++ b/docs-site/src/content/docs/fr/guides/claude-code.md @@ -368,8 +368,6 @@ La recherche web et la description d'images avec OAuth Anthropic réutilisent le d'empreintes précédent. Testez néanmoins ces fonctions avec votre compte et votre charge de travail avant de vous y fier pour de longues exécutions sans surveillance. - - ## Effort de raisonnement Le paramètre `/effort` de Claude Code est conservé sur l'ensemble de l'adaptateur : diff --git a/docs-site/src/content/docs/getting-started/how-it-works.mdx b/docs-site/src/content/docs/getting-started/how-it-works.mdx index 7ddff28cb9..c29d2f87d7 100644 --- a/docs-site/src/content/docs/getting-started/how-it-works.mdx +++ b/docs-site/src/content/docs/getting-started/how-it-works.mdx @@ -71,8 +71,9 @@ effort to use; v2 requests keep Codex's native multi-agent guidance. to continue if the required pool credential is unavailable. 4. **Vision sidecar (optional)** — if the routed model is listed in `provider.noVisionModels` and the - request carries an image, opencodex describes each image with the configured ChatGPT vision - sidecar and replaces it with text, so a text-only model can still reason about it. + request carries an image, opencodex uses the configured vision backend — OpenAI via ChatGPT login, + Anthropic via OAuth, or the vision-only Chat backend with an explicit `provider/model` target — + then replaces each image with text so a text-only model can still reason about it. See [Sidecars](/guides/sidecars/). 5. **Passthrough fast path** — if the adapter is a Responses passthrough (`openai-responses` or diff --git a/docs-site/src/content/docs/getting-started/installation.md b/docs-site/src/content/docs/getting-started/installation.md index 7c68e1b840..8a6439a1ab 100644 --- a/docs-site/src/content/docs/getting-started/installation.md +++ b/docs-site/src/content/docs/getting-started/installation.md @@ -5,7 +5,7 @@ description: Install the opencodex (ocx) proxy, its prerequisites, and verify it opencodex installs two equivalent command names, `ocx` and `opencodex`. Both launch the same small local HTTP server (built on Bun). Model requests go to the provider selected by routing; optional -vision and web-search sidecars can also use your ChatGPT login when a routed model needs them. +Vision sidecars can use OpenAI via your ChatGPT login, Anthropic via OAuth, or the vision-only Chat backend with an explicit `provider/model` target. See the [sidecar guide](/guides/sidecars/). ## Prerequisites diff --git a/docs-site/src/content/docs/guides/claude-code.md b/docs-site/src/content/docs/guides/claude-code.md index 81d0e0eaea..df1055b819 100644 --- a/docs-site/src/content/docs/guides/claude-code.md +++ b/docs-site/src/content/docs/guides/claude-code.md @@ -326,12 +326,19 @@ before the main model answers: - The **vision sidecar** describes an attached image before calling a model listed in `noVisionModels`, then replaces the image with that description. -Both sidecars can use either backend: +The web-search sidecar accepts only `openai` and `anthropic`; the vision sidecar also accepts +`chat`. Chat is vision-only and never runs web search. | Backend | How it runs | What it requires | | --- | --- | --- | | `openai` | A small GPT model through the ChatGPT `forward` provider | A ChatGPT login and an enabled `authMode: "forward"` provider | | `anthropic` | Claude through stored Anthropic OAuth; web search uses `web_search_20250305` and vision sends the image to Claude for description | An enabled `adapter: "anthropic"`, `authMode: "oauth"` provider whose active stored account is not marked `needsReauth` | +| `chat` (vision only) | An OpenAI-compatible Chat Completions request through the selected configured provider | An enabled `openai-chat` or `google` provider with an API key/key pool or supported active OAuth account; keyless is limited to `openai-chat` local/key-optional providers | + +For `chat`, set `visionSidecar.model` to a provider-qualified `provider/model` value to select +exactly one provider. A bare model is accepted only when one usable provider's configured +`defaultModel`/`models` list matches it; ambiguous and live-only matches fail closed. Google is never +keyless. An explicit `backend` always wins. When it is omitted, the **web-search** sidecar always selects `openai` (`anthropic` runs only when explicitly configured), while the **vision** sidecar selects @@ -365,13 +372,13 @@ images are cached by backend, model, detail, image bytes, and request context, s image-and-context pair is not described again on every replay. Remote `https:` images are never cached because their contents can change. -See the [configuration reference](/reference/configuration/#sidecars) for every key. +See the [configuration reference](/reference/configuration/#sidecars) for every key. The Dashboard +sidecar card exposes the same vision backend/model selection, **Off**, and advanced settings; Claude's +page adds separate overrides with **Use main setting** and **Auto** inheritance controls. Anthropic-OAuth web search and image description reuse the repository's existing Claude Code OAuth fingerprint precedent, but should still be soak-tested with your account and workload before you depend on them for long unattended runs. - - ## Reasoning effort Claude Code's `/effort` setting is preserved across the adapter: @@ -484,8 +491,11 @@ the search but Claude Code still counted zero. **A sidecar does not activate** — For `backend: "openai"`, confirm you are logged into ChatGPT and have an enabled `authMode: "forward"` provider. For `backend: "anthropic"`, confirm the active stored -Anthropic OAuth account is not marked `needsReauth`. An explicit Anthropic selection without that -credential intentionally fails closed. +Anthropic OAuth account is not marked `needsReauth`. For vision `backend: "chat"`, use a provider-qualified +`provider/model` and confirm that the selected enabled `openai-chat` or `google` provider has a key, +key pool, or supported active OAuth account (keyless is only for local/key-optional `openai-chat`). +Ambiguous bare models and live-only bare matches intentionally fail closed. An explicit Anthropic +selection without its credential also fails closed. **"claude.ai connectors are disabled"** — An `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` is set in your shell. `ocx claude` deliberately does NOT set `ANTHROPIC_API_KEY`; if you have it exported, diff --git a/docs-site/src/content/docs/guides/sidecars.md b/docs-site/src/content/docs/guides/sidecars.md index d92316d796..a3e6f1260c 100644 --- a/docs-site/src/content/docs/guides/sidecars.md +++ b/docs-site/src/content/docs/guides/sidecars.md @@ -1,13 +1,14 @@ --- title: "Sidecars: Web Search & Vision" -description: Give routed models real web search and text-only models image understanding through native ChatGPT sidecars. +description: Give routed models real web search and text-only models image understanding through explicit, provider-backed sidecars. --- Routed models do not all expose hosted **web search** or native **image input**. opencodex backfills those capabilities with two sidecars. Both support a ChatGPT-login (`forward`) provider or stored Anthropic OAuth provider; web search can additionally use stored Grok OAuth through the explicit -`xai` backend. Sidecar errors become bounded tool results or image markers instead of failing the -whole turn. +`xai` backend. Vision can additionally use the `chat` backend, which sends the image to a configured +chat-compatible provider. Sidecar errors become bounded tool results or image markers instead of +failing the whole turn. :::note[Automatic backend selection] Explicit `backend` config wins. The two sidecars default differently when `backend` is unset: @@ -15,7 +16,10 @@ Explicit `backend` config wins. The two sidecars default differently when `backe **Vision** defaults to `anthropic` if an enabled Anthropic OAuth provider has an active account not marked `needsReauth`, otherwise `openai`. Explicit `anthropic` without that credential fails closed. Explicit `xai` requires a usable stored Grok OAuth account and does not fall back. `openai` -requires both ChatGPT login auth and an enabled `forward` provider. +requires both ChatGPT login auth and an enabled `forward` provider. The `chat` backend is vision-only: +its model should be written as `provider/model` and must resolve to an enabled `openai-chat` or +`google` provider with usable credentials. Keyless operation is allowed only for local/key-optional +`openai-chat` providers; Google is never keyless. ::: ### Additional web-search backends (explicit-only) @@ -124,20 +128,41 @@ failures after response headers have started are delivered as `response.failed` ## Vision sidecar When the routed model is listed in its provider's `noVisionModels` and a request carries an image, -opencodex describes each image **before** the main call and replaces it with text. When -`visionSidecar.model` is absent or blank, the OpenAI execution path, Dashboard, and management API -use the `gpt-5.4-mini` fallback. Startup still migrates an explicitly persisted legacy -`gpt-5.4-mini` value to `gpt-5.6-luna`; that migration applies to a stored value, not to an absent -model field. +opencodex describes each image **before** the main call and replaces it with text. Vision is explicit: +choose a model and backend, or select **Off**. The `chat` backend is vision-only and uses the configured +provider-qualified `provider/model` target; it does not run web search. + +For `openai`/`anthropic`, when `visionSidecar.model` is absent or blank, the OpenAI execution path, +Dashboard, and management API use the `gpt-5.4-mini` fallback. Startup still migrates an explicitly +persisted legacy `gpt-5.4-mini` value to `gpt-5.6-luna`; that migration applies to a stored value, not +to an absent model field. + +For `chat`, the model is resolved as follows: + +- `provider/model` selects exactly that configured provider. The provider must be enabled, use + `openai-chat` or `google`, and have a usable API key/key pool or supported active OAuth account. +- A bare model id is accepted only when exactly one usable provider's configured `defaultModel` or + `models` list matches it. Bare matches are not inferred from live-only discovery, and ambiguous + matches fail closed rather than choosing an arbitrary provider. +- The selected provider's published model id is sent upstream. Keyless operation is limited to + `openai-chat` providers marked `authMode: "local"` or `keyOptional: true`; Google requires usable + credentials, either an API key/key pool or supported OAuth with a healthy active account. + +The provider-qualified form is recommended whenever more than one chat-compatible provider could +serve the same model id. - Images can come from user, developer, and tool-result messages, including Codex's `view_image`. - On the OpenAI path (ChatGPT-login passthrough), each image is sent to the configured vision model over the Responses endpoint with the selected `reasoning.effort` (`low` by default), and its description replaces the image part inline. The Anthropic path uses the Messages endpoint with its - own thinking-budget mapping and ignores this OpenAI-specific setting. + own thinking-budget mapping and ignores this OpenAI-specific setting. The chat path uses the + selected provider's wire: `openai-chat` sends OpenAI-compatible `POST /chat/completions`, while + `google` uses the native Gemini `streamGenerateContent` adapter (AI Studio, Vertex, or Cloud Code + Assist as configured). - For native models with known capability metadata, unsupported reasoning is normalized to the highest supported rung at or below the requested level; if none exists, the lowest supported rung is used. Unknown or custom models remain permissive when reliable capability metadata is absent. + The chat backend uses the selected provider/model's effective reasoning ladder. - Descriptions run with bounded concurrency (3 at a time, input order preserved). User context sent to the describer is capped at 800 characters, and each injected description is capped at 2,000 characters. The request does not send `max_output_tokens`, which the ChatGPT backend rejects. @@ -156,8 +181,11 @@ model field. The management API and Dashboard picker now list models that can actually accept image input. When the matching backend is available, `gpt-5.6-luna` (OpenAI) and `claude-haiku-4-5` (Anthropic) -are always offered as baseline options. `PUT /api/sidecar-settings` rejects a model known to be -text-only, but still accepts an unknown id so custom or ahead-of-catalog names keep working. +are always offered as baseline options. Chat options come from usable configured `openai-chat` or +`google` providers and are displayed as provider-qualified values such as `google/gemini-flash`. +`PUT /api/sidecar-settings` rejects a model known to be text-only, but still accepts an unknown id so +custom or ahead-of-catalog names keep working; a chat target that cannot resolve to a usable provider +is not executed. ```json { @@ -188,11 +216,16 @@ A model is marked text-only per provider: ## Dashboard controls and disabling -The Dashboard Vision sidecar card can enable or disable the sidecar, set -`maxDescriptionsPerTurn`, and set `timeoutMs`, along with the existing model, -backend, and reasoning controls. Disabling the sidecar does not delete those -settings; turning it back on keeps the previous model, backend, reasoning, -timeout, and limit. +The Dashboard Vision sidecar card uses the model picker as the activation control: choose a listed +OpenAI, Anthropic, or provider-qualified Chat model, or choose **Off**. The backend is inferred from +an eligible model; the advanced settings popover contains reasoning, `maxDescriptionsPerTurn`, and +`timeoutMs`. Web Search has its own model/backend controls and does not offer Chat, because Chat is +vision-only. Changes apply on the next request. + +Disabling the Vision sidecar does not delete its settings; turning it back on keeps the previous model, +backend, reasoning, timeout, and limit. The Claude Code page has separate per-surface overrides: **Use +main setting** inherits the global setting, **Auto** leaves backend selection unset, and an explicit +backend/model override is saved only for Claude-originated requests. `PUT /api/sidecar-settings` accepts the same fields. Partial updates leave omitted keys unchanged. `timeoutMs` uses the runtime integer bounds diff --git a/docs-site/src/content/docs/guides/web-dashboard.md b/docs-site/src/content/docs/guides/web-dashboard.md index 6c2ca81e05..4accc10718 100644 --- a/docs-site/src/content/docs/guides/web-dashboard.md +++ b/docs-site/src/content/docs/guides/web-dashboard.md @@ -40,7 +40,7 @@ the browser or password manager's decision. | --- | --- | | **Dashboard summary** | Multi-agent mode, online state, version, uptime, provider count, 30-day token total, active providers, and available native/routed models. | | **Sub-agent delegation** | Choose a native or routed model and optional reasoning effort shared by OpenCodex delegation guidance and the separate native-default opt-in. This is not a proxy-side per-spawn router; see below. | -| **Sidecars** | Choose the web-search model and effort plus the vision-description model. Changes apply on the next request. | +| **Sidecars** | Choose the web-search backend/model and effort plus the vision backend/model. Web search offers OpenAI or Anthropic; vision also offers provider-backed Chat, and **Off** disables vision without deleting its settings. Advanced vision controls include reasoning, description limit, and timeout. Changes apply on the next request. | | **Maintenance** | Resync the Codex model catalog, inspect project-local config bypass warnings, check the latest or preview release, and run an update with optional proxy restart. | | **Startup safety** | Show whether injected Codex routing survives a restart, with separate service and launcher-shim health plus exact repair commands. | | **Windows tray** | Install a per-user login tray for one-click proxy start, stop, restart, dashboard access, and status. The tray is a controller, not a proxy restart service. | @@ -203,7 +203,7 @@ The GUI is a thin client over the proxy's JSON management API. Useful endpoints | `GET` / `POST /api/windows-tray` | Read or change the Windows tray installation and visible-process state. POST accepts `install`, `start`, `stop`, or `uninstall`. | | `POST /api/sync` | Rebuild the shared model catalog and stale the Codex model cache. | | `GET /api/update/check` · `POST /api/update/run` · `GET /api/update/status` | Check, run, and monitor self-update jobs. Worker PIDs are persisted so a crashed job recovers automatically; legacy no-PID jobs recover after ten minutes. | -| `GET` / `PUT /api/sidecar-settings` | Read or set search/vision sidecar model settings. | +| `GET` / `PUT /api/sidecar-settings` | Read or set search/vision sidecar backend, model, and vision advanced settings. Web search accepts only OpenAI/Anthropic; vision may use provider-qualified Chat. | | `GET` / `PUT /api/injection-model` | Read or set the shared sub-agent model/effort selection and the independent guidance/native-default switches. | | `GET` / `PUT /api/v2` | Read or set the surface mode, Codex feature flag, and v2 thread limit. | | `GET /api/providers` · `POST /api/providers` · `PATCH /api/providers?name=...` · `DELETE /api/providers?name=...` | List, add/replace, enable/disable, set the default, or remove providers. `PATCH` uses standalone `{ "setDefault": true }` on an enabled provider; `POST` may include `setDefault` when creating/replacing (also enabled-only). Deleting the current default reassigns to the first remaining enabled provider when one exists; otherwise the API returns `409` with `code: "last_provider"` and keeps the current default. | diff --git a/docs-site/src/content/docs/ja/getting-started/how-it-works.mdx b/docs-site/src/content/docs/ja/getting-started/how-it-works.mdx index d19b222213..f3fa6aecff 100644 --- a/docs-site/src/content/docs/ja/getting-started/how-it-works.mdx +++ b/docs-site/src/content/docs/ja/getting-started/how-it-works.mdx @@ -69,8 +69,9 @@ Codex は OpenAI **Responses API** を使います。opencodex は HTTP と Serv パススルーアダプターは進みません。 4. **Vision サイドカー(任意)** — ルーティングされたモデルが `provider.noVisionModels` に列挙されており - リクエストに画像が含まれる場合、opencodex は設定された ChatGPT vision サイドカーで各画像を説明した - 後テキストに置き換えます。これによりテキスト専用モデルでも該当画像を推論できます。 + リクエストに画像が含まれる場合、opencodex は設定された vision backend（ChatGPT ログイン経由の OpenAI、 + OAuth 経由の Anthropic、または明示的な `provider/model` を使う Vision 専用の Chat backend）で各画像を + 説明した後、テキストに置き換えます。これによりテキスト専用モデルでも該当画像を推論できます。 [サイドカー](/ja/guides/sidecars/)を参照してください。 5. **パススルーの高速経路** — Responses パススルーアダプター(`openai-responses` または diff --git a/docs-site/src/content/docs/ja/getting-started/installation.md b/docs-site/src/content/docs/ja/getting-started/installation.md index 52ff9d1781..f72fdc64e0 100644 --- a/docs-site/src/content/docs/ja/getting-started/installation.md +++ b/docs-site/src/content/docs/ja/getting-started/installation.md @@ -5,7 +5,8 @@ description: opencodex(ocx)プロキシと前提条件をインストールし opencodex をインストールすると同じ実行ファイルを指す `ocx` と `opencodex` コマンドが一緒に提供されます。 どちらも Bun ベースの小さなローカル HTTP サーバーを実行します。モデルリクエストはルーティングで選ばれたプロバイダーに -転送され、必要に応じて vision とウェブ検索のサイドカーが ChatGPT ログインを使うこともあります。 +転送されます。Vision サイドカーは ChatGPT ログイン経由の OpenAI、OAuth 経由の Anthropic、または +明示的な `provider/model` を使う Vision 専用の Chat backend に対応します。 ## 前提条件 diff --git a/docs-site/src/content/docs/ja/guides/claude-code.md b/docs-site/src/content/docs/ja/guides/claude-code.md index bc7e84e39d..645f713d6a 100644 --- a/docs-site/src/content/docs/ja/guides/claude-code.md +++ b/docs-site/src/content/docs/ja/guides/claude-code.md @@ -246,8 +246,6 @@ Anthropic OAuth のウェブ検索と画像説明は保存所ですでに使っ fingerprint 方式をそのまま踏襲しますが、長時間の無人作業に使う前に自身のアカウントと実際の作業で 十分 soak test するのが無難です。 - - ## 推論負荷 Claude Code の `/effort` 設定はアダプターでも維持されます。 diff --git a/docs-site/src/content/docs/ko/getting-started/how-it-works.mdx b/docs-site/src/content/docs/ko/getting-started/how-it-works.mdx index 9cc23b7c34..218510dc3e 100644 --- a/docs-site/src/content/docs/ko/getting-started/how-it-works.mdx +++ b/docs-site/src/content/docs/ko/getting-started/how-it-works.mdx @@ -69,8 +69,9 @@ reasoning effort를 알려 줍니다. v2 요청은 Codex가 제공하는 멀티 passthrough adapter가 계속 진행하지 않습니다. 4. **Vision sidecar(선택)** — 라우팅된 모델이 `provider.noVisionModels`에 나열되어 있고 - 요청에 이미지가 포함된 경우, opencodex는 설정된 ChatGPT vision sidecar로 각 이미지를 설명한 - 뒤 텍스트로 치환합니다. 덕분에 텍스트 전용 모델도 해당 이미지에 대해 추론할 수 있습니다. + 요청에 이미지가 포함된 경우, opencodex는 설정된 vision backend(ChatGPT 로그인 기반 OpenAI, + OAuth 기반 Anthropic, 또는 명시적 `provider/model`을 사용하는 vision 전용 Chat backend)로 각 이미지를 + 설명한 뒤 텍스트로 치환합니다. 덕분에 텍스트 전용 모델도 해당 이미지에 대해 추론할 수 있습니다. [Sidecar](/ko/guides/sidecars/)를 참고하세요. 5. **Passthrough 빠른 경로** — Responses 패스스루 adapter(`openai-responses` 또는 diff --git a/docs-site/src/content/docs/ko/getting-started/installation.md b/docs-site/src/content/docs/ko/getting-started/installation.md index 1dfff4b156..0d2e5b3950 100644 --- a/docs-site/src/content/docs/ko/getting-started/installation.md +++ b/docs-site/src/content/docs/ko/getting-started/installation.md @@ -5,7 +5,8 @@ description: opencodex(ocx) 프록시와 사전 요구 사항을 설치하고, opencodex를 설치하면 같은 실행 파일을 가리키는 `ocx`와 `opencodex` 명령이 함께 제공됩니다. 둘 다 Bun 기반의 작은 로컬 HTTP 서버를 실행합니다. 모델 요청은 라우팅으로 선택된 프로바이더에 -전달되며, 필요할 때 vision 및 웹 검색 sidecar가 ChatGPT 로그인을 사용할 수도 있습니다. +전달됩니다. Vision sidecar는 ChatGPT 로그인 기반 OpenAI, OAuth 기반 Anthropic, 또는 명시적 +`provider/model`을 사용하는 vision 전용 Chat backend를 지원합니다. ## 사전 요구 사항 diff --git a/docs-site/src/content/docs/ko/guides/claude-code.md b/docs-site/src/content/docs/ko/guides/claude-code.md index 0183b5b72f..ccb75634b3 100644 --- a/docs-site/src/content/docs/ko/guides/claude-code.md +++ b/docs-site/src/content/docs/ko/guides/claude-code.md @@ -283,8 +283,6 @@ ChatGPT bearer는 메인 라우팅 프로바이더에는 전달하지 않아요. fingerprint 방식을 그대로 따르지만, 장시간 무인 작업에 쓰기 전에는 본인 계정과 실제 작업으로 충분히 soak test하는 편이 좋아요. - - ## 추론 강도 Claude Code의 `/effort` 설정은 어댑터에서도 유지돼요. diff --git a/docs-site/src/content/docs/reference/adapters.md b/docs-site/src/content/docs/reference/adapters.md index 29859992f7..41a2bfb779 100644 --- a/docs-site/src/content/docs/reference/adapters.md +++ b/docs-site/src/content/docs/reference/adapters.md @@ -106,8 +106,10 @@ of the HTTP retry loop. ## `google` -**Targets:** Google **Gemini**, **Vertex AI**, and Antigravity **Cloud Code Assist**. AI Studio uses -`/v1beta/models/{model}:streamGenerateContent`; the other modes use their native Google endpoints. +**Targets:** Google **Gemini**, **Vertex AI**, and Antigravity **Cloud Code Assist** through Google's +native adapter. AI Studio uses `/v1beta/models/{model}:streamGenerateContent`; the other modes use + their native Google endpoints. This is separate from `openai-chat`, which targets OpenAI-compatible +`POST {baseUrl}/chat/completions`. **Auth:** API key, Vertex ADC, or Google Antigravity OAuth, selected by `googleMode`. - System prompt → `systemInstruction`; messages → `contents[]` (assistant → `model`); tools → diff --git a/docs-site/src/content/docs/reference/architecture.md b/docs-site/src/content/docs/reference/architecture.md index 180c043a88..66fc5ffedc 100644 --- a/docs-site/src/content/docs/reference/architecture.md +++ b/docs-site/src/content/docs/reference/architecture.md @@ -62,7 +62,7 @@ the `server/responses.ts` facade and its `server/responses/*.ts` modules: seven adapters. Responses passthrough relays the native body, Cursor runs its bidirectional `runTurn` transport, and translated adapters build/fetch/parse an upstream request. 6. For routed models with a hosted `web_search` tool, `web-search/` exposes a synthetic function, - executes the real search through the configured backend (the OpenAI/ChatGPT sidecar or Anthropic), + executes the real search through the configured OpenAI/ChatGPT or Anthropic backend, feeds results back to the routed model, and repeats within the configured loop limit. This loop supports only the standard HTTP path; adapters that implement `runTurn`, such as Cursor, bypass it. 7. `bridge.ts` produces Responses SSE or JSON. `server/request-log.ts` and `usage/` collect terminal diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 842d9a6efa..2c7111b4cc 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -119,7 +119,7 @@ differing backup and rewrites known legacy namespaced selected ids to bare ids. | `requiresReasoningPlaceholderModels?` | `string[]` | Models whose upstream rejects a tool_call continuation missing `reasoning_content` (DeepSeek thinking mode); a minimal placeholder is injected when the replay cache misses. Defaults to `preserveReasoningContentModels`; set `[]` to opt out. | | `thinkingToggleModels?` | `string[]` | Chat models using `thinking.enabled` rather than an effort ladder. | | `thinkingBudgetModels?` | `string[]` | Chat models using integer `thinking_budget`; effort maps to a budget fraction. | -| `noVisionModels?` | `string[]` | Text-only models sent through the vision sidecar; matching tolerates an Ollama `:size` tag. | +| `noVisionModels?` | `string[]` | Text-only models sent through the vision sidecar; matching tolerates an Ollama `:size` tag. The vision `chat` backend can use configured `openai-chat` or `google` providers, but its target is provider-qualified (`provider/model`) when selecting a specific provider. | | `escapeBuiltinToolNames?` | `boolean` | Escape built-in tool names for Anthropic-compatible gateways and restore them in returned calls. | | `anthropicEofTolerance?` | `boolean` | Let an Anthropic-compatible gateway complete a stream that ends before `message_stop`, only when visible text or a complete JSON-object tool input was received. Off by default. | | `googleMode?` | `"ai-studio" \| "vertex" \| "cloud-code-assist"` | Google transport/auth mode. Default `ai-studio`. | diff --git a/docs-site/src/content/docs/reference/configuration/server.md b/docs-site/src/content/docs/reference/configuration/server.md index 5ddf5588bc..f5831cc773 100644 --- a/docs-site/src/content/docs/reference/configuration/server.md +++ b/docs-site/src/content/docs/reference/configuration/server.md @@ -205,7 +205,7 @@ Images API paths and response shape expected by Codex. | Field | Type | Default | Meaning | | --- | --- | --- | --- | | `enabled?` | `boolean` | on when usable | Master switch. | -| `backend?` | `"openai" \| "anthropic" \| "xai" \| "gemini" \| "exa"` | `openai` | Explicit wins; unset always resolves to `openai`. `anthropic` and `xai` run only when explicitly configured; `gemini` and `exa` remain reserved until their executors ship. | +| `backend?` | `"openai" \| "anthropic" \| "xai" \| "gemini" \| "exa"` | `openai` | Explicit wins; unset always resolves to `openai`. `anthropic`, `xai`, `gemini`, and `exa` run only when explicitly configured. `chat` is not a web-search backend. | | `model?` | `string` | backend-dependent | `gpt-5.6-luna` for OpenAI, `claude-sonnet-5` for Anthropic, or `grok-4.6` for xAI. Legacy explicit `gpt-5.4-mini` migrates on start. | | `exaApiKey?` | `string` | none | Operator key for the `exa` backend. Write-only: management reads never return the stored value. | | `xSearch?` | `object` | omitted | xAI-only opt-in for hosted `x_search`: `enabled`, mutually exclusive `allowedXHandles` / `excludedXHandles` arrays (maximum 20), and ISO `fromDate` / `toDate` (`YYYY-MM-DD`). | @@ -223,6 +223,7 @@ hosted `web_search`, and adds hosted `x_search` when `xSearch.enabled` is true. management input returns `400`; a malformed persisted block fails closed during planning. The `gemini` and `exa` lanes never activate from credential discovery or fallback; the operator must select them explicitly. `exaApiKey` is accepted on writes but omitted from management responses. +The `chat` backend belongs to the vision sidecar only. Four clocks govern search: base `stallTimeoutSec`, `connectTimeoutMs`, routed-model inactivity, and hosted-search timeout. The effective bridge watchdog is the maximum plus 30 seconds. Routed stall is @@ -233,16 +234,22 @@ an inactivity guard, not a total generation deadline. | Field | Type | Default | Meaning | | --- | --- | --- | --- | | `enabled?` | `boolean` | on when usable | Master image-description switch. | -| `backend?` | `"openai" \| "anthropic"` | auto | Explicit wins; unset prefers a usable stored Anthropic OAuth credential, else `openai`. | -| `model?` | `string` | backend-dependent | `gpt-5.4-mini` for OpenAI or `claude-sonnet-5` for Anthropic. | +| `backend?` | `"openai" \| "anthropic" \| "chat"` | auto | Explicit wins; unset prefers a usable stored Anthropic OAuth credential, else `openai`. `chat` is vision-only and uses a configured `openai-chat` or `google` provider. | +| `model?` | `string` | backend-dependent | `gpt-5.4-mini` for OpenAI, `claude-sonnet-5` for Anthropic, or provider-qualified `provider/model` for Chat. | | `maxDescriptionsPerTurn?` | `number` | `8` | New description cache misses admitted per main turn. `0` disables calls; invalid values use default. | | `timeoutMs?` | `number` | `45000` | Sidecar fetch timeout. Integer 1–2147483647. | Vision activates only for images sent to a model in its provider's `noVisionModels`. OpenAI has the same login/forward requirements as search; explicitly selected Anthropic fails closed without a usable -credential. Successful `data:` descriptions use a bounded cache keyed by backend, model, detail, -image bytes, and normalized message context. Hits and same-turn duplicates do not consume the limit. -Remote `https:` images and failed or empty descriptions are not cached. +credential. For `chat`, the selected provider must be enabled and use `openai-chat` or `google`; it +must have an API key/key pool or a supported OAuth account. Keyless operation is allowed only for +`openai-chat` with `authMode: "local"` or `keyOptional: true`, never for Google. A provider-qualified +`provider/model` target selects exactly that provider. Bare Chat ids resolve only through one unique +configured `defaultModel`/`models` match; ambiguous or live-only matches fail closed. + +Successful `data:` descriptions use a bounded cache keyed by backend, model, detail, image bytes, and +normalized message context. Hits and same-turn duplicates do not consume the limit. Remote `https:` +images and failed or empty descriptions are not cached. Anthropic OAuth sidecars reuse opencodex's existing Claude Code OAuth fingerprint. Soak-test the intended account and workload. diff --git a/docs-site/src/content/docs/reference/management-api.md b/docs-site/src/content/docs/reference/management-api.md index bef307c3d5..a387cdd02f 100644 --- a/docs-site/src/content/docs/reference/management-api.md +++ b/docs-site/src/content/docs/reference/management-api.md @@ -111,7 +111,7 @@ See [Combos](/guides/combos/) for target strategies, cooldowns, aliases, and rou | `GET /api/update/check` | Check the `latest` or `preview` update channel | 400 invalid tag | | `POST /api/update/run` | Start an update job, optionally followed by restart | 400 invalid body; job-specific conflict/error status | | `GET /api/update/status` | Poll an update job by id | 404 unknown job | -| `GET, PUT /api/sidecar-settings` | Read or update web-search and vision sidecar model/backend settings | 400 invalid shape, backend, or limit | +| `GET, PUT /api/sidecar-settings` | Read or update web-search and vision sidecar backend/model settings; vision also accepts `enabled`, `reasoning`, `maxDescriptionsPerTurn`, and `timeoutMs` | 400 invalid shape, backend, model, or limit | | `GET, PUT /api/shadow-call-settings` | Read or update shadow-call interception settings | 400 invalid shape or value | ### Logs, usage, and storage diff --git a/docs-site/src/content/docs/ru/getting-started/how-it-works.mdx b/docs-site/src/content/docs/ru/getting-started/how-it-works.mdx index 00c83a2ef5..56ce47d4c9 100644 --- a/docs-site/src/content/docs/ru/getting-started/how-it-works.mdx +++ b/docs-site/src/content/docs/ru/getting-started/how-it-works.mdx @@ -78,8 +78,9 @@ Codex даже не догадывается, что общается не с Op 4. **Vision-сайдкар (опционально)** — если маршрутизируемая модель указана в `provider.noVisionModels`, а запрос содержит изображение, opencodex описывает каждое - изображение с помощью настроенного vision-сайдкара ChatGPT и заменяет его текстом, чтобы - модель без поддержки изображений всё равно могла о нём рассуждать. + изображение через настроенный vision-бэкенд — OpenAI с входом ChatGPT, Anthropic через OAuth + или предназначенный только для vision Chat-бэкенд с явной целью `provider/model` — и заменяет его + текстом, чтобы модель без поддержки изображений всё равно могла о нём рассуждать. См. [Сайдкары](/ru/guides/sidecars/). 5. **Быстрый путь passthrough** — если адаптер является passthrough для Responses diff --git a/docs-site/src/content/docs/ru/getting-started/installation.md b/docs-site/src/content/docs/ru/getting-started/installation.md index 837178f7e4..a7506d1a30 100644 --- a/docs-site/src/content/docs/ru/getting-started/installation.md +++ b/docs-site/src/content/docs/ru/getting-started/installation.md @@ -5,8 +5,9 @@ description: Установите прокси opencodex (ocx) и необход opencodex устанавливает два эквивалентных имени команды: `ocx` и `opencodex`. Обе запускают один и тот же небольшой локальный HTTP-сервер (построенный на Bun). Запросы к моделям идут к провайдеру, -выбранному маршрутизацией; опциональные сайдкары для vision и веб-поиска также могут использовать -ваш вход в ChatGPT, когда они нужны маршрутизируемой модели. +выбранному маршрутизацией. Vision-сайдкар поддерживает OpenAI через вход ChatGPT, Anthropic через OAuth +или предназначенный только для vision Chat-бэкенд с явной целью `provider/model`; веб-поиск использует свои +отдельные бэкенды. ## Предварительные требования diff --git a/docs-site/src/content/docs/ru/guides/claude-code.md b/docs-site/src/content/docs/ru/guides/claude-code.md index b04612625b..0b807a8e3a 100644 --- a/docs-site/src/content/docs/ru/guides/claude-code.md +++ b/docs-site/src/content/docs/ru/guides/claude-code.md @@ -264,8 +264,6 @@ Claude Code — это лишь учётные данные для доступ подход к OAuth-отпечатку Claude Code, но, прежде чем полагаться на них в длительных автономных запусках, их стоит обкатать на вашем аккаунте и рабочей нагрузке. - - ## Уровень рассуждений Настройка `/effort` в Claude Code сохраняется при прохождении через адаптер: diff --git a/docs-site/src/content/docs/tr/getting-started/how-it-works.mdx b/docs-site/src/content/docs/tr/getting-started/how-it-works.mdx index 2da540397f..01df76b1f9 100644 --- a/docs-site/src/content/docs/tr/getting-started/how-it-works.mdx +++ b/docs-site/src/content/docs/tr/getting-started/how-it-works.mdx @@ -90,9 +90,10 @@ istekleri Codex'in yerel çoklu ajan rehberliğini korur. 4. **Vizyon sidecar'ı (isteğe bağlı)** — yönlendirilen model `provider.noVisionModels` listesinde yer alıyorsa ve istek bir görsel - taşıyorsa, opencodex her görseli yapılandırılmış ChatGPT vizyon sidecar'ı ile - tanımlar ve metinle değiştirir; böylece salt metin bir model bile görsel - hakkında akıl yürütebilir. Bkz. [Sidecar'lar](/tr/guides/sidecars/). + taşıyorsa, opencodex her görseli yapılandırılmış vision arka ucuyla — ChatGPT oturumuyla OpenAI, + OAuth ile Anthropic veya açık bir `provider/model` hedefi kullanan yalnızca vision için Chat backend'i — + tanımlar ve metinle değiştirir; böylece salt metin bir model bile görsel hakkında akıl yürütebilir. + Bkz. [Sidecar'lar](/tr/guides/sidecars/). 5. **Doğrudan geçiş hızlı yolu (Passthrough fast path)** — adaptör bir Responses doğrudan geçişi ise (`openai-responses` veya `azure-openai`), opencodex diff --git a/docs-site/src/content/docs/tr/getting-started/installation.md b/docs-site/src/content/docs/tr/getting-started/installation.md index e138735b81..5c9fce37c7 100644 --- a/docs-site/src/content/docs/tr/getting-started/installation.md +++ b/docs-site/src/content/docs/tr/getting-started/installation.md @@ -6,8 +6,8 @@ description: opencodex (ocx) proxy'sini ve ön koşullarını kurun, çalıştı opencodex, `ocx` ve `opencodex` olmak üzere iki eşdeğer komut adı kurar. Her ikisi de aynı küçük yerel HTTP sunucusunu (Bun üzerinde oluşturulmuştur) başlatır. Model istekleri yönlendirme tarafından seçilen sağlayıcıya gider; -isteğe bağlı vizyon ve web araması sidecar'ları, yönlendirilen bir model ihtiyaç -duyduğunda ChatGPT oturumunuzu da kullanabilir. +isteğe bağlı sidecar'lar yönlendirilen bir modelin ihtiyaç duyduğu yetenekleri sağlar. Vizyon; ChatGPT oturumuyla +OpenAI'yi, OAuth ile Anthropic'i veya açık bir `provider/model` hedefi kullanan yalnızca vision için Chat backend'ini destekler. ## Ön Koşullar diff --git a/docs-site/src/content/docs/tr/reference/architecture.md b/docs-site/src/content/docs/tr/reference/architecture.md index 131c85e3e1..7fe86f82c1 100644 --- a/docs-site/src/content/docs/tr/reference/architecture.md +++ b/docs-site/src/content/docs/tr/reference/architecture.md @@ -72,8 +72,8 @@ devreder: gövdeyi iletir, Cursor çift yönlü `runTurn` aktarımını çalıştırır ve çevrilen adaptörler bir yukarı akış isteği oluşturur/getirir/ayrıştırır. 6. Barındırılan bir `web_search` aracına sahip yönlendirilen modeller için - `web-search/` sentetik bir fonksiyon sunar, gerçek aramayı ChatGPT sidecar'ı - aracılığıyla yürütür, sonuçları yönlendirilen modele geri besler ve + `web-search/` sentetik bir fonksiyon sunar, gerçek aramayı yapılandırılmış OpenAI/ChatGPT veya Anthropic + backend'i aracılığıyla yürütür, sonuçları yönlendirilen modele geri besler ve yapılandırılmış döngü sınırı içinde tekrarlar. 7. `bridge.ts` Responses SSE veya JSON üretir. `server/request-log.ts` ve `usage/` yanıtı değiştirmeden uç durumu, gecikmeyi, sağlayıcı/model diff --git a/docs-site/src/content/docs/zh-cn/getting-started/how-it-works.mdx b/docs-site/src/content/docs/zh-cn/getting-started/how-it-works.mdx index 4aed1da4f4..272dbb9667 100644 --- a/docs-site/src/content/docs/zh-cn/getting-started/how-it-works.mdx +++ b/docs-site/src/content/docs/zh-cn/getting-started/how-it-works.mdx @@ -48,8 +48,9 @@ Sol/Terra/Luna 三个模型和 `gpt-5.4-mini`。仪表盘可以从原生或已 token 作为 bearer key,从而让现有的 adapter 无需改动即可完成认证。 4. **Vision sidecar(可选)** —— 如果已路由的模型被列在 `provider.noVisionModels` 中,且 - 请求携带了图像，opencodex 会通过你的 ChatGPT 登录凭据，使用已配置的 vision sidecar 描述 - 每张图像并将其替换为文本，让纯文本模型仍可对图像进行推理。 + 请求携带了图像，opencodex 会使用已配置的 vision 后端描述每张图像：OpenAI 使用 ChatGPT 登录， + Anthropic 使用 OAuth，或使用带明确 `provider/model` 目标、仅用于 Vision 的 Chat 后端；然后将图像替换为 + 文本，让纯文本模型仍可对图像进行推理。 参见 [Sidecar](/zh-cn/guides/sidecars/)。 5. **直通快速路径** —— 对于 Responses 直通 adapter（`openai-responses` 或 `azure-openai`）， diff --git a/docs-site/src/content/docs/zh-cn/getting-started/installation.md b/docs-site/src/content/docs/zh-cn/getting-started/installation.md index 8730fd20a4..e432fcff2b 100644 --- a/docs-site/src/content/docs/zh-cn/getting-started/installation.md +++ b/docs-site/src/content/docs/zh-cn/getting-started/installation.md @@ -5,7 +5,8 @@ description: 安装 opencodex(ocx)代理及其前置条件,并验证它能够运 安装 opencodex 后会得到 `ocx` 和 `opencodex` 两个等价命令，它们都指向同一个基于 Bun 的 小型本地 HTTP 服务器。模型请求会发往路由所选的 provider；当已路由模型需要时，可选的 -vision 和网络搜索 sidecar 也可以使用你的 ChatGPT 登录凭据。 +Vision sidecar 支持通过 ChatGPT 登录使用 OpenAI、通过 OAuth 使用 Anthropic，或使用带明确 +`provider/model` 目标、仅用于 Vision 的 Chat backend。 ## 前置条件 diff --git a/docs-site/src/content/docs/zh-cn/guides/claude-code.md b/docs-site/src/content/docs/zh-cn/guides/claude-code.md index e5140cb8a7..b0e59b1777 100644 --- a/docs-site/src/content/docs/zh-cn/guides/claude-code.md +++ b/docs-site/src/content/docs/zh-cn/guides/claude-code.md @@ -239,8 +239,6 @@ Claude 入站的路由重放会把主 ChatGPT 登录附加到内部请求，因 Search 和图像描述沿用仓库已有的 Claude Code OAuth fingerprint 先例，但在用于长时间无人值守任务前， 仍应使用你的账户和实际负载进行充分 soak test。 - - ## 推理强度 Claude Code 的 `/effort` 设置会完整保留并传递给适配器： diff --git a/docs-site/src/content/docs/zh-tw/getting-started/how-it-works.mdx b/docs-site/src/content/docs/zh-tw/getting-started/how-it-works.mdx index 14934a2968..dc57ef4a0a 100644 --- a/docs-site/src/content/docs/zh-tw/getting-started/how-it-works.mdx +++ b/docs-site/src/content/docs/zh-tw/getting-started/how-it-works.mdx @@ -65,8 +65,9 @@ Sol/Terra/Luna 三個模型和 `gpt-5.4-mini`。儀表板可以從原生或已 4. **Vision sidecar（可選）** —— 如果已路由的模型被列在 `provider.noVisionModels` 中，且 請求攜帶了圖像，opencodex 會透過已設定的 vision sidecar 描述每張圖像並替換為文字，讓純文字 - 模型仍可對圖像進行推理。後端可選 `openai`（ChatGPT 登入）或 `anthropic`（OAuth）；未設定時 - 會自動選擇，顯式 `anthropic` 但無可用憑證時會關閉失敗。 + 模型仍可對圖像進行推理。後端可選 OpenAI（ChatGPT 登入）、Anthropic（OAuth），或使用明確 + `provider/model` 目標、僅用於 Vision 的 Chat backend。未設定時會自動選擇，顯式 `anthropic` 但無可用 + 憑證時會關閉失敗。 參見 [Sidecar](/zh-tw/guides/sidecars/)。 5. **直通快速路徑** —— 對於 Responses 直通 adapter（`openai-responses` 或 `azure-openai`）， diff --git a/docs-site/src/content/docs/zh-tw/getting-started/installation.md b/docs-site/src/content/docs/zh-tw/getting-started/installation.md index 23b36b97da..4975989ed2 100644 --- a/docs-site/src/content/docs/zh-tw/getting-started/installation.md +++ b/docs-site/src/content/docs/zh-tw/getting-started/installation.md @@ -5,7 +5,8 @@ description: 安裝 opencodex(ocx)代理及其前置條件,並驗證它能夠執 安裝 opencodex 後會得到 `ocx` 和 `opencodex` 兩個等價命令，它們都指向同一個基於 Bun 的 小型本機 HTTP 伺服器。模型請求會發往路由所選的 provider；當已路由模型需要時，可選的 -vision 和網路搜尋 sidecar 也可以使用你的 ChatGPT 登入憑證。 +Vision sidecar 支援透過 ChatGPT 登入使用 OpenAI、透過 OAuth 使用 Anthropic，或使用帶有明確 +`provider/model` 目標、僅用於 Vision 的 Chat backend。 ## 前置條件 diff --git a/docs-site/src/content/docs/zh-tw/guides/claude-code.md b/docs-site/src/content/docs/zh-tw/guides/claude-code.md index be06746fba..d338d3a331 100644 --- a/docs-site/src/content/docs/zh-tw/guides/claude-code.md +++ b/docs-site/src/content/docs/zh-tw/guides/claude-code.md @@ -308,8 +308,6 @@ Claude 入站的路由重放會把主 ChatGPT 登入附加到內部請求，因 Search 和圖像描述沿用儲存庫已有的 Claude Code OAuth fingerprint 先例，但在用於長時間無人值守任務前， 仍應使用你的帳號和實際負載進行充分 soak test。 - - ## 推理強度 Claude Code 的 `/effort` 設定會完整保留並傳遞給適配器： diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index 8e64e29b97..8ec232cad3 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -302,6 +302,7 @@ export const de: Record = { "dash.backendAuto": "Automatisch", "dash.backendOpenAI": "OpenAI", "dash.backendAnthropic": "Anthropic", + "dash.backendChat": "Chat API", "dash.sidecarSaved": "Sidecar-Einstellungen gespeichert. Angewendet bei der nächsten Anfrage.", "dash.sidecarSaveFailed": "Sidecar-Einstellungen konnten nicht gespeichert werden.", "dash.injectionLabel": "Sub-Agent-Delegation", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index 262e913162..6b83a6f9bc 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -320,6 +320,7 @@ export const en = { "dash.backendAuto": "Auto", "dash.backendOpenAI": "OpenAI", "dash.backendAnthropic": "Anthropic", + "dash.backendChat": "Chat API", "dash.sidecarSaved": "Sidecar settings saved. Applied on the next request.", "dash.sidecarSaveFailed": "Failed to save sidecar settings.", "dash.injectionLabel": "Sub-agent delegation", diff --git a/gui/src/i18n/fr.ts b/gui/src/i18n/fr.ts index 147773fd93..b36a4f5dce 100644 --- a/gui/src/i18n/fr.ts +++ b/gui/src/i18n/fr.ts @@ -310,6 +310,7 @@ export const fr: Record = { "dash.backendAuto": "Auto", "dash.backendOpenAI": "OpenAI", "dash.backendAnthropic": "Anthropic", + "dash.backendChat": "API Chat", "dash.sidecarSaved": "Paramètres des services auxiliaires enregistrés. Ils s’appliqueront à la prochaine requête.", "dash.sidecarSaveFailed": "Échec de l’enregistrement des paramètres des services auxiliaires.", "dash.injectionLabel": "Délégation aux sous-agents", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index 11a85127ea..36c4dec1be 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -311,6 +311,7 @@ export const ja: Record = { "dash.backendAuto": "自動", "dash.backendOpenAI": "OpenAI", "dash.backendAnthropic": "Anthropic", + "dash.backendChat": "Chat API", "dash.sidecarSaved": "サイドカー設定を保存しました。次回リクエスト時に適用されます。", "dash.sidecarSaveFailed": "サイドカー設定の保存に失敗しました。", "dash.injectionLabel": "サブエージェント委任", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index 5bbc14ae7d..9477152fed 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -306,6 +306,7 @@ export const ko: Record = { "dash.backendAuto": "자동", "dash.backendOpenAI": "OpenAI", "dash.backendAnthropic": "Anthropic", + "dash.backendChat": "Chat API", "dash.sidecarSaved": "사이드카 설정이 저장됐습니다. 다음 요청부터 적용됩니다.", "dash.sidecarSaveFailed": "사이드카 설정 저장에 실패했습니다.", "dash.injectionLabel": "서브에이전트 위임", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index 396ccf3ed0..7bb9b4efb2 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -311,6 +311,7 @@ export const ru: Record = { "dash.backendAuto": "Авто", "dash.backendOpenAI": "OpenAI", "dash.backendAnthropic": "Anthropic", + "dash.backendChat": "Chat API", "dash.sidecarSaved": "Настройки сайдкара сохранены. Вступят в силу со следующего запроса.", "dash.sidecarSaveFailed": "Не удалось сохранить настройки сайдкара.", "dash.injectionLabel": "Делегирование подагентам", diff --git a/gui/src/i18n/tr.ts b/gui/src/i18n/tr.ts index f460bbeb36..72674c44bd 100644 --- a/gui/src/i18n/tr.ts +++ b/gui/src/i18n/tr.ts @@ -312,6 +312,7 @@ export const tr: Record = { "dash.backendAuto": "Otomatik", "dash.backendOpenAI": "OpenAI", "dash.backendAnthropic": "Anthropic", + "dash.backendChat": "Sohbet API", "dash.sidecarSaved": "Yan araç ayarları kaydedildi. Sonraki istekte uygulanacak.", "dash.sidecarSaveFailed": "Yan araç ayarları kaydedilemedi.", "dash.injectionLabel": "Alt ajan devri", diff --git a/gui/src/i18n/zh-TW.ts b/gui/src/i18n/zh-TW.ts index 942e21e61f..368a211983 100644 --- a/gui/src/i18n/zh-TW.ts +++ b/gui/src/i18n/zh-TW.ts @@ -205,6 +205,7 @@ export const zhTW: Record = { "dash.backendAuto": "自動", "dash.backendOpenAI": "OpenAI", "dash.backendAnthropic": "Anthropic", + "dash.backendChat": "聊天 API", "dash.sidecarSaved": "附屬設定已儲存。將在下一個請求時生效。", "dash.sidecarSaveFailed": "儲存附屬設定失敗。", "dash.injectionLabel": "子代理委託", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index b9cd4a3573..eab6d06ce6 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -306,6 +306,7 @@ export const zh: Record = { "dash.backendAuto": "自动", "dash.backendOpenAI": "OpenAI", "dash.backendAnthropic": "Anthropic", + "dash.backendChat": "Chat API", "dash.sidecarSaved": "附属设置已保存。将在下一个请求时生效。", "dash.sidecarSaveFailed": "保存附属设置失败。", "dash.injectionLabel": "子代理委托", diff --git a/gui/src/pages/claude-code-sections.tsx b/gui/src/pages/claude-code-sections.tsx index 70fa292fa4..183d23b3f5 100644 --- a/gui/src/pages/claude-code-sections.tsx +++ b/gui/src/pages/claude-code-sections.tsx @@ -170,6 +170,7 @@ export function ClaudeCodeSettingsCard({ { value: "auto", label: t("dash.backendAuto") }, { value: "openai", label: t("dash.backendOpenAI") }, { value: "anthropic", label: t("dash.backendAnthropic") }, + ...(key === "visionSidecar" ? [{ value: "chat", label: t("dash.backendChat") }] : []), ]} onChange={value => { // Auto may exist as an empty in-memory draft so the model input diff --git a/gui/src/pages/claude-manual-env.ts b/gui/src/pages/claude-manual-env.ts index 59f3360c83..65afc00a30 100644 --- a/gui/src/pages/claude-manual-env.ts +++ b/gui/src/pages/claude-manual-env.ts @@ -5,7 +5,7 @@ */ import { AUTO_COMPACT_WINDOW_DEFAULT } from "./claude-code-types"; -export type SidecarBackend = "openai" | "anthropic"; +export type SidecarBackend = "openai" | "anthropic" | "chat"; export interface SidecarOverride { backend?: SidecarBackend; model?: string } export interface ClaudeManualEnvState { diff --git a/gui/src/pages/dashboard-overview-sections.tsx b/gui/src/pages/dashboard-overview-sections.tsx index 6b944ed5eb..d312f8e6f9 100644 --- a/gui/src/pages/dashboard-overview-sections.tsx +++ b/gui/src/pages/dashboard-overview-sections.tsx @@ -557,7 +557,14 @@ export function DashboardSidecarPanels({ d }: { d: Dash }) { } const ladder = visionReasoningLadder(models, model); const reasoning = clampVisionReasoningToLadder(ladder, visionReasoning); - const patch: SidecarPatch = { vision: { model, backend: visionSidecarBackendForModel(models, visionModels, model), reasoning } }; + const backend = visionSidecarBackendForModel(models, visionModels, model); + const patch: SidecarPatch = { + vision: { + model, + ...(backend === "unresolved" ? { backend: null } : { backend }), + reasoning, + }, + }; // Choosing a model is the activation control: turning Vision back on from Off. if (!visionEnabled) patch.vision = { ...patch.vision, enabled: true }; void saveSidecar(patch); diff --git a/gui/src/pages/dashboard-shared.ts b/gui/src/pages/dashboard-shared.ts index 809d08c04f..713dbc27e6 100644 --- a/gui/src/pages/dashboard-shared.ts +++ b/gui/src/pages/dashboard-shared.ts @@ -59,10 +59,12 @@ export interface SettingsData { diagnosticStale: boolean; }; } -export type SidecarBackend = "openai" | "anthropic"; +export type SidecarBackend = "openai" | "anthropic" | "chat"; +/** Web-search sidecars accept only OpenAI/Anthropic — `chat` is vision-only. */ +export type WebSearchBackend = "openai" | "anthropic"; export type VisionReasoning = "low" | "medium" | "high" | "xhigh" | "max"; -export interface SidecarSetting { - backend?: SidecarBackend; +export interface SidecarSetting { + backend?: B; model: string; reasoning?: VisionReasoning; streamRoutedModelOutput?: boolean; @@ -74,18 +76,18 @@ export interface VisionModelOption { value: string; label: string; backend: Side export interface WebSearchModelOption { value: string; label: string; - backend: SidecarBackend; + backend: WebSearchBackend; model: string; authSlot?: boolean; } export interface WebSearchPickerOption { value: string; label: string; - backend?: SidecarBackend; + backend?: WebSearchBackend; model?: string; } export interface SidecarData { - webSearch: SidecarSetting; + webSearch: SidecarSetting; vision: SidecarSetting; /** Server-computed eligible describers. Optional: an older server omits it and * the client falls back to the legacy provider-name list rather than showing @@ -97,7 +99,7 @@ export interface SidecarData { webSearchModels?: WebSearchModelOption[]; } export interface SidecarPatch { - webSearch?: { backend?: SidecarBackend | null; model?: string; streamRoutedModelOutput?: boolean }; + webSearch?: { backend?: WebSearchBackend | null; model?: string; streamRoutedModelOutput?: boolean }; vision?: { backend?: SidecarBackend | null; model?: string; @@ -186,10 +188,10 @@ export function updateJobLabel(status: UpdateJobStatus, t: (key: TKey) => string } } -export function mergeSidecarSetting( - current: SidecarSetting, +export function mergeSidecarSetting( + current: SidecarSetting, update?: { - backend?: SidecarBackend | null; + backend?: B | null; model?: string; reasoning?: VisionReasoning; streamRoutedModelOutput?: boolean; @@ -197,7 +199,7 @@ export function mergeSidecarSetting( maxDescriptionsPerTurn?: number; timeoutMs?: number; }, -): SidecarSetting { +): SidecarSetting { const merged = { ...current }; if (update?.model !== undefined) merged.model = update.model; if (update?.backend === null) delete merged.backend; @@ -307,7 +309,7 @@ export function webSearchModelOptionsForPicker( serverOptions: WebSearchModelOption[] | undefined, models: ModelInfo[], current: string | undefined, - currentBackend?: SidecarBackend, + currentBackend?: WebSearchBackend, ): WebSearchPickerOption[] { if (serverOptions === undefined) { const legacy: WebSearchPickerOption[] = sidecarModelOptions(models); @@ -359,11 +361,23 @@ export function visionModelOptions( current: string | undefined, currentBackend?: SidecarBackend, ): Array<{ value: string; label: string; backend?: SidecarBackend }> { - const options = serverOptions + const options: Array<{ value: string; label: string; backend?: SidecarBackend }> = serverOptions ? serverOptions.map(option => ({ value: option.value, label: option.label, backend: option.backend })) : sidecarModelOptions(models); - if (current && !options.some(option => option.value === current)) { - options.unshift({ value: current, label: current, ...(currentBackend ? { backend: currentBackend } : {}) }); + if (current) { + const currentOption = options.find(option => option.value === current); + if (currentOption) { + // On the legacy path, the persisted backend is stronger evidence than a + // catalog provider match. A chat-backed model can share an id with a + // catalog OpenAI row, and saving that row must not silently switch it. + if (serverOptions === undefined && currentBackend && currentOption.backend === undefined) { + currentOption.backend = currentBackend; + } + } else { + // A legacy server has no authoritative option/backend to provide, so keep + // the configured backend on a grandfathered model rather than guessing. + options.unshift({ value: current, label: current, ...(currentBackend ? { backend: currentBackend } : {}) }); + } } return options; } @@ -375,8 +389,21 @@ export function shadowCallModelOptions(models: ModelInfo[], current: string | un return out; } -export function sidecarBackendForModel(models: ModelInfo[], modelId: string): SidecarBackend { - return models.find(model => model.id === modelId)?.provider === "anthropic" ? "anthropic" : "openai"; +export type SidecarBackendResolution = SidecarBackend | "unresolved"; + +/** Resolve a catalog model without guessing across namespaces or providers. */ +export function sidecarBackendForModel(models: ModelInfo[], modelId: string): SidecarBackendResolution { + // Exact namespaced matches win; otherwise a bare id must match exactly one row. + // Ambiguous and unknown ids stay unresolved instead of persisting a guessed backend. + const namespacedMatches = models.filter(model => model.namespaced === modelId); + const matches = namespacedMatches.length > 0 + ? namespacedMatches + : models.filter(model => model.id === modelId); + if (matches.length !== 1) return "unresolved"; + const provider = matches[0]!.provider; + if (provider === "openai") return "openai"; + if (provider === "anthropic") return "anthropic"; + return "chat"; } /** Server provenance wins; catalog inference supports only legacy option rows. */ @@ -384,10 +411,11 @@ export function webSearchSidecarSelectionForModel( models: ModelInfo[], options: WebSearchPickerOption[], modelId: string, -): { backend: SidecarBackend; model: string } { +): { backend: WebSearchBackend; model: string } { const option = options.find(entry => entry.value === modelId); + const inferred = sidecarBackendForModel(models, modelId); return { - backend: option?.backend ?? sidecarBackendForModel(models, modelId), + backend: option?.backend ?? (inferred === "anthropic" ? "anthropic" : "openai"), model: option?.model ?? modelId, }; } @@ -397,7 +425,7 @@ export function visionSidecarBackendForModel( models: ModelInfo[], options: Array<{ value: string; backend?: SidecarBackend }>, modelId: string, -): SidecarBackend { +): SidecarBackendResolution { return options.find(option => option.value === modelId)?.backend ?? sidecarBackendForModel(models, modelId); } diff --git a/gui/tests/claude-code-sidecar-draft.test.tsx b/gui/tests/claude-code-sidecar-draft.test.tsx index bfcc6cca85..8a4e55fd8c 100644 --- a/gui/tests/claude-code-sidecar-draft.test.tsx +++ b/gui/tests/claude-code-sidecar-draft.test.tsx @@ -117,6 +117,29 @@ test("Inherit → Auto stays Auto and enables the sidecar model input", async () await act(async () => { root.unmount(); }); }); +test("Chat is available only for the Vision override", async () => { + const { root, host } = await mountSettings(); + const [webSearchTrigger, visionTrigger] = sidecarBackendTriggers(host); + + await act(async () => { webSearchTrigger!.click(); }); + expect([...document.body.querySelectorAll('[role="option"]')] + .some(option => option.textContent?.includes("Chat API"))).toBe(false); + const webInherit = [...document.body.querySelectorAll('[role="option"]')] + .find(option => option.textContent?.includes("Use main setting")); + expect(webInherit).toBeTruthy(); + await act(async () => { webInherit!.click(); }); + + await act(async () => { visionTrigger!.click(); }); + expect([...document.body.querySelectorAll('[role="option"]')] + .some(option => option.textContent?.includes("Chat API"))).toBe(true); + const visionInherit = [...document.body.querySelectorAll('[role="option"]')] + .find(option => option.textContent?.includes("Use main setting")); + expect(visionInherit).toBeTruthy(); + await act(async () => { visionInherit!.click(); }); + + await act(async () => { root.unmount(); }); +}); + test("Empty Auto draft keeps model input enabled until Inherit is chosen", async () => { const { root, host, getState } = await mountSettings(); const [webSearchTrigger] = sidecarBackendTriggers(host); diff --git a/gui/tests/vision-model-options.test.ts b/gui/tests/vision-model-options.test.ts index 14c9cf4078..622603e5d6 100644 --- a/gui/tests/vision-model-options.test.ts +++ b/gui/tests/vision-model-options.test.ts @@ -70,9 +70,9 @@ test("a server-supplied vision backend wins when the catalog cannot identify its { value: model, label: model, backend: "anthropic" }, ], [], undefined); - // This documents the old save-path result: catalog inference alone defaults - // to OpenAI when there is no matching model row. - expect(sidecarBackendForModel([], model)).toBe("openai"); + // Catalog inference alone cannot name a backend for an unknown model. An + // unresolved result must never silently persist a guessed backend. + expect(sidecarBackendForModel([], model)).toBe("unresolved"); expect(visionSidecarBackendForModel([], options, model)).toBe("anthropic"); }); @@ -85,7 +85,19 @@ test("an older server with no option list cannot rewrite the persisted anthropic expect(options[0]).toMatchObject({ value: model, backend: "anthropic" }); expect(visionSidecarBackendForModel([], options, model)).toBe("anthropic"); - expect(sidecarBackendForModel([], model)).toBe("openai"); + expect(sidecarBackendForModel([], model)).toBe("unresolved"); +}); + +test("a legacy fallback keeps chat on the current option even when the catalog has the same id", () => { + const model = "shared-vision-model"; + const options = visionModelOptions(undefined, [ + { id: model, provider: "openai", namespaced: model }, + ], model, "chat"); + + // The catalog row is only a legacy fallback suggestion. The persisted chat backend must win + // for the current option, otherwise selecting it sends backend:null and reverts to auto. + expect(options[0]).toMatchObject({ value: model, backend: "chat" }); + expect(visionSidecarBackendForModel([], options, model)).toBe("chat"); }); test("an authoritative empty list still keeps the configured model and its backend", () => { @@ -109,3 +121,18 @@ test("a server web-search auth-slot option keeps its backend without a catalog r model, }); }); + +test("an ambiguous bare id is unresolved while an exact namespaced id wins", () => { + const shared: ModelInfo[] = [ + { id: "shared", provider: "openai", namespaced: "openai/shared" }, + { id: "shared", provider: "anthropic", namespaced: "anthropic/shared" }, + ]; + expect(sidecarBackendForModel(shared, "shared")).toBe("unresolved"); + expect(sidecarBackendForModel(shared, "anthropic/shared")).toBe("anthropic"); +}); + +test("a non-OpenAI and non-Anthropic catalog provider resolves to chat", () => { + const chat: ModelInfo = { id: "vision-chat", provider: "google", namespaced: "google/vision-chat" }; + expect(sidecarBackendForModel([chat], "google/vision-chat")).toBe("chat"); + expect(sidecarBackendForModel([chat], "vision-chat")).toBe("chat"); +}); diff --git a/readme/README.fr.md b/readme/README.fr.md index f4630952b9..5e46ecfdd2 100644 --- a/readme/README.fr.md +++ b/readme/README.fr.md @@ -145,9 +145,9 @@ bloqué les scripts d'installation de l'environnement inclus, consultez la [guide des sous-agents](https://opencodex.me/fr/guides/sub-agent-surface/). - **Connectez-vous une fois, oubliez la clé API** — OAuth pour xAI, Anthropic et Kimi ; ou transmettez `codex login`, collez une clé ou utilisez des références `${ENV_VAR}`. -- **Modules complémentaires de recherche web et de vision** — les modèles non-OpenAI bénéficient d'une - véritable recherche web et de la compréhension d'images grâce à un module complémentaire utilisant - votre connexion ChatGPT. +- **Modules complémentaires de recherche web et de vision** — la recherche web reste OpenAI/Anthropic ; + la vision peut aussi utiliser un backend Chat explicite via un fournisseur configuré (`provider/model`). + Consultez le [guide canonique des sidecars](https://opencodex.me/guides/sidecars/). - **Voyez ce qui se passe** — le tableau de bord affiche les fournisseurs, l'état OAuth, la sélection des modèles et un journal des requêtes en direct avec le nombre de jetons de cache. - **Arrêt propre, aucun résidu** — `ocx stop` restaure la configuration d'origine de Codex. diff --git a/readme/README.ja.md b/readme/README.ja.md index ff20cd2c90..6df581d0da 100644 --- a/readme/README.ja.md +++ b/readme/README.ja.md @@ -207,7 +207,7 @@ opencodex は 2 つの動作を分離して保持します: - **履歴セーフな注入。** ローカルインストールではプロキシは Codex 自身の組み込み `openai` プロバイダーを単一の `openai_base_url` 行で自身に向けるため、新しいスレッドはネイティブのプロバイダータグを維持し、進行中のチャット履歴が再マッピングされることはなく、クリーンでないシャットダウンでも隠せません。(古いバージョンで再タグ付けされたスレッドは初回起動時に一度だけマイグレートされます; リモート/LAN バインドは API キーヘッダーが必要なため、専用のプロバイダーエントリを使用します。) - **適切なモデルに委任。** ダッシュボードや config から最大 5 つのルーティング/ネイティブモデルを Codex サブエージェントピッカーに公開し、複雑なタスクは推論モデルへ、高速なタスクは安価なモデルへ送れます。v2 マルチエージェントサーフェス(GPT-5.6 Sol/Terra)ではプロキシが簡潔な委任ガイダンスを注入します。推奨サブエージェントモデル・負荷(`injectionModel` / `injectionEffort`)、公開モデルロスターと各モデルが対応する負荷ラダー、そしてクロスモデル `spawn_agent` オーバーライドを適用する `fork_turns` ルールまで。既知の制限: ネイティブの親がルーティング子をスポーンすると、タスク本文がバックエンド暗号化状態で到着し失われることがあります([#92](https://github.com/lidge-jun/opencodex/issues/92)) — 安定したクロスプロバイダー委任には v1 サーフェスを使ってください。表現を自分で書きたい場合は `injectionPrompt` に `{{model}}` / `{{effort}}` / `{{roster}}` プレースホルダーを入れてください。 - **preview gate された OpenAI ロールアウトに備える。** GPT-5.6 Sol/Terra/Luna の負荷ラダーを保存します。Direct/Multi は 372k Codex 契約を、OpenAI API と OpenRouter は 1.05M メタデータを使います。 -- **任意のモデルに超能力を。** OpenAI 以外のモデルも ChatGPT ログイン上で動く `gpt-5.4-mini` サイドカーで本当のウェブ検索と画像理解を得られます。 +- **Web 検索と Vision サイドカー。** Web 検索は OpenAI/Anthropic を使い、Vision は設定済みプロバイダー経由の明示的な Chat backend（`provider/model`）にも対応します。詳しくは[サイドカーガイド](https://opencodex.me/guides/sidecars/)を参照してください。 - **画像をネイティブに生成。** Codex の独立型 `image_gen` ツールは生成時に `POST /v1/images/generations`、編集時に `POST /v1/images/edits` を使います。Responses のホスト型 `image_generation` ツールとは別物です。 - **何が起きているかを可視化。** ウェブダッシュボードがプロバイダー、OAuth 状態、モデル選択、upstream が報告した cached/cache-write トークン数を含むライブリクエストログを表示します — なぜリクエストが失敗したか推測する必要はもうありません。 - **バックグラウンド実行。** システムサービス(launchd / systemd / Task Scheduler)としてインストールすれば起動時に自動開始され、気にする必要がありません。 diff --git a/readme/README.ko.md b/readme/README.ko.md index f7f11aef0a..ceee42fa78 100644 --- a/readme/README.ko.md +++ b/readme/README.ko.md @@ -198,7 +198,7 @@ opencodex는 두 가지 동작을 분리해서 유지합니다: - **Codex가 동작하는 모든 곳에서.** Codex CLI, TUI, App, SDK에 자동으로 주입됩니다. 라우팅된 모델이 네이티브 모델처럼 Codex 모델 선택기에 나타납니다. - **알맞은 모델에 위임.** 대시보드나 config에서 최대 5개의 라우팅/네이티브 모델을 Codex 서브에이전트 선택기에 노출해, 복잡한 작업은 reasoning 모델로, 빠른 작업은 저렴한 모델로 보낼 수 있습니다. v2 멀티에이전트 표면(GPT-5.6 Sol/Terra)에서는 프록시가 간결한 위임 가이드를 주입합니다. 선호 서브에이전트 모델·effort(`injectionModel` / `injectionEffort`), 노출된 모델 로스터와 각 모델이 지원하는 effort 사다리, 그리고 크로스모델 `spawn_agent` 오버라이드를 적용하는 `fork_turns` 규칙까지. 알려진 제한: 네이티브 부모가 라우팅 자식을 스폰하면 작업 본문이 백엔드 암호화 상태로 도착해 유실될 수 있습니다([#92](https://github.com/lidge-jun/opencodex/issues/92)) — 안정적인 크로스 프로바이더 위임에는 v1 표면을 쓰세요. 문구를 직접 쓰고 싶다면 `injectionPrompt`에 `{{model}}` / `{{effort}}` / `{{roster}}` 플레이스홀더를 넣으면 됩니다. - **프리뷰 게이트된 OpenAI rollout에 대비.** GPT-5.6 Sol/Terra/Luna의 effort 사다리를 보존합니다. Direct/Multi는 372k Codex 계약을, OpenAI API와 OpenRouter는 1.05M metadata를 사용합니다. -- **어떤 모델에도 초능력을.** OpenAI가 아닌 모델도 ChatGPT 로그인 위에서 도는 `gpt-5.4-mini` sidecar로 실제 웹 검색과 이미지 이해를 사용합니다. +- **Web 검색과 vision sidecar.** Web 검색은 OpenAI/Anthropic을 사용하며, vision은 구성된 provider를 통한 명시적 Chat backend(`provider/model`)도 지원합니다. 자세한 내용은 [sidecar 가이드](https://opencodex.me/guides/sidecars/)를 참조하세요. - **이미지를 네이티브로 생성.** Codex의 독립형 `image_gen` 도구는 생성할 때 `POST /v1/images/generations`, 편집할 때 `POST /v1/images/edits`를 사용합니다. Responses의 hosted `image_generation` 도구와는 별개입니다. - **무슨 일이 일어나는지 보이게.** 웹 대시보드가 프로바이더, OAuth 상태, 모델 선택, upstream이 보고한 cached/cache-write 토큰 수를 포함한 실시간 요청 로그를 보여줍니다 — 왜 요청이 실패했는지 더는 추측하지 않아도 됩니다. - **백그라운드 실행.** 시스템 서비스(launchd / systemd / Task Scheduler)로 설치하면 부팅 시 자동 시작되어 신경 쓸 필요가 없습니다. diff --git a/readme/README.ru.md b/readme/README.ru.md index 478c8772e9..d4c1e4d78e 100644 --- a/readme/README.ru.md +++ b/readme/README.ru.md @@ -233,7 +233,8 @@ OpenAI API-ключа и OpenRouter (`gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-lu - **Встраивание без риска для истории.** При локальной установке прокси перенаправляет встроенный провайдер Codex `openai` на себя одной строкой `openai_base_url` — новые треды сохраняют нативный тег провайдера, поэтому текущая история чатов никогда не перепривязывается, и даже некорректное завершение работы не может её скрыть. (Треды, перетегированные старыми версиями, однократно мигрируются обратно при первом запуске; при удалённой/LAN-привязке вместо этого используется отдельная запись провайдера, поскольку ей нужен заголовок с API-ключом.) - **Делегируйте задачи подходящей модели.** Через панель управления или конфигурацию можно вывести до пяти маршрутизируемых или нативных моделей в селектор подагентов Codex — сложные задачи отправляйте модели с развитыми рассуждениями, быстрые — дешёвой. На мультиагентной поверхности v2 (GPT-5.6 Sol/Terra) прокси внедряет компактные указания по делегированию: предпочтительную модель и уровень рассуждений подагента (`injectionModel` / `injectionEffort`), список отобранных моделей со шкалой уровней, которую поддерживает каждая из них, и правила `fork_turns`, позволяющие кросс-модельным вызовам `spawn_agent` применять свои переопределения. Известное ограничение: когда нативный родитель порождает маршрутизируемого потомка, тело задачи в настоящий момент может прийти зашифрованным на бэкенде и потеряться ([#92](https://github.com/lidge-jun/opencodex/issues/92)) — для надёжного делегирования между провайдерами используйте поверхность v1. Хотите свои формулировки? Задайте `injectionPrompt` с плейсхолдерами `{{model}}` / `{{effort}}` / `{{roster}}`. - **Готовность к превью-релизам OpenAI.** Записи GPT-5.6 Sol/Terra/Luna сохраняют исходные шкалы уровней рассуждений. Direct/Multi используют контракт Codex на 372k токенов; OpenAI API и OpenRouter — метаданные на 1.05M, когда открыт вышестоящий доступ. -- **Суперспособности для любой модели.** Модели не от OpenAI получают настоящий веб-поиск и понимание изображений через сайдкар `gpt-5.4-mini`, работающий поверх вашего входа ChatGPT. +- **Веб-поиск и vision-сайдкары.** Веб-поиск использует OpenAI/Anthropic; vision также поддерживает + Chat-бэкенд через настроенного провайдера (`provider/model`). См. [руководство по сайдкарам](https://opencodex.me/guides/sidecars/). - **Нативная генерация изображений.** Автономный инструмент Codex `image_gen` использует `POST /v1/images/generations` для генерации и `POST /v1/images/edits` для правок; он не связан с размещённым инструментом Responses `image_generation`. - **Видно, что происходит.** Веб-панель управления показывает провайдеров, статус OAuth, выбор моделей и живой журнал запросов, включая количество кэшированных и записанных в кэш токенов, когда вышестоящий провайдер их сообщает, — больше не нужно гадать, почему запрос не прошёл. - **Работает в фоне.** Установите как системную службу (launchd / systemd / Task Scheduler) и забудьте о ней. На macOS/Linux прокси стартует при входе в систему; на Windows бэкенд Task Scheduler по умолчанию запускается при входе (без окон), либо используйте `ocx service install --native`, чтобы получить полноценную службу Windows, стартующую при загрузке. diff --git a/readme/README.tr.md b/readme/README.tr.md index a0388863cb..8bfaf8b838 100644 --- a/readme/README.tr.md +++ b/readme/README.tr.md @@ -85,7 +85,7 @@ ocx init # etkileşimli kurulum: ~/.opencodex/config.json yazar ve Codex'i - **Kombolar (Combos)** — Sağlayıcılar arasında yedekli (failover) veya ağırlıklı round-robin ile çalışan tek bir sanal model kimliği. [Kombo rehberine](https://opencodex.me/guides/combos/) bakın. - **Herhangi bir modelde alt ajanlar** — v1/v2 yüzey kontrolü ve yedekleme zincirleriyle Codex'in alt ajan seçicisinde yönlendirilen modelleri öne çıkarın. [Alt ajan rehberine](https://opencodex.me/guides/sub-agent-surface/) bakın. - **Bir kez giriş yapın, API anahtarını atlayın** — xAI, Anthropic ve Kimi için OAuth; veya `codex login` iletme, anahtar yapıştırma veya `${ENV_VAR}` referansları kullanma. -- **Web araması ve görsel yan araçları (sidecars)** — OpenAI dışı modeller, ChatGPT girişiniz üzerinden bir sidecar aracılığıyla gerçek web araması ve görsel anlama yeteneği kazanır. +- **Web araması ve görsel sidecar'ları** — Web araması OpenAI/Anthropic kullanır; vision ayrıca yapılandırılmış bir sağlayıcı üzerinden açık Chat backend'ini (`provider/model`) destekler. [Sidecar kılavuzuna](https://opencodex.me/guides/sidecars/) bakın. - **Ne olduğunu görün** — Panel; sağlayıcıları, OAuth durumunu, model seçimini ve önbellek jeton sayılarıyla canlı istek günlüğünü gösterir. - **Temiz çıkış, sıfır kalıntı** — `ocx stop` Codex'i orijinal yapılandırmasına geri döndürür. diff --git a/readme/README.zh-CN.md b/readme/README.zh-CN.md index fa7cc35c3a..1df9a1ddb8 100644 --- a/readme/README.zh-CN.md +++ b/readme/README.zh-CN.md @@ -133,7 +133,7 @@ npm 警告里给出的缩写命令缺少包名，会把当前目录重新安装 - **Codex 在哪里能用，它就在哪里能用。** 自动注入 Codex CLI、TUI、App 和 SDK。路由模型像原生模型一样出现在 Codex 的模型选择器里。 - **委派给合适的模型。** 在仪表盘或 config 中把最多 5 个路由/原生模型放进 Codex 的 subagent 选择器 —— 复杂任务交给 reasoning 模型，快速任务交给便宜模型。在 v2 多智能体表面（GPT-5.6 Sol/Terra）上，代理会注入精简的委派指引：首选子智能体模型与 effort（`injectionModel` / `injectionEffort`）、featured 模型清单及各自支持的 effort 阶梯，以及让跨模型 `spawn_agent` 覆盖得以应用的 `fork_turns` 规则。已知限制：原生父代理 spawn 路由子代理时，任务正文可能以后端加密形式到达而丢失（[#92](https://github.com/lidge-jun/opencodex/issues/92)）—— 需要可靠的跨 provider 委派请使用 v1 表面。想自定义文案，可在 `injectionPrompt` 中使用 `{{model}}` / `{{effort}}` / `{{roster}}` 占位符。 - **为 preview-gated OpenAI rollout 做好准备。** GPT-5.6 Sol/Terra/Luna 保留 upstream effort 阶梯。Direct/Multi 使用 372k Codex 契约，OpenAI API 与 OpenRouter 使用 1.05M 元数据。 -- **给任意模型超能力。** 非 OpenAI 模型也能通过你的 ChatGPT 登录上运行的 `gpt-5.4-mini` sidecar 获得真正的网页搜索和图片理解。 +- **Web Search 与 Vision sidecar。** Web Search 仍使用 OpenAI/Anthropic；Vision 也支持通过已配置 provider 的显式 Chat backend（`provider/model`）。详见 [sidecar 文档](https://opencodex.me/guides/sidecars/)。 - **原生生成图片。** Codex 的独立 `image_gen` 工具通过 `POST /v1/images/generations` 生成图片、通过 `POST /v1/images/edits` 编辑图片；它独立于 hosted Responses 的 `image_generation` 工具。 - **看清正在发生什么。** Web 仪表盘展示 provider、OAuth 状态、模型选择和实时请求日志；当上游返回时，也会包含 cached/cache-write token 计数 —— 不必再猜测请求为何失败。 - **后台运行。** 安装为系统服务（launchd / systemd / Task Scheduler）后开机自启，无需操心。 diff --git a/readme/README.zh-TW.md b/readme/README.zh-TW.md index d587a908cd..db322a1f87 100644 --- a/readme/README.zh-TW.md +++ b/readme/README.zh-TW.md @@ -120,7 +120,7 @@ npm 警告給的縮寫指令少了套件名，會把目前目錄重裝進去， - **Codex 在哪裡能用，它就在哪裡能用。** 自動注入 Codex CLI、TUI、App 和 SDK。路由模型像原生模型一樣出現在 Codex 的模型選擇器裡。 - **委派給合適的模型。** 在儀表板或 config 中把最多 5 個路由/原生模型放進 Codex 的 subagent 選擇器 —— 複雜任務交給 reasoning 模型，快速任務交給便宜模型。在 v2 多智慧體表面（GPT-5.6 Sol/Terra）上，代理會注入精簡的委派指引：首選子智慧體模型與 effort（`injectionModel` / `injectionEffort`）、featured 模型清單及各自支援的 effort 階梯，以及讓跨模型 `spawn_agent` 覆蓋得以應用的 `fork_turns` 規則。已知限制：原生父代理 spawn 路由子代理時，任務本文可能以後端加密形式到達而丟失（[#92](https://github.com/lidge-jun/opencodex/issues/92)）—— 需要可靠的跨 provider 委派請使用 v1 表面。想自訂文案，可在 `injectionPrompt` 中使用 `{{model}}` / `{{effort}}` / `{{roster}}` 預留位置。 - **為 preview-gated OpenAI rollout 做好準備。** GPT-5.6 Sol/Terra/Luna 保留 upstream effort 階梯。Direct/Multi 使用 372k Codex 契約，OpenAI API 與 OpenRouter 使用 1.05M 後設資料。 -- **給任意模型超能力。** 非 OpenAI 模型可透過 `gpt-5.4-mini` sidecar（使用你的 ChatGPT 登入）獲得真正的網頁搜尋與圖片理解。 +- **Web Search 與 Vision sidecar。** Web Search 仍使用 OpenAI/Anthropic；Vision 也支援透過已設定 provider 的明確 Chat backend（`provider/model`）。詳見 [sidecar 文件](https://opencodex.me/guides/sidecars/)。 - **原生生成圖片。** Codex 的獨立 `image_gen` 工具透過 `POST /v1/images/generations` 生成圖片、透過 `POST /v1/images/edits` 編輯圖片；它獨立於 hosted Responses 的 `image_generation` 工具。 - **看清正在發生什麼。** Web 儀表板展示 provider、OAuth 狀態、模型選擇和即時請求日誌；當上遊回傳時，也會包含 cached/cache-write token 計數 —— 不用再猜請求為何失敗。 - **背景執行。** 安裝為系統服務（launchd / systemd / Task Scheduler）後開機自啟，無需操心。 diff --git a/src/providers/api-keys.ts b/src/providers/api-keys.ts index 88a0d72238..a543903944 100644 --- a/src/providers/api-keys.ts +++ b/src/providers/api-keys.ts @@ -7,7 +7,7 @@ * A provider with a legacy bare `apiKey` is seeded into a one-entry pool on first touch. */ import { createHash } from "node:crypto"; -import { saveConfigPreservingClaudeCode } from "../config"; +import { resolveEnvValue, saveConfigPreservingClaudeCode } from "../config"; import type { OcxConfig, OcxProviderConfig } from "../types"; export interface ProviderApiKeyInfo { @@ -61,6 +61,24 @@ function activeEntryId(provider: OcxProviderConfig): string | null { return (pool.find(e => e.key === provider.apiKey) ?? pool[0]!).id; } +/** + * Resolve the non-mutating active credential for a provider. + * + * `provider.apiKey` is the active mirror, while `apiKeyPool` may contain the + * complete set of slots. Match that mirror to its pool entry instead of + * assuming the first slot is active. Environment references are resolved only + * for the returned value; the provider config and pool are never changed. + */ +export function resolveActiveProviderApiKey(provider: OcxProviderConfig): string | undefined { + const pool = provider.apiKeyPool ?? []; + const active = provider.apiKey !== undefined + ? pool.find(entry => entry.key === provider.apiKey) + : undefined; + const candidate = active?.key ?? provider.apiKey ?? pool[0]?.key; + const resolved = resolveEnvValue(candidate); + return sanitizeApiKeyValue(resolved); +} + export function listProviderApiKeys(config: OcxConfig, name: string): { activeId: string | null; keys: ProviderApiKeyInfo[] } { const provider = config.providers[name]; if (!provider || !isKeyAuthProvider(provider)) return { activeId: null, keys: [] }; diff --git a/src/server/management/agent-settings-routes.ts b/src/server/management/agent-settings-routes.ts index 2fbee7d5a9..080ac1b561 100644 --- a/src/server/management/agent-settings-routes.ts +++ b/src/server/management/agent-settings-routes.ts @@ -1073,13 +1073,10 @@ export async function handleAgentSettingsRoutes(ctx: ManagementContext): Promise const section = body[field]; if (section === undefined || section === null) continue; if (!isPlainObject(section)) return jsonResponse({ error: `${field} must be an object or null` }, 400); - // The widened union applies to the WEB-SEARCH override only (roadmap 060). - // Vision keeps its two-backend contract — accepting a wider id there would - // persist a backend the vision resolver reads as unset, silently activating - // a backend the operator never chose (review F1). + // Web search and vision intentionally expose different backend sets. const allowedBackends = field === "webSearchSidecar" ? ["openai", "anthropic", "xai", "gemini", "exa"] - : ["openai", "anthropic"]; + : ["openai", "anthropic", "chat"]; if (section.backend !== undefined && section.backend !== null && !allowedBackends.includes(section.backend as string)) { return jsonResponse({ error: `${field}.backend must be ${allowedBackends.join(", ")}, or null` }, 400); @@ -1138,9 +1135,11 @@ export async function handleAgentSettingsRoutes(ctx: ManagementContext): Promise delete next[field]; continue; } - // The per-field validation above guarantees vision only ever carries the two-member - // union; the cast is the loop's shared-shape compromise, not a wider write path. - const requested = section as { backend?: "openai" | "anthropic" | "xai" | "gemini" | "exa" | null; model?: string }; + // Validation above guarantees the backend belongs to this field's union. + const requested = section as { + backend?: "openai" | "anthropic" | "xai" | "gemini" | "exa" | "chat" | null; + model?: string; + }; const override = { ...next[field] } as NonNullable; if (requested.backend === null) delete override.backend; else if (requested.backend !== undefined) override.backend = requested.backend as never; diff --git a/src/server/management/config-routes.ts b/src/server/management/config-routes.ts index 1e5e1ad2c6..b3af0ede4a 100644 --- a/src/server/management/config-routes.ts +++ b/src/server/management/config-routes.ts @@ -592,8 +592,8 @@ export async function handleConfigRoutes(ctx: ManagementContext): Promise 0) backends.push("openai"); if (anthropicSidecar) backends.push("anthropic"); + // The chat describer needs a configured openai-chat/google provider with usable + // auth — same predicate the runtime chat sidecar resolver uses. + if (hasUsableChatVisionProvider(config)) backends.push("chat"); // Neither side resolvable (fresh install, no login): fall back to both so the // picker is populated rather than empty, matching the permissive-unknown rule. return backends.length > 0 ? backends : ["openai", "anthropic"]; } +/** Any configured chat/google provider the chat sidecar can actually execute. */ +function hasUsableChatVisionProvider(config: OcxConfig): boolean { + return Object.entries(config.providers ?? {}).some(([providerName, provider]) => + (provider.adapter === "openai-chat" || provider.adapter === "google") + && isChatVisionProviderUsable(providerName, provider), + ); +} + /** * Visible catalog rows in the shape the eligibility predicate consumes. * Sourced from the unified picker set (#2188): picker-visible rows ∪ auth diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index 22bf3c18c3..3fe28cf0ae 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -2728,6 +2728,15 @@ async function handleResponsesInner( const visionPlan = planVisionSidecar(config, route.provider, route.modelId, parsed, openAiSidecar); const recordSidecarOutcome = openAiSidecar?.recordOutcome; if (visionPlan) { + const visionProviderFetch = visionPlan.backend === "chat" && visionPlan.chatSidecar + ? providerFetch(visionPlan.chatSidecar.provider, options.codexWsRuntimeIdentity, { + providerName: visionPlan.chatSidecar.providerName, + modelId: visionPlan.chatSidecar.model, + }) + : providerFetch(route.provider, options.codexWsRuntimeIdentity, { + providerName: route.providerName, + modelId: route.modelId, + }); await describeImagesInPlace( parsed, visionPlan, @@ -2735,6 +2744,7 @@ async function handleResponsesInner( options.abortSignal, recordSidecarOutcome, translatorBudget, + visionProviderFetch, ); } else if (isModelTextOnly(route.provider, route.modelId)) { // Sidecar-covered model but NO plan (no forward provider / missing forwarded auth / sidecar diff --git a/src/types/config.ts b/src/types/config.ts index de84457743..b0a5df0153 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -130,7 +130,7 @@ export interface OcxClaudeCodeConfig { /** Claude-originated web-search override. Unset fields inherit the global sidecar settings. */ webSearchSidecar?: { backend?: "openai" | "anthropic" | "xai" | "gemini" | "exa"; model?: string }; /** Claude-originated vision override. Unset fields inherit the global sidecar settings. */ - visionSidecar?: { backend?: "openai" | "anthropic"; model?: string }; + visionSidecar?: { backend?: "openai" | "anthropic" | "chat"; model?: string }; /** Persisted Claude Desktop four-family routing profile. */ desktopProfile?: OcxClaudeDesktopProfile; /** Auto-reconcile Desktop 3P config when provider catalog changes. Default: enabled. */ @@ -775,7 +775,7 @@ export interface OcxVisionSidecarConfig { /** Master switch. Default: enabled when the selected backend has a usable credential. */ enabled?: boolean; /** Description backend. Unset prefers a usable stored Anthropic OAuth credential, else OpenAI. */ - backend?: "openai" | "anthropic"; + backend?: "openai" | "anthropic" | "chat"; /** Vision model that describes images. */ model?: string; /** Max description cache misses admitted in one main-model turn. Zero disables description calls. */ diff --git a/src/vision/describe-chat.ts b/src/vision/describe-chat.ts new file mode 100644 index 0000000000..b4bf3ece80 --- /dev/null +++ b/src/vision/describe-chat.ts @@ -0,0 +1,311 @@ +import { createGoogleAdapter } from "../adapters/google"; +import type { OcxParsedRequest, OcxProviderConfig } from "../types"; +import { signalWithTimeout, cancelBodyOnAbort } from "../lib/abort"; +import { sidecarEnter } from "../lib/sidecar-tracker"; +import { createTranslatorBudget } from "../lib/translator-budget"; +import type { SidecarOutcomeRecorder } from "../web-search/executor"; +import { getOAuthCredentialProjectId, getValidAccessToken, publicOAuthAuthenticationErrorMessage } from "../oauth"; +import { resolveActiveProviderApiKey } from "../providers/api-keys"; + +export interface ChatVisionSettings { + model: string; + timeoutMs: number; + detail?: string; + reasoning?: string; +} + +export type DescribeChatOutcome = { text: string; error?: string }; + +/** + * Fail before any credential acquisition or network activity when the destination + * is not HTTPS. The vision sidecar sends user images, so an `http:` baseUrl must + * never see a bearer token or an outbound request — checked BEFORE the OAuth token + * lookup, BEFORE Authorization construction, and BEFORE any fetch. + */ +function httpsGuardError(baseUrl: string, allowLoopbackHttp: boolean): string | null { + let parsed: URL; + try { + parsed = new URL(baseUrl); + } catch { + return "invalid provider baseUrl"; + } + if (parsed.protocol === "https:") return null; + // Cleartext is acceptable only when the bytes never leave the host: a local + // OpenAI-compatible server (e.g. http://127.0.0.1:1234/v1) is a legitimate + // chat-vision destination. Remote http: destinations stay rejected before + // any token fetch or network call. + const host = parsed.hostname.replace(/^\[|]$/g, "").toLowerCase(); + const loopback = host === "localhost" || host === "127.0.0.1" || host === "::1"; + if (allowLoopbackHttp && parsed.protocol === "http:" && loopback) return null; + return `provider baseUrl must use HTTPS (got ${parsed.protocol})`; +} + +const ALLOWED_IMAGE_MIME = new Set(["image/png", "image/jpeg", "image/jpg", "image/webp", "image/gif"]); +const MAX_IMAGE_BYTES = 20 * 1024 * 1024; + +function validateImageInput(imageUrl: string): { mime?: string; base64Data?: string; error?: string } { + if (imageUrl.startsWith("data:")) { + const match = /^data:([^;,]+?)(;base64)?,(.*)$/s.exec(imageUrl); + if (!match || !match[2]) return { error: "malformed data URL" }; + const mime = match[1].toLowerCase(); + if (!ALLOWED_IMAGE_MIME.has(mime)) return { error: `unsupported image type "${mime}"` }; + const bytes = Math.floor((match[3].length * 3) / 4); + if (bytes > MAX_IMAGE_BYTES) return { error: `image too large (~${Math.round(bytes / 1024 / 1024)}MB)` }; + return { mime, base64Data: match[3] }; + } + if (imageUrl.startsWith("https://")) { + return {}; + } + return { error: "unsupported image URL scheme (expected data: or https:)" }; +} + +function toChatImagePart(imageUrl: string, detail?: string): Record { + return { type: "image_url", image_url: { url: imageUrl, detail: detail ?? "high" } }; +} + +async function describeImageGoogle( + imageUrl: string, + detail: string | undefined, + contextText: string, + provider: OcxProviderConfig, + providerName: string, + settings: ChatVisionSettings, + abortSignal?: AbortSignal, + recordOutcome?: SidecarOutcomeRecorder, + executor: typeof globalThis.fetch = globalThis.fetch, +): Promise { + const imageValidation = validateImageInput(imageUrl); + if (imageValidation.error) return { text: "", error: imageValidation.error }; + const destinationError = httpsGuardError(provider.baseUrl, provider.authMode !== "oauth"); + if (destinationError) return { text: "", error: destinationError }; + let requestProvider = provider; + if (provider.authMode === "oauth") { + try { + const token = await getValidAccessToken(providerName); + const project = provider.googleMode === "cloud-code-assist" + ? getOAuthCredentialProjectId(providerName) + : provider.project; + requestProvider = { ...provider, apiKey: token, ...(project ? { project } : {}) }; + } catch (e) { + return { text: "", error: `google oauth token failed: ${publicOAuthAuthenticationErrorMessage(e)}` }; + } + } else { + const apiKey = resolveActiveProviderApiKey(provider); + if (!apiKey) return { text: "", error: "provider has no API key" }; + requestProvider = { ...provider, apiKey }; + } + + const parsed = { + modelId: settings.model, + context: { + messages: [{ + role: "user", + content: [ + { type: "text", text: contextText || "Describe this image." }, + { type: "image", imageUrl, detail }, + ], + timestamp: Date.now(), + }], + }, + stream: true, + options: { maxOutputTokens: 1024, reasoning: settings.reasoning ?? "low" }, + } as OcxParsedRequest; + const budget = createTranslatorBudget(); + const adapter = createGoogleAdapter(requestProvider); + const sidecarExit = sidecarEnter("vision"); + // Bound the AI Studio fallback below: created lazily, cleaned up in finally. + let fallbackSignal: ReturnType | undefined; + try { + const request = await adapter.buildRequest(parsed, { + headers: new Headers(), + translatorBudget: budget, + abortSignal, + }); + const response = adapter.fetchResponse + ? await adapter.fetchResponse(request, { + abortSignal, + timeoutMs: settings.timeoutMs, + stream: true, + returnRawErrors: true, + executor, + }) + : (fallbackSignal = signalWithTimeout(settings.timeoutMs, abortSignal), await executor(request.url, { + method: request.method, + headers: request.headers, + body: request.body, + // AI Studio leaves fetchResponse undefined, so this fallback must bound + // the request itself: without a timeout a stalled upstream holds a + // VISION_CONCURRENCY worker open indefinitely. signalWithTimeout + // merges settings.timeoutMs with the caller's abortSignal; cleanup() + // in finally releases the merged listener. + signal: fallbackSignal.signal, + })); + recordOutcome?.(response.status); + if (!response.ok) { + return { text: "", error: `google sidecar HTTP ${response.status}` }; + } + let text = ""; + for await (const event of adapter.parseStream(response, budget)) { + if (event.type === "text_delta") text += event.text; + if (event.type === "error") return { text: "", error: "google sidecar stream error" }; + } + const trimmed = text.trim(); + if (!trimmed) return { text: "", error: "google sidecar produced no description" }; + return { text: trimmed }; + } catch (e) { + const kind = e instanceof Error && e.name === "TimeoutError" ? "timeout" : "connect_error"; + recordOutcome?.(kind); + return { text: "", error: `google sidecar ${kind}` }; + } finally { + fallbackSignal?.cleanup(); + sidecarExit(); + } +} + +export async function describeImageChat( + imageUrl: string, + detail: string | undefined, + contextText: string, + provider: OcxProviderConfig, + providerName: string, + settings: ChatVisionSettings, + abortSignal?: AbortSignal, + recordOutcome?: SidecarOutcomeRecorder, + executor: typeof globalThis.fetch = globalThis.fetch, +): Promise { + const imageValidation = validateImageInput(imageUrl); + if (imageValidation.error) return { text: "", error: imageValidation.error }; + if (provider.adapter === "google") { + return describeImageGoogle( + imageUrl, + detail, + contextText, + provider, + providerName, + settings, + abortSignal, + recordOutcome, + executor, + ); + } + const destinationError = httpsGuardError(provider.baseUrl, provider.authMode !== "oauth"); + if (destinationError) return { text: "", error: destinationError }; + let authHeader = ""; + if (provider.authMode === "oauth") { + try { + const token = await getValidAccessToken(providerName); + authHeader = `Bearer ${token}`; + } catch (e) { + return { text: "", error: `oauth token failed: ${publicOAuthAuthenticationErrorMessage(e)}` }; + } + } else { + const apiKey = resolveActiveProviderApiKey(provider); + if (apiKey) { + authHeader = `Bearer ${apiKey}`; + } else if (provider.apiKey !== undefined || (provider.apiKeyPool?.length ?? 0) > 0) { + return { text: "", error: "provider has no API key or OAuth token" }; + } else { + // Preserve legacy static-header auth only when no configured key exists; + // unresolved env references must never fall through to raw placeholders. + const fallback = provider.headers?.Authorization?.match(/^Bearer\s+(.+)$/i)?.[1]?.trim(); + if (fallback) authHeader = `Bearer ${fallback}`; + else if (provider.authMode !== "local" && provider.keyOptional !== true) { + return { text: "", error: "provider has no API key or OAuth token" }; + } + } + // authMode "local" / keyOptional: keyless destination, no Authorization header. + } + + const content: unknown[] = [ + { type: "text", text: contextText || "Describe this image." }, + toChatImagePart(imageUrl, detail), + ]; + const body: Record = { + model: settings.model, + messages: [{ role: "user", content }], + stream: true, + max_tokens: 1024, + }; + // Forward the planned reasoning through the provider-aware wire mapping, + // mirroring the openai-chat adapter: OpenAI-compatible targets carry + // reasoning_effort, gateway-object targets carry reasoning.enabled/effort. + if (settings.reasoning && settings.reasoning !== "none") { + if (provider.reasoningWireFormat === "gateway-object") { + body.reasoning = { enabled: true, effort: settings.reasoning }; + } else { + body.reasoning_effort = settings.reasoning; + } + } + + const headers: Record = { + "Content-Type": "application/json", + ...(provider.headers ?? {}), + // The resolved credential must win over any static provider header: a + // configured Authorization would otherwise replace a freshly rotated OAuth + // token / API key and break every authenticated request with 401. + ...(authHeader ? { Authorization: authHeader } : {}), + }; + + const linkedSignal = signalWithTimeout(settings.timeoutMs ?? 30_000, abortSignal); + const sidecarExit = sidecarEnter("vision"); + const t0 = Date.now(); + try { + const baseUrl = provider.baseUrl.replace(/\/$/, ""); + const url = new URL(`${baseUrl}/chat/completions`); + const res = await executor(url.toString(), { + method: "POST", + headers, + body: JSON.stringify(body), + signal: linkedSignal.signal, + }); + recordOutcome?.(res.status); + if (!res.ok) { + console.warn(`[vision] chat sidecar HTTP ${res.status} (${Date.now() - t0}ms)`); + return { text: "", error: `chat sidecar HTTP ${res.status}` }; + } + if (!res.body) return { text: "", error: "chat sidecar returned no response body" }; + const detachBodyGuard = cancelBodyOnAbort(res.body, linkedSignal.signal); + let text = ""; + try { + const reader = res.body.getReader(); + const decoder = new TextDecoder(); + let buf = ""; + const consumeLine = (line: string) => { + const s = line.replace(/^data:\s*/, "").trim(); + if (!s || s === "[DONE]") return; + try { + const j = JSON.parse(s); + const delta = j.choices?.[0]?.delta?.content; + if (typeof delta === "string") text += delta; + else if (Array.isArray(delta)) { + for (const part of delta) if (typeof part?.text === "string") text += part.text; + } + const message = j.choices?.[0]?.message?.content; + if (typeof message === "string") text += message; + } catch { /* ignore non-JSON data lines */ } + }; + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + buf += decoder.decode(value, { stream: true }); + const lines = buf.split(/\r?\n/); + buf = lines.pop() ?? ""; + for (const line of lines) consumeLine(line); + } + consumeLine(buf); + const trimmed = text.trim(); + if (!trimmed) return { text: "", error: "chat sidecar produced no description" }; + return { text: trimmed }; + } finally { + detachBodyGuard(); + } + } catch (e) { + const kind = e instanceof Error && e.name === "TimeoutError" ? "timeout" : "connect_error"; + recordOutcome?.(kind); + console.warn(`[vision] chat sidecar error (${Date.now() - t0}ms)`); + return { text: "", error: `chat sidecar ${kind}` }; + } finally { + sidecarExit(); + linkedSignal.cleanup(); + } +} diff --git a/src/vision/eligibility.ts b/src/vision/eligibility.ts index 7737f67844..50c9ca0819 100644 --- a/src/vision/eligibility.ts +++ b/src/vision/eligibility.ts @@ -25,16 +25,23 @@ import { getModelMetadataCaseInsensitive, resolveMetadataProvider } from "../gen import { nativeInputModalities } from "../codex/catalog/metadata"; import { SUPPORTED_NATIVE_OPENAI_SLUGS } from "../codex/catalog/native-models"; import { enrichProviderFromRegistry } from "../providers/derive"; +import { resolveActiveProviderApiKey } from "../providers/api-keys"; +import { isOAuthProvider } from "../oauth"; +import { getAccountSet } from "../oauth/store"; -/** The two wire protocols `planVisionSidecar` can actually dispatch to. */ -export type VisionSidecarBackend = "openai" | "anthropic"; +/** The wire protocols `planVisionSidecar` can actually dispatch to. */ +export type VisionSidecarBackend = "openai" | "anthropic" | "chat"; /** * Default entry per backend: cheap, image-capable, and present in every deployment. Offered * whenever its side is enabled, and withheld only when that provider explicitly lists it as a * model the sidecar describes FOR — never merely because a metadata table stayed silent. + * + * The chat side has no universal baseline: it dispatches through a CONFIGURED chat/google + * provider, and no canonical provider exists to name. Chat rows therefore come only from + * catalog/configured models, never from a fabricated default. */ -export const BASELINE_VISION_MODELS: Record = { +export const BASELINE_VISION_MODELS: Record<"openai" | "anthropic", string> = { openai: "gpt-5.6-luna", anthropic: "claude-haiku-4-5", }; @@ -56,6 +63,33 @@ export interface VisionModelOption { type EnrichedProviderCache = Map; +/** + * Whether a configured provider can actually execute a chat vision sidecar. + * OAuth is usable only for a supported OAuth provider with a healthy active account; + * keyless mode is intentionally limited to openai-chat local-compatible servers. + */ +export function isChatVisionProviderUsable(providerName: string, provider: OcxProviderConfig): boolean { + if (provider.disabled === true) return false; + if (provider.adapter !== "openai-chat" && provider.adapter !== "google") return false; + if (provider.authMode === "oauth") { + if (!isOAuthProvider(providerName)) return false; + const accountSet = getAccountSet(providerName); + const active = accountSet?.accounts.find(account => account.id === accountSet.activeAccountId); + return active !== undefined && active.needsReauth !== true; + } + if (resolveActiveProviderApiKey(provider)) return true; + if (provider.adapter === "openai-chat" && (provider.authMode === "local" || provider.keyOptional === true)) return true; + return false; +} + +export function isChatVisionCandidateUsable( + config: Pick, + providerName: string, +): boolean { + const provider = config.providers?.[providerName]; + return provider !== undefined && isChatVisionProviderUsable(providerName, provider); +} + function advertisesImageInput(modalities: readonly string[] | undefined): boolean | undefined { if (!modalities || modalities.length === 0) return undefined; return modalities.includes("image"); @@ -163,12 +197,19 @@ export function visionBackendForCandidate( // executor to name, no catalog row qualifies; the side's baseline is added separately and // keeps the picker populated. Narrowing here never widens the write gate, which is a // different predicate (`modelAcceptsImageInput`) and still treats unknown as allowed. - if (anthropicProviderName === undefined) return undefined; - return candidate.provider === anthropicProviderName ? "anthropic" : undefined; + if (anthropicProviderName !== undefined && candidate.provider === anthropicProviderName) return "anthropic"; + // A chat-side row is reachable only through its configured chat/google adapter with + // usable auth — the same predicate the runtime chat sidecar resolver uses. A routed + // row whose provider is absent, disabled, or keyless must not be offered, because + // selecting it would fail at describe time rather than at pick time. + const configured = config.providers?.[candidate.provider]; + if (!configured) return undefined; + const chatLike = configured.adapter === "openai-chat" || configured.adapter === "google"; + return chatLike && isChatVisionProviderUsable(candidate.provider, configured) ? "chat" : undefined; } function baselineCandidate( - backend: VisionSidecarBackend, + backend: "openai" | "anthropic", anthropicProviderName: string | undefined, ): VisionCandidateModel { return { @@ -194,9 +235,9 @@ function baselineCandidate( * and is the only input to rejection. Absence from this list must never imply * rejection — an unknown id stays eligible via the undefined → eligible fallback. * - * De-duplication is by BARE model id, first eligible row wins. Two providers of - * the same adapter family can expose the same id; they resolve to the same - * backend, and only `value` reaches the client, so first-wins costs nothing. + * Chat rows are provider-qualified because the selected provider is part of the + * runtime resolution identity. This also makes same-id rows from different + * providers distinct instead of silently first-wins. */ export function visionEligibleModelOptions( config: Pick, @@ -219,8 +260,13 @@ export function visionEligibleModelOptions( const backend = visionBackendForCandidate(config, candidate, anthropicProviderName); if (!backend || !enabled.has(backend)) continue; if (!isVisionEligibleModelWithCache(config, candidate, enrichedProviders)) continue; - if (byValue.has(candidate.id)) continue; - byValue.set(candidate.id, { value: candidate.id, label: candidate.id, backend }); + const value = backend === "chat" ? `${candidate.provider}/${candidate.id}` : candidate.id; + if (byValue.has(value)) continue; + byValue.set(value, { + value, + label: backend === "chat" ? `${candidate.provider} / ${candidate.id}` : candidate.id, + backend, + }); } const order = (option: VisionModelOption) => diff --git a/src/vision/index.ts b/src/vision/index.ts index 792e5db1b1..60bd2789dd 100644 --- a/src/vision/index.ts +++ b/src/vision/index.ts @@ -5,9 +5,11 @@ import { modelRecordValue } from "../reasoning-effort"; import type { VisionReasoningEffort } from "../reasoning-effort"; import { describeImage, type DescribeOutcome, type VisionSettings } from "./describe"; import { describeImageAnthropic } from "./anthropic-describe"; +import { describeImageChat } from "./describe-chat"; import { normalizeVisionReasoningForModel } from "./reasoning"; import type { CodexAuthContext } from "../codex/auth-context"; import { resolveSidecarAuth } from "../sidecar/auth"; +import { isChatVisionProviderUsable } from "./eligibility"; import type { ResolvedOpenAiForwardSidecar } from "../providers/openai-sidecar"; import type { SidecarOutcomeRecorder } from "../web-search/executor"; import { enforceAppOwnedMemoryBudget } from "../lib/app-owned-memory"; @@ -38,6 +40,7 @@ export function isModelTextOnly( export { describeImageAnthropic, parseAnthropicVisionSSE } from "./anthropic-describe"; export { BASELINE_VISION_MODELS, + isChatVisionProviderUsable, isVisionEligibleModel, isVisionSidecarConsumer, modelAcceptsImageInput, @@ -225,11 +228,80 @@ export function findAnthropicVisionProvider(config: OcxConfig): AnthropicVisionP return { providerName: auth.anthropicProviderName, provider: auth.anthropicProvider }; } +/** Chat-adapter or google-adapter provider selected to run the chat vision sidecar. */ +export interface ChatVisionProvider { + providerName: string; + provider: OcxProviderConfig; + model: string; +} + +/** + * Resolve the provider that runs the chat vision sidecar for `model`. + * + * A provider-qualified `provider/model` selects exactly that provider when it is + * enabled, chat-like, and has usable auth. A bare model resolves ONLY through a + * unique configured match (defaultModel or the provider's model list): an + * ambiguous match (two providers listing the same model) and a provider that only + * has live-discovered models (no configured list) both produce NO provider — + * routing a user's image to an arbitrary first authenticated provider is exactly + * the mis-send the qualified form exists to prevent. + */ +export function findChatVisionProvider(config: OcxConfig, model: string): ChatVisionProvider | undefined { + let providerName = ""; + let bareModel = model; + let qualified = false; + if (model.includes("/")) { + const sep = model.indexOf("/"); + const prefix = model.slice(0, sep); + bareModel = model.slice(sep + 1); + // The prefix is a provider qualifier ONLY when such a provider actually + // exists. A namespaced catalog id like "anthropic/claude-3-haiku" where + // "anthropic" is a model namespace (not a configured provider) must fall + // through to bare resolution against the published namespaced ids. + if (config.providers[prefix]) { + providerName = prefix; + qualified = true; + } + } + const isChatLike = (p: OcxProviderConfig) => p.adapter === "openai-chat" || p.adapter === "google"; + if (qualified) { + const provider = config.providers[providerName]; + if (provider && isChatLike(provider) && isChatVisionProviderUsable(providerName, provider)) { + return { provider, providerName, model: bareModel }; + } + return undefined; + } + const matches: Array<{ provider: OcxProviderConfig; providerName: string; model: string }> = []; + for (const [name, provider] of Object.entries(config.providers)) { + if (!isChatLike(provider)) continue; + if (!isChatVisionProviderUsable(name, provider)) continue; + const models = provider.models ?? []; + const defaultModel = provider.defaultModel; + if (bareModel === defaultModel) { + matches.push({ provider, providerName: name, model: bareModel }); + continue; + } + if (models.includes(bareModel)) { + matches.push({ provider, providerName: name, model: bareModel }); + continue; + } + // A bare id that is the suffix of a namespaced published id ("gemini-flash" + // matching "google/gemini-flash") resolves to the PUBLISHED id, not the + // bare one — the bare id is not routable upstream. + const published = models.find(m => m.endsWith("/" + bareModel)); + if (published) { + matches.push({ provider, providerName: name, model: published }); + } + } + if (matches.length === 1) return { ...matches[0]! }; + return undefined; +} + export function resolveVisionBackend( - explicit: "openai" | "anthropic" | undefined, + explicit: "openai" | "anthropic" | "chat" | undefined, anthropicSidecar: AnthropicVisionProvider | undefined, -): "openai" | "anthropic" { - if (explicit === "openai" || explicit === "anthropic") return explicit; +): "openai" | "anthropic" | "chat" { + if (explicit === "openai" || explicit === "anthropic" || explicit === "chat") return explicit; return anthropicSidecar ? "anthropic" : "openai"; } @@ -241,8 +313,11 @@ export function resolveOpenAiVisionModel(config: Pick, - backend: "openai" | "anthropic", + backend: "openai" | "anthropic" | "chat", ): string { + // The chat sidecar's model is provider-qualified and resolved separately via + // findChatVisionProvider; the generic default only applies to the forward sides. + if (backend === "chat") return resolveOpenAiVisionModel(config); return backend === "anthropic" ? config.visionSidecar?.model || DEFAULT_ANTHROPIC_VISION_MODEL : resolveOpenAiVisionModel(config); @@ -261,8 +336,8 @@ function messagesHaveImage(parsed: OcxParsedRequest): boolean { export function shouldResolveOpenAiVisionSidecar( config: OcxConfig, provider: OcxProviderConfig, - modelId: string, - parsed: OcxParsedRequest, + modelId: string, + parsed: OcxParsedRequest, ): boolean { if (!isModelTextOnly(provider, modelId) || !messagesHaveImage(parsed)) return false; const cfg = config.visionSidecar ?? {}; @@ -271,17 +346,18 @@ export function shouldResolveOpenAiVisionSidecar( } export interface VisionPlan { - backend: "openai" | "anthropic"; + backend: "openai" | "anthropic" | "chat"; forwardSidecar?: ResolvedOpenAiForwardSidecar; anthropicSidecar?: AnthropicVisionProvider; + chatSidecar?: ChatVisionProvider; settings: VisionSettings; maxDescriptionsPerTurn: number; } /** * Decide whether the vision sidecar should pre-describe images for this request, returning the plan - * if so. Active when: the routed model is in `provider.noVisionModels`, the request actually carries - * an image, the sidecar isn't disabled, and the selected backend has usable auth. Returns undefined + * if so. Active when: the routed model is explicitly text-only, the request actually carries an + * image, the sidecar isn't disabled, and the selected backend has usable auth. Returns undefined * otherwise (the caller strips images before sending to a text-only model). */ export function planVisionSidecar( @@ -300,6 +376,21 @@ export function planVisionSidecar( const model = resolveEffectiveVisionModel(config, backend); const maxDescriptionsPerTurn = resolveMaxDescriptionsPerTurn(cfg.maxDescriptionsPerTurn); + if (backend === "chat") { + const chatProvider = findChatVisionProvider(config, cfg.model ?? ""); + if (!chatProvider) return undefined; + return { + backend, + chatSidecar: chatProvider, + settings: { + model: chatProvider.model, + reasoning: normalizeVisionReasoningForModel(chatProvider.model, cfg.reasoning) ?? DEFAULT_REASONING, + timeoutMs: resolveVisionTimeoutMs(cfg.timeoutMs), + }, + maxDescriptionsPerTurn, + }; + } + if (backend === "anthropic") { if (!anthropicSidecar) return undefined; return { @@ -417,6 +508,35 @@ function normalizedContext(contextText: string): string { return contextText.trim().replace(/\s+/g, " "); } +function normalizedProviderBaseUrl(baseUrl: string): string { + try { + const url = new URL(baseUrl); + url.hash = ""; + url.search = ""; + url.pathname = url.pathname.replace(/\/+$/, "") || "/"; + return url.toString(); + } catch { + return baseUrl.trim().replace(/\/+$/, ""); + } +} + +function selectedProviderIdentity(plan: VisionPlan): Record { + const selected = plan.backend === "openai" + ? plan.forwardSidecar + : plan.backend === "anthropic" + ? plan.anthropicSidecar + : plan.chatSidecar; + const provider = selected?.provider; + return { + providerName: selected?.providerName ?? "", + adapter: provider?.adapter ?? "", + baseUrl: provider ? normalizedProviderBaseUrl(provider.baseUrl) : "", + ...(provider?.googleMode ? { googleMode: provider.googleMode } : {}), + ...(provider?.project ? { project: provider.project } : {}), + ...(provider?.location ? { location: provider.location } : {}), + }; +} + function descriptionIdentity(job: ImageJob, plan: VisionPlan): { key: string; persistent: boolean } { let imageHash: string; let persistent = false; @@ -430,6 +550,7 @@ function descriptionIdentity(job: ImageJob, plan: VisionPlan): { key: string; pe return { key: JSON.stringify([ plan.backend, + selectedProviderIdentity(plan), plan.settings.model, ...(plan.backend === "openai" ? [plan.settings.reasoning] : []), job.detail ?? "high", @@ -446,6 +567,7 @@ async function executeDescription( selectedForwardHeaders: Headers, abortSignal?: AbortSignal, recordSidecarOutcome?: SidecarOutcomeRecorder, + providerFetch?: typeof globalThis.fetch, ): Promise { if (plan.backend === "anthropic") { const sidecar = plan.anthropicSidecar; @@ -460,6 +582,26 @@ async function executeDescription( abortSignal, ); } + if (plan.backend === "chat") { + const sidecar = plan.chatSidecar; + if (!sidecar) return { text: "", error: "chat vision sidecar is unavailable" }; + return describeImageChat( + job.imageUrl, + job.detail, + job.contextText, + sidecar.provider, + sidecar.providerName, + { + model: sidecar.model, + timeoutMs: plan.settings.timeoutMs, + detail: job.detail, + reasoning: plan.settings.reasoning, + }, + abortSignal, + recordSidecarOutcome, + providerFetch, + ); + } if (!plan.forwardSidecar) return { text: "", error: "OpenAI vision sidecar is unavailable" }; return describeImage( job.imageUrl, @@ -486,6 +628,7 @@ export async function describeImagesInPlace( abortSignal?: AbortSignal, recordSidecarOutcome?: SidecarOutcomeRecorder, translatorBudget?: TranslatorBudget, + providerFetch?: typeof globalThis.fetch, ): Promise { const jobs: ImageJob[] = []; const targets: { msg: OcxMessage; parts: OcxContentPart[] }[] = []; @@ -542,7 +685,7 @@ export async function describeImagesInPlace( executions.push(async () => { let outcome: DescribeOutcome; try { - outcome = await executeDescription(job, plan, selectedForwardHeaders, abortSignal, recordSidecarOutcome); + outcome = await executeDescription(job, plan, selectedForwardHeaders, abortSignal, recordSidecarOutcome, providerFetch); } catch (error) { outcome = { text: "", error: error instanceof Error ? error.message : String(error) }; } diff --git a/tests/vision-cache.test.ts b/tests/vision-cache.test.ts index b2369bc00f..6831c08678 100644 --- a/tests/vision-cache.test.ts +++ b/tests/vision-cache.test.ts @@ -73,6 +73,12 @@ test("vision sidecar auth stays lazy for no-image and disabled branches", () => withImage, )).toBe(false); expect(shouldResolveOpenAiVisionSidecar(cfg, textOnlyProvider, "text-model", withImage)).toBe(true); + expect(shouldResolveOpenAiVisionSidecar( + cfg, + { ...textOnlyProvider, noVisionModels: undefined, modelInputModalities: { "text-model": ["text"] } }, + "text-model", + withImage, + )).toBe(true); }); function parsed(parts: Array>) { @@ -255,6 +261,35 @@ describe("vision description cache and per-turn cap", () => { expect(textParts(request, 1).join("\n")).toContain("caption-a"); }); + test("separates cache keys by selected provider identity", async () => { + let calls = 0; + globalThis.fetch = (async (_url, init) => { + calls += 1; + const body = JSON.parse(String(init?.body)); + return new Response(`data: ${JSON.stringify({ choices: [{ delta: { content: `provider-${body.model}-${calls}` } }] })}\n\ndata: [DONE]\n\n`, { + headers: { "content-type": "text/event-stream" }, + }); + }) as typeof fetch; + const headers = new Headers({ authorization: "Bearer test" }); + const sameImage = () => parsed([{ type: "input_text", text: "same context" }, { type: "input_image", image_url: DATA_A }]); + const providerA: OcxProviderConfig = { adapter: "openai-chat", baseUrl: "https://provider-a.test/v1", apiKey: "key-a" }; + const providerB: OcxProviderConfig = { adapter: "openai-chat", baseUrl: "https://provider-b.test/v1", apiKey: "key-b" }; + const chatPlan = (provider: OcxProviderConfig, providerName: string): VisionPlan => ({ + backend: "chat", + chatSidecar: { providerName, provider, model: "same-model" }, + settings: { model: "same-model", timeoutMs: 5000 }, + maxDescriptionsPerTurn: 8, + }); + + const first = sameImage(); + await describeImagesInPlace(first, chatPlan(providerA, "provider-a"), headers); + const second = sameImage(); + await describeImagesInPlace(second, chatPlan(providerB, "provider-b"), headers); + + expect(calls).toBe(2); + expect(textParts(second).join("\n")).toContain("provider-same-model-2"); + }); + test("separates cache keys by backend, model, detail, and normalized context", async () => { let calls = 0; globalThis.fetch = (async (url, init) => { diff --git a/tests/vision-chat.test.ts b/tests/vision-chat.test.ts new file mode 100644 index 0000000000..9f122d0b09 --- /dev/null +++ b/tests/vision-chat.test.ts @@ -0,0 +1,441 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { describeImageChat } from "../src/vision/describe-chat"; +import { planVisionSidecar } from "../src/vision"; +import { parseRequest } from "../src/responses/parser"; +import type { OcxConfig, OcxProviderConfig } from "../src/types"; +import { resolveActiveProviderApiKey } from "../src/providers/api-keys"; + +const originalFetch = globalThis.fetch; +const image = "data:image/png;base64,aGVsbG8="; +const settings = { model: "vision-test", timeoutMs: 5000 }; + +afterEach(() => { globalThis.fetch = originalFetch; }); + +function chatSse(text: string): Response { + const body = [ + `data: ${JSON.stringify({ choices: [{ delta: { content: text } }] })}`, + "", + "data: [DONE]", + "", + ].join("\n"); + return new Response(body, { headers: { "content-type": "text/event-stream" } }); +} + +function geminiSse(text: string): Response { + const body = [ + `data: ${JSON.stringify({ candidates: [{ content: { parts: [{ text }] } }] })}`, + "", + `data: ${JSON.stringify({ candidates: [{ finishReason: "STOP" }] })}`, + "", + ].join("\n"); + return new Response(body, { headers: { "content-type": "text/event-stream" } }); +} + +describe("chat vision sidecar", () => { + test("resolves env-backed active keys and matches the active non-first pool entry without mutation", () => { + const previous = process.env.VISION_CHAT_KEY; + process.env.VISION_CHAT_KEY = " env-active-key "; + try { + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "https://vision.example/v1", + apiKey: "${VISION_CHAT_KEY}", + apiKeyPool: [ + { id: "first", key: "first-key" }, + { id: "active", key: "${VISION_CHAT_KEY}" }, + ], + }; + const before = structuredClone(provider); + expect(resolveActiveProviderApiKey(provider)).toBe("env-active-key"); + expect(provider).toEqual(before); + } finally { + if (previous === undefined) delete process.env.VISION_CHAT_KEY; + else process.env.VISION_CHAT_KEY = previous; + } + }); + + test("does not advertise or send an unresolved env placeholder", async () => { + const previous = process.env.VISION_CHAT_MISSING; + delete process.env.VISION_CHAT_MISSING; + try { + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "https://vision.example/v1", + apiKey: "$VISION_CHAT_MISSING", + }; + expect(resolveActiveProviderApiKey(provider)).toBeUndefined(); + expect(await describeImageChat(image, "high", "describe this", provider, "mimo", settings)).toEqual({ + text: "", + error: "provider has no API key or OAuth token", + }); + } finally { + if (previous === undefined) delete process.env.VISION_CHAT_MISSING; + else process.env.VISION_CHAT_MISSING = previous; + } + }); + + test("sends an image_url through an OpenAI-compatible provider", async () => { + let capturedUrl = ""; + let capturedBody: Record | undefined; + globalThis.fetch = (async (url, init) => { + capturedUrl = String(url); + capturedBody = JSON.parse(String(init?.body)); + return chatSse("Mimo description"); + }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "https://vision.example/v1", + authMode: "key", + apiKey: "test-key", + }; + + const result = await describeImageChat(image, "high", "describe this", provider, "mimo", settings); + + expect(result).toEqual({ text: "Mimo description" }); + expect(capturedUrl).toBe("https://vision.example/v1/chat/completions"); + expect(capturedBody?.model).toBe("vision-test"); + expect(capturedBody?.messages[0].content).toEqual([ + { type: "text", text: "describe this" }, + { type: "image_url", image_url: { url: image, detail: "high" } }, + ]); + }); + + test("uses the active non-first pool key for Google adapter auth", async () => { + let capturedKey = ""; + globalThis.fetch = (async (_url, init) => { + capturedKey = new Headers(init?.headers).get("x-goog-api-key") ?? ""; + return geminiSse("Gemini pool description"); + }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "google", + baseUrl: "https://generativelanguage.googleapis.com", + authMode: "key", + apiKey: "active-google-key", + apiKeyPool: [ + { id: "first", key: "first-google-key" }, + { id: "active", key: "active-google-key" }, + ], + }; + + const result = await describeImageChat(image, "high", "describe this", provider, "gemini", { model: "gemini-test", timeoutMs: 5000 }); + + expect(result).toEqual({ text: "Gemini pool description" }); + expect(capturedKey).toBe("active-google-key"); + }); + + test("uses the native Google adapter wire format", async () => { + let capturedUrl = ""; + let capturedBody: Record | undefined; + globalThis.fetch = (async (url, init) => { + capturedUrl = String(url); + capturedBody = JSON.parse(String(init?.body)); + return geminiSse("Gemini description"); + }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "google", + baseUrl: "https://generativelanguage.googleapis.com", + authMode: "key", + apiKey: "test-key", + }; + + const result = await describeImageChat(image, "high", "describe this", provider, "gemini", { model: "gemini-test", timeoutMs: 5000 }); + + expect(result).toEqual({ text: "Gemini description" }); + expect(capturedUrl).toBe("https://generativelanguage.googleapis.com/v1beta/models/gemini-test:streamGenerateContent?alt=sse"); + expect(capturedBody?.contents?.[0]?.parts).toEqual([ + { text: "describe this" }, + { inline_data: { mime_type: "image/png", data: "aGVsbG8=" } }, + ]); + }); + + test("the resolved credential wins over a static provider Authorization header", async () => { + let capturedAuth = ""; + globalThis.fetch = (async (_url, init) => { + capturedAuth = String((init?.headers as Record)?.Authorization ?? (init?.headers as Headers)?.get?.("Authorization") ?? ""); + return chatSse("authed description"); + }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "https://vision.example/v1", + authMode: "key", + apiKey: "fresh-key", + headers: { Authorization: "Bearer stale-static" }, + }; + + const result = await describeImageChat(image, "high", "describe this", provider, "mimo", settings); + + expect(result.error).toBeUndefined(); + expect(capturedAuth).toBe("Bearer fresh-key"); + }); + + test("a keyless local provider (authMode local) sends no Authorization header", async () => { + let capturedAuth: string | undefined = "unset"; + globalThis.fetch = (async (_url, init) => { + const h = init?.headers as Record; + capturedAuth = h?.Authorization ?? (init?.headers as Headers)?.get?.("Authorization") ?? undefined; + return chatSse("local description"); + }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1234/v1", + authMode: "local", + }; + + const result = await describeImageChat(image, "high", "describe this", provider, "local", settings); + + expect(result.error).toBeUndefined(); + expect(capturedAuth).toBeUndefined(); + }); + + test("forwards the planned reasoning through the provider-aware wire mapping", async () => { + let capturedBody: Record | undefined; + globalThis.fetch = (async (_url, init) => { + capturedBody = JSON.parse(String(init?.body)); + return chatSse("reasoned description"); + }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "https://vision.example/v1", + authMode: "key", + apiKey: "k", + }; + + await describeImageChat(image, "high", "describe this", provider, "mimo", { ...settings, reasoning: "high" }); + + expect(capturedBody?.reasoning_effort).toBe("high"); + expect(capturedBody?.reasoning).toBeUndefined(); + + // gateway-object providers emit reasoning.enabled/effort instead. + const gateway: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "https://gateway.example/v1", + authMode: "key", + apiKey: "k", + reasoningWireFormat: "gateway-object", + }; + globalThis.fetch = (async (_url, init) => { + capturedBody = JSON.parse(String(init?.body)); + return chatSse("gateway description"); + }) as typeof fetch; + await describeImageChat(image, "high", "describe this", gateway, "gateway", { ...settings, reasoning: "medium" }); + + expect(capturedBody?.reasoning).toEqual({ enabled: true, effort: "medium" }); + expect(capturedBody?.reasoning_effort).toBeUndefined(); + }); +}); + +describe("chat vision destination guard", () => { + afterEach(() => { globalThis.fetch = originalFetch; }); + + test("an http: openai-compatible destination fails without any network call", async () => { + let fetchCalled = false; + globalThis.fetch = (async () => { fetchCalled = true; return chatSse("unreachable"); }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "http://vision.example/v1", + authMode: "key", + apiKey: "test-key", + }; + + const result = await describeImageChat(image, "high", "describe this", provider, "mimo", settings); + + expect(result.error).toContain("must use HTTPS"); + expect(result.text).toBe(""); + expect(fetchCalled).toBe(false); + }); + + test("an http: OAuth destination fails BEFORE token acquisition and before any network call", async () => { + let fetchCalled = false; + globalThis.fetch = (async () => { fetchCalled = true; return chatSse("unreachable"); }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "http://vision.example/v1", + authMode: "oauth", + }; + + const result = await describeImageChat(image, "high", "describe this", provider, "mimo", settings); + + // The HTTPS error, not an "oauth token failed" error: the guard must run before + // getValidAccessToken is reached. + expect(result.error).toContain("must use HTTPS"); + expect(result.error).not.toContain("oauth"); + expect(fetchCalled).toBe(false); + }); + + test("an http: Google destination fails before token acquisition and before any network call", async () => { + let fetchCalled = false; + globalThis.fetch = (async () => { fetchCalled = true; return geminiSse("unreachable"); }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "google", + baseUrl: "http://generativelanguage.googleapis.com", + authMode: "oauth", + }; + + const result = await describeImageChat(image, "high", "describe this", provider, "gemini", settings); + + expect(result.error).toContain("must use HTTPS"); + expect(result.error).not.toContain("oauth"); + expect(fetchCalled).toBe(false); + }); + + test("a loopback http: destination is allowed (cleartext never leaves the host)", async () => { + let fetchCalled = false; + globalThis.fetch = (async () => { fetchCalled = true; return chatSse("local-description"); }) as typeof fetch; + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1234/v1", + authMode: "key", + apiKey: "local-key", + }; + + const result = await describeImageChat(image, "high", "describe this", provider, "local", settings); + + expect(result.error).toBeUndefined(); + expect(fetchCalled).toBe(true); + }); +}); + +describe("chat vision plan provider resolution", () => { + const routed: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "https://routed.test/v1", + apiKey: "routed-key", + noVisionModels: ["text-model"], + }; + const request = parseRequest({ + model: "routed/text-model", + input: [{ + type: "message", + role: "user", + content: [ + { type: "input_text", text: "what is in this picture?" }, + { type: "input_image", image_url: "data:image/png;base64,aGVsbG8=" }, + ], + }], + }); + const chatConfig = (providers: Record, model: string): OcxConfig => + ({ port: 10100, defaultProvider: "routed", providers, visionSidecar: { enabled: true, backend: "chat", model } }) as OcxConfig; + + test("a bare model with two live-only providers produces NO plan (no arbitrary first-provider pick)", () => { + const config = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", apiKey: "k1", liveModels: true }, + p2: { adapter: "openai-chat", baseUrl: "https://p2.test/v1", apiKey: "k2", liveModels: true }, + }, "gemini-flash"); + + expect(planVisionSidecar(config, routed, "text-model", request)).toBeUndefined(); + }); + + test("a bare model listed by TWO configured providers is ambiguous and produces NO plan", () => { + const config = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", apiKey: "k1", models: ["gemini-flash"] }, + p2: { adapter: "openai-chat", baseUrl: "https://p2.test/v1", apiKey: "k2", models: ["gemini-flash"] }, + }, "gemini-flash"); + + expect(planVisionSidecar(config, routed, "text-model", request)).toBeUndefined(); + }); + + test("a provider-qualified model selects exactly that provider", () => { + const config = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", apiKey: "k1", liveModels: true }, + p2: { adapter: "openai-chat", baseUrl: "https://p2.test/v1", apiKey: "k2", liveModels: true }, + }, "p2/gemini-flash"); + + const plan = planVisionSidecar(config, routed, "text-model", request); + expect(plan?.backend).toBe("chat"); + expect(plan?.chatSidecar?.providerName).toBe("p2"); + expect(plan?.chatSidecar?.model).toBe("gemini-flash"); + }); + + test("a uniquely configured bare model resolves to its provider", () => { + const config = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", apiKey: "k1", defaultModel: "gemini-flash" }, + }, "gemini-flash"); + + const plan = planVisionSidecar(config, routed, "text-model", request); + expect(plan?.backend).toBe("chat"); + expect(plan?.chatSidecar?.providerName).toBe("p1"); + }); + + test("modelInputModalities text-only evidence triggers chat planning", () => { + const config = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", apiKey: "k1", defaultModel: "gemini-flash" }, + }, "gemini-flash"); + const modalitiesOnly = { + ...routed, + noVisionModels: undefined, + modelInputModalities: { "text-model": ["text"] }, + }; + + expect(planVisionSidecar(config, modalitiesOnly, "text-model", request)?.backend).toBe("chat"); + }); + + test("disabled or unauthenticated chat providers produce no plan", () => { + const disabled = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", apiKey: "k1", disabled: true, models: ["gemini-flash"] }, + }, "gemini-flash"); + expect(planVisionSidecar(disabled, routed, "text-model", request)).toBeUndefined(); + + const unauthenticated = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", models: ["gemini-flash"] }, + }, "gemini-flash"); + expect(planVisionSidecar(unauthenticated, routed, "text-model", request)).toBeUndefined(); + }); + + test("a namespaced model id resolves through its unique configured provider", () => { + // "anthropic" here is a MODEL NAMESPACE inside the published id, not a + // configured provider: the prefix must not be read as a provider qualifier. + const config = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", apiKey: "k1", models: ["anthropic/claude-3-haiku"] }, + }, "anthropic/claude-3-haiku"); + + const plan = planVisionSidecar(config, routed, "text-model", request); + expect(plan?.backend).toBe("chat"); + expect(plan?.chatSidecar?.providerName).toBe("p1"); + expect(plan?.chatSidecar?.model).toBe("anthropic/claude-3-haiku"); + }); + + test("a bare model matched by namespace suffix keeps the id the provider publishes", () => { + // Bare "gemini-flash" matches the suffix of "google/gemini-flash": the plan + // must carry the PUBLISHED namespaced id, not the bare (unroutable) one. + const config = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", apiKey: "k1", models: ["google/gemini-flash"] }, + }, "gemini-flash"); + + const plan = planVisionSidecar(config, routed, "text-model", request); + expect(plan?.backend).toBe("chat"); + expect(plan?.chatSidecar?.providerName).toBe("p1"); + expect(plan?.chatSidecar?.model).toBe("google/gemini-flash"); + }); + + test("an ambiguous namespaced-suffix match produces no plan", () => { + const config = chatConfig({ + routed, + p1: { adapter: "openai-chat", baseUrl: "https://p1.test/v1", apiKey: "k1", models: ["google/gemini-flash"] }, + p2: { adapter: "openai-chat", baseUrl: "https://p2.test/v1", apiKey: "k2", models: ["vertex/gemini-flash"] }, + }, "gemini-flash"); + + expect(planVisionSidecar(config, routed, "text-model", request)).toBeUndefined(); + }); + + test("rejects malformed data URL or unsupported image scheme", async () => { + const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "https://vision.example/v1", + apiKey: "k1", + }; + const badScheme = await describeImageChat("http://insecure.example/img.png", "high", "desc", provider, "p", settings); + expect(badScheme.error).toContain("unsupported image URL scheme"); + + const badData = await describeImageChat("data:image/unsupported;base64,123", "high", "desc", provider, "p", settings); + expect(badData.error).toContain("unsupported image type"); + }); +}); diff --git a/tests/vision-eligibility.test.ts b/tests/vision-eligibility.test.ts index 5e77cda24a..4427ae56d9 100644 --- a/tests/vision-eligibility.test.ts +++ b/tests/vision-eligibility.test.ts @@ -4,6 +4,7 @@ import type { OcxConfig } from "../src/types"; import { BASELINE_VISION_MODELS, isVisionEligibleModel, + isChatVisionProviderUsable, modelAcceptsImageInput, visionBackendForCandidate, visionEligibleModelOptions, @@ -245,6 +246,53 @@ describe("vision eligibility core", () => { expect(matches[0]?.baseline).toBe(true); }); + test("chat usability rejects missing env-backed keys", () => { + const previous = process.env.VISION_ELIGIBILITY_MISSING; + delete process.env.VISION_ELIGIBILITY_MISSING; + try { + expect(isChatVisionProviderUsable("missing-env", { + adapter: "openai-chat", baseUrl: "https://example.test", apiKey: "${VISION_ELIGIBILITY_MISSING}", + })).toBe(false); + expect(isChatVisionProviderUsable("missing-google-env", { + adapter: "google", baseUrl: "https://example.test", apiKey: "$VISION_ELIGIBILITY_MISSING", + })).toBe(false); + } finally { + if (previous === undefined) delete process.env.VISION_ELIGIBILITY_MISSING; + else process.env.VISION_ELIGIBILITY_MISSING = previous; + } + }); + + test("chat usability only allows supported OAuth accounts, API keys, and openai-chat keyless local", () => { + expect(isChatVisionProviderUsable("xai", { + adapter: "openai-chat", baseUrl: "https://xai.example", authMode: "oauth", + })).toBe(false); + expect(isChatVisionProviderUsable("not-oauth", { + adapter: "openai-chat", baseUrl: "https://example.test", authMode: "oauth", apiKey: "stale", + })).toBe(false); + expect(isChatVisionProviderUsable("google", { + adapter: "google", baseUrl: "https://google.example", authMode: "local", keyOptional: true, + })).toBe(false); + expect(isChatVisionProviderUsable("local", { + adapter: "openai-chat", baseUrl: "http://127.0.0.1:1234/v1", authMode: "local", + })).toBe(true); + expect(isChatVisionProviderUsable("local", { + adapter: "openai-chat", baseUrl: "https://example.test", apiKeyPool: [{ key: "pool-key" }], + })).toBe(true); + }); + + test("chat options preserve provider identity and qualify every chat candidate", () => { + const config = configWithProviders({ + first: { adapter: "openai-chat", baseUrl: "https://first.test/v1", apiKey: "first-key" }, + second: { adapter: "openai-chat", baseUrl: "https://second.test/v1", apiKey: "second-key" }, + }); + const options = visionEligibleModelOptions(config, [ + { provider: "first", id: "same-model", inputModalities: ["text", "image"] }, + { provider: "second", id: "same-model", inputModalities: ["text", "image"] }, + ], ["chat"]); + expect(options.map(option => option.value)).toEqual(["first/same-model", "second/same-model"]); + expect(options.map(option => option.label)).toEqual(["first / same-model", "second / same-model"]); + }); + test("8. backend routing excludes image-capable rows with no executor", () => { // cursor has no vision sidecar executor — backend is undefined and the row is absent // from the options list even when it is image-capable. diff --git a/tests/vision-sidecar-e2e.test.ts b/tests/vision-sidecar-e2e.test.ts index 9de4bef38b..ab19202a63 100644 --- a/tests/vision-sidecar-e2e.test.ts +++ b/tests/vision-sidecar-e2e.test.ts @@ -66,6 +66,23 @@ function serveSidecar(onRequest: (req: Request, bodyText: string) => void) { }); } +/** Fake chat-completions vision sidecar: records the planned provider request. */ +function serveChatVisionSidecar(record: (bodyText: string) => void) { + return Bun.serve({ + hostname: "127.0.0.1", port: 0, + async fetch(req) { + record(await req.text()); + const sse = [ + `data: ${JSON.stringify({ choices: [{ delta: { content: CAPTION } }] })}`, + "", + "data: [DONE]", + "", + ].join("\n"); + return new Response(sse, { headers: { "content-type": "text/event-stream" } }); + }, + }); +} + /** Fake text-only upstream (openai-chat wire): records the forwarded body. */ function serveUpstream(record: (bodyText: string) => void) { return Bun.serve({ @@ -149,6 +166,49 @@ describe("vision sidecar fallback (issue #88, end-to-end)", () => { expect(JSON.stringify(parsed._rawBody)).toContain("[image omitted:"); }); + test("chat vision sidecar uses the chat provider executor identity, not the routed provider", async () => { + let routedBody = ""; + let chatBody = ""; + upstream = serveUpstream(b => { routedBody = b; }); + sidecar = serveChatVisionSidecar(b => { chatBody = b; }); + const config: OcxConfig = { + port: 0, hostname: "127.0.0.1", defaultProvider: "routed", openaiProviderTierVersion: 2, + providers: { + routed: { + adapter: "openai-chat", + baseUrl: `http://127.0.0.1:${upstream.port}/v1`, + allowPrivateNetwork: true, + apiKey: "routed-key", + noVisionModels: ["blind-model"], + }, + chatvision: { + adapter: "openai-chat", + baseUrl: `http://127.0.0.1:${sidecar.port}/v1`, + allowPrivateNetwork: true, + apiKey: "chat-key", + models: ["vision-model"], + }, + }, + visionSidecar: { enabled: true, backend: "chat", model: "chatvision/vision-model" }, + } as OcxConfig; + saveConfig(config); + const server = startServer(0); + try { + const res = await fetch(new URL("/v1/responses", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(baseRequest("routed/blind-model")), + }); + expect(res.status).toBe(200); + expect(JSON.parse(chatBody).model).toBe("vision-model"); + expect(chatBody).toContain("aGVsbG8taW1hZ2UtYnl0ZXM="); + expect(routedBody).toContain(CAPTION); + expect(routedBody).not.toContain("aGVsbG8taW1hZ2UtYnl0ZXM="); + } finally { + await server.stop(true); + } + }); + test("noVisionModels request fires the sidecar and forwards the caption instead of the image", async () => { let upstreamBody = ""; let sidecarBody = "";