diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/000_master_plan.md b/devlog/_plan/260813_routed_tool_discovery_profiles/000_master_plan.md new file mode 100644 index 000000000..5f27a142a --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/000_master_plan.md @@ -0,0 +1,181 @@ +# 000 - Master plan: routed tool discovery profiles + +Status: PLAN / PATCH-DRAFT / PROTOTYPE-VERIFIED / LANDED-AND-RE-VERIFIED +Created: 2026-08-13 +Target repository: `lidge-jun/opencodex` +Target branch: `dev` +Packaging-time `dev` head: `2cdbf66a23f9fd8f2f38dcc702ccd3f2e60ac535` +Tool-discovery semantic base: `5703473041a9f4f415743652de5d86d51fd66db5` (PR #1596 parent of the packaging-time head) +Base change: PR #1596, `fix(codex): restore deferred tool discovery for non-Cursor routed rows` + +> **Read `094_landing_verification_pass.md` before implementing anything.** This +> bundle was authored without a mounted checkout; every claim below was +> re-verified on 2026-08-13 against a real worktree, the upstream `codex-rs` +> source and live GitHub state. Eight corrections were recorded, the most +> consequential being that an **eligible** MCP tool stays callable in **both** +> discovery modes under code mode — so for those tools `direct` is a +> comprehension/compatibility lever with a payload cost, not a reachability fix. +> "Eligible" excludes `direct_only_tool_namespaces`, `excluded_tool_namespaces`, +> and anything removed by MCP/App policy filtering; see `094` for the exclusion +> table and the differential test that must prove the claim. + +## 1. Objective + +The immediate objective is to preserve the good part of PR #1596—small turn-1 payloads and Code Mode access through `exec`/`tools`/`ALL_TOOLS`—while adding a narrow, evidence-driven escape hatch for exact client/provider/model combinations where deferred discovery is proven unusable. + +This plan does **not** revert non-Cursor routed rows to blanket `supports_search_tool: false`. That would recreate the measured full-catalog payload tax. It also does **not** claim that one catalog boolean can represent every tool lifecycle. The work is split into four layers: + +1. **Catalog policy**: resolve whether a routed row advertises deferred discovery or direct discovery. +2. **Protocol conformance**: prove that `additional_tools`, custom tools, namespaces and tool-search history survive each adapter. +3. **Fallback architecture**: add bounded meta-tools for routes that cannot preserve native discovery. +4. **Live validation**: test exact Codex App + Browser plugin + external-model combinations. + +## 2. Verified current state + +At the verified `dev` head: + +- `normalizeRoutedCatalogEntry()` lives in `src/codex/catalog/parsing.ts`. +- Every routed row receives `tool_mode = "code_mode_only"` through `applyRoutedCodexToolMode()`. +- Cursor rows receive `supports_search_tool = false` and no hosted web-search metadata. +- Every other routed row receives `supports_search_tool = true` and `web_search_tool_type = "text_and_image"`. +- The template-less fallback in `src/codex/catalog/sync.ts` reproduces the same Cursor/non-Cursor split. +- `tests/catalog-cursor-search.test.ts` pins the template and template-less paths. +- PR #1596 reports a measured request-size change of 96,699 → 258,929 characters when deferred discovery is disabled under Code Mode. +- PR #1596 also records a live canary where routed `kimi/k3` called `tools.mcp__node_repl__js` successfully. +- The exact #1522 pairing—Codex App + DeepSeek-compatible routed model + Browser plugin—remains the material evidence gap. + +## 3. Recommended PR stack + +### PR A — profile resolver and explicit escape hatch + +Default behavior remains byte-for-byte equivalent to #1596: + +- non-Cursor: deferred +- Cursor: direct + +New provider-level and model-level settings allow a proven-bad route to opt into direct discovery without changing unrelated routes. + +Suggested fields: + +```ts +export type OcxRoutedToolDiscoveryMode = "auto" | "deferred" | "direct"; + +interface OcxProviderConfig { + routedToolDiscovery?: OcxRoutedToolDiscoveryMode; + modelRoutedToolDiscovery?: Record; +} +``` + +Resolution precedence: + +```text +Cursor hard fence + > exact model override + > provider override + > auto default +``` + +`auto` resolves to `deferred` for non-Cursor and `direct` for Cursor. + +### PR B — Responses tool conformance + +Build fixture-driven tests for: + +- top-level `tools` +- `input[].type == "additional_tools"` +- function/custom/namespace conversion +- tool-search call/output history +- streaming and non-streaming output +- continuation, compaction and resume + +No route may claim native discovery unless this matrix passes for its active adapter. + +### PR C — bounded meta-tool fallback + +Introduce a sidecar/multiplexer surface: + +```text +ocx_tool_search +ocx_tool_describe +ocx_tool_call +``` + +This is the long-term fallback for incompatible routes. It avoids both silent tool loss and full eager schemas. + +### PR D — exact live E2E and rollout telemetry + +Automate or manually certify the #1522 pairing and representative variants. Record payload size, discovery path, actual tool call, adapter, client surface and model. + +## 4. Files in this unit + +### Research and decisions + +- `001_verified_dev_baseline.md` +- `002_incident_history_1522_1529_1596.md` +- `003_current_code_map.md` +- `004_upstream_codex_code_mode.md` +- `005_comparator_findings.md` +- `006_architecture_invariants.md` +- `007_scenario_matrix.md` +- `008_risk_register.md` +- `009_open_questions_and_evidence_gaps.md` +- `094_landing_verification_pass.md` — worktree/upstream/GitHub re-verification + +### Implementation roadmap + +- `010_phase1_profile_resolver.md` +- `011_phase1_types_and_config.md` +- `012_phase1_catalog_patch.md` +- `013_phase1_sync_and_fingerprint.md` +- `014_phase1_diagnostics.md` +- `020_phase2_unit_tests.md` +- `021_catalog_test_cases.md` +- `022_config_and_precedence_tests.md` +- `023_backward_compatibility_tests.md` +- `030_phase3_protocol_conformance.md` +- `031_responses_lite_additional_tools.md` +- `032_custom_namespace_roundtrip.md` +- `033_tool_search_history_and_compaction.md` +- `040_phase4_live_e2e.md` +- `041_code_mode_all_tools_canary.md` +- `042_codex_app_deepseek_browser.md` +- `043_cursor_and_direct_bounded.md` +- `044_weak_model_meta_tool_fallback.md` +- `050_phase5_payload_cache_benchmarks.md` +- `051_benchmark_methodology.md` +- `052_acceptance_thresholds.md` +- `060_phase6_meta_tool_design.md` +- `061_meta_tool_contract.md` +- `062_meta_tool_security.md` +- `070_rollout_plan.md` +- `071_observability.md` +- `072_canary_matrix.md` +- `080_rollback_plan.md` +- `081_failure_triage_runbook.md` +- `090_final_recommendation.md` +- `091_pr_stack_and_commits.md` +- `092_definition_of_done.md` + +### Executable supplements + +- `prototype/mvp-resolver.mjs` +- `prototype/profile-resolver.mjs` +- `prototype/tool-discovery-profile.test.mjs` +- `prototype/payload-benchmark.mjs` +- `patches/0001-routed-tool-discovery-profile.patch` +- `patches/0002-focused-tests.patch` +- `scripts/run_repo_validation.sh` +- `results/prototype-test-output.txt` +- `results/payload-benchmark.json` + +## 5. Decision summary + +The first code change should be a **small resolver and override**, not the full meta-tool system. It gives operators a safe emergency lever while preserving the current default. The full fallback belongs in a separate PR because it adds a new execution surface, security boundary and search-quality problem. + +The minimum acceptable outcome is: + +- zero default behavior change from PR #1596; +- explicit direct override only for a named provider/model; +- exact unit coverage for precedence and fallback catalog paths; +- payload warning for direct mode; +- no claim of universal native tool-search support until adapter conformance passes. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/001_verified_dev_baseline.md b/devlog/_plan/260813_routed_tool_discovery_profiles/001_verified_dev_baseline.md new file mode 100644 index 000000000..6fcd36e6e --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/001_verified_dev_baseline.md @@ -0,0 +1,80 @@ +# 001 - Verified `dev` baseline + +Status: VERIFIED FROM GITHUB +Branch: `dev` +Packaging-time head: `2cdbf66a23f9fd8f2f38dcc702ccd3f2e60ac535` +Packaging-time commit: PR #1600, CI timeout/watchdog scaling +Tool-discovery semantic base: `5703473041a9f4f415743652de5d86d51fd66db5`, PR #1596 + +The packaging-time head has PR #1596 as a direct parent. Relevant source files were fetched again from `dev` after the CI-only head advance; the catalog policy seams documented below remained present. + +## Source locations + +| Concern | Current location | Verified behavior | +|---|---|---| +| Routed normalization | `src/codex/catalog/parsing.ts` | `tool_mode=code_mode_only`; Cursor false, non-Cursor true for `supports_search_tool` | +| Template-less fallback | `src/codex/catalog/sync.ts` | Repeats Cursor/non-Cursor discovery split | +| Provider metadata → catalog | `src/codex/catalog/provider-fetch.ts` | `applyProviderConfigHints()` already carries per-provider/model capability hints such as context, modalities, reasoning and parallel tools | +| Provider config type | `src/types.ts` | `OcxProviderConfig` contains provider-level and model-level capability overrides; no routed discovery override yet | +| Config schema | `src/config.ts` | `providerConfigSchema` validates known provider fields and passes unknown fields through | +| Facade exports | `src/codex/catalog.ts` | Re-exports catalog parsing/provider-fetch/sync surfaces | +| Focused tests | `tests/catalog-cursor-search.test.ts` | Pins template and fallback search advertising | +| Broad catalog tests | `tests/codex-catalog.test.ts` | Pins normalized and combo rows | +| Native-parity smoke | `tests/e2e-style/phase100-native-parity.test.ts` | Expects routed DeepSeek row to advertise search | + +## Current normalization seam + +The load-bearing current code is conceptually: + +```ts +const isCursorEntry = typeof entry.slug === "string" + && entry.slug.startsWith("cursor/"); + +if (isCursorEntry) { + delete entry.web_search_tool_type; +} else { + entry.web_search_tool_type = "text_and_image"; +} +entry.supports_search_tool = !isCursorEntry; +``` + +This is not yet a capability resolver. It is a surface-name special case. + +## Current provider hint seam + +`applyProviderConfigHints(name, prov, model, providerCap)` is the correct place to resolve a provider/model override because it already applies: + +- `modelContextWindows` +- `modelInputModalities` +- `modelMaxInputTokens` +- reasoning ladders and defaults +- reasoning-summary support +- parallel-tool-call support + +A resolved `toolDiscoveryMode` carried on `CatalogModel` follows the existing architecture instead of making `normalizeRoutedCatalogEntry()` reach back into global config. + +## Current gather identity seam + +`providerCatalogFingerprint()` includes fields that influence catalog output. A discovery override must be added there so two different configs do not share a stale gather key. The newer `providerGraphIdentity` hashes the whole admitted provider row as a second fence, but the explicit fingerprint should still record the field because it is part of the catalog contract and testable identity. + +## Baseline test commands from the repository + +```bash +bun run typecheck +bun test tests/catalog-cursor-search.test.ts \ + tests/codex-catalog.test.ts \ + tests/e2e-style/phase100-native-parity.test.ts +bun run test +``` + +## Baseline constraint + +This bundle was built without a mounted full repository checkout. The executable tests included here validate the resolver and synthetic payload model independently. The repository commands above are the required integration gate when applied in the actual OpenCodex worktree. + +**Superseded on 2026-08-13:** that gate has now been run. See +`094_landing_verification_pass.md` for the re-verification of every claim above +against a real worktree, the upstream `codex-rs` source, and live GitHub state. +Eight corrections were recorded there; the ones touching this document are: +`CatalogModel` lives in `src/codex/catalog/parsing.ts`, not `src/types.ts`, and +`supports_search_tool` is gated upstream by a second conjunct +(`namespace_tools_enabled`), so it is not the sole switch for deferred discovery. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/002_incident_history_1522_1529_1596.md b/devlog/_plan/260813_routed_tool_discovery_profiles/002_incident_history_1522_1529_1596.md new file mode 100644 index 000000000..5b9ed85d7 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/002_incident_history_1522_1529_1596.md @@ -0,0 +1,72 @@ +# 002 - Incident history: #1522 → #1529 → #1596 + +## Timeline + +### Issue #1522 + +Reported behavior: + +- Codex App +- OpenAI-compatible third-party provider +- DeepSeek-compatible routed model +- Browser plugin enabled +- `supports_search_tool=true` +- Browser deferred tools such as `mcp__node_repl__js` did not appear to the model + +The reporter changed only the catalog flag to false and observed direct MCP tools become visible and callable. + +This proved a real user-visible failure, but it did **not** prove that all non-Cursor routed models require direct discovery. It proved one exact client/provider/model/plugin combination was broken. + +### PR #1529 + +Response: + +- set `supports_search_tool=false` for every routed row; +- kept hosted web-search metadata separate for non-Cursor rows; +- updated template and template-less catalog paths; +- added regression expectations for direct discovery. + +Benefit: + +- restored the reporter's direct MCP surface. + +Cost discovered later: + +- under `code_mode_only`, disabling deferred discovery causes Codex to include every MCP declaration in `exec.description`; +- request size was measured at approximately 96,699 versus 258,929 characters; +- the fix applied the cost globally to solve one unverified scope. + +### PR #1596 + +Response: + +- restored `supports_search_tool=true` for non-Cursor routed rows; +- kept Cursor false; +- pinned `code_mode_only + supports_search_tool=true` as a pair; +- measured payload size; +- ran a live Kimi Code Mode canary through `tools.mcp__node_repl__js`; +- added template and template-less regression tests. + +Residual risk explicitly recorded: + +- the exact #1522 DeepSeek-compatible Browser pairing was not reproduced on the development machine; +- the canary proved the Code Mode mechanism, not every model's compliance or every Codex App plugin lifecycle. + +## Root lesson + +The mistake was not “true” versus “false.” The mistake was treating a route-dependent capability as a global Boolean default with no escape hatch and no evidence record. + +## Required correction + +Keep #1596 as the default. Add: + +1. a provider/model override for exact proven failures; +2. a diagnostic explaining the resolved mode and reason; +3. payload warnings when direct mode is selected; +4. an E2E evidence matrix that can later drive automatic profile selection. + +## Source links + +- Issue #1522: https://github.com/lidge-jun/opencodex/issues/1522 +- PR #1529: https://github.com/lidge-jun/opencodex/pull/1529 +- PR #1596: https://github.com/lidge-jun/opencodex/pull/1596 diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/003_current_code_map.md b/devlog/_plan/260813_routed_tool_discovery_profiles/003_current_code_map.md new file mode 100644 index 000000000..2ddb90595 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/003_current_code_map.md @@ -0,0 +1,175 @@ +# 003 - Current code map and patch insertion points + +> **Amended 2026-08-13** after re-verification in a real worktree +> (`094_landing_verification_pass.md`). Three fixes apply to this document: +> `CatalogModel` is declared in `src/codex/catalog/parsing.ts:94`, not +> `src/types.ts`; the combo rule needs +> `src/codex/catalog/aggregation.ts` (`deriveComboCatalogModel`, lines 125-177), +> which was missing from the file list below; and `providerConfigSchema` is a +> `.passthrough()` object that would not validate a new field at all. + +## 1. Data model + +### `src/types.ts` + +Add a reusable mode type: + +```ts +export type OcxRoutedToolDiscoveryMode = "auto" | "deferred" | "direct"; +``` + +Add provider fields beside other Codex/catalog capability fields: + +```ts +routedToolDiscovery?: OcxRoutedToolDiscoveryMode; +modelRoutedToolDiscovery?: Record; +``` + +Why provider + model: + +- one gateway can front multiple upstream models; +- one model may need a direct fallback while siblings stay deferred; +- the codebase already uses this provider/model override pattern for adapters, context, modalities and reasoning. + +## 2. Catalog carrier + +### `src/codex/catalog/parsing.ts` + +Extend `CatalogModel` (declared here at line 94, beside `parallelToolCalls` and +`supportsReasoningSummaries` — it is **not** in `src/types.ts`): + +```ts +toolDiscoveryMode?: "deferred" | "direct"; +``` + +The catalog row should contain the **resolved** mode, never `auto`. + +## 3. Resolution + +### `src/codex/catalog/provider-fetch.ts` + +Add: + +```ts +export function configuredRoutedToolDiscoveryMode( + name: string, + prov: OcxProviderConfig, + modelId: string, +): "deferred" | "direct"; +``` + +Resolution: + +```text +if Cursor adapter/provider -> direct +else model override +else provider override +else deferred +``` + +`auto` means current default: deferred for non-Cursor. + +Apply the result inside `applyProviderConfigHints()` so configured, live-discovered, cached and combo-derived rows receive the same mode. + +## 4. Catalog serialization + +### `src/codex/catalog/parsing.ts` + +Change: + +```ts +normalizeRoutedCatalogEntry(entry, parallelToolCalls) +``` + +into an options object — the fence needs the provider identity as well as the +mode, and the existing positional arguments must stay put for the public callers +(`src/codex/catalog.ts` re-export, `tests/parallel-tool-calls-optin.test.ts`): + +```ts +normalizeRoutedCatalogEntry( + entry, + parallelToolCalls, + { toolDiscoveryMode, providerId }, +) +``` + +See `012` for the resolved snippet and the shared `isCursorRoute()` helper. + +The function still hard-fences Cursor to direct. + +### `src/codex/catalog/sync.ts` + +Pass `model?.toolDiscoveryMode` on the template path (line 287) and use the same +resolved value on the template-less path (lines 312-343). Both paths must emit +explicitly: the template-less fallback never calls `normalizeRoutedCatalogEntry`, +and `ensureStrictCatalogFields` silently defaults missing capability flags, so +policy must never be left to a strict-field default. + +### `src/codex/catalog/aggregation.ts` + +`deriveComboCatalogModel()` (lines 125-177) owns combo capability derivation. The +"any member direct → combo direct" rule from `012` is implemented here. + +Avoid duplicating a second independent policy expression. A small helper such as `applyRoutedToolDiscoveryMetadata()` should own: + +- `supports_search_tool` +- `web_search_tool_type` +- Cursor exception + +## 5. Config admission + +### `src/config.ts` + +`providerConfigSchema` (lines 616-645) declares only a minority of +`OcxProviderConfig` fields and ends with `.passthrough()`; it contains no +`.catch()`. A field added only to the TypeScript interface is passed through +**unvalidated on both paths**. Both halves must be built explicitly. + +Add schema fields and a strict write-boundary validator. Recommended behavior: + +- malformed hand edit: degrade the optional field and warn; +- live config write/import: reject with a path-specific error; +- unknown future values: preserve only if the project intentionally chooses forward compatibility for this enum. + +## 6. Gather identity + +### `src/codex/catalog/provider-fetch.ts` + +Add the two fields to `providerCatalogFingerprint()`: + +```ts +rtd: prov.routedToolDiscovery ?? null, +mrtd: prov.modelRoutedToolDiscovery ?? null, +``` + +The whole-row `providerGraphIdentity` already provides a broad safety net. The explicit fingerprint keeps the older join key semantically complete and makes regression intent clear. + +## 7. Tests + +Note: no `tests/config*.test.ts` currently asserts routed discovery flags at all, +so the config coverage below is new work rather than an extension. + +Update: + +- `tests/catalog-cursor-search.test.ts` +- `tests/codex-catalog.test.ts` +- `tests/e2e-style/phase100-native-parity.test.ts` + +Add: + +- `tests/codex-tool-discovery-mode.test.ts` +- config degradation and write-boundary cases in `tests/config.test.ts` / `tests/config-user-edits.test.ts` + +## 8. Diagnostics + +A later management DTO can expose: + +```json +{ + "resolvedToolDiscovery": "deferred", + "source": "default", + "reason": "non-Cursor Code Mode default" +} +``` + +Do not encode this diagnostic-only source/reason into Codex's model catalog unless the client explicitly tolerates unknown extension keys. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/004_upstream_codex_code_mode.md b/devlog/_plan/260813_routed_tool_discovery_profiles/004_upstream_codex_code_mode.md new file mode 100644 index 000000000..07ed06f49 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/004_upstream_codex_code_mode.md @@ -0,0 +1,118 @@ +# 004 - Upstream Codex Code Mode findings + +> **Amended 2026-08-13** against the upstream source on disk +> (`/Users/jun/Developer/codex/120_codex-cli`, `main` @ `4462b9dee`, 2026-07-23). +> See `094_landing_verification_pass.md` Correction 2. The mechanism described +> below is confirmed, but its consequence was understated: under +> `tool_mode = code_mode_only`, an **eligible** MCP tool is installed on the V8 +> `tools` / `ALL_TOOLS` globals in **both** deferred and direct mode. For such a +> tool the flag changes where the schemas live, not whether it is callable. +> "Eligible" excludes `direct_only_tool_namespaces`, `excluded_tool_namespaces`, +> and anything removed by MCP/App visibility or policy filtering — see the +> exclusion table in `094`. + +## Verified mechanism + +OpenAI Codex's Code Mode runtime installs: + +- `tools`: callable functions for enabled nested tools; +- `ALL_TOOLS`: `{name, description}` metadata for enabled nested tools; +- helpers such as `text`, `image`, `store`, `load`, `yield_control` and `exit`. + +The public `exec` description states that nested tools are available through the global `tools` object and listed in `ALL_TOOLS`. When some nested tools are deferred, the description tells the model that omitted tools still exist in those globals. + +Relevant upstream files: + +- `codex-rs/code-mode/src/runtime/globals.rs` +- `codex-rs/code-mode-protocol/src/description.rs` +- `codex-rs/core/tests/suite/code_mode.rs` +- `codex-rs/core/src/tools/spec_plan.rs` +- `codex-rs/core/src/tools/spec_plan_tests.rs` + +Path note (2026-08-13): the runtime crate is `codex-rs/code-mode/`, not +`code-mode-runtime/`. The load-bearing exposure and executor logic lives in +`spec_plan.rs`; `spec_plan_tests.rs` also exists and was correctly named in the +original list. Verified in the checkout at `4462b9dee`. + +Repository: https://github.com/openai/codex + +## The exposure switch, verbatim + +```rust +let exposure = if search_tool_enabled { ToolExposure::Deferred } else { ToolExposure::Direct }; +``` + +`codex-rs/core/src/mcp_tool_exposure.rs:35`, where + +```rust +pub(crate) fn search_tool_enabled(turn_context: &TurnContext) -> bool { + turn_context.model_info.supports_search_tool && namespace_tools_enabled(turn_context) +} +``` + +`codex-rs/core/src/tools/spec_plan.rs:330`. + +Two consequences: + +1. `supports_search_tool` is not the sole switch — a provider without Responses + namespace tools resolves to `Direct` whatever the catalog says. +2. `Direct` under code mode means every MCP declaration is embedded in + `exec.description` (they enter `enabled_tools` instead of `deferred_tools`). + That is the measured 96,699 → 258,929 character regression. For an eligible + tool the schema placement is the material difference — it stays equally + callable either way. + +The flag additionally drives `tool_search` construction and the deferred-guidance +text, and it does **not** govern three independent removal paths +(`direct_only_tool_namespaces` → `DirectModelOnly`, `excluded_tool_namespaces`, +and pre-classification MCP/App policy filtering). An override cannot repair any of +those. + +The isolate is built for `ToolMode::CodeMode` as well as `CodeModeOnly` +(`spec_plan.rs:459`), so the reachability reasoning covers both; `code_mode_only` +additionally hides ordinary nested tools from the top-level list. Only under +`ToolMode::Direct` is there no isolate, and there the flag does govern direct +declaration versus `tool_search`. Every routed OpenCodex row is stamped +`code_mode_only` today. + +So for eligible tools under code mode the `direct` override is a +model-comprehension lever (full schemas inline rather than via `ALL_TOOLS`) with a +payload cost, not a reachability fix. E1/E2 in `009` remain open: this reading +comes from a 2026-07-23 clone, while #1522 reported against CLI +`0.147.0-alpha.6.5` on 2026-08-12, and app-layer gating outside this clone is +unverified. `020`/`021` owe a controlled single-variable test before this claim is +treated as settled. + +## Architectural implication + +For a non-Cursor routed model running under Codex Code Mode, provider-native `tool_search` is not the only reachability path. The client-side isolate can expose tools independently of whether the upstream provider knows anything about MCP. + +Therefore: + +```text +supports_search_tool=true ++ tool_mode=code_mode_only +``` + +is a client-harness profile, not a claim that Kimi, DeepSeek or Claude hosts implement OpenAI tool search. + +## What this mechanism does not prove + +It does not prove: + +- every external model will choose `exec` reliably; +- Codex App and CLI have identical plugin initialization timing; +- Browser plugin namespaces always enter the Code Mode runtime; +- dynamic `tools/list_changed` updates refresh the active isolate; +- compaction and resume preserve the same tool roster; +- a weak model can discover a semantically relevant tool from `ALL_TOOLS`. + +## Test consequence + +Tests must distinguish: + +1. **mechanical availability**: the function exists on `tools`; +2. **catalog discoverability**: metadata appears in `ALL_TOOLS`; +3. **model acquisition**: the model actually selects the function; +4. **round-trip execution**: Codex executes it and sends the result back; +5. **lifecycle durability**: the same remains true after changes, compaction and resume. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/005_comparator_findings.md b/devlog/_plan/260813_routed_tool_discovery_profiles/005_comparator_findings.md new file mode 100644 index 000000000..65d2179ef --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/005_comparator_findings.md @@ -0,0 +1,77 @@ +# 005 - Comparator findings + +## CLIProxyAPI + +Useful evidence: + +- Responses Lite may put tools in `input[].type == "additional_tools"` instead of top-level `tools`. +- A translator that reads only top-level tools can silently erase terminal, custom and namespace tools. +- The model may then emit ordinary text such as “I will run it” and complete normally, making the failure look like model behavior rather than protocol loss. +- Full round-trip support requires request conversion, history replay, response restoration, stable call IDs and namespace preservation. + +Relevant issues: + +- https://github.com/router-for-me/CLIProxyAPI/issues/4798 +- https://github.com/router-for-me/CLIProxyAPI/issues/3361 + +## LiteLLM + +LiteLLM added two virtual tools for large MCP catalogs: + +```text +mcp_tool_search +mcp_tool_call +``` + +This bounds the always-visible schema surface, but its simple token-overlap search and default `top_k=5` can miss the correct tool. A reported 113-tool test observed recall 0.70 at 5 and 1.00 at 10. + +Relevant items: + +- https://github.com/BerriAI/litellm/pull/31777 +- https://github.com/BerriAI/litellm/issues/33440 + +## Cloudflare Code Mode + +Cloudflare exposes a huge API through three general tools (`docs`, `search`, `execute`) and keeps the full specification server-side. Its published comparison shows the central advantage: tool count becomes approximately constant in the model context. + +Repository: + +- https://github.com/cloudflare/mcp + +## cc-switch + +cc-switch's useful principle is capability bundling by verified host/protocol profile rather than model brand alone. Its limitation is that conservative templates can still disable discovery broadly. OpenCodex should adopt the profile idea but retain Code Mode-aware defaults. + +Repository: + +- https://github.com/farion1231/cc-switch + +## Claude Code public issue evidence + +Deferred loading introduces lifecycle risks: + +- tool-array mutation can invalidate large prompt-cache prefixes; +- a model may under-use deferred capabilities; +- dynamic tool-list updates can leave a stale search index; +- transient tool references can poison resumed sessions; +- large search batches can trigger expensive cache rebuilds. + +Representative issues: + +- https://github.com/anthropics/claude-code/issues/81967 +- https://github.com/anthropics/claude-code/issues/84312 +- https://github.com/anthropics/claude-code/issues/66084 +- https://github.com/anthropics/claude-code/issues/79970 +- https://github.com/anthropics/claude-code/issues/83756 + +## Composite conclusion + +No comparator provides a complete drop-in answer. The strongest composite is: + +```text +Codex local Code Mode by default ++ route-specific conformance evidence ++ bounded meta-tool fallback ++ stable session manifest ++ exact live E2E certification +``` diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/006_architecture_invariants.md b/devlog/_plan/260813_routed_tool_discovery_profiles/006_architecture_invariants.md new file mode 100644 index 000000000..43f416185 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/006_architecture_invariants.md @@ -0,0 +1,71 @@ +# 006 - Architecture invariants + +## INV-1 — Default behavior stays #1596-compatible + +Without a new config field, generated catalogs must be byte-equivalent for the affected discovery fields: + +- Cursor: direct +- non-Cursor: deferred + +## INV-2 — Code Mode and deferred discovery move together + +A non-Cursor `deferred` row must have: + +```json +{ + "tool_mode": "code_mode_only", + "supports_search_tool": true +} +``` + +A regression changing only one side must fail tests. + +## INV-3 — Hosted web search is independent + +`web_search_tool_type` describes the OpenCodex hosted-search sidecar. It is not the same capability as MCP/plugin discovery. Direct discovery on a non-Cursor provider may still advertise hosted web search. + +## INV-4 — Cursor remains hard-fenced + +Cursor's custom transport bypasses the relevant sidecar/deferred path. A config attempting to force Cursor deferred should either: + +- be rejected at the write boundary, or +- resolve to direct with an explicit warning. + +Silent acceptance is not allowed. + +## INV-5 — Model override wins provider override + +This follows existing `modelXxx` configuration conventions and allows one incompatible model on a mixed gateway to fall back without penalizing siblings. + +## INV-6 — `auto` is resolved before serialization + +`CatalogModel.toolDiscoveryMode` carries only `deferred` or `direct`. Codex never sees the internal `auto` state. + +## INV-7 — Catalog gather identity includes the policy + +Changing discovery mode must invalidate or separate cached catalog gathers. + +## INV-8 — Direct mode is observable and bounded + +Because direct mode can inflate `exec.description`, the runtime or management surface must report: + +- resolved mode; +- provider/model source; +- approximate visible tool/schema count when available; +- a warning when configured bounds are exceeded. + +## INV-9 — Protocol converters never silently drop tool types + +Unsupported tool shapes must produce an explicit compatibility error or select a fallback profile. A normal `response.completed` after stripping tools is unacceptable. + +## INV-10 — Session-visible top-level tool manifests should remain stable + +Dynamic changes should update a side index or local Code Mode registry where possible, not mutate the prompt-prefix tool array repeatedly. + +## INV-11 — Transient tools do not become durable references + +Startup-only or connection-only tools must be filtered from persisted search results or marked non-durable and reconciled before replay. + +## INV-12 — Exact E2E evidence outranks model-brand assumptions + +A model name, provider label or marketing claim is not sufficient to select native discovery. Selection requires either a client-side Code Mode guarantee or a passing adapter/client conformance suite. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/007_scenario_matrix.md b/devlog/_plan/260813_routed_tool_discovery_profiles/007_scenario_matrix.md new file mode 100644 index 000000000..fea27ecaa --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/007_scenario_matrix.md @@ -0,0 +1,36 @@ +# 007 - Scenario matrix + +| ID | Surface | Provider/adapter | Tool path | Expected profile | Required proof | +|---|---|---|---|---|---| +| S01 | Codex CLI | non-Cursor routed | Code Mode `tools`/`ALL_TOOLS` | deferred | tool exists, model calls it, payload bounded | +| S02 | Codex App | DeepSeek-compatible routed + Browser plugin | Code Mode plugin tool | deferred if passing; direct override if failing | exact #1522 reproduction | +| S03 | Codex App | Kimi routed + Browser/plugin | Code Mode plugin tool | deferred | live canary parity with App surface | +| S04 | Codex CLI/App | Cursor | custom runTurn transport | direct | no false hosted/deferred advertisement | +| S05 | Responses Lite | Anthropic translator | `additional_tools` | native/conformant | function/custom/namespace retained | +| S06 | Chat Completions bridge | generic external model | flattened functions | meta-tools or direct bounded | no silent type loss | +| S07 | weak external model | many MCP tools | Code Mode | deferred or meta-tools | acquisition-rate A/B test | +| S08 | dynamic MCP server | `tools/list_changed` | local index refresh | deferred/meta-tools | new tool callable without restart | +| S09 | compaction/resume | any routed model | restored history | same profile | no dangling references; tool callable | +| S10 | transient startup tool | any surface | early discovery | filtered | no persisted dead reference | +| S11 | 500+ tools | any non-Cursor Code Mode | deferred | deferred | turn-1 payload sublinear in schema bytes | +| S12 | explicit provider direct override | named route | eager/direct | direct | only target provider changes | +| S13 | explicit model direct override | mixed gateway | eager/direct | direct for one model | sibling stays deferred | +| S14 | invalid config value | config load/write | n/a | degrade on load; reject write | no provider loss | +| S15 | provider config changes mid-flight | catalog gather | cache identity | new result | no stale shared promise | +| S16 | meta-tool search | 100+ similar names | bounded search | meta-tools | exact/prefix/BM25 recall and pagination | +| S17 | meta-tool call collision | same short name in namespaces | qualified call | meta-tools | no wrong namespace fallback | +| S18 | hosted web search + direct MCP | non-Cursor | independent surfaces | direct + hosted search | both available independently | + +## Priority + +### P0 + +S01, S02, S04, S05, S09, S11, S12, S13, S14. + +### P1 + +S07, S08, S10, S15, S18. + +### P2 + +S16 and S17 belong to the meta-tool PR, not the initial resolver PR. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/008_risk_register.md b/devlog/_plan/260813_routed_tool_discovery_profiles/008_risk_register.md new file mode 100644 index 000000000..253d07259 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/008_risk_register.md @@ -0,0 +1,30 @@ +# 008 - Risk register + +| Risk | Likelihood | Impact | Detection | Mitigation | +|---|---:|---:|---|---| +| Direct override recreates large payloads | High when used | High cost/latency | request-size telemetry | warn, bound, scope to exact model | +| Resolver defaults accidentally change #1596 | Medium | High regression | snapshot/unit tests | explicit auto-default tests | +| Cursor forced into deferred path | Low | High functional loss | Cursor catalog tests | hard fence in resolver | +| Config typo invalidates entire file | Medium | High data loss | config user-edit tests | load sanitizer + strict write boundary | +| Catalog gather reuses stale policy | Medium | Medium | concurrency/fingerprint test | include policy in gather identity | +| Model override key fails date/alias matching | Medium | Medium | modelRecordValue fixtures | reuse existing helper | +| Combo derives conflicting member modes | Medium | Medium | combo tests | conservative intersection: direct wins | +| Responses Lite drops `additional_tools` | Known class | High silent failure | request translation fixtures | merge both tool sources | +| Tool type restored as wrong event | Medium | High agent-loop failure | streaming/non-streaming fixtures | preserve original type map | +| Deferred tool not acquired by weak model | Medium | Medium | A/B acquisition eval | meta-tools or direct override | +| Tool list changes invalidate cache | Medium | High cost | manifest hash telemetry | stable top-level manifest | +| Search index stale after list change | Medium | Medium | dynamic MCP E2E | refresh side index | +| Transient tool poisons resume | Low/Medium | High | resume fixture | filter/reconcile durable references | +| Meta-tool search misses correct tool | Medium | Medium | recall benchmark | exact match, top-k≥10, pagination | +| Meta-tool call bypasses permissions | Low if designed poorly | Critical | security tests | reuse normal authorization path | +| Huge tool result bloats active context | High over long sessions | Medium | result-byte metrics | result caps and compact references | + +## Stop-ship conditions + +- default non-Cursor row no longer resolves deferred; +- Cursor row advertises deferred or hosted search; +- direct override leaks to sibling providers/models; +- invalid optional config resets providers or API keys; +- translator returns success after deleting all tools; +- meta-tool call can invoke an unauthorized server/tool; +- full suite introduces unexplained catalog snapshot churn. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/009_open_questions_and_evidence_gaps.md b/devlog/_plan/260813_routed_tool_discovery_profiles/009_open_questions_and_evidence_gaps.md new file mode 100644 index 000000000..2a22fd6d6 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/009_open_questions_and_evidence_gaps.md @@ -0,0 +1,74 @@ +# 009 - Open questions and evidence gaps + +## E1 — Exact #1522 pairing + +Still required: + +```text +Codex App ++ Browser plugin ++ DeepSeek-compatible external model ++ current dev catalog ++ supports_search_tool=true ++ code_mode_only +``` + +Need to capture: + +- client version; +- catalog row; +- first request tool metadata; +- `ALL_TOOLS` listing; +- whether `mcp__node_repl__js` exists on `tools`; +- actual model call; +- result event; +- payload bytes. + +## E2 — App versus CLI parity + +The live canary in #1596 used routed Kimi through a CLI-style execution path. Codex App plugin initialization and task lifecycle may differ. This must not be inferred away. + +## E3 — Dynamic tool updates + +Determine whether the current Codex build refreshes Code Mode `ALL_TOOLS` after MCP `tools/list_changed`, or whether a new session/isolate is required. + +## E4 — Compaction and resume + +Determine which items persist: + +- tool-search outputs; +- loaded tool definitions; +- namespace metadata; +- Code Mode registry state; +- transient tool references. + +## E5 — Acquisition quality by model + +Run identical task batteries with: + +- required tool already visible; +- required tool deferred in `ALL_TOOLS`; +- required tool behind `ocx_tool_search`. + +Measure tool acquisition, user-question fallback, invented workaround and false impossibility rates. + +## E6 — Direct-mode practical bounds + +Direct mode is currently an escape hatch, but safe limits are not yet defined. Measure realistic plugin sets and establish warning/failure thresholds for: + +- tool count; +- schema bytes; +- `exec.description` bytes; +- total first-request bytes. + +## E7 — Automatic profile promotion + +Do not implement automatic profile selection until evidence has: + +- a versioned suite id; +- client and adapter scope; +- last-verified timestamp; +- pass/fail result; +- expiry policy. + +Until then, defaults plus explicit override are more trustworthy. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/010_phase1_profile_resolver.md b/devlog/_plan/260813_routed_tool_discovery_profiles/010_phase1_profile_resolver.md new file mode 100644 index 000000000..1d67e07f5 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/010_phase1_profile_resolver.md @@ -0,0 +1,82 @@ +# 010 - Phase 1: route-scoped discovery resolver + +Status: IMPLEMENTATION-READY +Scope: first PR only + +## Goal + +Replace the hard-coded Cursor/non-Cursor Boolean with one pure resolver while preserving PR #1596 as the default. + +The first PR is deliberately small. It does not implement a new tool protocol, mutate the Code Mode runtime, or auto-classify providers. It only introduces a typed route policy and a narrow compatibility escape hatch. + +## Public configuration + +```ts +export type OcxRoutedToolDiscoveryMode = "auto" | "deferred" | "direct"; + +interface OcxProviderConfig { + routedToolDiscovery?: OcxRoutedToolDiscoveryMode; + modelRoutedToolDiscovery?: Record; +} +``` + +## Resolution order + +```text +Cursor hard fence + > exact model override + > provider override + > auto default +``` + +> **Amended 2026-08-13:** the original text said "exact/date-compatible model +> override". `modelRecordValue()` does not match `-YYYYMMDD` variants +> (`094_landing_verification_pass.md` Correction 4), and the decision is to keep +> its semantics rather than invent a bespoke matcher. An operator pinning an +> emergency escape hatch names the exact model id that failed. + +Resolved values are only: + +```ts +type ResolvedRoutedToolDiscoveryMode = "deferred" | "direct"; +``` + +`auto` resolves as follows: + +| Route | Result | +|---|---| +| Cursor provider name or `adapter: "cursor"` | `direct` | +| every other routed provider | `deferred` | + +## Why this is the first PR + +- It keeps current users on the #1596 shape. +- It gives #1522-style reproductions a route-local remediation. +- It creates a stable seam for later conformance evidence. +- It is reversible: deleting the two optional fields restores current behavior. + +## Non-goals + +- no automatic switch based on model brand; +- no full MCP multiplexer; +- no client-version database; +- no per-session mode mutation; +- no claim that `direct` is cheaper or preferred. +- no claim that `direct` restores *reachability* for an **eligible** tool under + code mode. Upstream installs such tools on the `tools`/`ALL_TOOLS` globals in + both modes; `direct` moves their full schemas into `exec.description` at a + measured payload cost and changes `tool_search` construction. It cannot repair + a tool removed by `direct_only_tool_namespaces`, `excluded_tool_namespaces`, or + MCP/App policy filtering (`004`, `094` Correction 2). + +## Acceptance + +1. No new config produces current catalog flags. +2. One provider override changes only that provider. +3. One model override changes only that model. +4. Cursor remains direct even when configured deferred. +5. Hosted web search remains independent. +6. The resolved mode participates in catalog gather identity. +7. The Cursor fence resolves identically on the template and template-less paths + (provider identity first, slug prefix only as fallback) — closing the + unresolved #1596 P2 rather than reproducing it (`094` Correction 8). diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/011_phase1_types_and_config.md b/devlog/_plan/260813_routed_tool_discovery_profiles/011_phase1_types_and_config.md new file mode 100644 index 000000000..2b99494e9 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/011_phase1_types_and_config.md @@ -0,0 +1,101 @@ +# 011 - Phase 1: types, config parsing and precedence + +## Type changes + +### `src/types.ts` + +Add: + +```ts +export type OcxRoutedToolDiscoveryMode = "auto" | "deferred" | "direct"; +``` + +Add to `OcxProviderConfig` near the existing MCP/catalog capability fields: + +```ts +/** Codex routed-row discovery policy. Default auto. */ +routedToolDiscovery?: OcxRoutedToolDiscoveryMode; + +/** Exact/modelRecordValue-compatible per-model override. */ +modelRoutedToolDiscovery?: Record; +``` + +### `CatalogModel` + +Carry only the resolved value: + +```ts +toolDiscoveryMode?: "deferred" | "direct"; +``` + +This follows existing `parallelToolCalls`, `reasoningEfforts` and modality propagation. `normalizeRoutedCatalogEntry()` should not read global config. + +## Config load and write behavior + +OpenCodex intentionally distinguishes hand-edited load recovery from live write validation. + +### Load path + +Malformed optional discovery fields should degrade to undefined, preserving providers, credentials and ports. Recommended schema: + +```ts +const routedToolDiscoveryModeSchema = z.enum(["auto", "deferred", "direct"]); + +routedToolDiscovery: routedToolDiscoveryModeSchema + .optional() + .catch(undefined), +modelRoutedToolDiscovery: z.record(z.string(), routedToolDiscoveryModeSchema) + .optional() + .catch(undefined), +``` + +A warning should state that the invalid policy was ignored. + +### Write path + +> **Amended 2026-08-13 (`094` Correction 5 + audit blocker 3).** Two gaps between +> this section's promises and the draft in `patches/`: +> +> 1. The draft adds `.catch(undefined)` but **no load-path warning**. Silent +> degradation is exactly the observability failure `014` exists to prevent — +> an operator whose emergency override was dropped by a typo must be told. +> The implementation must emit the warning, and a test must assert it. +> 2. The accessor/prototype-pollution rejection below must be established +> **before** the validator enumerates or reads properties. A validator that +> reaches for `candidate.providers[name].modelRoutedToolDiscovery` and only +> then checks for getters has already run attacker-controlled code. Inspect +> with `Object.getOwnPropertyDescriptor` and reject non-data descriptors and +> non-own/prototype-sourced keys first, then read values. +> +> Both are first-PR requirements, not follow-ups. + +`validateConfigCandidate()` must inspect the raw candidate before `.catch()` can erase the invalid value. Reject: + +- unknown mode; +- non-object model map; +- blank model key; +- accessor/prototype-polluted object; +- non-string map value. + +Example error: + +```text +schema_invalid: providers.deepseek.modelRoutedToolDiscovery.glm-5.2: +must be auto, deferred, or direct +``` + +## Precedence tests + +| Provider | Model | Expected | Source | +|---|---|---|---| +| unset | unset | deferred | default | +| auto | unset | deferred | provider | +| direct | unset | direct | provider | +| direct | deferred | deferred | model | +| deferred | direct | direct | model | +| deferred | auto | deferred | model auto | +| any | any on Cursor | direct | hard fence | + +## Compatibility note + +Do not name the field `supportsSearchTool`. That would preserve the original ambiguity between hosted web search and tool discovery. The config name must describe the actual product decision. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/012_phase1_catalog_patch.md b/devlog/_plan/260813_routed_tool_discovery_profiles/012_phase1_catalog_patch.md new file mode 100644 index 000000000..bdd6fd419 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/012_phase1_catalog_patch.md @@ -0,0 +1,130 @@ +# 012 - Phase 1: catalog patch + +## New pure module + +Recommended file: + +```text +src/codex/catalog/tool-discovery.ts +``` + +Owned functions: + +```ts +resolveConfiguredRoutedToolDiscoveryMode(providerName, provider, modelId) +applyRoutedToolDiscoveryPolicy(entry, resolvedMode) +deriveComboToolDiscoveryMode(memberModes) +``` + +The executable reference implementation is included at: + +```text +prototype/mvp-resolver.mjs +``` + +## Template path + +Current: + +```ts +normalizeRoutedCatalogEntry(e, model?.parallelToolCalls === true); +``` + +Proposed. The third argument becomes an options object rather than another +positional boolean, because the fence now needs the provider identity as well as +the mode, and `normalizeRoutedCatalogEntry` has public callers (`src/codex/catalog.ts` +re-exports it, and `tests/parallel-tool-calls-optin.test.ts` calls it with one and +two arguments). Keeping both existing positions intact preserves those callers: + +```ts +normalizeRoutedCatalogEntry( + e, + model?.parallelToolCalls === true, + { + toolDiscoveryMode: model?.toolDiscoveryMode ?? "deferred", + providerId: model?.provider, + }, +); +``` + +Inside normalization: + +```ts +// Provider identity first; the slug prefix is only a fallback for callers that +// have no CatalogModel (see the fence note below). +const isCursorEntry = isCursorRoute(entry, options?.providerId); +const effective = isCursorEntry ? "direct" : (options?.toolDiscoveryMode ?? "deferred"); + +applyRoutedCodexToolMode(entry); +entry.supports_search_tool = effective === "deferred"; +``` + +The shared helper, used by both construction paths: + +```ts +export function isCursorRoute(entry: RawEntry, providerId?: string): boolean { + if (providerId !== undefined) return providerId === "cursor"; + return typeof entry.slug === "string" && entry.slug.startsWith("cursor/"); +} +``` + +With no `CatalogModel` the slug prefix remains the only available signal, which is +why it stays as the fallback rather than being deleted. + +> **Amended 2026-08-13 (`094` Correction 8).** The original draft here reproduced +> `entry.slug.startsWith("cursor/")` verbatim. That is exactly the asymmetry the +> unresolved #1596 P2 review flags: the template path fences on the public slug +> while the template-less path fences on `model?.provider === "cursor"`, so a +> `cursor/`-aliased combo whose canonical provider is `combo` is classified +> differently depending on whether a template happened to be available. +> +> Since this unit already unifies both paths behind one resolver, it must close +> the asymmetry rather than inherit it. Both paths call one shared helper that +> resolves from provider identity when a `CatalogModel` is present and falls back +> to the slug prefix only when it is not. + +Hosted search stays independent: + +```ts +if (isCursorEntry) delete entry.web_search_tool_type; +else entry.web_search_tool_type = "text_and_image"; +``` + +## Template-less path + +The fallback in `src/codex/catalog/sync.ts` currently duplicates policy. It must +consume the same resolved value **and the same fence helper** — using +`isCursorFallback` here is what created the asymmetry in the first place: + +```ts +const isCursorEntry = isCursorRoute(entry, model?.provider); +const mode = isCursorEntry + ? "direct" + : model?.toolDiscoveryMode ?? "deferred"; +``` + +`isCursorRoute()` collapses the old `isCursorFallback` +(`model?.provider === "cursor"`) and the old template-path slug check into one +expression, which is the point of Correction 8: both paths must reach the same +verdict for the same row. + +Regression tests must cover both paths because #1596 already established this as a dual seam. + +## Combo path + +A single public combo row cannot switch catalog capabilities after selecting a target. Use the conservative composition rule: + +```text +if any member is direct -> combo direct +otherwise -> combo deferred +``` + +This prevents a combo from advertising deferred discovery when one possible target was explicitly marked incompatible. + +## Native rows + +No change. Native OpenAI rows keep the upstream snapshot and their existing capability metadata. + +## Serialized extensions + +Do not serialize an OpenCodex-only `tool_discovery_mode` field into the Codex model catalog unless the client explicitly tolerates it. The internal `CatalogModel` field is enough; only standard Codex fields are emitted. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/013_phase1_sync_and_fingerprint.md b/devlog/_plan/260813_routed_tool_discovery_profiles/013_phase1_sync_and_fingerprint.md new file mode 100644 index 000000000..ca31fde7c --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/013_phase1_sync_and_fingerprint.md @@ -0,0 +1,80 @@ +# 013 - Phase 1: propagation, synchronization and cache identity + +## Propagation chain + +> **Amended 2026-08-13:** this chain is modeled on `parallelToolCalls`, which is a +> complete precedent for the **template** path only. The template-less fallback +> never passes it through normalization, and `ensureStrictCatalogFields` +> (`parsing.ts:293-306`) defaults a missing flag to `true`. The new field must +> emit explicitly on both construction paths and must never rely on a +> strict-field default (`094_landing_verification_pass.md` Correction 7). + +```text +config.providers[name] + -> applyProviderConfigHints(name, provider, model) + -> CatalogModel.toolDiscoveryMode + -> combo derivation if applicable + -> deriveEntry() + -> normalizeRoutedCatalogEntry() + -> supports_search_tool +``` + +Each hop must be deterministic and side-effect free. + +## Provider hint seam + +`applyProviderConfigHints()` is the preferred resolver call site because it already handles provider and model overrides. Add: + +```ts +const toolDiscoveryMode = resolveConfiguredRoutedToolDiscoveryMode( + name, + prov, + model.id, +); +``` + +and include it in the returned `CatalogModel`. + +Use `modelRecordValue()` for the per-model map so the override matches exactly +the way every sibling model-keyed override does. + +> **Amended 2026-08-13:** the original sentence claimed dated variants follow the +> same behavior. They do not — `modelRecordValue()` (`src/reasoning-effort.ts:49-62`) +> matches exact own-property, the prefix before a `:`, and case-insensitively; it +> does **not** match `-YYYYMMDD` suffixes (that is `isDatedVariantId()` at +> `provider-fetch.ts:805-808`). A `glm-5.2` key will not cover `glm-5.2-20260813`. +> See `094_landing_verification_pass.md` Correction 4. + +## Catalog gather identity + +Add both fields to `providerCatalogFingerprint()`: + +```ts +toolDiscovery: prov.routedToolDiscovery ?? null, +modelToolDiscovery: prov.modelRoutedToolDiscovery ?? null, +``` + +The newer provider-graph identity already hashes the admitted provider row, but the explicit fingerprint remains valuable because: + +- it documents output-affecting state; +- it protects older/isolated paths; +- it creates a focused regression assertion; +- it avoids a future refactor accidentally omitting the field. + +## Cache test + +Start two concurrent gathers with identical transport/model lists but different discovery overrides. They must not join the same in-flight promise and must produce different catalog flags. + +## Sync test + +Run catalog sync twice: + +1. provider default deferred; +2. change only one model to direct. + +Expected: + +- one row changes `supports_search_tool`; +- native rows are byte-identical; +- sibling routed rows are byte-identical; +- no stale `models_cache.json` result survives invalidation. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/014_phase1_diagnostics.md b/devlog/_plan/260813_routed_tool_discovery_profiles/014_phase1_diagnostics.md new file mode 100644 index 000000000..9570e3809 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/014_phase1_diagnostics.md @@ -0,0 +1,68 @@ +# 014 - Phase 1: diagnostics and operator feedback + +## Why diagnostics are required + +A direct override solves compatibility at the cost of potentially large first requests. Silent operation would repeat the same observability failure as #1529, only at smaller scope. + +## Recommended management DTO + +> **Amended 2026-08-13:** the DTO reports the mode OpenCodex *advertises*, which +> is not necessarily the mode the client *applies*. Upstream gates deferred +> discovery on `supports_search_tool && namespace_tools_enabled` +> (`spec_plan.rs:330`), so a route can run direct without any override. Word the +> diagnostic as advertised policy, never as effective runtime state +> (`094_landing_verification_pass.md` Correction 1). + +Expose a derived, non-secret object per catalog row/provider model: + +```json +{ + "resolvedToolDiscovery": { + "mode": "direct", + "source": "model-override", + "reason": "providers.deepseek.modelRoutedToolDiscovery.glm-5.2", + "warning": "Direct discovery may include full MCP declarations" + } +} +``` + +This is management-only metadata; it need not be written into the Codex catalog. + +## CLI output + +Suggested command extension: + +```text +ocx models explain deepseek/glm-5.2 +``` + +Minimum output: + +```text +Tool discovery: direct +Resolved from: providers.deepseek.modelRoutedToolDiscovery.glm-5.2 +Code mode: code_mode_only +Hosted web search: enabled +Warning: direct discovery can increase the first-request payload +``` + +## Warning thresholds + +When request-build instrumentation is available, warn when direct mode sees either: + +- more than 100 nested tools; +- more than 128 KiB of declaration/schema text; +- more than 256 KiB total first-request body. + +These are initial operational thresholds, not API limits. Calibrate from real captures before enforcing a hard failure. + +## Privacy + +Do not log: + +- full tool schemas; +- tool arguments; +- credentials or headers; +- user prompt content. + +Safe telemetry is counts, byte sizes, hashes, selected mode and source. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/015_phase1_review_checklist.md b/devlog/_plan/260813_routed_tool_discovery_profiles/015_phase1_review_checklist.md new file mode 100644 index 000000000..a94d287c7 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/015_phase1_review_checklist.md @@ -0,0 +1,40 @@ +# 015 - Phase 1 review checklist + +## Scope + +- [ ] Only route policy, config, catalog propagation, tests and docs. +- [ ] No adapter behavior change. +- [ ] No MCP execution path change. +- [ ] No new network request. + +## Correctness + +- [ ] Non-Cursor default remains deferred. +- [ ] Cursor remains direct under every override. +- [ ] Model override wins provider override. +- [ ] `auto` is resolved before serialization. +- [ ] Hosted web-search metadata remains independent. +- [ ] Combo direct member forces direct combo row. +- [ ] Template and fallback paths share one policy. + +## Config safety + +- [ ] Invalid hand edit does not reset providers or API keys. +- [ ] Live write rejects invalid values. +- [ ] Prototype-polluted maps fail closed. +- [ ] Display/diagnostic surfaces redact provider/model keys when needed. + +## Cache safety + +- [ ] Policy fields participate in gather identity. +- [ ] Changing an override invalidates relevant catalog cache. +- [ ] Unrelated rows do not churn. + +## Tests + +- [ ] typecheck. +- [ ] focused catalog tests. +- [ ] config load/write tests. +- [ ] combo tests. +- [ ] full suite. +- [ ] privacy scan. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/020_phase2_unit_tests.md b/devlog/_plan/260813_routed_tool_discovery_profiles/020_phase2_unit_tests.md new file mode 100644 index 000000000..b46fe4110 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/020_phase2_unit_tests.md @@ -0,0 +1,69 @@ +# 020 - Phase 2: unit and integration test plan + +Status: READY TO IMPLEMENT AFTER PR A + +> **Added 2026-08-13 (audit blocker 1).** One case in this phase is load-bearing +> for the whole unit's rationale and must not be skipped: +> +> **Single-variable code-mode differential.** Under otherwise byte-identical +> `tool_mode = code_mode_only` conditions, toggle **only** `supports_search_tool` +> and assert an eligible MCP tool remains callable in both states. The expected +> delta is `exec.description` content and size **plus** `tool_search` +> construction and the deferred-guidance text — do not assert the difference is +> confined to `exec.description`. Until this runs, the claim in `004`/`094` that +> `direct` is a comprehension lever rather than a reachability fix rests on a +> source reading of a 2026-07-23 upstream clone rather than on an executed +> differential. +> +> Two further cases from the same audit: +> +> - the load-path warning fires when a malformed discovery value is degraded; +> - the write-boundary validator rejects accessor/prototype-polluted model maps +> *without* first reading the attacker-controlled property. + +## Test pyramid + +### Layer 1 — pure resolver + +Fast table-driven tests with no filesystem or network. + +### Layer 2 — catalog serialization + +Exercise native template, routed template, template-less fallback, combo and Cursor rows. + +### Layer 3 — config load/write + +Prove load degradation and strict write rejection. + +### Layer 4 — catalog gather/cache + +Exercise provider hints, model matching, concurrency and cache invalidation. + +### Layer 5 — repository E2E-style smoke + +Build a catalog from a representative OpenCodex config and assert only intended rows change. + +## Proposed test files + +```text +tests/codex-tool-discovery-mode.test.ts +tests/catalog-cursor-search.test.ts # extend +tests/codex-catalog.test.ts # extend +tests/config.test.ts # extend +tests/config-user-edits.test.ts # extend +tests/e2e-style/phase100-native-parity.test.ts +``` + +## Local executable proof in this bundle + +```bash +node --test prototype/tool-discovery-profile.test.mjs +``` + +Recorded result: + +```text +17 tests, 17 passed, 0 failed +``` + +This validates the policy prototype only. It does not replace the Bun repository suite. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/021_catalog_test_cases.md b/devlog/_plan/260813_routed_tool_discovery_profiles/021_catalog_test_cases.md new file mode 100644 index 000000000..a421f596e --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/021_catalog_test_cases.md @@ -0,0 +1,38 @@ +# 021 - Catalog test cases + +| ID | Input | Expected | +|---|---|---| +| C01 | non-Cursor, no override | `code_mode_only`, search true, hosted search present | +| C02 | non-Cursor provider direct | `code_mode_only`, search false, hosted search present | +| C03 | provider direct + model deferred | target model true; siblings false | +| C04 | provider deferred + model direct | target model false; siblings true | +| C05 | Cursor no override | search false; hosted search absent | +| C06 | Cursor configured deferred | still false; warning/diagnostic | +| C07 | native OpenAI row | unchanged from snapshot | +| C08 | template-less non-Cursor | same policy as template path | +| C09 | template-less Cursor | same hard fence | +| C10 | combo all deferred | combo true | +| C11 | combo one direct | combo false | +| C12 | bare combo alias | still treated as routed and policy applied | +| C13 | account-qualified native row | unchanged | +| C14 | model date variant | override resolved via existing model helper | +| C15 | model alias mismatch | no accidental sibling match | + +## Exact assertion style + +Avoid broad snapshots as the only fence. Pin the load-bearing pair explicitly: + +```ts +expect(row.tool_mode).toBe("code_mode_only"); +expect(row.supports_search_tool).toBe(true); +``` + +For direct mode: + +```ts +expect(row.tool_mode).toBe("code_mode_only"); +expect(row.supports_search_tool).toBe(false); +expect(row.web_search_tool_type).toBe("text_and_image"); +``` + +The third assertion prevents hosted search from being accidentally coupled to discovery mode. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/022_config_and_precedence_tests.md b/devlog/_plan/260813_routed_tool_discovery_profiles/022_config_and_precedence_tests.md new file mode 100644 index 000000000..d58031373 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/022_config_and_precedence_tests.md @@ -0,0 +1,59 @@ +# 022 - Config and precedence tests + +## Valid candidates + +```json +{ "routedToolDiscovery": "auto" } +{ "routedToolDiscovery": "deferred" } +{ "routedToolDiscovery": "direct" } +{ "modelRoutedToolDiscovery": { "glm-5.2": "direct" } } +``` + +## Invalid live writes + +Reject: + +```json +{ "routedToolDiscovery": "eager" } +{ "modelRoutedToolDiscovery": [] } +{ "modelRoutedToolDiscovery": { "": "direct" } } +{ "modelRoutedToolDiscovery": { "glm-5.2": false } } +``` + +## Hand-edited load recovery + +A malformed optional field should be ignored while preserving: + +- provider adapter and base URL; +- API key pool; +- default provider; +- port and hostname; +- all unrelated providers. + +Test both the raw diagnostics path and load→mutate→save round trip. + +## Precedence fixture + +Use one provider with three models: + +```json +{ + "routedToolDiscovery": "direct", + "modelRoutedToolDiscovery": { + "a": "deferred", + "b": "auto" + } +} +``` + +Expected: + +| Model | Result | +|---|---| +| a | deferred | +| b | deferred (`auto` default) | +| c | direct (provider) | + +## Security fixture + +Construct a null-prototype map and explicit own `__proto__` key. Validation must not permit prototype pollution or silently rewrite the target model set. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/023_backward_compatibility_tests.md b/devlog/_plan/260813_routed_tool_discovery_profiles/023_backward_compatibility_tests.md new file mode 100644 index 000000000..3ba92ee43 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/023_backward_compatibility_tests.md @@ -0,0 +1,31 @@ +# 023 - Backward-compatibility tests + +## Zero-config byte compatibility + +Build the same catalog on current `dev` and patched code with no new fields. Compare normalized JSON. + +Allowed differences: + +- none in discovery-owned fields; +- no ordering change; +- no new serialized OpenCodex extension. + +## Existing direct Cursor behavior + +Existing Cursor tests must remain unchanged: + +- no `web_search_tool_type`; +- `supports_search_tool=false`; +- parallel tool calls remain as currently advertised. + +## PR #1596 regression fence + +The original focused tests must still pass. Add one explicit test proving that an ordinary provider with no override remains true even after resolver introduction. + +## Existing configs + +Load fixtures from before the new fields existed. Their parsed output should not gain persisted fields on a no-op read. Saving an unrelated field should not write an explicit default unless OpenCodex normally materializes optional defaults. + +## Downgrade behavior + +An older OpenCodex binary sees unknown provider fields through `.passthrough()` and should preserve them during unrelated config saves. Verify against the current schema strategy where practical. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/024_catalog_cache_identity_tests.md b/devlog/_plan/260813_routed_tool_discovery_profiles/024_catalog_cache_identity_tests.md new file mode 100644 index 000000000..7601c6b26 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/024_catalog_cache_identity_tests.md @@ -0,0 +1,41 @@ +# 024 - Catalog cache and concurrency tests + +## Why + +`gatherRoutedModels()` deduplicates concurrent work. If discovery policy is omitted from its identity, a direct request can receive a deferred catalog—or the reverse—depending on which call entered first. + +## Test A — fingerprint divergence + +Create two configs differing only in: + +```json +"routedToolDiscovery": "deferred" +``` + +versus: + +```json +"routedToolDiscovery": "direct" +``` + +Assert gather keys or returned rows differ. + +## Test B — model-map divergence + +Same provider-wide mode; change one model override. Assert only the matching row changes. + +## Test C — concurrent flights + +Block provider discovery behind a test promise, start both configs concurrently, then release. They must execute distinct admissions and publish their own policy. + +## Test D — cache refresh + +1. gather deferred; +2. cache result; +3. mutate config to direct; +4. gather again; +5. ensure stale cached row is not reused. + +## Test E — warning memo lifecycle + +If direct-mode payload warnings are memoized, config generation reconciliation must clear stale signatures when the effective policy changes. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/025_combo_policy_tests.md b/devlog/_plan/260813_routed_tool_discovery_profiles/025_combo_policy_tests.md new file mode 100644 index 000000000..23c899949 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/025_combo_policy_tests.md @@ -0,0 +1,41 @@ +# 025 - Combo policy tests + +## Composition rule + +```ts +members.some(member => member.toolDiscoveryMode === "direct") + ? "direct" + : "deferred"; +``` + +## Cases + +| Members | Expected combo | +|---|---| +| deferred + deferred | deferred | +| deferred + direct | direct | +| direct + direct | direct | +| undefined + deferred | deferred during migration | +| undefined + direct | direct | + +## Rationale + +A combo catalog row is selected before the concrete target is known. Advertising deferred while a possible target requires direct can make that target unusable. Direct is more expensive but compatible with both member classes, so it is the conservative intersection. + +## Diagnostics + +The combo explain surface should say which member forced direct mode: + +```text +combo/mixed -> direct +reason: member deepseek/glm-5.2 is configured direct +``` + +## Alias coverage + +Run the same cases for: + +- normal `combo/` slug; +- bare alias; +- slashed alias; +- explicit native alias where allowed. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/029_phase2_exit_gate.md b/devlog/_plan/260813_routed_tool_discovery_profiles/029_phase2_exit_gate.md new file mode 100644 index 000000000..f924c19ca --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/029_phase2_exit_gate.md @@ -0,0 +1,15 @@ +# 029 - Phase 2 exit gate + +Phase 2 is complete only when: + +- [ ] pure resolver tests pass; +- [ ] template and fallback catalog tests pass; +- [ ] config load/write tests pass; +- [ ] combo composition tests pass; +- [ ] concurrent gather identity tests pass; +- [ ] zero-config catalog comparison is clean; +- [ ] full Bun suite passes; +- [ ] no privacy scan regression; +- [ ] direct mode warning is observable. + +A passing unit suite without the zero-config comparison is insufficient: the primary promise of PR A is that #1596 remains the default. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/030_phase3_protocol_conformance.md b/devlog/_plan/260813_routed_tool_discovery_profiles/030_phase3_protocol_conformance.md new file mode 100644 index 000000000..7a7fa6763 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/030_phase3_protocol_conformance.md @@ -0,0 +1,50 @@ +# 030 - Phase 3: protocol conformance programme + +Status: SEPARATE PR STACK + +## Objective + +Determine whether each adapter can support native tool discovery rather than inferring capability from a model name or provider label. + +## Conformance identity + +Every result must be scoped by: + +```text +client surface + client version + adapter + upstream protocol + OpenCodex version +``` + +Example: + +```text +codex-app/26.803.61601 +openai-responses -> anthropic translator +opencodex/2.13.x +``` + +## Required dimensions + +- request tool declaration; +- model tool call; +- tool result replay; +- next-turn activation; +- streaming/non-streaming; +- continuation; +- compaction; +- resume; +- namespace and custom type preservation; +- transient tool handling. + +## Result states + +```text +UNPROBED -> PROBED -> VERIFIED + \-> DEGRADED + \-> FAILED +``` + +No automatic profile selection should depend on less than VERIFIED evidence. + +## Artifact output + +Each suite writes a machine-readable JSON result and a human-readable MD report. The JSON includes fixture hash, versions, pass counts, failure stage and expiry timestamp. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/031_responses_lite_additional_tools.md b/devlog/_plan/260813_routed_tool_discovery_profiles/031_responses_lite_additional_tools.md new file mode 100644 index 000000000..287dcc5d3 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/031_responses_lite_additional_tools.md @@ -0,0 +1,51 @@ +# 031 - Responses Lite `additional_tools` scenarios + +## Regression class + +Codex Responses Lite can place tool definitions under: + +```json +{ + "input": [ + { + "type": "additional_tools", + "role": "developer", + "tools": [] + } + ] +} +``` + +A translator that reads only top-level `tools` silently strips the complete execution surface. + +## Request tests + +1. top-level tools only; +2. `additional_tools` only; +3. both sources; +4. multiple `additional_tools` items; +5. duplicate names across sources; +6. custom + function + namespace in one item; +7. empty items; +8. malformed item must fail explicitly. + +## Merge rule + +- stable declaration order; +- top-level definition wins an exact collision; +- deduplicate by translated qualified name; +- preserve original tool kind in a side map. + +## End-to-end assertion + +A prompt requesting `pwd` must produce: + +```text +assistant tool call -> client execution -> tool output -> second model request -> final answer +``` + +A normal `response.completed` after only “I will run pwd” is a failure, not success. + +## Fixtures + +Use the exact Responses Lite shape from the CLIProxyAPI regression as one fixture, with secrets and provider identifiers removed. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/032_custom_namespace_roundtrip.md b/devlog/_plan/260813_routed_tool_discovery_profiles/032_custom_namespace_roundtrip.md new file mode 100644 index 000000000..ab7b40269 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/032_custom_namespace_roundtrip.md @@ -0,0 +1,47 @@ +# 032 - Custom and namespace round-trip scenarios + +## Tool kinds + +### Function + +Structured JSON arguments. Must return a normal function call. + +### Custom/freeform + +Example: `exec` or `apply_patch`. The translator may wrap freeform text as: + +```json +{ "input": "raw source text" } +``` + +but must remember the original type and restore a custom tool call to Codex. + +### Namespace + +Must preserve both: + +```text +namespace + function name +``` + +Flattening to one string is acceptable only inside protocols that require it, and the reverse mapping must be exact. + +## Collision tests + +- plain `search` and namespace `github.search`; +- two namespaces with the same short name; +- custom and function sharing a translated alias; +- exact qualified name versus suffix fallback. + +Plain tools should win plain-name resolution. Namespace calls require their original namespace; never guess from a suffix when two candidates exist. + +## Response tests + +For streaming and non-streaming: + +- original custom -> custom event; +- original function -> function event; +- namespace metadata restored; +- stable call id; +- fragmented arguments reconstructed once; +- no duplicate tool execution. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/033_tool_search_history_and_compaction.md b/devlog/_plan/260813_routed_tool_discovery_profiles/033_tool_search_history_and_compaction.md new file mode 100644 index 000000000..f6348fe83 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/033_tool_search_history_and_compaction.md @@ -0,0 +1,38 @@ +# 033 - Tool-search history, compaction and resume + +## History items to preserve + +- `tool_search_call`; +- `tool_search_output`; +- discovered qualified names; +- original tool kinds; +- call ids; +- namespace metadata. + +## Continuation scenario + +1. model searches for Browser screenshot tool; +2. client returns search output; +3. model calls discovered tool; +4. client returns result; +5. model answers; +6. next user turn calls the same tool again. + +The second turn must not lose the discovered identity or mis-serialize it as an unrelated function. + +## Compaction scenario + +Compact after step 3 and continue. Either: + +- the discovered schema/reference is restored safely; or +- the client re-describes the exact tool from a stable index. + +It must not reconstruct an argument schema from model memory. + +## Resume scenario + +Restart the process and resume the transcript. Reconcile persisted references against the current tool manifest. Missing tools should return a structured unavailable result, not poison every later request. + +## Transient tools + +Startup-only tools such as connection waiters must never become durable search references. Filter them before persistence or tag/reconcile them explicitly. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/034_streaming_nonstreaming_matrix.md b/devlog/_plan/260813_routed_tool_discovery_profiles/034_streaming_nonstreaming_matrix.md new file mode 100644 index 000000000..4848212c5 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/034_streaming_nonstreaming_matrix.md @@ -0,0 +1,30 @@ +# 034 - Streaming and non-streaming matrix + +| Stage | Streaming assertions | Non-streaming assertions | +|---|---|---| +| declaration | same translated tool set | same translated tool set | +| call start | one output item/call id | one call object/call id | +| arguments | ordered fragments, valid final JSON/freeform | exact arguments | +| call end | exactly one terminal event | completed call | +| tool output | paired by call id | paired by call id | +| final response | loop continues | loop continues | +| error | structured, no fake completion | structured, no fake completion | + +## Fragmentation cases + +- UTF-8 character split across chunks; +- JSON string escape split; +- two parallel calls interleaved; +- one custom freeform call; +- reasoning/text before tool call; +- tool call followed by transport error. + +## Equality check + +Normalize both modes into canonical `OcxToolCall` objects and compare: + +```text +name, namespace, custom type, arguments, call id +``` + +Differences in event timing are allowed. Differences in semantic tool identity are not. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/035_failure_injection.md b/devlog/_plan/260813_routed_tool_discovery_profiles/035_failure_injection.md new file mode 100644 index 000000000..7ee623b6d --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/035_failure_injection.md @@ -0,0 +1,30 @@ +# 035 - Protocol failure injection + +## Injected failures + +1. translator drops `additional_tools`; +2. translator maps custom to function; +3. namespace map missing on replay; +4. tool-search output omitted on next turn; +5. duplicate call id; +6. tool result arrives before declaration; +7. transient reference disappears; +8. compaction removes loaded schema; +9. upstream returns ordinary text instead of tool call; +10. SSE closes after call start. + +## Required behavior + +OpenCodex must distinguish: + +- model chose not to use a tool; +- route could not advertise the tool; +- translator deleted the tool; +- protocol call failed; +- tool execution failed. + +Only the first is a normal model completion. The others require explicit diagnostics or fallback. + +## Fail-closed rule + +If the route has deferred declarations but no usable discovery mechanism, select a safer profile before sending the request. Do not forward an internally inconsistent request and hope the model compensates. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/039_phase3_exit_gate.md b/devlog/_plan/260813_routed_tool_discovery_profiles/039_phase3_exit_gate.md new file mode 100644 index 000000000..123d0f9a1 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/039_phase3_exit_gate.md @@ -0,0 +1,17 @@ +# 039 - Phase 3 exit gate + +A route may be marked native-tool-search compatible only when all are true: + +- [ ] top-level and Responses Lite declarations preserved; +- [ ] custom/function/namespace types round-trip; +- [ ] search call/output history replays; +- [ ] discovered tool activates on the next request; +- [ ] streaming and non-streaming are semantically equal; +- [ ] continuation passes; +- [ ] compaction passes; +- [ ] process resume passes; +- [ ] transient references are safe; +- [ ] failures are explicit; +- [ ] exact adapter/client version recorded. + +One successful first-turn tool call is not sufficient certification. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/040_phase4_live_e2e.md b/devlog/_plan/260813_routed_tool_discovery_profiles/040_phase4_live_e2e.md new file mode 100644 index 000000000..cdd874493 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/040_phase4_live_e2e.md @@ -0,0 +1,32 @@ +# 040 - Phase 4: live E2E programme + +Status: REQUIRES REAL CODEX CLIENTS AND CREDENTIALS + +## Purpose + +Unit tests prove serialization. Live E2E proves the client constructs the expected Code Mode registry and the selected model actually acquires and calls the tool. + +## Capture bundle per run + +```text +run.json # versions, profile, result +catalog-row.json # redacted selected model entry +request-metrics.json # counts/bytes/hashes, no prompt or credentials +all-tools.txt # names and bounded descriptions +transcript.redacted.jsonl +client.log +proxy.log +``` + +## Success definition + +- requested capability is reachable; +- model calls it without the user naming an internal function when testing discoverability; +- output is returned to the model; +- final answer uses the result; +- request size stays inside threshold; +- no silent fallback to a different model/provider. + +## Test hygiene + +Use a fresh task/session for every arm. Keep prompt, plugin set, model effort and repository constant. Record absolute versions and timestamps. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/041_code_mode_all_tools_canary.md b/devlog/_plan/260813_routed_tool_discovery_profiles/041_code_mode_all_tools_canary.md new file mode 100644 index 000000000..30cd43037 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/041_code_mode_all_tools_canary.md @@ -0,0 +1,38 @@ +# 041 - Code Mode `tools` / `ALL_TOOLS` canary + +## Preconditions + +- non-Cursor routed model; +- catalog row has `tool_mode=code_mode_only` and search true; +- at least one test MCP server exposing a harmless deterministic tool. + +Recommended tool: + +```text +mcp__ocx_canary__echo({"value":"CANARY_42"}) +``` + +## Arm A — exact internal name supplied + +Prompt directly names the tool. This verifies reachability, not discovery. + +## Arm B — capability-only prompt + +```text +Use the available canary capability to echo CANARY_42. Do not simulate it. +``` + +This verifies acquisition from the visible Code Mode metadata. + +## Arm C — search within `ALL_TOOLS` + +Ask the model to inspect the tool index by name/description and invoke the match. Capture whether it uses `exec` and the correct `tools.*` function. + +## Assertions + +- `ALL_TOOLS` contains the canary; +- `tools.mcp__ocx_canary__echo` exists; +- call executes once; +- result contains exact sentinel; +- final response includes sentinel; +- no direct/eager schema dump appears in the first request. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/042_codex_app_deepseek_browser.md b/devlog/_plan/260813_routed_tool_discovery_profiles/042_codex_app_deepseek_browser.md new file mode 100644 index 000000000..f547c8ab9 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/042_codex_app_deepseek_browser.md @@ -0,0 +1,58 @@ +# 042 - Exact #1522 scenario: Codex App + DeepSeek + Browser + +Priority: P0 + +## Fixed setup + +- current stable Codex App and bundled CLI version recorded; +- current OpenCodex `dev` build; +- Browser plugin enabled and authenticated; +- one DeepSeek-compatible routed model; +- same workspace and prompt for both arms. + +## Arm A — current default + +```json +{ + "tool_mode": "code_mode_only", + "supports_search_tool": true +} +``` + +## Arm B — exact model override + +```json +{ + "modelRoutedToolDiscovery": { + "": "direct" + } +} +``` + +## Prompt battery + +1. navigate to a deterministic local/static URL; +2. read page title; +3. take screenshot; +4. report one DOM fact; +5. perform one second browser action in the same session. + +Do not name `mcp__node_repl__js` in the discoverability run. + +## Capture + +- whether Browser tools appear in `ALL_TOOLS`; +- whether they exist on `tools`; +- first-request bytes; +- tool call sequence; +- model text when it fails; +- second-turn behavior. + +## Decision + +| Outcome | Action | +|---|---| +| A passes and is materially smaller | keep deferred; close evidence gap | +| A fails, B passes | document exact model/client direct override | +| both fail | problem is not discovery flag; inspect plugin/client lifecycle | +| A calls wrong tool | acquisition issue; evaluate meta-tool profile | diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/043_cursor_and_direct_bounded.md b/devlog/_plan/260813_routed_tool_discovery_profiles/043_cursor_and_direct_bounded.md new file mode 100644 index 000000000..130a3cb79 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/043_cursor_and_direct_bounded.md @@ -0,0 +1,40 @@ +# 043 - Cursor and bounded direct scenarios + +## Cursor regression + +Verify current behavior remains: + +- no deferred discovery advertisement; +- no hosted-search advertisement through the OpenCodex sidecar; +- existing Cursor MCP caps apply; +- parallel tool behavior unchanged. + +## Direct-mode bounds + +For a non-Cursor explicit direct override, test tool surfaces at: + +```text +10 / 50 / 100 / 250 / 500 tools +``` + +Record: + +- visible tool count; +- schema bytes; +- `exec.description` bytes; +- total request bytes; +- model latency to first tool call; +- whether the model chooses the correct tool. + +## Safety policy + +Initial PR may warn only. A later bounded-direct profile should enforce configured caps such as: + +```json +{ + "mcpMaxTools": 100, + "mcpMaxSchemaBytes": 131072 +} +``` + +When the catalog exceeds the bound, fail explicitly or route through meta-tools. Never silently truncate without `has_more`/diagnostics. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/044_weak_model_meta_tool_fallback.md b/devlog/_plan/260813_routed_tool_discovery_profiles/044_weak_model_meta_tool_fallback.md new file mode 100644 index 000000000..27226a8ae --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/044_weak_model_meta_tool_fallback.md @@ -0,0 +1,43 @@ +# 044 - Weak-model acquisition scenario + +## Hypothesis + +Some external models can call tools but do not reliably search `ALL_TOOLS` or compose Code Mode JavaScript. For them, a purpose-built search tool may outperform both local Code Mode discovery and an eager catalog. + +## Three-arm eval + +### A — loaded tool + +Required tool is directly visible. + +### B — Code Mode index + +Tool is available only through `ALL_TOOLS`/`tools`. + +### C — proxy meta-tools + +Tool is available through `ocx_tool_search` → `ocx_tool_describe` → `ocx_tool_call`. + +## Task battery + +Use at least 30 tasks across: + +- exact tool-name clue; +- semantic description only; +- namespace ambiguity; +- permission-blocked tool; +- two-step workflow; +- similar-name decoys. + +## Metrics + +- correct acquisition rate; +- wrong-tool rate; +- unnecessary user-question rate; +- invented workaround rate; +- false “not possible” rate; +- turns and bytes to successful call. + +## Selection rule + +Do not switch a model to meta-tools based on one anecdote. Require statistically meaningful improvement or a deterministic conformance failure. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/045_dynamic_mcp_refresh.md b/devlog/_plan/260813_routed_tool_discovery_profiles/045_dynamic_mcp_refresh.md new file mode 100644 index 000000000..fb8568bcd --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/045_dynamic_mcp_refresh.md @@ -0,0 +1,28 @@ +# 045 - Dynamic MCP `tools/list_changed` scenario + +## Setup + +Use an MCP server that starts with two tools and registers a third tool after the session begins, then emits `notifications/tools/list_changed`. + +## Sequence + +1. start session; +2. verify initial index; +3. register `dynamic_echo`; +4. emit list-changed; +5. search by exact name; +6. call tool; +7. remove tool; +8. search and call again. + +## Expected + +- side index refreshes without changing the top-level prompt manifest where possible; +- new tool becomes discoverable; +- removed tool produces structured unavailable result; +- no dangling transcript reference poisons later turns; +- cache prefix does not rebuild solely because the tool registry changed. + +## Instrumentation + +Record manifest hash before/after, index generation and cache-read/write token metrics when the client exposes them. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/046_compaction_resume_live.md b/devlog/_plan/260813_routed_tool_discovery_profiles/046_compaction_resume_live.md new file mode 100644 index 000000000..c3e5ad801 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/046_compaction_resume_live.md @@ -0,0 +1,26 @@ +# 046 - Live compaction and resume scenario + +## Sequence + +1. discover and call one deferred Browser/MCP tool; +2. grow context with deterministic text/tool results; +3. trigger compaction; +4. call the same tool; +5. close client; +6. resume session; +7. call the tool again; +8. reconnect or change MCP tool list; +9. issue one normal user turn. + +## Success + +- no schema reconstruction error; +- no missing namespace; +- no invalid call id; +- no permanent 400 loop; +- missing tool is reported as unavailable, not retained as a dead reference; +- the session can continue after a tool availability change. + +## Recovery drill + +Keep an offline copy of the transcript. If the session becomes poisoned, identify the exact durable reference that caused validation failure and document whether OpenCodex can reconcile it automatically. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/049_phase4_exit_gate.md b/devlog/_plan/260813_routed_tool_discovery_profiles/049_phase4_exit_gate.md new file mode 100644 index 000000000..55c1d8ee0 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/049_phase4_exit_gate.md @@ -0,0 +1,14 @@ +# 049 - Phase 4 exit gate + +- [ ] CLI Code Mode canary passes. +- [ ] App Code Mode canary passes. +- [ ] Exact #1522 A/B is captured. +- [ ] Cursor remains unchanged. +- [ ] direct-mode size curve is measured. +- [ ] weak-model three-arm eval is complete for target models. +- [ ] dynamic list change does not poison the session. +- [ ] compaction and resume pass. +- [ ] all captures are redacted and reproducible. +- [ ] each decision records exact versions and expiry. + +Until this gate passes, automatic route promotion remains out of scope. Explicit overrides are the safe control plane. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/050_phase5_payload_cache_benchmarks.md b/devlog/_plan/260813_routed_tool_discovery_profiles/050_phase5_payload_cache_benchmarks.md new file mode 100644 index 000000000..5b0a10b75 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/050_phase5_payload_cache_benchmarks.md @@ -0,0 +1,40 @@ +# 050 - Phase 5: payload and cache benchmarks + +## Questions + +1. How much first-request text does each profile add? +2. Does cost scale with tool count, description bytes or schema bytes? +3. Does loading a tool mutate the cached prefix? +4. What happens after compaction and dynamic tool changes? + +## Profiles + +- eager/direct full declarations; +- Code Mode name/description index; +- native tool search; +- proxy meta-tools. + +## Sizes + +Test 0, 10, 50, 100, 250, 500 and 1,000 tools. Use multiple schema shapes: + +- small flat function; +- nested object; +- enum-heavy; +- long descriptions; +- namespace groups; +- custom/freeform. + +## Metrics + +- request UTF-8 bytes; +- estimated/tokenizer tokens when available; +- cache read/write tokens; +- time to first model byte; +- time to first tool call; +- total turns to completion; +- acquisition success. + +## Included synthetic benchmark + +`prototype/payload-benchmark.mjs` compares structural UTF-8 bytes. It is not a live Codex capture or token estimate. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/051_benchmark_methodology.md b/devlog/_plan/260813_routed_tool_discovery_profiles/051_benchmark_methodology.md new file mode 100644 index 000000000..70503f280 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/051_benchmark_methodology.md @@ -0,0 +1,41 @@ +# 051 - Benchmark methodology and recorded synthetic result + +## Command + +```bash +node prototype/payload-benchmark.mjs 250 +node prototype/payload-benchmark.mjs 1000 +``` + +## 250-tool result + +| Shape | UTF-8 bytes | +|---|---:| +| eager full schemas | 179,218 | +| Code Mode names + descriptions | 52,393 | +| three fixed meta-tools | 744 | + +Ratios: + +- eager / Code Mode index: 3.421×; +- eager / meta-tools: 240.884×. + +## 1,000-tool result + +| Shape | UTF-8 bytes | +|---|---:| +| eager full schemas | 716,969 | +| Code Mode names + descriptions | 209,894 | +| three fixed meta-tools | 744 | + +## Interpretation limits + +These figures demonstrate scaling shape only. They do not reproduce: + +- Codex's complete instructions; +- actual plugin descriptions; +- OpenAI tokenizer behavior; +- prompt compression/caching; +- the exact #1596 measurement harness. + +Use the included JSON files as a local sanity check, then replace them with live fixed-harness captures before making product thresholds. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/052_acceptance_thresholds.md b/devlog/_plan/260813_routed_tool_discovery_profiles/052_acceptance_thresholds.md new file mode 100644 index 000000000..71335e523 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/052_acceptance_thresholds.md @@ -0,0 +1,35 @@ +# 052 - Initial acceptance thresholds + +## Default deferred profile + +- first-request declaration growth should be dominated by names/descriptions, not full schemas; +- 500 tools must not force all JSON Schemas into `exec.description`; +- a simple canary must remain callable. + +## Direct override + +Warning threshold proposal: + +```text +>100 tools OR >128 KiB schema bytes OR >256 KiB request body +``` + +Stop-ship threshold for an unbounded default: + +```text +Any zero-config change that makes request size proportional to full MCP schema bytes +``` + +## Meta-tools + +- always-visible declaration budget below 4 KiB; +- search default at least 10 results; +- exact qualified-name recall 100%; +- pagination signal required for incomplete results; +- describe maximum 3 schemas/call by default. + +## Acquisition + +For target models, deferred/meta profile should achieve at least 95% of loaded-tool success on deterministic tasks, or materially outperform direct mode on cost without unacceptable correctness loss. + +Thresholds are provisional until live data is collected. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/053_prompt_cache_scenarios.md b/devlog/_plan/260813_routed_tool_discovery_profiles/053_prompt_cache_scenarios.md new file mode 100644 index 000000000..89d4c9a2d --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/053_prompt_cache_scenarios.md @@ -0,0 +1,35 @@ +# 053 - Prompt-cache scenarios + +## Stable-manifest invariant + +Hash the model-visible top-level tool list for every request in one session. It should not change when: + +- an MCP server connects late; +- one deferred tool is loaded; +- a tool is temporarily unavailable; +- list-changed fires. + +Where a protocol mandates dynamic declarations, record the exact append/mutation and cache effect. + +## Cache chain assertion + +When provider usage exposes cache metrics: + +```text +expected next cache read ≈ previous cache read + previous cache creation +``` + +Large unexplained collapse indicates prefix mutation or TTL loss. + +## Scenarios + +- late LSP/tool availability; +- one-tool discovery; +- batch discovery of 1/3/5/10 tools; +- dynamic server reconnect; +- compaction boundary; +- idle inside short versus long cache TTL. + +## Product rule + +Do not optimize only turn-1 bytes while causing repeated full-prefix cache rebuilds later. Evaluate total session cost. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/059_phase5_exit_gate.md b/devlog/_plan/260813_routed_tool_discovery_profiles/059_phase5_exit_gate.md new file mode 100644 index 000000000..85895c842 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/059_phase5_exit_gate.md @@ -0,0 +1,10 @@ +# 059 - Phase 5 exit gate + +- [ ] fixed tool fixtures checked into tests; +- [ ] request-byte curves captured; +- [ ] tokenizer/token estimates captured where supported; +- [ ] cache mutation scenarios captured; +- [ ] total-session cost compared; +- [ ] warning thresholds justified; +- [ ] #1596 fixed-harness measurement reproduced or superseded; +- [ ] synthetic and live results clearly separated. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/060_phase6_meta_tool_design.md b/devlog/_plan/260813_routed_tool_discovery_profiles/060_phase6_meta_tool_design.md new file mode 100644 index 000000000..360dab503 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/060_phase6_meta_tool_design.md @@ -0,0 +1,36 @@ +# 060 - Phase 6: bounded MCP meta-tool fallback + +Status: DESIGN; DO NOT MIX INTO PR A + +## Purpose + +Provide a constant-size discovery surface for routes where native deferred discovery is unavailable or model compliance with Code Mode is poor. + +## Always-visible tools + +```text +ocx_tool_search +ocx_tool_describe +ocx_tool_call +``` + +The complete authorized catalog remains server-side. + +## Flow + +```text +model -> search short metadata + -> describe 1-3 selected schemas + -> call exact qualified name + -> normal authorization/execution/logging +``` + +## Why three tools + +- search output stays small and cache-stable; +- describe makes schema loading explicit and bounded; +- call can preserve permissions and tool identity without mutating the top-level tool array. + +## Non-goal + +This is not a second unrestricted code execution environment. `ocx_tool_call` dispatches through existing MCP authorization and validation. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/061_meta_tool_contract.md b/devlog/_plan/260813_routed_tool_discovery_profiles/061_meta_tool_contract.md new file mode 100644 index 000000000..0ec3a2a67 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/061_meta_tool_contract.md @@ -0,0 +1,56 @@ +# 061 - Meta-tool contract + +## `ocx_tool_search` + +Input: + +```json +{ + "query": "browser screenshot", + "namespace": "browser", + "limit": 10, + "cursor": null +} +``` + +Output: + +```json +{ + "matches": [ + { + "name": "mcp__browser__take_screenshot", + "namespace": "mcp__browser", + "shortName": "take_screenshot", + "description": "Capture the current page" + } + ], + "hasMore": false, + "nextCursor": null, + "indexGeneration": 7 +} +``` + +No full JSON Schema in search output. + +## `ocx_tool_describe` + +Input contains one to three exact qualified names. Output returns schemas plus manifest/index generation. Unknown names return per-item errors. + +## `ocx_tool_call` + +Input: + +```json +{ + "name": "mcp__browser__take_screenshot", + "arguments": { "fullPage": true }, + "expectedIndexGeneration": 7 +} +``` + +Output uses the normal MCP result shape, bounded by existing result limits. + +## Versioning + +Every response carries contract version and index generation. Breaking changes require a new versioned tool name or negotiated field. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/062_meta_tool_security.md b/devlog/_plan/260813_routed_tool_discovery_profiles/062_meta_tool_security.md new file mode 100644 index 000000000..158684470 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/062_meta_tool_security.md @@ -0,0 +1,36 @@ +# 062 - Meta-tool security model + +## Authorization + +Search, describe and call must all operate on the caller's filtered authorized catalog. A search result must never reveal a tool the same caller cannot describe or invoke. + +## Dispatch + +`ocx_tool_call` must reuse the existing execution path after authorization. Do not create a local-registry shortcut that bypasses: + +- per-key server allowlists; +- per-tool permissions; +- IP/network policy; +- pre-call hooks/guardrails; +- logging and rate limits. + +## Name handling + +- exact qualified name required for call; +- no suffix fallback on ambiguous names; +- normalize once, then compare canonical names; +- reject control characters and oversized names; +- protect against `__proto__`, constructor and prototype keys. + +## Input/result bounds + +- search query length cap; +- maximum result count; +- describe maximum tools and schema bytes; +- call argument byte/depth limits; +- tool-result byte/token limit; +- timeouts and cancellation. + +## Prompt injection + +Tool descriptions and results are untrusted data. The meta-tool wrapper should label them as data, preserve provenance and avoid presenting server text as system authority. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/063_meta_tool_ranking.md b/devlog/_plan/260813_routed_tool_discovery_profiles/063_meta_tool_ranking.md new file mode 100644 index 000000000..1542a688b --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/063_meta_tool_ranking.md @@ -0,0 +1,38 @@ +# 063 - Meta-tool ranking, recall and pagination + +## Ranking stages + +1. exact qualified name; +2. exact short name when unique; +3. namespace + prefix; +4. token/BM25 lexical score; +5. optional semantic rerank over a bounded candidate set. + +Exact matches always outrank semantic similarity. + +## Default result count + +Use at least 10, not 5. Similar tool families commonly push the correct result below five. + +## Pagination + +Every incomplete result must state: + +```json +{ "hasMore": true, "nextCursor": "..." } +``` + +The model must not interpret an unmarked partial result as an exhaustive catalog. + +## Evaluation + +Create a 100+ tool corpus with: + +- image/get/view/generate collisions; +- repeated `search` names across namespaces; +- abbreviations; +- Korean and English descriptions; +- exact qualified-name queries; +- semantic queries with no name overlap. + +Measure recall@5, @10, @20 and wrong-namespace rate. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/064_meta_tool_integration.md b/devlog/_plan/260813_routed_tool_discovery_profiles/064_meta_tool_integration.md new file mode 100644 index 000000000..a19e6f672 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/064_meta_tool_integration.md @@ -0,0 +1,25 @@ +# 064 - Meta-tool integration points + +## Registry + +Build one session-scoped filtered index from the same registered tools used by normal execution. Keep a generation counter and stable canonical names. + +## Code Mode + +The meta-tools can themselves be nested under `exec` or directly visible, depending on the client profile. Do not expose both the full direct catalog and meta-tools unless intentionally evaluating them. + +## Tool choice + +Honor existing `tool_choice` and allowed-tool predicates. A denied underlying tool must not become callable through `ocx_tool_call`. + +## History + +Persist only exact selected names and normal tool-call/result records. Search result bodies can be compacted or reissued from the current index. + +## Dynamic updates + +On `tools/list_changed`, rebuild/patch the side index and increment generation without mutating the top-level model tool manifest. + +## Observability + +Log query count, candidate count, selected qualified name, latency and result bytes. Do not log full descriptions, schemas or arguments by default. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/065_meta_tool_tests.md b/devlog/_plan/260813_routed_tool_discovery_profiles/065_meta_tool_tests.md new file mode 100644 index 000000000..fdfc1ab61 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/065_meta_tool_tests.md @@ -0,0 +1,38 @@ +# 065 - Meta-tool test plan + +## Search + +- exact name first; +- namespace filter; +- unique short name; +- ambiguous short name; +- top-k and pagination; +- stable order under ties; +- authorized subset only; +- dynamic generation update. + +## Describe + +- one and three names; +- over-limit request; +- schema-byte cap; +- removed tool; +- mixed valid/invalid names; +- custom/freeform schema representation. + +## Call + +- normal success; +- invalid arguments; +- denied server; +- denied tool; +- IP restriction; +- pre-call hook rejection; +- cancellation/timeout; +- result cap; +- namespace collision; +- stale generation. + +## E2E + +Run the same task through direct, Code Mode and meta-tool profiles. Assert final external side effect/result is identical and authorization decisions are identical. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/069_phase6_exit_gate.md b/devlog/_plan/260813_routed_tool_discovery_profiles/069_phase6_exit_gate.md new file mode 100644 index 000000000..b716fca19 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/069_phase6_exit_gate.md @@ -0,0 +1,13 @@ +# 069 - Phase 6 exit gate + +- [ ] constant declaration budget demonstrated; +- [ ] exact-name recall 100%; +- [ ] pagination implemented; +- [ ] describe bounded; +- [ ] normal authorization path reused; +- [ ] pre-call hooks verified; +- [ ] dynamic index refresh verified; +- [ ] compaction/resume verified; +- [ ] no ambiguous name fallback; +- [ ] direct and meta-tool results are behaviorally equivalent; +- [ ] security review completed. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/070_rollout_plan.md b/devlog/_plan/260813_routed_tool_discovery_profiles/070_rollout_plan.md new file mode 100644 index 000000000..ce43d4a22 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/070_rollout_plan.md @@ -0,0 +1,29 @@ +# 070 - Rollout plan + +## Stage 0 — docs and prototype + +Ship no runtime change. Review the mode vocabulary, resolver and test matrix. + +## Stage 1 — explicit override, default unchanged + +Land PR A. Non-Cursor remains deferred; Cursor remains direct. Overrides are config-only and diagnostics warn on direct mode. + +## Stage 2 — protocol conformance + +Land adapter fixtures and evidence reporting. Do not auto-select yet. + +## Stage 3 — live canaries + +Certify target client/adapter combinations, especially the exact #1522 pairing. + +## Stage 4 — bounded meta-tools + +Opt in only for models/routes with demonstrated acquisition benefit or native incompatibility. + +## Stage 5 — evidence-driven auto mode + +Only after versioned evidence exists may `auto` choose among profiles. Keep explicit override as final authority. + +## Compatibility promise + +Every stage is additive. Removing optional config/evidence restores #1596 defaults. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/071_observability.md b/devlog/_plan/260813_routed_tool_discovery_profiles/071_observability.md new file mode 100644 index 000000000..95be69b3f --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/071_observability.md @@ -0,0 +1,41 @@ +# 071 - Observability plan + +## Per catalog row + +- resolved mode/profile; +- resolution source; +- policy/evidence revision; +- client/surface scope when known. + +## Per request + +- profile; +- top-level visible tool count; +- nested/indexed tool count; +- declaration bytes; +- total request bytes; +- manifest hash; +- search/describe counts; +- first tool-call latency; +- final success/failure stage. + +## Per session + +- manifest changes; +- compactions; +- cache read/write totals; +- unavailable/dangling tool references; +- profile switches—normally zero. + +## Redaction + +Store counts/hashes, not schemas, arguments, credentials or user prompts. Provide an explicit diagnostic capture command for users who consent to a redacted bundle. + +## Alert candidates + +- zero tools after translation when input declared tools; +- direct mode over warning thresholds; +- top-level manifest hash changes mid-session; +- repeated tool-search misses; +- normal completion immediately after a declared mandatory tool task; +- resume validation loop. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/072_canary_matrix.md b/devlog/_plan/260813_routed_tool_discovery_profiles/072_canary_matrix.md new file mode 100644 index 000000000..a7a81a200 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/072_canary_matrix.md @@ -0,0 +1,25 @@ +# 072 - Canary matrix + +| Client | Adapter | Model family | Plugin/MCP | Profile | Cadence | +|---|---|---|---|---|---| +| Codex CLI | openai-chat | Kimi | canary MCP | Code Mode | every release | +| Codex App | openai-chat | DeepSeek | Browser | Code Mode | every App/CLI change | +| Codex App | openai-chat | DeepSeek | Browser | direct override | control arm | +| Codex CLI | anthropic | Claude-compatible | canary MCP | Code Mode | every translator change | +| Codex CLI | Responses Lite bridge | GLM | exec/custom | native/conformance | every bridge change | +| Codex App | Cursor | Browser | direct | every Cursor transport change | +| Codex CLI | weak model set | synthetic catalog | meta-tools | candidate releases | + +## Sentinel rules + +Use harmless, deterministic operations and unique sentinels. A canary passes only when the external tool produces the sentinel; model text alone is insufficient. + +## Evidence expiry + +Expire certification when any scoped component changes materially: + +- client minor version/tool runtime; +- adapter implementation; +- upstream protocol version; +- meta-tool contract; +- model family behavior if acquisition-dependent. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/073_config_migration_and_docs.md b/devlog/_plan/260813_routed_tool_discovery_profiles/073_config_migration_and_docs.md new file mode 100644 index 000000000..7d7989efb --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/073_config_migration_and_docs.md @@ -0,0 +1,41 @@ +# 073 - Config migration and documentation + +## Migration + +No migration is necessary. Missing fields mean `auto`. + +## Example: exact model fallback + +```json +{ + "providers": { + "deepseek": { + "adapter": "openai-chat", + "baseUrl": "https://example.invalid/v1", + "modelRoutedToolDiscovery": { + "deepseek-v4-pro": "direct" + } + } + } +} +``` + +## Example: provider-wide diagnostic override + +```json +{ + "routedToolDiscovery": "direct" +} +``` + +Document that provider-wide direct mode is a broad compatibility fallback and can materially increase context/cost. + +## Required docs + +- configuration reference; +- troubleshooting: plugin missing versus translator missing tools; +- explanation of hosted search versus tool discovery; +- payload warning; +- Cursor hard fence; +- how to collect a redacted canary bundle; +- how to remove override after a client fix. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/074_support_runbook.md b/devlog/_plan/260813_routed_tool_discovery_profiles/074_support_runbook.md new file mode 100644 index 000000000..47d9eb677 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/074_support_runbook.md @@ -0,0 +1,27 @@ +# 074 - Support runbook + +## Symptom: plugin tool missing + +1. record client/OpenCodex versions; +2. inspect catalog row; +3. check resolved mode and source; +4. inspect `ALL_TOOLS`/local tools if Code Mode; +5. compare a fresh session; +6. apply exact model direct override as A/B—not as permanent global fix; +7. capture payload and call sequence. + +## Symptom: model says it will run a tool, then stops + +Inspect translation for lost tool declarations, especially Responses Lite `additional_tools`. Do not classify as streaming failure until tool presence is proven. + +## Symptom: very large first request + +Check whether direct mode or search false caused full schemas to enter `exec.description`. Report tool count/schema bytes. + +## Symptom: session dies after resume + +Search for durable references to transient or removed tools. Reconcile rather than clearing the entire session where possible. + +## Escalation bundle + +Use the capture list in `040_phase4_live_e2e.md`; redact secrets and prompt content. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/079_rollout_exit_gate.md b/devlog/_plan/260813_routed_tool_discovery_profiles/079_rollout_exit_gate.md new file mode 100644 index 000000000..3373792e7 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/079_rollout_exit_gate.md @@ -0,0 +1,11 @@ +# 079 - Rollout exit gate + +- [ ] defaults documented and unchanged; +- [ ] direct override warning shipped; +- [ ] diagnostics identify source; +- [ ] canary automation in CI/release workflow where feasible; +- [ ] support runbook published; +- [ ] evidence expiry policy implemented before auto-selection; +- [ ] rollback tested; +- [ ] no migration required; +- [ ] release notes distinguish compatibility override from performance optimization. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/080_rollback_plan.md b/devlog/_plan/260813_routed_tool_discovery_profiles/080_rollback_plan.md new file mode 100644 index 000000000..043da9c8a --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/080_rollback_plan.md @@ -0,0 +1,29 @@ +# 080 - Rollback plan + +## PR A rollback + +Code rollback: + +- remove resolver module and propagation; +- restore `normalizeRoutedCatalogEntry()` default Boolean; +- remove optional fields from active docs/schema. + +Config rollback is simpler: delete `routedToolDiscovery` and `modelRoutedToolDiscovery`. Missing fields return #1596 behavior. + +## Runtime emergency rollback + +If a release causes broad plugin loss: + +1. disable auto evidence selection if present; +2. restore non-Cursor deferred default; +3. retain exact direct overrides for confirmed affected routes; +4. publish client/version-specific advisory; +5. avoid blanket direct unless payload impact is accepted explicitly. + +## Meta-tool rollback + +Disable the profile flag and return to Code Mode/default. Because underlying tools remain in the normal registry, removing meta-tools should not alter authorization state. + +## Data compatibility + +Evidence and diagnostics are derived state. They may be discarded. User configuration must remain preserved by older binaries through passthrough behavior. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/081_failure_triage_runbook.md b/devlog/_plan/260813_routed_tool_discovery_profiles/081_failure_triage_runbook.md new file mode 100644 index 000000000..cdfec32ab --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/081_failure_triage_runbook.md @@ -0,0 +1,32 @@ +# 081 - Failure triage decision tree + +```text +Tool declared by client? + no -> client/plugin initialization issue + yes + Survives OpenCodex inbound parse? + no -> Responses/Chat parser bug + yes + Reaches adapter request? + no -> translator/filter bug + yes + Model emits tool call? + no + Tool discoverable in selected profile? + no -> profile/catalog/index bug + yes -> model acquisition/compliance issue + yes + Call restored to correct type/name/namespace? + no -> response translator bug + yes + Tool executes? + no -> authorization/runtime/tool failure + yes + Output replayed and loop continues? + no -> history/stream/continuation bug + yes -> success +``` + +## Required classification + +Every issue should end with one primary stage, not “tools broken.” This prevents catalog flags from being used to mask translator or execution defects. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/082_configuration_examples.md b/devlog/_plan/260813_routed_tool_discovery_profiles/082_configuration_examples.md new file mode 100644 index 000000000..c292ed9e9 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/082_configuration_examples.md @@ -0,0 +1,58 @@ +# 082 - Configuration examples + +## Default—recommended + +No field: + +```json +{ + "adapter": "openai-chat", + "baseUrl": "https://provider.example/v1" +} +``` + +Result: non-Cursor deferred, Cursor direct. + +## One known-bad model + +```json +{ + "modelRoutedToolDiscovery": { + "glm-5.2": "direct" + } +} +``` + +## Broad provider diagnostic + +```json +{ + "routedToolDiscovery": "direct" +} +``` + +Use temporarily while isolating a provider-wide compatibility issue. + +## Explicit return to default + +```json +{ + "routedToolDiscovery": "auto" +} +``` + +or remove the key. + +## Mixed gateway + +```json +{ + "routedToolDiscovery": "deferred", + "modelRoutedToolDiscovery": { + "legacy-model": "direct", + "fixed-model": "auto" + } +} +``` + +`fixed-model:auto` resolves to the non-Cursor default, not the provider's direct/deferred value. This is intentional if model-level `auto` is defined as “re-evaluate default.” The implementation and docs must pin this semantic; an alternative is to treat model `auto` as inheritance. The recommended design here uses default re-evaluation. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/083_incident_report_template.md b/devlog/_plan/260813_routed_tool_discovery_profiles/083_incident_report_template.md new file mode 100644 index 000000000..36ed14bd4 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/083_incident_report_template.md @@ -0,0 +1,46 @@ +# 083 - Incident report template + +```markdown +# Routed tool incident + +Date/time: +OpenCodex commit/version: +Client surface/version: +Provider/adapter/base URL class: +Model id: +Plugin/MCP server: +Resolved discovery mode/profile/source: + +## Expected + +## Actual + +## Minimal prompt + +## Tool lifecycle +- declared by client: +- parsed by OpenCodex: +- sent upstream: +- model call received: +- restored call type/name/namespace: +- executed: +- output replayed: +- final response: + +## Metrics +- top-level tools: +- indexed/nested tools: +- declaration bytes: +- total request bytes: +- manifest hash changes: +- cache read/write: + +## A/B result +- deferred: +- direct: +- meta-tools: + +## Redaction statement + +## Proposed disposition +``` diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/089_rollback_exit_gate.md b/devlog/_plan/260813_routed_tool_discovery_profiles/089_rollback_exit_gate.md new file mode 100644 index 000000000..f86381bac --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/089_rollback_exit_gate.md @@ -0,0 +1,10 @@ +# 089 - Rollback readiness gate + +- [ ] removing config fields restores #1596 defaults; +- [ ] old binary preserves unknown fields; +- [ ] derived evidence can be deleted safely; +- [ ] no transcript migration required for PR A; +- [ ] meta-tool disable path tested; +- [ ] direct override can be narrowed/removed without restart surprises; +- [ ] emergency advisory template ready; +- [ ] rollback does not alter credentials, provider routing or model lists. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/090_final_recommendation.md b/devlog/_plan/260813_routed_tool_discovery_profiles/090_final_recommendation.md new file mode 100644 index 000000000..f0ae41ab9 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/090_final_recommendation.md @@ -0,0 +1,39 @@ +# 090 - Final recommendation + +## Decision + +Implement the route-scoped resolver first. Keep: + +```text +non-Cursor -> code_mode_only + deferred +Cursor -> direct +``` + +Add provider/model direct overrides only for evidence-backed compatibility failures. + +## Why + +- PR #1596 established the lowest measured default among the tested shapes. +- Code Mode keeps nested tools locally available without full schema injection. +- #1522 still deserves a precise remediation path. +- One Boolean cannot represent catalog exposure, protocol conformance and model acquisition quality. + +## PR order + +1. resolver/config/tests/diagnostics; +2. adapter conformance fixtures; +3. exact live E2E certification; +4. bounded meta-tools where justified; +5. optional evidence-driven auto profile. + +## What not to do + +- do not restore blanket `supports_search_tool=false`; +- do not infer capability from model brand alone; +- do not silently drop unsupported tool kinds; +- do not dynamically mutate the top-level tool manifest without measuring cache impact; +- do not call an unmarked partial search result exhaustive. + +## Immediate maintainer action + +Review the field names and semantics in PR A, especially whether model-level `auto` means default re-evaluation or provider inheritance. Once fixed, implement the pure resolver and focused tests before touching protocol adapters. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/091_pr_stack_and_commits.md b/devlog/_plan/260813_routed_tool_discovery_profiles/091_pr_stack_and_commits.md new file mode 100644 index 000000000..92a0585ac --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/091_pr_stack_and_commits.md @@ -0,0 +1,50 @@ +# 091 - Proposed PR stack and commits + +## PR A — route policy + +Title: + +```text +feat(codex): add routed tool-discovery compatibility profiles +``` + +Commits: + +1. `feat(config): add routed tool discovery overrides` +2. `refactor(catalog): resolve discovery mode through provider hints` +3. `test(codex): cover discovery precedence, Cursor fence and combos` +4. `docs(codex): document route-scoped direct fallback` + +## PR B — conformance + +Title: + +```text +fix(responses): preserve complete tool discovery round trips +``` + +Commits organized by declaration, history, streaming and compaction fixtures. + +## PR C — meta-tools + +Title: + +```text +feat(mcp): add bounded search, describe and call fallback +``` + +Security review required. + +## PR D — live evidence and auto selection + +Title: + +```text +feat(codex): select tool profile from versioned compatibility evidence +``` + +Do not open until exact live matrix is complete. + +## Branch discipline + +Each PR starts from the then-current `dev`, remains independently reviewable and does not mix GUI cleanup or unrelated adapter changes. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/092_definition_of_done.md b/devlog/_plan/260813_routed_tool_discovery_profiles/092_definition_of_done.md new file mode 100644 index 000000000..f2486ef76 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/092_definition_of_done.md @@ -0,0 +1,31 @@ +# 092 - Definition of done + +## Phase 1 done + +- typed resolver merged; +- defaults unchanged; +- exact overrides work; +- diagnostics present; +- full suite green. + +## Programme done + +- adapter conformance is fixture-driven; +- exact #1522 scenario has a recorded disposition; +- live canaries cover CLI and App; +- dynamic tools, compaction and resume are safe; +- payload and cache thresholds are data-backed; +- meta-tools, if shipped, reuse authorization and have high recall; +- auto selection, if shipped, uses versioned expiring evidence; +- rollback is rehearsed; +- documentation separates hosted search, discovery and execution. + +## Evidence standard + +A claim is complete only when it states: + +```text +what was tested + exact versions + fixed inputs + observed outputs + limitations +``` + +“No error” is not sufficient. A tool test must prove actual execution and result use. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/093_execution_order.md b/devlog/_plan/260813_routed_tool_discovery_profiles/093_execution_order.md new file mode 100644 index 000000000..e131de894 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/093_execution_order.md @@ -0,0 +1,33 @@ +# 093 - Suggested execution order + +```text +Day/PR 1 + resolver module + config fields and validation + provider-hint propagation + template/fallback/combo serialization + focused tests + +Day/PR 2 + Responses Lite fixtures + custom/namespace history + streaming parity + explicit failure diagnostics + +Lab run + CLI canary + App canary + #1522 exact A/B + payload/cache captures + +PR 3 if required + meta-tool registry + search/describe/call + security and authorization tests + +Later + evidence store + automatic profile selection +``` + +The sequence is risk-ordered: first add control without changing defaults, then prove the protocol, then add a new execution surface only when evidence justifies it. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/094_landing_verification_pass.md b/devlog/_plan/260813_routed_tool_discovery_profiles/094_landing_verification_pass.md new file mode 100644 index 000000000..486a5f031 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/094_landing_verification_pass.md @@ -0,0 +1,293 @@ +# 094 - Landing verification pass + +Status: VERIFIED IN A REAL WORKTREE +Date: 2026-08-13 +Worktree: `/Users/jun/.codex/worktrees/9d46/opencodex` +Worktree HEAD at landing: `1849b947b32f9d909e06552582a63b6842db246b` +`origin/dev` at landing: `2cdbf66a23f9fd8f2f38dcc702ccd3f2e60ac535` + +## Why this document exists + +`001_verified_dev_baseline.md` closes with an honest admission: + +> This bundle was built without a mounted full repository checkout. + +Every "verified" claim in documents `001`–`013` was therefore verified against +GitHub file reads, not against a working tree, a compiler, or the upstream Rust +client. This pass re-checks the load-bearing claims with the repository actually +mounted, the upstream `codex-rs` source on disk, and authenticated GitHub access. + +The result: the plan's **architecture is sound and its default-preserving stance +is correct**, but six specific claims are wrong or incomplete in ways that would +have produced a broken or under-scoped first PR. Corrections are recorded here +and applied in place to the affected documents. + +## Verification sources + +| Lane | Source | Scope | +|---|---|---| +| Upstream semantics | `/Users/jun/Developer/codex/120_codex-cli`, `main` @ `4462b9deef211723b781b426f5e5d36a5777115f` (2026-07-23) | What `supports_search_tool` actually does in the client | +| Repository history | authenticated `gh`, `origin/dev` @ `2cdbf66a2` | #1522 / #1529 / #1596 / #1587, open-PR collisions, CI gates | +| Code seams | this worktree, `rg` + numbered reads | every file the first PR must touch | + +## Confirmed claims + +These bundle statements survived contact with the real tree unchanged: + +- `normalizeRoutedCatalogEntry()` lives in `src/codex/catalog/parsing.ts` and + stamps `tool_mode = code_mode_only`, Cursor `supports_search_tool = false`, + non-Cursor `true`. Verified at `parsing.ts:381-418`. +- `src/codex/catalog/sync.ts` carries an independent template-less fallback that + duplicates the same policy. Verified at `sync.ts:312-343`. +- `applyProviderConfigHints()` is the right resolver call site: it already + resolves context windows, modalities, max input tokens, reasoning ladders, + summary support and `parallelToolCalls`. Verified at `provider-fetch.ts:610-658`. +- `providerCatalogFingerprint()` is an explicit allow-list and would silently omit + a new field. Verified at `provider-fetch.ts:536-559`. +- No commit on `origin/dev` after #1596 (`5703473041a9f4f415743652de5d86d51fd66db5`) + has touched `parsing.ts`, `sync.ts`, `config.ts` or `types.ts`. The semantic base + the plan targets is still the live head's behavior. +- Hosted web search really is a separate capability upstream: `web_search_tool_type` + feeds only the hosted tool's content types, and hosted availability is a distinct + provider capability. INV-3 is correct. + +## Correction 1 — `supports_search_tool` is gated by a second condition + +The plan treats the flag as the sole switch for deferred discovery. Upstream it is +one of two conjuncts: + +```rust +pub(crate) fn search_tool_enabled(turn_context: &TurnContext) -> bool { + turn_context.model_info.supports_search_tool && namespace_tools_enabled(turn_context) +} +``` + +`codex-rs/core/src/tools/spec_plan.rs:330`. + +A route whose provider does not support Responses namespace tools resolves to +`Direct` MCP exposure **regardless of what the catalog advertises**. That has two +consequences the plan must absorb: + +- `direct` mode is sometimes already in effect without any override, so an + operator's "it did not work" report is not automatically evidence that the + catalog flag is wrong. +- The diagnostic in `014` must not claim the resolved catalog mode is the + effective runtime mode. It can only report what OpenCodex advertises. + +## Correction 2 — under `code_mode_only`, an *eligible* MCP tool stays reachable in BOTH modes + +This is the most important correction, and it strengthens the plan's default. + +**Scope, stated first.** This correction holds for an **ordinary eligible MCP +tool** — one that already survived MCP/App visibility and policy filtering, is not +in `direct_only_tool_namespaces`, and is not in `excluded_tool_namespaces`. It is +not an unconditional statement about every tool on every surface. The exclusions +are enumerated below and each is independent of `supports_search_tool`. + +`004` implies deferred discovery is what keeps tools reachable. In fact, under +`ToolMode::CodeModeOnly` the nested tool specs are handed to the `exec` handler +and installed as V8 globals in either mode: + +```rust +set_global(scope, global, "tools", tools.into())?; +set_global(scope, global, "ALL_TOOLS", all_tools)?; +``` + +`codex-rs/code-mode/src/runtime/globals.rs:15`, fed from +`spec_plan.rs:463` → `execute_handler.rs:29` → `runtime/mod.rs:89`. + +The exposure switch changes classification, not reachability: + +```rust +let exposure = if search_tool_enabled { ToolExposure::Deferred } else { ToolExposure::Direct }; +``` + +`codex-rs/core/src/mcp_tool_exposure.rs:35`. + +With `supports_search_tool = false`, the tools' **full declarations** move into +`exec.description` because they enter `enabled_tools` rather than `deferred_tools`. +That is precisely the 96,699 → 258,929 character regression #1596 measured. So for +an eligible tool the `direct` escape hatch buys **no additional callability under +code mode** — it buys schema text inside `exec.description`. + +Confirmed by the upstream suite from both directions: a direct-exposure MCP tool +reaches the globals (`core/tests/suite/code_mode.rs:3248`), and a deferred one does +too (`core/tests/suite/code_mode.rs:644`). + +### What else the flag changes, and what it never controls + +Schema placement is the main difference but not the only one. Deferred exposure +also drives `tool_search` construction and the deferred-tool guidance text +(`spec_plan.rs:932`, `description.rs:10`). + +And three mechanisms remove a tool from the Code Mode globals **independently of +this flag** — an override cannot fix any of them, and a reporter hitting one will +look like a discovery failure: + +| Mechanism | Effect | Citation | +|---|---|---| +| `direct_only_tool_namespaces` | exposure becomes `DirectModelOnly`; the namespace stays top-level and is **excluded from Code Mode** | `spec_plan.rs:210` | +| `excluded_tool_namespaces` | nested tools removed outright | `spec_plan.rs:444` | +| MCP/App visibility and policy filtering | applied *before* exposure classification, so a filtered tool never reaches either path | `mcp_tool_exposure.rs:20` | + +The isolate is not exclusive to `code_mode_only`. `build_code_mode_executors()` +returns early only when the mode is neither: + +```rust +if !matches!(tool_mode, ToolMode::CodeMode | ToolMode::CodeModeOnly) { + return vec![]; +} +``` + +`spec_plan.rs:459`. So plain `ToolMode::CodeMode` also builds the `tools`/ +`ALL_TOOLS` globals, and the reachability reasoning holds there too; what +`code_mode_only` adds is hiding ordinary nested tools from the **top-level** tool +list. Only under `ToolMode::Direct` is there no isolate — and there +`supports_search_tool` genuinely governs whether MCP tools are declared directly +or must be found through `tool_search`. OpenCodex stamps every routed row +`code_mode_only`, so `Direct` is not a routed-row concern today, but the +distinction matters if that stamp is ever relaxed. + +### Required before the claim is load-bearing + +`020`/`021` must add one controlled test that toggles **only** +`supports_search_tool` under otherwise identical `code_mode_only` conditions and +asserts the tool remains callable in both, so this correction rests on an +executed differential rather than on a source reading. The expected delta is +`exec.description` content/size **plus** `tool_search` construction and the +deferred-guidance text — not `exec.description` alone. + +This reframes the override honestly: for an eligible tool under code mode it is a +*model-comprehension* lever (the model sees full schemas inline instead of having +to consult `ALL_TOOLS`), not a *reachability* lever. `010`'s non-goal list already +says "no claim that `direct` is cheaper or preferred"; it must also say direct is +not a reachability fix for eligible tools under code mode. `044`'s weak-model +fallback rationale is the honest use case. + +Caveat recorded rather than assumed away: the inspected clone is dated 2026-07-23, +while #1522 reported against CLI `0.147.0-alpha.6.5` on 2026-08-12. Whether the +shipped App build has additional app-layer gating is **not** established by this +pass. E1/E2 in `009` stay open. + +## Correction 3 — `CatalogModel` is not in `src/types.ts` + +`003` and `011` place the `toolDiscoveryMode` carrier under "Data model / +`src/types.ts`". The interface actually lives in `src/codex/catalog/parsing.ts:94`, +alongside `parallelToolCalls`, `supportsVerbosity` and `supportsReasoningSummaries`. + +Only the **config** fields (`routedToolDiscovery`, `modelRoutedToolDiscovery`) +belong in `src/types.ts` on `OcxProviderConfig`. The resolved carrier belongs in +`parsing.ts`. Following the documents literally would have created a second +`CatalogModel` or an import cycle. + +## Correction 4 — `modelRecordValue()` does not match dated variants + +`010` promises an "exact/date-compatible model override" and `013` instructs: + +> Use `modelRecordValue()` for the per-model map so dated or normalized variants +> follow the same matching behavior as existing model metadata. + +`modelRecordValue` (`src/reasoning-effort.ts:49-62`) matches exactly three ways: +exact own-property, the prefix before a `:`, and a case-insensitive full-id match. +It does **not** match `-YYYYMMDD` suffixes. Dated matching is a separate helper, +`isDatedVariantId()` at `provider-fetch.ts:805-808`. + +So a `glm-5.2` override will not apply to `glm-5.2-20260813` through +`modelRecordValue` alone. Either the resolver composes both helpers, or the +documents must stop promising date compatibility. **Decision: keep +`modelRecordValue` semantics only** — matching every sibling model-keyed override +in the codebase is worth more than a bespoke matcher, and an operator pinning an +emergency escape hatch should name the exact model id that failed. `010` and `013` +are amended to say "exact model override" without the date claim. + +## Correction 5 — the config schema will not validate a new provider field + +`011` proposes a `.catch()`-bearing zod field and assumes it slots into the +provider schema. In reality `providerConfigSchema` (`src/config.ts:616-645`) +declares only a minority of `OcxProviderConfig` fields and ends with +`.passthrough()`. It contains no `.catch()` anywhere. + +A field added only to the TypeScript interface is therefore **passed through +entirely unvalidated** — neither degraded on load nor rejected on write. Both +halves of `011`'s config contract have to be built explicitly: + +- declare the fields in `providerConfigSchema` with `.catch(undefined)` for + load-path degradation; +- add a boundary validator to the `validateConfigCandidate` chain + (`config.ts:2298-2312`) that inspects the raw candidate before `.catch()` erases + it, emitting the house format `schema_invalid: providers..: `. + +## Correction 6 — the touch map is missing `aggregation.ts` + +`012` specifies a combo rule ("any member direct → combo direct") but `003`'s file +list never names the file that would implement it. Combo capability derivation +lives in `deriveComboCatalogModel()` at `src/codex/catalog/aggregation.ts:125-177`. + +The proposed rule is directionally consistent with house precedent — the existing +flags use `members.every(...)` to grant a capability, and "direct wins" is the same +shape as "deferred only if every member is deferred". The file simply has to be in +the plan. + +## Correction 7 — the `parallelToolCalls` precedent is incomplete + +`013`'s propagation chain models itself on `parallelToolCalls`. That precedent is +correct for the **template** path only. The template-less fallback never passes +`parallelToolCalls` into normalization at all, and `ensureStrictCatalogFields` +defaults a missing `supports_parallel_tool_calls` to `true` +(`parsing.ts:293-306`). + +The new field must therefore emit explicitly on **both** construction paths and +must never rely on a strict-field default to carry policy. + +## Correction 8 — the Cursor fence is inconsistent across the two paths, today + +This is an unresolved P2 review finding from #1596, not a new defect, but it lands +directly on this unit's surface: + +- template path: `entry.slug.startsWith("cursor/")` (`parsing.ts:395`) +- template-less path: `model?.provider === "cursor"` (`sync.ts:313`) + +A combo whose public alias begins `cursor/` but whose canonical provider is +`combo` is classified **differently depending on whether a template was +available**. Discovery mode and payload size then depend on template availability +— a real behavior defect. + +`012` preserves the slug check verbatim and so would inherit the inconsistency. +Since this unit is already unifying both paths behind one resolver, it should +close the asymmetry rather than reproduce it: the fence resolves from provider +identity, with the slug prefix retained only as a fallback when no `CatalogModel` +is available. Recorded as a first-PR requirement. + +## Repository-state findings the bundle could not have known + +- **Issue #1522 is closed**, at `2026-08-12T20:22:45Z`, with the comment "Fixed by + #1515". That attribution is wrong — #1515 concerns account-scoped native model + ids. The incident is closed under a false cause, which is worth noting before + citing it as live justification. +- **Issue #1587** ("routed first-turn tool catalog is 3–5x native Sol input + tokens") is open and is the standing argument against any return to a blanket + `false` default. It supports this unit's default-preserving stance. +- **#1596's three inline review threads are all still unresolved**: the + provider-identity/alias P2 (Correction 8), a request to publish the referenced + measurement record — `devlog/_plan/260813_tool_catalog_deferral/010` does not + exist anywhere in history — and a CodeRabbit note that + `structure/03_catalog-and-subagents.md` wrongly says Cursor "advertises neither + flag" when the implementation does emit `supports_search_tool: false`. +- **Open PR collisions** the PR stack must be sequenced around: + - #1604 (draft) touches `parsing.ts` and `sync.ts` — direct textual collision. + - #1521 (draft) touches `parsing.ts`, `config.ts`, `types.ts` and provider + capability configuration — the strongest architectural collision. + - #1602 (draft) promotes client `tool_search_output` definitions into active + upstream tool declarations — adjacent semantic collision with any discovery + override. + +## Net effect on the roadmap + +The phase structure stands. The first PR's scope grows by one file +(`aggregation.ts`), gains one requirement (unify the Cursor fence on provider +identity), loses one promise (date-compatible model keys), and must build config +validation from scratch rather than extending an existing validated field. + +Correction 2 changes what the feature is *sold as*, which is the most consequential +edit in this pass: the override is a comprehension and compatibility lever with a +measured payload cost, not a fix for tools being unreachable. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/KOREAN_SUMMARY.md b/devlog/_plan/260813_routed_tool_discovery_profiles/KOREAN_SUMMARY.md new file mode 100644 index 000000000..722a03f1a --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/KOREAN_SUMMARY.md @@ -0,0 +1,74 @@ +# 한국어 요약 + +## 결론 + +1차 패치는 PR #1596의 기본동작을 유지합니다. + +```text +비-Cursor 외부모델: code_mode_only + deferred discovery +Cursor: direct discovery +``` + +그 위에 정확한 provider/model 조합에만 적용되는 예외 설정을 추가합니다. + +```json +{ + "routedToolDiscovery": "auto | deferred | direct", + "modelRoutedToolDiscovery": { + "model-id": "auto | deferred | direct" + } +} +``` + +모델별 설정이 provider 설정보다 우선하며, Cursor는 잘못 `deferred`로 설정해도 `direct`로 고정합니다. `direct`는 플러그인 호환성용 탈출구이지 기본 최적화가 아닙니다. 전체 MCP 스키마가 첫 요청에 들어가 컨텍스트가 다시 커질 수 있기 때문입니다. + +## 실제로 포함한 것 + +- 000–009: 현재 `dev`, #1522/#1529/#1596, Codex Code Mode, 비교 프로젝트 조사 +- 010–015: 파일·함수 단위 1차 패치 설계 +- 020–029: 단위·통합·설정·캐시·combo 테스트 +- 030–039: Responses Lite/custom/namespace/tool-search 이력 conformance +- 040–049: Codex CLI/App, DeepSeek+Browser 정확 재현, Cursor, compaction/resume +- 050–059: payload·prompt cache 벤치마크 +- 060–069: `search/describe/call` 3개 meta-tool 폴백 +- 070–079: 단계적 배포·관측·canary +- 080–089: 롤백·장애 분류·설정 예시 +- 090–094: 최종 권고·PR 스택·완료 기준·실행 순서·랜딩 검증 +- `patches/apply-draft.mjs`: seam 위치를 읽기 위한 보관용 스크립트 (실행 금지, 아래 참고) +- `prototype/`: 의존성 없는 정책 프로토타입과 벤치마크 +- `results/`: 실행 결과와 한계 + +## 이 환경에서 실제 실행한 검증 + +- 정책 프로토타입: 17개 테스트 전부 통과 +- 패치 적용 스크립트: Node 문법 검사 통과 +- 현재 코드 seam을 흉내 낸 합성 작업트리에서 `--check`와 적용 모두 통과 +- 합성 250-tool payload: + - 전체 스키마 직접 노출: 179,218 bytes + - 이름·설명 인덱스: 52,393 bytes + - 고정 meta-tool 3개: 744 bytes + +이 수치는 실제 Codex 요청 토큰값이 아니라 구조 비교용 UTF-8 바이트입니다. + +## 아직 실행하지 못한 것 + +이 실행환경은 외부 DNS가 차단되어 전체 OpenCodex 저장소를 복제하지 못했고 Bun도 없었습니다. 따라서 다음은 완료했다고 주장하지 않습니다. + +- 전체 TypeScript typecheck +- OpenCodex 집중 Bun 테스트 +- 전체 테스트 스위트 +- 실제 Codex CLI/App + Browser 플러그인 E2E + +실제 작업트리에서 실행할 명령은 `scripts/run-repo-validation.sh`에 묶었습니다. + +## 적용 순서 (2026-08-13 철회됨) + +`patches/apply-draft.mjs` 실행 절차는 삭제했습니다. 이 초안은 `094` 정정 8이 +지적한 Cursor 판정 불일치를 그대로 재현하면서도 `draft seams: OK`를 출력합니다. +seam 위치를 읽는 참고 자료로만 쓰고, 커밋할 저장소에는 적용하지 마십시오. +자세한 내용은 `patches/README.md`에 있습니다. + +실제 구현은 provider 신원을 먼저 보는 공용 헬퍼 하나로 템플릿 경로와 +템플릿 없는 경로를 모두 처리하므로, 이 초안과 diff가 일치하지 않습니다. + +적용 뒤에는 반드시 `git diff`를 리뷰하고, `042_codex_app_deepseek_browser.md`의 A/B 시나리오로 #1522의 정확한 조합을 확인해야 합니다. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/MANIFEST.txt b/devlog/_plan/260813_routed_tool_discovery_profiles/MANIFEST.txt new file mode 100644 index 000000000..31504940f --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/MANIFEST.txt @@ -0,0 +1,95 @@ +000_master_plan.md +001_verified_dev_baseline.md +002_incident_history_1522_1529_1596.md +003_current_code_map.md +004_upstream_codex_code_mode.md +005_comparator_findings.md +006_architecture_invariants.md +007_scenario_matrix.md +008_risk_register.md +009_open_questions_and_evidence_gaps.md +010_phase1_profile_resolver.md +011_phase1_types_and_config.md +012_phase1_catalog_patch.md +013_phase1_sync_and_fingerprint.md +014_phase1_diagnostics.md +015_phase1_review_checklist.md +020_phase2_unit_tests.md +021_catalog_test_cases.md +022_config_and_precedence_tests.md +023_backward_compatibility_tests.md +024_catalog_cache_identity_tests.md +025_combo_policy_tests.md +029_phase2_exit_gate.md +030_phase3_protocol_conformance.md +031_responses_lite_additional_tools.md +032_custom_namespace_roundtrip.md +033_tool_search_history_and_compaction.md +034_streaming_nonstreaming_matrix.md +035_failure_injection.md +039_phase3_exit_gate.md +040_phase4_live_e2e.md +041_code_mode_all_tools_canary.md +042_codex_app_deepseek_browser.md +043_cursor_and_direct_bounded.md +044_weak_model_meta_tool_fallback.md +045_dynamic_mcp_refresh.md +046_compaction_resume_live.md +049_phase4_exit_gate.md +050_phase5_payload_cache_benchmarks.md +051_benchmark_methodology.md +052_acceptance_thresholds.md +053_prompt_cache_scenarios.md +059_phase5_exit_gate.md +060_phase6_meta_tool_design.md +061_meta_tool_contract.md +062_meta_tool_security.md +063_meta_tool_ranking.md +064_meta_tool_integration.md +065_meta_tool_tests.md +069_phase6_exit_gate.md +070_rollout_plan.md +071_observability.md +072_canary_matrix.md +073_config_migration_and_docs.md +074_support_runbook.md +079_rollout_exit_gate.md +080_rollback_plan.md +081_failure_triage_runbook.md +082_configuration_examples.md +083_incident_report_template.md +089_rollback_exit_gate.md +090_final_recommendation.md +091_pr_stack_and_commits.md +092_definition_of_done.md +093_execution_order.md +094_landing_verification_pass.md +KOREAN_SUMMARY.md +README.md +patches/0001-add-tool-discovery-module.patch +patches/0002-focused-test-plan.patch +patches/0003-route-scoped-tool-discovery.review.diff +patches/README.md +patches/apply-draft.mjs +patches/proposed/src/codex/catalog/tool-discovery.ts +patches/proposed/tests/codex-tool-discovery-mode.test.ts +prototype/mvp-resolver.mjs +prototype/payload-benchmark.mjs +prototype/profile-resolver.mjs +prototype/tool-discovery-profile.test.mjs +results/TEST_SCOPE.md +results/apply-draft-synthetic-test.txt +results/benchmark-summary.md +results/current-dev-head.json +results/patch-0001-numstat.txt +results/patch-0002-numstat.txt +results/payload-benchmark-1000.json +results/payload-benchmark-250.json +results/prototype-summary.json +results/prototype-test.txt +results/repository-clone-attempt.txt +scripts/inspect-generated-catalog.mjs +scripts/run-prototype-tests.sh +scripts/run-repo-validation.sh +scripts/validate-bundle.sh +sources/SOURCE_INDEX.md diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/README.md b/devlog/_plan/260813_routed_tool_discovery_profiles/README.md new file mode 100644 index 000000000..e447fad36 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/README.md @@ -0,0 +1,47 @@ +# Routed tool discovery profile investigation bundle + +This folder follows the OpenCodex `devlog/_plan` convention. + +## Reading order + +1. `000_master_plan.md` +2. `094_landing_verification_pass.md` — **read before implementing anything** +3. `001`–`009`: verified baseline, incident history and research +4. `010`–`015`: minimal implementation patch +5. `020`–`029`: repository test plan +6. `030`–`039`: protocol conformance +7. `040`–`049`: live CLI/App scenarios +8. `050`–`059`: payload/cache benchmarks +9. `060`–`069`: bounded meta-tool fallback +10. `070`–`079`: rollout and operations +11. `080`–`089`: rollback and incident response +12. `090`–`094`: final decision, PR stack and landing verification + +## Included executable artifacts + +```text +prototype/mvp-resolver.mjs +prototype/profile-resolver.mjs +prototype/tool-discovery-profile.test.mjs +prototype/payload-benchmark.mjs +``` + +Recorded results are under `results/`. + +## Verification status + +- GitHub source inspection: relevant files re-fetched from packaging-time `dev` head `2cdbf66a...`; PR #1596 commit `570347304...` is its direct parent and remains the tool-discovery semantic base. +- **Worktree re-verification (2026-08-13):** performed after landing. Confirmed the + seam map and the default-preserving stance; recorded eight corrections in + `094_landing_verification_pass.md`, including that `CatalogModel` lives in + `parsing.ts` rather than `types.ts`, that `aggregation.ts` was missing from the + touch map, and that under `code_mode_only` MCP tools are callable in both + discovery modes. +- Independent Node policy tests: executed, 17/17 passed. +- Synthetic payload benchmark: executed for 250 and 1,000 tools. +- Full OpenCodex Bun suite: not executed in this environment because the complete repository could not be cloned/materialized; the exact commands are in `scripts/run-repo-validation.sh`. +- Remote repository: not modified. + +## Patch status + +The `patches/` directory contains an implementation-oriented draft and proposed source/test files. It is not represented as a merged or full-suite-verified patch. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/SHA256SUMS b/devlog/_plan/260813_routed_tool_discovery_profiles/SHA256SUMS new file mode 100644 index 000000000..b4741547f --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/SHA256SUMS @@ -0,0 +1,95 @@ +4e5dd4520224a37fddfd8fb0dc532cd62c9daa9f703d2037c2875a44f8073b58 ./000_master_plan.md +7e853a53e344412cc4aa2fcbe64b140195e9825a1988c5e245204c19a8f22449 ./001_verified_dev_baseline.md +32d7a7ffff41bd0c036cef153f53047ce416624032dae9dff43e88ff24d653fc ./002_incident_history_1522_1529_1596.md +343ee877dde67978b7ffd6a1e6f7825751c5fb113fd51dead6554036f92fa904 ./003_current_code_map.md +f89df146f9c053ea3686492e52c621081b0a517e0438e9400273ac168959cc20 ./004_upstream_codex_code_mode.md +d0b48db849bdaa61c76e21a1eb78e28329e842f2589895fa0d81960fc74bd72f ./005_comparator_findings.md +2efb16f68b94cf7ca00bf7786ff1acb244f366b81004bdfb518c220b27815944 ./006_architecture_invariants.md +582a31ad7cc1d9842204c5e82e2b6c93889f59bca3fcc8c6853a1446787e0a94 ./007_scenario_matrix.md +ea00a33d3486ead3aab43fefabe341c65924bb9c43e01bc39dfefe99dbd341b4 ./008_risk_register.md +7bf595a2c99827e99f07a423dfdc34e8a8019082eff77ca4db476a612da5b9a4 ./009_open_questions_and_evidence_gaps.md +59c693fe63a3eaef7326e8efcdf26f757b64ee3ab322cc663fe3a647b63f5965 ./010_phase1_profile_resolver.md +1d9ff0e274599d1f85220026e3cb0e5c0be920e3e670ddb93e0d81bfc567dd26 ./011_phase1_types_and_config.md +a5147da15f8306343c25e5c47c4bf66bda696fd645d2fa3a36f2241ce8e4898f ./012_phase1_catalog_patch.md +81d11a2fcaf9481eb7995b83a8be0b34c83e542c827870c955a21c7ccf4d32ac ./013_phase1_sync_and_fingerprint.md +2fe731e78947cf69054e7a4d1ffa5376e81ee872d27dd6b0bab9efc37ac95d59 ./014_phase1_diagnostics.md +27315c12d0a4e8c7f4740be923a20cd3adf81b4b6b6cd380f52202c5abf7249b ./015_phase1_review_checklist.md +a4877785c6970d0eba172657aff894976683d2225e1cc48096b78aa4082832a4 ./020_phase2_unit_tests.md +a8ea01e49ccb8c964c56581455e45288651d84a13f12a1ad9fcbdfad41b8641f ./021_catalog_test_cases.md +99fe77bff87f97b3465b5aa11f697252b7e3809ec19f66d446cbfa1c54e3ff4d ./022_config_and_precedence_tests.md +bd18baa3e16b54fe262435a6ee28c1ca5b389ec8b54433be1b05466559df58b1 ./023_backward_compatibility_tests.md +9542600beb52c958f6065a2c76a18c104c791c01b2035860b9730296ae45b4de ./024_catalog_cache_identity_tests.md +4fdb03fa022bf684257f95752e545bd4e9541b9258361bd9f5da5fc137b822cc ./025_combo_policy_tests.md +00c0913b15591630290c4a08429393570467cdc6799a1433b0eb16532b0630b0 ./029_phase2_exit_gate.md +1f49005dbac81b2c3f3548d97f9aa32d6c8ada99f533d0fe197bcb047c553464 ./030_phase3_protocol_conformance.md +5e450a7557f7be2fad3f06c3c35288eb8654f156c402f826ef9c9b9703a52bf6 ./031_responses_lite_additional_tools.md +7bc16b20b8abe848726cb662865d542f2a8a19d6acb400a5473b0e9e3bc003e1 ./032_custom_namespace_roundtrip.md +47d6bd2282ef635d237c12740551f118064e2d2396e4bf2905cc4078e9bc65b6 ./033_tool_search_history_and_compaction.md +9e433b4cccff0cc935bf6fc75622820459f2f2dccfe89d9b07bc978424090afb ./034_streaming_nonstreaming_matrix.md +d75c3ed8cd224daf3e917bc55c6a8451afdc5ea4b5ac1e78ef51a2491617729b ./035_failure_injection.md +53190507b3f4147d44dbb7841af10048ce1a31dc0d4e168223d3407e28f3f14a ./039_phase3_exit_gate.md +c70e00bf1850d383409c24f0e7e6a347069e742c0a842dc438b3c33fbd2d5674 ./040_phase4_live_e2e.md +fd9bb9da5b142290d6abb99619e0792e4ce286e17a4c70782248696980d8b102 ./041_code_mode_all_tools_canary.md +f493c9e99375bd8d221931a86ccc036eab1d2e14c12618ba39e9c731749f2556 ./042_codex_app_deepseek_browser.md +9cfaf4ddd2ba96ff10255f206af3e7b06154ed00a26b229e2602fa1eab9d1856 ./043_cursor_and_direct_bounded.md +185888918b9dbed71644bf8da1097ff615018d4b4a3e47c10069c65292f39cb2 ./044_weak_model_meta_tool_fallback.md +5d3315f222b229bdb4481710da5139374b56c0ab16a1e9d7fa9db326aa0151b0 ./045_dynamic_mcp_refresh.md +2a8211dda7fc5c4570ba1b3d14c7371351d5252c9fe563141e13e9a31f6834b8 ./046_compaction_resume_live.md +18d6668a06b8efcc1acb56860735c6fa40529d325053eef1b70f6aea52be62a1 ./049_phase4_exit_gate.md +4cd3c62d6ace57a35de44449fef9b6a569e109edd34958327a48275b9fbd7518 ./050_phase5_payload_cache_benchmarks.md +19a27a668114f65a0f3a63029d841c596232e10a986dc87143b4e97e5bae5385 ./051_benchmark_methodology.md +95845bbfecbbe612c0842282e3387c4e51f3b7b39bb084d6e93a8ac4cef156f9 ./052_acceptance_thresholds.md +a39df766cd1b7dca6efb00cd8ea70d9781bcdab3ac271ca861a0084c63b221f0 ./053_prompt_cache_scenarios.md +acbbffbb2f310359066cd26c8bb5b61de07d5171ea3c1763518a54544a80ac27 ./059_phase5_exit_gate.md +966fa3d42f35e68eb297ed185588cfcab283b269b12689fe638dcbcc596c9ebe ./060_phase6_meta_tool_design.md +5a85f852d9a6135a7850716a74337c0a35d7ddf1b474c23c1e3e3a5ac3f13429 ./061_meta_tool_contract.md +7cedc0763c928a06e75a2197f30271e5114a889e8dadd1dc6a89932e73686050 ./062_meta_tool_security.md +8a2f75e78d1be501515ac935539307ff5c5316f4602a8d1e5c7a88e7938ed6d5 ./063_meta_tool_ranking.md +a0fb831f39348ca8f9775e8788fe7190188d0232e6f252e9d38c947716c9dfdf ./064_meta_tool_integration.md +a7cbcc0258bd2727ded2dddda97f85fa55fb73480ea10a653309ba15f58423ab ./065_meta_tool_tests.md +e2b608da313bd1149aab979b615652748bcdb7457581b2ae629369ac3dc02407 ./069_phase6_exit_gate.md +c73c82c0c7c430bd929b66dd1ae2d1da04c04f32efdfa9a686d5e91c72b5a1e6 ./070_rollout_plan.md +9cd1656fbc710b1f2d86eb5cbec4c776e15652dad3bed761cc56309fdcc6ce6e ./071_observability.md +b912d366c91d479477002e476a6c49849885ad8125e2ec91dc8151ebe7b7be9f ./072_canary_matrix.md +4fc17779ed6cffff1b5ed52880f5e65139bd21566d6fde72216d1507f5edeb98 ./073_config_migration_and_docs.md +d5285237f65d973f57e94cc44ec1db97f2a9b3c0426f0d0cb4bd5e74ca9bb807 ./074_support_runbook.md +e76d43befdbec894e49b0c48fc4a90818c5c392600cf908275fe775db9efa8f1 ./079_rollout_exit_gate.md +818c0be7a2051ab0726ad7120fc3470f2f5a74f235c6c13d6446e6ec4da43e4d ./080_rollback_plan.md +44a5d22f3829a4fb10d5e394c4a939946b78fba4042e30048040663becb255ac ./081_failure_triage_runbook.md +8939976307d6b35cc2e0276a942644fb4e6860c2a66cebdb9a6ed496404eabde ./082_configuration_examples.md +e4d78f0db15cfe7819c98c5fde166c863a74d3a751ffe8196ddb6492691dc799 ./083_incident_report_template.md +155f42241018ca5e96672f04bee61c6ee0b5770917acebba6bed7ed8a0762c27 ./089_rollback_exit_gate.md +2ded6ef14069124beb69a67cc5bf8f15fca89a999c48f1b2fd3892700d90db33 ./090_final_recommendation.md +28cb35f98ab0827a18f7ab52f28ad8cfd3f21ef475142c38ff8d49d41b43c106 ./091_pr_stack_and_commits.md +5bf3d16bea1a3ef12d68e3a467b271d3fd6631202fa43fb5f862f4ed505a43a3 ./092_definition_of_done.md +4cd63329dc69b6af90d2bdd31e64926ebc4acc916e1b34ccbef66e44f9a10937 ./093_execution_order.md +d68e52650b5b4dd7a6047b77f7942148f9ee36414a193828847a65b239411124 ./094_landing_verification_pass.md +383ed01bca1fda60f5b57fa13e0f9cc8408d1ed02a8eab802600f6afe4097c73 ./KOREAN_SUMMARY.md +61be1a43ef84b272e6d04170dbd3e0a617aac36d9d3fb0fc2c4d96da2a62f775 ./README.md +82522bf74dd85254d7162ce103886b2a483d1c274a241afb80fa488a8a7d4356 ./patches/0001-add-tool-discovery-module.patch +2dde3abf8bdf91e30d4583de4251422c5fd9efc17f6aea1cebd2e961a7c7c522 ./patches/0002-focused-test-plan.patch +0bd0d33c05e86467b9443fce11781b996f6890812e3dee249ba6e43f13aa239c ./patches/0003-route-scoped-tool-discovery.review.diff +e395da194d34c1fbbb0d5087a3262b03a9248434458daa900c8f9c99a140d344 ./patches/README.md +51bc6fcfea9dd6729c233571d400f5c6bd496c7b52bd6074fcf0c57d55e66565 ./patches/apply-draft.mjs +eab4d01b4123aa7b1c676dc3b52bdd37750fd6a8e8dd714ef1a324624374d01f ./patches/proposed/src/codex/catalog/tool-discovery.ts +399dcaaf3e8a50a11a2883b4074b8b91f30c169f587ba198ef8f23ef39ab1ca3 ./patches/proposed/tests/codex-tool-discovery-mode.test.ts +db28d1ca42f8408bd0fc529942047217a36142e64137460f7721ab73c5e3506e ./prototype/mvp-resolver.mjs +57b07940e8956e2de6ca9d7e66ebea00c5271a8c7d82543fcd49f3d25fbf1a48 ./prototype/payload-benchmark.mjs +a650497d5b5ac187ce2b871ddbdcb7706618dd756e93692d2a6c1ee9552ab8a5 ./prototype/profile-resolver.mjs +4c1df71867f1bfefb195ff24ec8ed11e153bbf18f18ae6d44d1de5c45886cc2b ./prototype/tool-discovery-profile.test.mjs +511217f0124b3e361338c998a35e5a6aa10da2909e0d1da8d24cf440a878ed16 ./results/TEST_SCOPE.md +bb2cc53b714f09ff36a5f3bbf7d2e51a3808f0cdf01570dd629af20951b5aea8 ./results/apply-draft-synthetic-test.txt +4b6b6c97330f227c6dd9a53c3801e10d5e46042c89124f8d7a3d370c465f3afe ./results/benchmark-summary.md +fa8caea17fa7cf83e7c58fe933e3da0882734cdd910f661cf9373e095e76c43a ./results/current-dev-head.json +06c4fb742eda0fca025d59f9c36633fd74700a02f04b66799f3c7d7fd6565446 ./results/patch-0001-numstat.txt +b97a5d03101c722e34c4fd4774d17cf9125c0d13ab3f083b29f81119537cda5b ./results/patch-0002-numstat.txt +c650d839f049203c93cb70990f1b3f2ba157f9bcf761be45061182ca8a71fef2 ./results/payload-benchmark-1000.json +74621a7a5a2921dd241a55c698da59dc54800f8f324ce247389b54fe07ea4c6c ./results/payload-benchmark-250.json +c86436f1482f50e380acc8bf7c1f42ef51bacd5ac687c5c8862f4ca8c31b7f53 ./results/prototype-summary.json +61605ec685b8d1688906014f67468bc8dead2cd98dd7e9a29a53ea2501fa9713 ./results/prototype-test.txt +af6741eaf44df981f506407024ae8ba4692db8f03680ecb7d6b8a2cf36738de0 ./results/repository-clone-attempt.txt +2f5932d8a481a25a0cca0e9bcfe39c99a84f5dab4702a96a2e487d2222c458f9 ./scripts/inspect-generated-catalog.mjs +e6d4998cc0a5a2a995c4b4590698424d7f992db8260184fa9185b1fb94e74c8f ./scripts/run-prototype-tests.sh +a67a5234f6e96e88595126ccd72eb84ca02765e2713d0240932382985b1b3b66 ./scripts/run-repo-validation.sh +c413909354adaaef33fcc82a4b3e7f7fc35187f72cd3ca769a77f1e5cd01b774 ./scripts/validate-bundle.sh +5ceb92ba799a84aa4f18b1825505c7307906b4f94bcb535628baec5abaef5970 ./sources/SOURCE_INDEX.md diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/patches/0001-add-tool-discovery-module.patch b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/0001-add-tool-discovery-module.patch new file mode 100644 index 000000000..3c2132cca --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/0001-add-tool-discovery-module.patch @@ -0,0 +1,71 @@ +diff --git a/src/codex/catalog/tool-discovery.ts b/src/codex/catalog/tool-discovery.ts +new file mode 100644 +--- /dev/null ++++ b/src/codex/catalog/tool-discovery.ts +@@ -0,0 +1,65 @@ ++import type { ++ OcxProviderConfig, ++ OcxRoutedToolDiscoveryMode, ++} from "../../types"; ++import { modelRecordValue } from "../../reasoning-effort"; ++ ++export type ResolvedRoutedToolDiscoveryMode = "deferred" | "direct"; ++ ++export interface ResolvedRoutedToolDiscovery { ++ readonly mode: ResolvedRoutedToolDiscoveryMode; ++ readonly source: "cursor-hard-fence" | "model-override" | "provider-override" | "default"; ++ readonly configured: OcxRoutedToolDiscoveryMode; ++ readonly warning?: string; ++} ++ ++export const ROUTED_TOOL_DISCOVERY_MODES: readonly OcxRoutedToolDiscoveryMode[] = [ ++ "auto", ++ "deferred", ++ "direct", ++] as const; ++ ++export function resolveRoutedToolDiscoveryMode( ++ providerName: string, ++ provider: Pick< ++ OcxProviderConfig, ++ "adapter" | "routedToolDiscovery" | "modelRoutedToolDiscovery" ++ >, ++ modelId: string, ++): ResolvedRoutedToolDiscovery { ++ const modelConfigured = modelRecordValue(provider.modelRoutedToolDiscovery, modelId); ++ const configured = modelConfigured ?? provider.routedToolDiscovery ?? "auto"; ++ const source: ResolvedRoutedToolDiscovery["source"] = modelConfigured !== undefined ++ ? "model-override" ++ : provider.routedToolDiscovery !== undefined ++ ? "provider-override" ++ : "default"; ++ ++ if (providerName === "cursor" || provider.adapter === "cursor") { ++ return { ++ mode: "direct", ++ source: "cursor-hard-fence", ++ configured, ++ ...(configured === "deferred" ++ ? { warning: "Configured deferred discovery was ignored for Cursor." } ++ : {}), ++ }; ++ } ++ ++ if (configured === "direct") { ++ return { ++ mode: "direct", ++ source, ++ configured, ++ warning: "Direct discovery may expand the first request with full MCP declarations.", ++ }; ++ } ++ ++ return { mode: "deferred", source, configured }; ++} ++ ++export function deriveComboToolDiscoveryMode( ++ memberModes: readonly (ResolvedRoutedToolDiscoveryMode | undefined)[], ++): ResolvedRoutedToolDiscoveryMode { ++ return memberModes.some(mode => mode === "direct") ? "direct" : "deferred"; ++} + diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/patches/0002-focused-test-plan.patch b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/0002-focused-test-plan.patch new file mode 100644 index 000000000..358dc1dcb --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/0002-focused-test-plan.patch @@ -0,0 +1,77 @@ +diff --git a/tests/codex-tool-discovery-mode.test.ts b/tests/codex-tool-discovery-mode.test.ts +new file mode 100644 +--- /dev/null ++++ b/tests/codex-tool-discovery-mode.test.ts +@@ -0,0 +1,72 @@ ++import { describe, expect, test } from "bun:test"; ++import { ++ deriveComboToolDiscoveryMode, ++ resolveRoutedToolDiscoveryMode, ++} from "../src/codex/catalog/tool-discovery"; ++ ++const provider = { ++ adapter: "openai-chat", ++ baseUrl: "https://example.invalid/v1", ++} as const; ++ ++describe("routed tool discovery resolver", () => { ++ test("preserves PR #1596 as the non-Cursor default", () => { ++ expect(resolveRoutedToolDiscoveryMode("external", provider, "model")).toMatchObject({ ++ mode: "deferred", ++ source: "default", ++ configured: "auto", ++ }); ++ }); ++ ++ test("model override wins provider override", () => { ++ expect(resolveRoutedToolDiscoveryMode("external", { ++ ...provider, ++ routedToolDiscovery: "direct", ++ modelRoutedToolDiscovery: { model: "deferred" }, ++ }, "model")).toMatchObject({ mode: "deferred", source: "model-override" }); ++ }); ++ ++ test("direct model override is scoped", () => { ++ const configured = { ++ ...provider, ++ routedToolDiscovery: "deferred" as const, ++ modelRoutedToolDiscovery: { broken: "direct" as const }, ++ }; ++ expect(resolveRoutedToolDiscoveryMode("external", configured, "broken").mode).toBe("direct"); ++ expect(resolveRoutedToolDiscoveryMode("external", configured, "sibling").mode).toBe("deferred"); ++ }); ++ ++ test("Cursor is hard-fenced even when deferred is configured", () => { ++ expect(resolveRoutedToolDiscoveryMode("custom", { ++ ...provider, ++ adapter: "cursor", ++ routedToolDiscovery: "deferred", ++ }, "auto")).toMatchObject({ mode: "direct", source: "cursor-hard-fence" }); ++ }); ++ ++ test("combo direct member wins", () => { ++ expect(deriveComboToolDiscoveryMode(["deferred", "direct"])).toBe("direct"); ++ expect(deriveComboToolDiscoveryMode(["deferred", "deferred"])).toBe("deferred"); ++ }); ++}); ++ ++// Extend tests/catalog-cursor-search.test.ts with: ++// - non-Cursor provider direct -> search false, hosted search kept ++// - model override scoped to one row ++// - Cursor configured deferred -> still false/no hosted search ++// - template-less fallback parity ++ ++// Extend tests/codex-catalog.test.ts with: ++// - combo direct-member composition ++// - bare/slashed combo alias parity ++// - zero-config current-dev behavior ++ ++// Extend tests/config.test.ts and tests/config-user-edits.test.ts with: ++// - valid modes ++// - invalid live-write rejection ++// - invalid hand-edit degradation without provider/key loss ++// - model map prototype/accessor safety ++ ++// Extend tests/e2e-style/phase100-native-parity.test.ts with: ++// - default DeepSeek row remains supports_search_tool:true ++// - exact DeepSeek model override produces false without changing siblings diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/patches/0003-route-scoped-tool-discovery.review.diff b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/0003-route-scoped-tool-discovery.review.diff new file mode 100644 index 000000000..b6954fd8e --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/0003-route-scoped-tool-discovery.review.diff @@ -0,0 +1,288 @@ +diff --git a/src/types.ts b/src/types.ts +--- a/src/types.ts ++++ b/src/types.ts +@@ -1,6 +1,9 @@ + export interface ProviderCostOverlay { + input: number; + output: number; + cacheRead: number; + cacheWrite: number; + } ++ ++/** Catalog policy for MCP/plugin discovery on Codex-routed models. */ ++export type OcxRoutedToolDiscoveryMode = "auto" | "deferred" | "direct"; +@@ -12,4 +15,11 @@ + export interface OcxProviderConfig { + adapter: string; ++ /** ++ * Route-wide Codex discovery policy. `auto` preserves the current default: ++ * non-Cursor deferred, Cursor direct. ++ */ ++ routedToolDiscovery?: OcxRoutedToolDiscoveryMode; ++ /** Exact/modelRecordValue-compatible per-model override. */ ++ modelRoutedToolDiscovery?: Record; + /** Cursor MCP compatibility bounds; positive integers when configured. */ + mcpMaxTools?: number; + +diff --git a/src/codex/catalog/tool-discovery.ts b/src/codex/catalog/tool-discovery.ts +new file mode 100644 +--- /dev/null ++++ b/src/codex/catalog/tool-discovery.ts +@@ -0,0 +1,65 @@ ++import type { ++ OcxProviderConfig, ++ OcxRoutedToolDiscoveryMode, ++} from "../../types"; ++import { modelRecordValue } from "../../reasoning-effort"; ++ ++export type ResolvedRoutedToolDiscoveryMode = "deferred" | "direct"; ++ ++export interface ResolvedRoutedToolDiscovery { ++ readonly mode: ResolvedRoutedToolDiscoveryMode; ++ readonly source: "cursor-hard-fence" | "model-override" | "provider-override" | "default"; ++ readonly configured: OcxRoutedToolDiscoveryMode; ++ readonly warning?: string; ++} ++ ++export const ROUTED_TOOL_DISCOVERY_MODES: readonly OcxRoutedToolDiscoveryMode[] = [ ++ "auto", ++ "deferred", ++ "direct", ++] as const; ++ ++export function resolveRoutedToolDiscoveryMode( ++ providerName: string, ++ provider: Pick< ++ OcxProviderConfig, ++ "adapter" | "routedToolDiscovery" | "modelRoutedToolDiscovery" ++ >, ++ modelId: string, ++): ResolvedRoutedToolDiscovery { ++ const modelConfigured = modelRecordValue(provider.modelRoutedToolDiscovery, modelId); ++ const configured = modelConfigured ?? provider.routedToolDiscovery ?? "auto"; ++ const source: ResolvedRoutedToolDiscovery["source"] = modelConfigured !== undefined ++ ? "model-override" ++ : provider.routedToolDiscovery !== undefined ++ ? "provider-override" ++ : "default"; ++ ++ if (providerName === "cursor" || provider.adapter === "cursor") { ++ return { ++ mode: "direct", ++ source: "cursor-hard-fence", ++ configured, ++ ...(configured === "deferred" ++ ? { warning: "Configured deferred discovery was ignored for Cursor." } ++ : {}), ++ }; ++ } ++ ++ if (configured === "direct") { ++ return { ++ mode: "direct", ++ source, ++ configured, ++ warning: "Direct discovery may expand the first request with full MCP declarations.", ++ }; ++ } ++ ++ return { mode: "deferred", source, configured }; ++} ++ ++export function deriveComboToolDiscoveryMode( ++ memberModes: readonly (ResolvedRoutedToolDiscoveryMode | undefined)[], ++): ResolvedRoutedToolDiscoveryMode { ++ return memberModes.some(mode => mode === "direct") ? "direct" : "deferred"; ++} + +diff --git a/src/codex/catalog/parsing.ts b/src/codex/catalog/parsing.ts +--- a/src/codex/catalog/parsing.ts ++++ b/src/codex/catalog/parsing.ts +@@ -1,1 +1,2 @@ + import { CODEX_NATIVE_ALIAS_CATALOG_KIND } from "./kinds"; ++import type { ResolvedRoutedToolDiscoveryMode } from "./tool-discovery"; +@@ -7,2 +8,4 @@ + /** Provider opted into parallel tool calls (OcxProviderConfig.parallelToolCalls). */ + parallelToolCalls?: boolean; ++ /** Resolved before catalog serialization; `auto` never reaches this layer. */ ++ toolDiscoveryMode?: ResolvedRoutedToolDiscoveryMode; +@@ -14,1 +17,5 @@ +-export function normalizeRoutedCatalogEntry(entry: RawEntry, parallelToolCalls = false): RawEntry { ++export function normalizeRoutedCatalogEntry( ++ entry: RawEntry, ++ parallelToolCalls = false, ++ toolDiscoveryMode: ResolvedRoutedToolDiscoveryMode = "deferred", ++): RawEntry { +@@ -20,6 +27,9 @@ + if (isCursorEntry) { + delete entry.web_search_tool_type; + } else { + entry.web_search_tool_type = "text_and_image"; + } +- entry.supports_search_tool = !isCursorEntry; ++ // Cursor is a hard fence. For every other routed row, an explicit direct ++ // override is a compatibility escape hatch; hosted search stays independent. ++ const effectiveToolDiscoveryMode = isCursorEntry ? "direct" : toolDiscoveryMode; ++ entry.supports_search_tool = effectiveToolDiscoveryMode === "deferred"; + +diff --git a/src/codex/catalog/provider-fetch.ts b/src/codex/catalog/provider-fetch.ts +--- a/src/codex/catalog/provider-fetch.ts ++++ b/src/codex/catalog/provider-fetch.ts +@@ -1,1 +1,2 @@ + import { deriveComboCatalogModel, normalizedOpenAiApiSignature, openAiApiCollisionWarnings, replaceLastComboCatalogOmissions, warnUncataloguedComboOnce } from "./aggregation"; ++import { resolveRoutedToolDiscoveryMode } from "./tool-discovery"; +@@ -7,2 +8,2 @@ + function providerCatalogFingerprint(name: string, prov: OcxProviderConfig): Record { + return { +@@ -14,4 +15,6 @@ + ptc: prov.parallelToolCalls ?? null, ++ routedToolDiscovery: prov.routedToolDiscovery ?? null, ++ modelRoutedToolDiscovery: prov.modelRoutedToolDiscovery ?? null, + gMode: prov.googleMode ?? null, + }; + } +@@ -23,3 +26,2 @@ + export function applyProviderConfigHints(name: string, prov: OcxProviderConfig, model: CatalogModel, providerCap?: number): CatalogModel { +- void name; + const configuredCap = configuredContextWindow(prov, model.id); +@@ -31,2 +33,3 @@ + const supportsReasoningSummaries = configuredReasoningSummarySupport(prov, model.id); ++ const toolDiscovery = resolveRoutedToolDiscoveryMode(name, prov, model.id); + const hinted = { +@@ -38,4 +41,5 @@ + ...(prov.parallelToolCalls === true || (prov.adapter === "openai-chat" && prov.parallelToolCalls !== false) + ? { parallelToolCalls: true } + : {}), ++ toolDiscoveryMode: toolDiscovery.mode, + }; + +diff --git a/src/codex/catalog/aggregation.ts b/src/codex/catalog/aggregation.ts +--- a/src/codex/catalog/aggregation.ts ++++ b/src/codex/catalog/aggregation.ts +@@ -1,1 +1,2 @@ + import type { CatalogModel } from "./parsing"; ++import { deriveComboToolDiscoveryMode } from "./tool-discovery"; +@@ -7,4 +8,7 @@ + const defaultReasoningEffort = effectiveComboDefault( + combo.defaultEffort, + reasoningEfforts, + ); ++ const toolDiscoveryMode = deriveComboToolDiscoveryMode( ++ members.map(member => member.toolDiscoveryMode), ++ ); +@@ -16,6 +20,7 @@ + ...(members.every(member => member.parallelToolCalls === true) + ? { parallelToolCalls: true } + : {}), ++ toolDiscoveryMode, + ...(members.some(member => member.supportsReasoningSummaries === false) ? { supportsReasoningSummaries: false } : {}), + }; + } +@@ -27,2 +32,3 @@ + parallelToolCalls: member?.parallelToolCalls === true, ++ toolDiscoveryMode: member?.toolDiscoveryMode ?? "deferred", + supportsReasoningSummaries: member?.supportsReasoningSummaries !== false, + +diff --git a/src/codex/catalog/sync.ts b/src/codex/catalog/sync.ts +--- a/src/codex/catalog/sync.ts ++++ b/src/codex/catalog/sync.ts +@@ -1,1 +1,5 @@ +- normalizeRoutedCatalogEntry(e, model?.parallelToolCalls === true); ++ normalizeRoutedCatalogEntry( ++ e, ++ model?.parallelToolCalls === true, ++ model?.toolDiscoveryMode ?? "deferred", ++ ); +@@ -7,2 +11,5 @@ + const isCursorFallback = isRouted && model?.provider === "cursor"; ++ const fallbackToolDiscoveryMode = isCursorFallback ++ ? "direct" ++ : model?.toolDiscoveryMode ?? "deferred"; + const entry: RawEntry = { +@@ -14,6 +21,9 @@ +- ...(isRouted +- ? isCursorFallback +- ? { supports_search_tool: false } +- : { web_search_tool_type: "text_and_image", supports_search_tool: true } ++ ...(isRouted ++ ? isCursorFallback ++ ? { supports_search_tool: false } ++ : { ++ web_search_tool_type: "text_and_image", ++ supports_search_tool: fallbackToolDiscoveryMode === "deferred", ++ } + : {}), + }; + +diff --git a/src/config.ts b/src/config.ts +--- a/src/config.ts ++++ b/src/config.ts +@@ -1,1 +1,1 @@ + const retryOn429PolicySchema = z.object({ +@@ -7,1 +7,3 @@ + }).strict(); ++ ++const routedToolDiscoveryModeSchema = z.enum(["auto", "deferred", "direct"]); +@@ -13,3 +15,9 @@ + const providerConfigSchema = z.object({ + adapter: z.string().min(1), + baseUrl: z.string().min(1), ++ // Invalid hand edits degrade on load; validateConfigCandidate inspects the ++ // raw candidate first so live writes still reject invalid modes. ++ routedToolDiscovery: routedToolDiscoveryModeSchema.optional().catch(undefined), ++ modelRoutedToolDiscovery: z.record(z.string(), routedToolDiscoveryModeSchema) ++ .optional() ++ .catch(undefined), +@@ -21,1 +29,32 @@ + }).passthrough(); ++ ++function routedToolDiscoveryBoundaryError(value: unknown): string | null { ++ const root = rawConfigRecord(value); ++ if (!root) return null; ++ const providers = root.providers; ++ if (!providers || typeof providers !== "object" || Array.isArray(providers)) return null; ++ const allowed = new Set(["auto", "deferred", "direct"]); ++ ++ for (const [providerName, rawProvider] of Object.entries(providers as Record)) { ++ if (!rawProvider || typeof rawProvider !== "object" || Array.isArray(rawProvider)) continue; ++ const provider = rawProvider as Record; ++ const mode = provider.routedToolDiscovery; ++ if (mode !== undefined && (typeof mode !== "string" || !allowed.has(mode))) { ++ return `schema_invalid: providers.${redactSecretString(providerName)}.routedToolDiscovery: must be auto, deferred, or direct`; ++ } ++ const modelModes = provider.modelRoutedToolDiscovery; ++ if (modelModes === undefined) continue; ++ if (!modelModes || typeof modelModes !== "object" || Array.isArray(modelModes)) { ++ return `schema_invalid: providers.${redactSecretString(providerName)}.modelRoutedToolDiscovery: must be a plain object`; ++ } ++ for (const [modelId, modelMode] of Object.entries(modelModes as Record)) { ++ if (!modelId.trim()) { ++ return `schema_invalid: providers.${redactSecretString(providerName)}.modelRoutedToolDiscovery: model ids must be nonblank`; ++ } ++ if (typeof modelMode !== "string" || !allowed.has(modelMode)) { ++ return `schema_invalid: providers.${redactSecretString(providerName)}.modelRoutedToolDiscovery.${redactSecretString(modelId)}: must be auto, deferred, or direct`; ++ } ++ } ++ } ++ return null; ++} +@@ -27,2 +66,2 @@ + export function validateConfigCandidate(value: unknown): { ok: true; config: OcxConfig } | { ok: false; error: string } { + const boundaryError = blankHostnameError(value) +@@ -34,1 +73,2 @@ +- ?? loopbackListenerPortError(value); ++ ?? loopbackListenerPortError(value) ++ ?? routedToolDiscoveryBoundaryError(value); +diff --git a/src/codex/catalog.ts b/src/codex/catalog.ts +--- a/src/codex/catalog.ts ++++ b/src/codex/catalog.ts +@@ -1,1 +1,9 @@ + export type { CatalogModel, MultiAgentMode } from "./catalog/parsing"; ++export { ++ deriveComboToolDiscoveryMode, ++ resolveRoutedToolDiscoveryMode, ++} from "./catalog/tool-discovery"; ++export type { ++ ResolvedRoutedToolDiscovery, ++ ResolvedRoutedToolDiscoveryMode, ++} from "./catalog/tool-discovery"; diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/patches/README.md b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/README.md new file mode 100644 index 000000000..e66f13402 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/README.md @@ -0,0 +1,58 @@ +# Patch drafts + +> **Do not apply these drafts as-is (2026-08-13).** An independent audit of the +> landing verification pass found that `apply-draft.mjs` and +> `0003-route-scoped-tool-discovery.review.diff` faithfully reproduce the +> **template/fallback Cursor-fence asymmetry** described in +> `094_landing_verification_pass.md` Correction 8 — the template path keeps +> `entry.slug.startsWith("cursor/")` while the fallback keeps +> `model?.provider === "cursor"`. The applicator still reports `draft seams: OK`, +> because it only asserts that the seams it expects are present; it does not know +> the policy is inconsistent. +> +> These files remain in the unit as the *seam inventory* that made the real +> implementation cheap to plan. The implementation PR resolves the fence from +> provider identity through one shared helper used by both paths, per the amended +> `012`, and its diff will therefore not match these drafts. + +Target inspected: OpenCodex packaging-time `dev` at `2cdbf66a23f9fd8f2f38dcc702ccd3f2e60ac535`. +The tool-discovery semantic base is parent commit `5703473041a9f4f415743652de5d86d51fd66db5` (PR #1596). + +## Status: ARCHIVAL — do not apply + +These drafts are retained as the **seam inventory** that made the implementation +cheap to plan. They are not an application path, and the commands that used to be +recommended here have been withdrawn, because `apply-draft.mjs` writes the +Cursor-fence asymmetry described in `094` Correction 8 while reporting +`draft seams: OK`. + +`apply-draft.mjs` remains useful for reading: it resolves every then-current `dev` +seam before its first write and refuses missing or duplicate seams, so its +`replaceOnce` anchors are an accurate map of where the real change lands. Read it; +do not run it against a repository you intend to commit from. + +The implementation PR resolves the fence from provider identity through one shared +helper used by both the template and template-less paths, so its diff will not +match these drafts. + +## Files + +- `apply-draft.mjs` — assertion-heavy draft applicator. **Archival: do not run.** + Reproduces the `094` Correction 8 fence asymmetry while reporting seams OK. +- `0001-add-tool-discovery-module.patch` — syntactically validated new-module patch. +- `0002-focused-test-plan.patch` — syntactically validated focused test patch/plan. +- `0003-route-scoped-tool-discovery.review.diff` — multi-file review diff. It is intentionally + marked `review.diff`: it is for reading, not for `git apply`, and it carries the same + fence asymmetry as the applicator. +- `proposed/src/codex/catalog/tool-discovery.ts` — complete proposed pure module. +- `proposed/tests/codex-tool-discovery-mode.test.ts` — complete proposed focused test. + +## Verification boundary + +- Node syntax check for `apply-draft.mjs`: passed. +- Synthetic current-seam `--check` and apply exercise: passed; see + `results/apply-draft-synthetic-test.txt`. + Note: that pass proves seam resolution only. It does **not** evaluate policy + correctness, which is why the asymmetry survived a green `--check`. +- Pure resolver prototype: 17/17 passed. +- Full OpenCodex typecheck/Bun suite: not run in this artifact environment. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/patches/apply-draft.mjs b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/apply-draft.mjs new file mode 100755 index 000000000..de5a931ff --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/apply-draft.mjs @@ -0,0 +1,279 @@ +#!/usr/bin/env node +/** + * ARCHIVAL — KNOWN-DEFECTIVE — DO NOT EXECUTE. + * + * This applicator is retained as a SEAM MAP, not as an application path. It + * writes the Cursor-fence asymmetry described in + * ../094_landing_verification_pass.md Correction 8: the template path keeps + * `entry.slug.startsWith("cursor/")` while the template-less path keeps + * `model?.provider === "cursor"`, so a `cursor/`-aliased combo whose canonical + * provider is `combo` is classified differently depending on whether a template + * happened to be available. That is the unresolved #1596 P2 review finding. + * + * It also encodes the obsolete positional third argument to + * `normalizeRoutedCatalogEntry`; the plan now specifies an options object + * `{ toolDiscoveryMode, providerId }` and one shared `isCursorRoute()` helper + * used by BOTH construction paths (see ../012_phase1_catalog_patch.md). + * + * Crucially, `--check` still prints `draft seams: OK` while all of the above is + * true: it asserts only that the seams it expects are present, and knows nothing + * about policy correctness. A green run here is not a verdict. + * + * Read it for the `replaceOnce` anchors — they are an accurate map of where the + * real change lands. Do not run it against a repository you intend to commit + * from. The implementation PR's diff will not match this draft. + * + * Withdrawn usage (kept only so the anchors below read in context): + * node apply-draft.mjs --check /path/to/opencodex + * node apply-draft.mjs /path/to/opencodex + */ + +import fs from "node:fs"; +import path from "node:path"; +import process from "node:process"; +import { fileURLToPath } from "node:url"; + +const args = process.argv.slice(2); +const checkOnly = args.includes("--check"); +const positional = args.filter(arg => arg !== "--check"); +const repo = path.resolve(positional[0] ?? process.cwd()); +const bundle = path.dirname(fileURLToPath(import.meta.url)); +const pendingWrites = []; + +function fail(message) { + console.error(`error: ${message}`); + process.exit(2); +} + +if (!fs.existsSync(path.join(repo, "package.json")) || !fs.existsSync(path.join(repo, "src/codex/catalog/parsing.ts"))) { + fail(`not an OpenCodex worktree: ${repo}`); +} + +function countOccurrences(source, needle) { + let count = 0; + let offset = 0; + while ((offset = source.indexOf(needle, offset)) !== -1) { + count += 1; + offset += needle.length; + } + return count; +} + +function replaceOnce(source, needle, replacement, label) { + const count = countOccurrences(source, needle); + if (count !== 1) { + throw new Error(`${label}: expected one seam, found ${count}`); + } + return source.replace(needle, replacement); +} + +function edit(rel, transform) { + const target = path.join(repo, rel); + const before = fs.readFileSync(target, "utf8"); + const after = transform(before); + if (after === before) throw new Error(`${rel}: transform produced no change`); + pendingWrites.push({ rel, target, content: after, action: "EDIT" }); + if (checkOnly) console.log(`CHECK ${rel}`); +} + +function addFromBundle(rel, sourceRel) { + const target = path.join(repo, rel); + const content = fs.readFileSync(path.join(bundle, sourceRel), "utf8"); + if (fs.existsSync(target)) { + const existing = fs.readFileSync(target, "utf8"); + if (existing !== content) throw new Error(`${rel}: file already exists with different content`); + console.log(`KEEP ${rel}`); + return; + } + pendingWrites.push({ rel, target, content, action: "ADD" }); + if (checkOnly) console.log(`CHECK ${rel}`); +} + +function writeAtomically(target, content) { + fs.mkdirSync(path.dirname(target), { recursive: true }); + const temp = `${target}.ocx-tool-discovery.${process.pid}.tmp`; + fs.writeFileSync(temp, content, "utf8"); + fs.renameSync(temp, target); +} + +try { + addFromBundle( + "src/codex/catalog/tool-discovery.ts", + "proposed/src/codex/catalog/tool-discovery.ts", + ); + addFromBundle( + "tests/codex-tool-discovery-mode.test.ts", + "proposed/tests/codex-tool-discovery-mode.test.ts", + ); + + edit("src/types.ts", source => { + let out = replaceOnce( + source, + "export interface OcxProviderConfig {\n adapter: string;", + "export type OcxRoutedToolDiscoveryMode = \"auto\" | \"deferred\" | \"direct\";\n\nexport interface OcxProviderConfig {\n adapter: string;\n /** Route-wide Codex discovery policy. Default: auto. */\n routedToolDiscovery?: OcxRoutedToolDiscoveryMode;\n /** Per-model override; modelRecordValue matching applies. */\n modelRoutedToolDiscovery?: Record;", + "OcxProviderConfig declaration", + ); + return out; + }); + + edit("src/codex/catalog/parsing.ts", source => { + let out = replaceOnce( + source, + 'import { CODEX_NATIVE_ALIAS_CATALOG_KIND } from "./kinds";', + 'import { CODEX_NATIVE_ALIAS_CATALOG_KIND } from "./kinds";\nimport type { ResolvedRoutedToolDiscoveryMode } from "./tool-discovery";', + "parsing import", + ); + out = replaceOnce( + out, + " /** Provider opted into parallel tool calls (OcxProviderConfig.parallelToolCalls). */\n parallelToolCalls?: boolean;", + " /** Provider opted into parallel tool calls (OcxProviderConfig.parallelToolCalls). */\n parallelToolCalls?: boolean;\n /** Resolved before catalog serialization; auto never reaches this layer. */\n toolDiscoveryMode?: ResolvedRoutedToolDiscoveryMode;", + "CatalogModel field", + ); + out = replaceOnce( + out, + "export function normalizeRoutedCatalogEntry(entry: RawEntry, parallelToolCalls = false): RawEntry {", + "export function normalizeRoutedCatalogEntry(\n entry: RawEntry,\n parallelToolCalls = false,\n toolDiscoveryMode: ResolvedRoutedToolDiscoveryMode = \"deferred\",\n): RawEntry {", + "normalizeRoutedCatalogEntry signature", + ); + out = replaceOnce( + out, + " entry.supports_search_tool = !isCursorEntry;", + " const effectiveToolDiscoveryMode = isCursorEntry ? \"direct\" : toolDiscoveryMode;\n entry.supports_search_tool = effectiveToolDiscoveryMode === \"deferred\";", + "supports_search_tool assignment", + ); + return out; + }); + + edit("src/codex/catalog/provider-fetch.ts", source => { + let out = replaceOnce( + source, + 'import type { CatalogModel } from "./parsing";', + 'import type { CatalogModel } from "./parsing";\nimport { resolveRoutedToolDiscoveryMode } from "./tool-discovery";', + "provider-fetch tool-discovery import", + ); + out = replaceOnce( + out, + " ptc: prov.parallelToolCalls ?? null,\n gMode: prov.googleMode ?? null,", + " ptc: prov.parallelToolCalls ?? null,\n routedToolDiscovery: prov.routedToolDiscovery ?? null,\n modelRoutedToolDiscovery: prov.modelRoutedToolDiscovery ?? null,\n gMode: prov.googleMode ?? null,", + "provider catalog fingerprint", + ); + out = replaceOnce( + out, + "export function applyProviderConfigHints(name: string, prov: OcxProviderConfig, model: CatalogModel, providerCap?: number): CatalogModel {\n void name;", + "export function applyProviderConfigHints(name: string, prov: OcxProviderConfig, model: CatalogModel, providerCap?: number): CatalogModel {", + "applyProviderConfigHints name seam", + ); + out = replaceOnce( + out, + " const supportsReasoningSummaries = configuredReasoningSummarySupport(prov, model.id);\n const hinted = {", + " const supportsReasoningSummaries = configuredReasoningSummarySupport(prov, model.id);\n const toolDiscovery = resolveRoutedToolDiscoveryMode(name, prov, model.id);\n const hinted = {", + "provider hint resolver", + ); + out = replaceOnce( + out, + " ...(prov.parallelToolCalls === true || (prov.adapter === \"openai-chat\" && prov.parallelToolCalls !== false)\n ? { parallelToolCalls: true }\n : {}),\n };", + " ...(prov.parallelToolCalls === true || (prov.adapter === \"openai-chat\" && prov.parallelToolCalls !== false)\n ? { parallelToolCalls: true }\n : {}),\n toolDiscoveryMode: toolDiscovery.mode,\n };", + "provider hint output", + ); + return out; + }); + + edit("src/codex/catalog/aggregation.ts", source => { + let out = replaceOnce( + source, + 'import type { CatalogModel } from "./parsing";', + 'import type { CatalogModel } from "./parsing";\nimport { deriveComboToolDiscoveryMode } from "./tool-discovery";', + "aggregation tool-discovery import", + ); + out = replaceOnce( + out, + " const defaultReasoningEffort = effectiveComboDefault(\n combo.defaultEffort,\n reasoningEfforts,\n );", + " const defaultReasoningEffort = effectiveComboDefault(\n combo.defaultEffort,\n reasoningEfforts,\n );\n const toolDiscoveryMode = deriveComboToolDiscoveryMode(\n members.map(member => member.toolDiscoveryMode),\n );", + "combo policy derivation", + ); + out = replaceOnce( + out, + " ...(members.every(member => member.parallelToolCalls === true)\n ? { parallelToolCalls: true }\n : {}),\n ...(members.some(member => member.supportsReasoningSummaries === false) ? { supportsReasoningSummaries: false } : {}),", + " ...(members.every(member => member.parallelToolCalls === true)\n ? { parallelToolCalls: true }\n : {}),\n toolDiscoveryMode,\n ...(members.some(member => member.supportsReasoningSummaries === false) ? { supportsReasoningSummaries: false } : {}),", + "combo return field", + ); + out = replaceOnce( + out, + " parallelToolCalls: member?.parallelToolCalls === true,\n supportsReasoningSummaries: member?.supportsReasoningSummaries !== false,", + " parallelToolCalls: member?.parallelToolCalls === true,\n toolDiscoveryMode: member?.toolDiscoveryMode ?? \"deferred\",\n supportsReasoningSummaries: member?.supportsReasoningSummaries !== false,", + "combo warning signature", + ); + return out; + }); + + edit("src/codex/catalog/sync.ts", source => { + let out = replaceOnce( + source, + " normalizeRoutedCatalogEntry(e, model?.parallelToolCalls === true);", + " normalizeRoutedCatalogEntry(\n e,\n model?.parallelToolCalls === true,\n model?.toolDiscoveryMode ?? \"deferred\",\n );", + "template catalog normalization call", + ); + out = replaceOnce( + out, + " const isCursorFallback = isRouted && model?.provider === \"cursor\";\n const entry: RawEntry = {", + " const isCursorFallback = isRouted && model?.provider === \"cursor\";\n const fallbackToolDiscoveryMode = isCursorFallback\n ? \"direct\"\n : model?.toolDiscoveryMode ?? \"deferred\";\n const entry: RawEntry = {", + "fallback resolver", + ); + out = replaceOnce( + out, + " : { web_search_tool_type: \"text_and_image\", supports_search_tool: true }", + " : {\n web_search_tool_type: \"text_and_image\",\n supports_search_tool: fallbackToolDiscoveryMode === \"deferred\",\n }", + "fallback search metadata", + ); + return out; + }); + + edit("src/config.ts", source => { + let out = replaceOnce( + source, + ").strict();\n\n/**\n * Zod schema for one provider entry:", + ").strict();\n\nconst routedToolDiscoveryModeSchema = z.enum([\"auto\", \"deferred\", \"direct\"]);\n\n/**\n * Zod schema for one provider entry:", + "provider schema preamble", + ); + out = replaceOnce( + out, + "const providerConfigSchema = z.object({\n adapter: z.string().min(1),\n baseUrl: z.string().min(1),", + "const providerConfigSchema = z.object({\n adapter: z.string().min(1),\n baseUrl: z.string().min(1),\n routedToolDiscovery: routedToolDiscoveryModeSchema.optional().catch(undefined),\n modelRoutedToolDiscovery: z.record(z.string(), routedToolDiscoveryModeSchema)\n .optional()\n .catch(undefined),", + "provider discovery schema fields", + ); + const boundary = `\nfunction routedToolDiscoveryBoundaryError(value: unknown): string | null {\n const root = rawConfigRecord(value);\n if (!root) return null;\n const providers = root.providers;\n if (!providers || typeof providers !== \"object\" || Array.isArray(providers)) return null;\n const allowed = new Set([\"auto\", \"deferred\", \"direct\"]);\n for (const [providerName, rawProvider] of Object.entries(providers as Record)) {\n if (!rawProvider || typeof rawProvider !== \"object\" || Array.isArray(rawProvider)) continue;\n const provider = rawProvider as Record;\n const mode = provider.routedToolDiscovery;\n if (mode !== undefined && (typeof mode !== \"string\" || !allowed.has(mode))) {\n return \`schema_invalid: providers.\${redactSecretString(providerName)}.routedToolDiscovery: must be auto, deferred, or direct\`;\n }\n const modelModes = provider.modelRoutedToolDiscovery;\n if (modelModes === undefined) continue;\n if (!modelModes || typeof modelModes !== \"object\" || Array.isArray(modelModes)) {\n return \`schema_invalid: providers.\${redactSecretString(providerName)}.modelRoutedToolDiscovery: must be a plain object\`;\n }\n for (const [modelId, modelMode] of Object.entries(modelModes as Record)) {\n if (!modelId.trim()) return \`schema_invalid: providers.\${redactSecretString(providerName)}.modelRoutedToolDiscovery: model ids must be nonblank\`;\n if (typeof modelMode !== \"string\" || !allowed.has(modelMode)) {\n return \`schema_invalid: providers.\${redactSecretString(providerName)}.modelRoutedToolDiscovery.\${redactSecretString(modelId)}: must be auto, deferred, or direct\`;\n }\n }\n }\n return null;\n}\n`; + out = replaceOnce( + out, + "/** Validate an in-memory config candidate without touching disk. Used by headless CLI import/set. */", + `${boundary}\n/** Validate an in-memory config candidate without touching disk. Used by headless CLI import/set. */`, + "write-boundary insertion", + ); + out = replaceOnce( + out, + " ?? loopbackListenerPortError(value);", + " ?? loopbackListenerPortError(value)\n ?? routedToolDiscoveryBoundaryError(value);", + "write-boundary chain", + ); + return out; + }); + + edit("src/codex/catalog.ts", source => replaceOnce( + source, + 'export type { CatalogModel, MultiAgentMode } from "./catalog/parsing";', + 'export type { CatalogModel, MultiAgentMode } from "./catalog/parsing";\nexport { deriveComboToolDiscoveryMode, resolveRoutedToolDiscoveryMode } from "./catalog/tool-discovery";\nexport type { ResolvedRoutedToolDiscovery, ResolvedRoutedToolDiscoveryMode } from "./catalog/tool-discovery";', + "catalog facade export", + )); + + if (checkOnly) { + console.log("draft seams: OK"); + } else { + // All seam assertions above succeeded before the first write. + for (const item of pendingWrites) { + writeAtomically(item.target, item.content); + console.log(`${item.action.padEnd(5)} ${item.rel}`); + } + console.log("draft applied; review git diff before testing"); + } +} catch (error) { + fail(error instanceof Error ? error.message : String(error)); +} diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/patches/proposed/src/codex/catalog/tool-discovery.ts b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/proposed/src/codex/catalog/tool-discovery.ts new file mode 100644 index 000000000..f89062718 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/proposed/src/codex/catalog/tool-discovery.ts @@ -0,0 +1,71 @@ +import type { + OcxProviderConfig, + OcxRoutedToolDiscoveryMode, +} from "../../types"; +import { modelRecordValue } from "../../reasoning-effort"; + +export type ResolvedRoutedToolDiscoveryMode = "deferred" | "direct"; + +export interface ResolvedRoutedToolDiscovery { + readonly mode: ResolvedRoutedToolDiscoveryMode; + readonly source: "cursor-hard-fence" | "model-override" | "provider-override" | "default"; + readonly configured: OcxRoutedToolDiscoveryMode; + readonly warning?: string; +} + +export const ROUTED_TOOL_DISCOVERY_MODES: readonly OcxRoutedToolDiscoveryMode[] = [ + "auto", + "deferred", + "direct", +] as const; + +/** + * Resolve one routed provider/model to a catalog-facing mode. `auto` never reaches + * serialization. Cursor is a hard fence because its custom runTurn transport has + * no verified deferred/sidecar route. + */ +export function resolveRoutedToolDiscoveryMode( + providerName: string, + provider: Pick< + OcxProviderConfig, + "adapter" | "routedToolDiscovery" | "modelRoutedToolDiscovery" + >, + modelId: string, +): ResolvedRoutedToolDiscovery { + const modelConfigured = modelRecordValue(provider.modelRoutedToolDiscovery, modelId); + const configured = modelConfigured ?? provider.routedToolDiscovery ?? "auto"; + const source: ResolvedRoutedToolDiscovery["source"] = modelConfigured !== undefined + ? "model-override" + : provider.routedToolDiscovery !== undefined + ? "provider-override" + : "default"; + + if (providerName === "cursor" || provider.adapter === "cursor") { + return { + mode: "direct", + source: "cursor-hard-fence", + configured, + ...(configured === "deferred" + ? { warning: "Configured deferred discovery was ignored for Cursor." } + : {}), + }; + } + + if (configured === "direct") { + return { + mode: "direct", + source, + configured, + warning: "Direct discovery may expand the first request with full MCP declarations.", + }; + } + + return { mode: "deferred", source, configured }; +} + +/** A combo uses one catalog row, so one direct-only member forces direct. */ +export function deriveComboToolDiscoveryMode( + memberModes: readonly (ResolvedRoutedToolDiscoveryMode | undefined)[], +): ResolvedRoutedToolDiscoveryMode { + return memberModes.some(mode => mode === "direct") ? "direct" : "deferred"; +} diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/patches/proposed/tests/codex-tool-discovery-mode.test.ts b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/proposed/tests/codex-tool-discovery-mode.test.ts new file mode 100644 index 000000000..a5944e213 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/patches/proposed/tests/codex-tool-discovery-mode.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, test } from "bun:test"; +import { + deriveComboToolDiscoveryMode, + resolveRoutedToolDiscoveryMode, +} from "../src/codex/catalog/tool-discovery"; + +const provider = { + adapter: "openai-chat", + baseUrl: "https://example.invalid/v1", +} as const; + +describe("routed tool discovery resolver", () => { + test("preserves PR #1596 as the non-Cursor default", () => { + expect(resolveRoutedToolDiscoveryMode("external", provider, "model")).toMatchObject({ + mode: "deferred", + source: "default", + configured: "auto", + }); + }); + + test("model override wins provider override", () => { + expect(resolveRoutedToolDiscoveryMode("external", { + ...provider, + routedToolDiscovery: "direct", + modelRoutedToolDiscovery: { model: "deferred" }, + }, "model")).toMatchObject({ mode: "deferred", source: "model-override" }); + }); + + test("direct model override is scoped", () => { + const configured = { + ...provider, + routedToolDiscovery: "deferred" as const, + modelRoutedToolDiscovery: { broken: "direct" as const }, + }; + expect(resolveRoutedToolDiscoveryMode("external", configured, "broken").mode).toBe("direct"); + expect(resolveRoutedToolDiscoveryMode("external", configured, "sibling").mode).toBe("deferred"); + }); + + test("Cursor is hard-fenced even when deferred is configured", () => { + expect(resolveRoutedToolDiscoveryMode("custom", { + ...provider, + adapter: "cursor", + routedToolDiscovery: "deferred", + }, "auto")).toMatchObject({ + mode: "direct", + source: "cursor-hard-fence", + }); + }); + + test("combo direct member wins", () => { + expect(deriveComboToolDiscoveryMode(["deferred", "direct"])).toBe("direct"); + expect(deriveComboToolDiscoveryMode(["deferred", "deferred"])).toBe("deferred"); + }); +}); diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/mvp-resolver.mjs b/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/mvp-resolver.mjs new file mode 100644 index 000000000..25dfdae7d --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/mvp-resolver.mjs @@ -0,0 +1,115 @@ +/** + * Executable, dependency-free prototype of the proposed phase-1 resolver. + * It is intentionally separate from the OpenCodex source tree so the policy can + * be tested in this artifact bundle without claiming a full repository build. + */ + +export const ROUTED_TOOL_DISCOVERY_MODES = Object.freeze([ + "auto", + "deferred", + "direct", +]); + +export function isRoutedToolDiscoveryMode(value) { + return ROUTED_TOOL_DISCOVERY_MODES.includes(value); +} + +function configuredMode({ providerMode, modelMode }) { + if (modelMode !== undefined) { + if (!isRoutedToolDiscoveryMode(modelMode)) { + throw new TypeError(`invalid model routed-tool discovery mode: ${String(modelMode)}`); + } + return { configured: modelMode, source: "model-override" }; + } + if (providerMode !== undefined) { + if (!isRoutedToolDiscoveryMode(providerMode)) { + throw new TypeError(`invalid provider routed-tool discovery mode: ${String(providerMode)}`); + } + return { configured: providerMode, source: "provider-override" }; + } + return { configured: "auto", source: "default" }; +} + +/** + * Resolve the catalog-facing mode. `auto` never reaches serialization. + * + * Precedence: + * Cursor hard fence > exact model override > provider override > auto default. + */ +export function resolveRoutedToolDiscovery({ + providerName, + adapter, + providerMode, + modelMode, +}) { + const isCursor = providerName === "cursor" || adapter === "cursor"; + const selected = configuredMode({ providerMode, modelMode }); + + if (isCursor) { + return Object.freeze({ + mode: "direct", + source: "cursor-hard-fence", + configured: selected.configured, + reason: "Cursor's runTurn transport has no verified deferred/sidecar path.", + warning: selected.configured === "deferred" + ? "Configured deferred mode was ignored for Cursor." + : undefined, + }); + } + + if (selected.configured === "direct") { + return Object.freeze({ + mode: "direct", + source: selected.source, + configured: selected.configured, + reason: "An explicit route-scoped compatibility override selected direct discovery.", + warning: "Direct discovery can expand the first request in proportion to MCP schema bytes.", + }); + } + + return Object.freeze({ + mode: "deferred", + source: selected.source, + configured: selected.configured, + reason: selected.configured === "deferred" + ? "The route explicitly selected Codex deferred discovery." + : "Non-Cursor auto mode preserves the PR #1596 Code Mode default.", + warning: undefined, + }); +} + +/** Apply only the catalog fields owned by this policy. */ +export function applyRoutedToolDiscoveryPolicy(entry, resolved) { + if (!entry || typeof entry !== "object" || Array.isArray(entry)) { + throw new TypeError("entry must be a plain object"); + } + if (!resolved || (resolved.mode !== "deferred" && resolved.mode !== "direct")) { + throw new TypeError("resolved mode must be deferred or direct"); + } + + const clone = structuredClone(entry); + const isCursor = typeof clone.slug === "string" && clone.slug.startsWith("cursor/"); + clone.tool_mode = "code_mode_only"; + + // Hosted search remains independent from MCP/plugin discovery. + if (isCursor) { + delete clone.web_search_tool_type; + } else { + clone.web_search_tool_type = "text_and_image"; + } + clone.supports_search_tool = !isCursor && resolved.mode === "deferred"; + return clone; +} + +/** Direct wins for a combo because one public row cannot vary after target selection. */ +export function deriveComboToolDiscoveryMode(memberModes) { + if (!Array.isArray(memberModes) || memberModes.length === 0) { + return "deferred"; + } + for (const mode of memberModes) { + if (mode !== undefined && mode !== "deferred" && mode !== "direct") { + throw new TypeError(`invalid resolved member mode: ${String(mode)}`); + } + } + return memberModes.some(mode => mode === "direct") ? "direct" : "deferred"; +} diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/payload-benchmark.mjs b/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/payload-benchmark.mjs new file mode 100644 index 000000000..5a158dd5d --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/payload-benchmark.mjs @@ -0,0 +1,87 @@ +import { Buffer } from "node:buffer"; + +const TOOL_COUNT = Number.parseInt(process.argv[2] ?? "250", 10); +if (!Number.isSafeInteger(TOOL_COUNT) || TOOL_COUNT < 1 || TOOL_COUNT > 10_000) { + throw new TypeError("tool count must be an integer from 1 to 10000"); +} + +function tool(index) { + const namespace = `mcp__service_${String(index % 17).padStart(2, "0")}`; + const name = `${namespace}__operation_${String(index).padStart(4, "0")}`; + return { + type: "function", + name, + description: `Operate service resource ${index}. This synthetic description represents a realistic MCP routing hint and intentionally repeats domain terms for retrieval.`, + strict: true, + parameters: { + type: "object", + additionalProperties: false, + properties: { + resource_id: { type: "string", description: "Stable resource identifier." }, + query: { type: "string", description: "User-provided search or mutation query." }, + options: { + type: "object", + additionalProperties: false, + properties: { + limit: { type: "integer", minimum: 1, maximum: 100 }, + include_archived: { type: "boolean" }, + fields: { type: "array", items: { type: "string" } }, + }, + }, + }, + required: ["resource_id"], + }, + }; +} + +const tools = Array.from({ length: TOOL_COUNT }, (_, index) => tool(index + 1)); +const eagerExecDescription = JSON.stringify({ + exec: "Run JavaScript to call nested tools.", + nested_tool_declarations: tools, +}); +const allToolsIndex = JSON.stringify(tools.map(({ name, description }) => ({ name, description }))); +const metaTools = JSON.stringify([ + { + name: "ocx_tool_search", + description: "Search the authorized MCP tool index by exact name, namespace, prefix, or terms.", + parameters: { type: "object", properties: { query: { type: "string" }, limit: { type: "integer" } }, required: ["query"] }, + }, + { + name: "ocx_tool_describe", + description: "Return bounded schemas for one to three exact qualified tool names.", + parameters: { type: "object", properties: { names: { type: "array", maxItems: 3, items: { type: "string" } } }, required: ["names"] }, + }, + { + name: "ocx_tool_call", + description: "Call one exact qualified tool through the normal authorization and logging path.", + parameters: { type: "object", properties: { name: { type: "string" }, arguments: { type: "object" } }, required: ["name", "arguments"] }, + }, +]); + +const bytes = value => Buffer.byteLength(value, "utf8"); +const eagerBytes = bytes(eagerExecDescription); +const indexBytes = bytes(allToolsIndex); +const metaBytes = bytes(metaTools); + +const result = { + generatedAt: new Date().toISOString(), + unit: "UTF-8 bytes", + note: "Synthetic structural benchmark; it is not a tokenizer or a capture of a live Codex request.", + toolCount: TOOL_COUNT, + modes: { + eagerFullSchemas: eagerBytes, + codeModeNameDescriptionIndex: indexBytes, + threeMetaTools: metaBytes, + }, + ratios: { + eagerToCodeModeIndex: Number((eagerBytes / indexBytes).toFixed(3)), + eagerToMetaTools: Number((eagerBytes / metaBytes).toFixed(3)), + codeModeIndexToMetaTools: Number((indexBytes / metaBytes).toFixed(3)), + }, + perToolBytes: { + eagerFullSchemas: Number((eagerBytes / TOOL_COUNT).toFixed(2)), + codeModeNameDescriptionIndex: Number((indexBytes / TOOL_COUNT).toFixed(2)), + }, +}; + +process.stdout.write(`${JSON.stringify(result, null, 2)}\n`); diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/profile-resolver.mjs b/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/profile-resolver.mjs new file mode 100644 index 000000000..75d1fce37 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/profile-resolver.mjs @@ -0,0 +1,51 @@ +/** Future four-profile resolver used by phases 3-6 of the roadmap. */ + +export const TOOL_DISCOVERY_PROFILES = Object.freeze([ + "codex-local-code-mode", + "native-tool-search", + "proxy-meta-tools", + "direct-bounded", +]); + +export function resolveToolDiscoveryProfile(capabilities) { + const c = capabilities ?? {}; + + if (c.isCursorSurface === true) { + return { + profile: "direct-bounded", + reason: "Cursor is hard-fenced until its transport proves a deferred path.", + }; + } + + if (c.hasCodeModeRuntime === true && c.hasAllToolsIndex === true) { + return { + profile: "codex-local-code-mode", + reason: "The client owns the tools registry and exposes tools/ALL_TOOLS locally.", + }; + } + + if ( + c.supportsNativeToolSearch === true + && c.preservesResponsesLiteAdditionalTools === true + && c.preservesCustomTools === true + && c.preservesNamespaceTools === true + && c.preservesToolSearchHistory === true + ) { + return { + profile: "native-tool-search", + reason: "The active protocol path passed the complete native discovery round trip.", + }; + } + + if (c.hasMetaToolSidecar === true) { + return { + profile: "proxy-meta-tools", + reason: "Native discovery is unverified, but a bounded search/describe/call sidecar exists.", + }; + } + + return { + profile: "direct-bounded", + reason: "No safe deferred or meta-tool route is available.", + }; +} diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/tool-discovery-profile.test.mjs b/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/tool-discovery-profile.test.mjs new file mode 100644 index 000000000..378b56fab --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/prototype/tool-discovery-profile.test.mjs @@ -0,0 +1,146 @@ +import test from "node:test"; +import assert from "node:assert/strict"; + +import { + applyRoutedToolDiscoveryPolicy, + deriveComboToolDiscoveryMode, + resolveRoutedToolDiscovery, +} from "./mvp-resolver.mjs"; +import { resolveToolDiscoveryProfile } from "./profile-resolver.mjs"; + +const base = { + providerName: "opencode-go", + adapter: "openai-chat", +}; + +test("non-Cursor default preserves deferred discovery", () => { + const result = resolveRoutedToolDiscovery(base); + assert.equal(result.mode, "deferred"); + assert.equal(result.source, "default"); + assert.equal(result.configured, "auto"); +}); + +test("provider auto resolves to deferred", () => { + assert.equal(resolveRoutedToolDiscovery({ ...base, providerMode: "auto" }).mode, "deferred"); +}); + +test("provider deferred resolves to deferred", () => { + assert.equal(resolveRoutedToolDiscovery({ ...base, providerMode: "deferred" }).mode, "deferred"); +}); + +test("provider direct resolves to direct and emits payload warning", () => { + const result = resolveRoutedToolDiscovery({ ...base, providerMode: "direct" }); + assert.equal(result.mode, "direct"); + assert.match(result.warning, /first request/i); +}); + +test("model direct wins over provider deferred", () => { + const result = resolveRoutedToolDiscovery({ + ...base, + providerMode: "deferred", + modelMode: "direct", + }); + assert.equal(result.mode, "direct"); + assert.equal(result.source, "model-override"); +}); + +test("model deferred wins over provider direct", () => { + const result = resolveRoutedToolDiscovery({ + ...base, + providerMode: "direct", + modelMode: "deferred", + }); + assert.equal(result.mode, "deferred"); + assert.equal(result.source, "model-override"); +}); + +test("Cursor provider name is hard-fenced to direct", () => { + const result = resolveRoutedToolDiscovery({ + providerName: "cursor", + adapter: "openai-responses", + }); + assert.equal(result.mode, "direct"); + assert.equal(result.source, "cursor-hard-fence"); +}); + +test("custom Cursor adapter is hard-fenced even under another provider name", () => { + const result = resolveRoutedToolDiscovery({ + providerName: "my-cursor", + adapter: "cursor", + modelMode: "deferred", + }); + assert.equal(result.mode, "direct"); + assert.match(result.warning, /ignored/i); +}); + +test("non-Cursor direct mode keeps hosted search independent", () => { + const resolved = resolveRoutedToolDiscovery({ ...base, providerMode: "direct" }); + const row = applyRoutedToolDiscoveryPolicy({ slug: "opencode-go/glm-5.2" }, resolved); + assert.equal(row.tool_mode, "code_mode_only"); + assert.equal(row.supports_search_tool, false); + assert.equal(row.web_search_tool_type, "text_and_image"); +}); + +test("non-Cursor deferred mode pins code mode and search together", () => { + const resolved = resolveRoutedToolDiscovery(base); + const row = applyRoutedToolDiscoveryPolicy({ slug: "opencode-go/glm-5.2" }, resolved); + assert.equal(row.tool_mode, "code_mode_only"); + assert.equal(row.supports_search_tool, true); +}); + +test("Cursor catalog row never advertises hosted or deferred search", () => { + const resolved = resolveRoutedToolDiscovery({ + providerName: "cursor", + adapter: "cursor", + modelMode: "deferred", + }); + const row = applyRoutedToolDiscoveryPolicy({ + slug: "cursor/auto", + web_search_tool_type: "text_and_image", + }, resolved); + assert.equal(row.supports_search_tool, false); + assert.ok(!Object.hasOwn(row, "web_search_tool_type")); +}); + +test("invalid override fails loudly in the prototype", () => { + assert.throws( + () => resolveRoutedToolDiscovery({ ...base, providerMode: "eager" }), + /invalid provider/i, + ); +}); + +test("combo direct mode wins over deferred members", () => { + assert.equal(deriveComboToolDiscoveryMode(["deferred", "direct"]), "direct"); + assert.equal(deriveComboToolDiscoveryMode(["deferred", "deferred"]), "deferred"); + assert.equal(deriveComboToolDiscoveryMode([undefined, "deferred"]), "deferred"); +}); + +test("future resolver prefers local Code Mode", () => { + assert.equal(resolveToolDiscoveryProfile({ + hasCodeModeRuntime: true, + hasAllToolsIndex: true, + hasMetaToolSidecar: true, + }).profile, "codex-local-code-mode"); +}); + +test("future resolver chooses verified native tool search only with complete round trip", () => { + assert.equal(resolveToolDiscoveryProfile({ + supportsNativeToolSearch: true, + preservesResponsesLiteAdditionalTools: true, + preservesCustomTools: true, + preservesNamespaceTools: true, + preservesToolSearchHistory: true, + }).profile, "native-tool-search"); +}); + +test("future resolver falls back to proxy meta-tools", () => { + assert.equal(resolveToolDiscoveryProfile({ + supportsNativeToolSearch: true, + preservesResponsesLiteAdditionalTools: false, + hasMetaToolSidecar: true, + }).profile, "proxy-meta-tools"); +}); + +test("future resolver finally falls back to bounded direct mode", () => { + assert.equal(resolveToolDiscoveryProfile({}).profile, "direct-bounded"); +}); diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/TEST_SCOPE.md b/devlog/_plan/260813_routed_tool_discovery_profiles/results/TEST_SCOPE.md new file mode 100644 index 000000000..f2049571b --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/TEST_SCOPE.md @@ -0,0 +1,31 @@ +# Test scope and limitations + +## Executed here + +```bash +node --test prototype/tool-discovery-profile.test.mjs +``` + +Result: 17 passed, 0 failed. + +Executed synthetic structural benchmarks: + +```bash +node prototype/payload-benchmark.mjs 250 +node prototype/payload-benchmark.mjs 1000 +``` + +## Not executed here + +- `bun run typecheck`; +- focused OpenCodex Bun tests; +- full OpenCodex test suite; +- Codex CLI/App live canary; +- Browser plugin E2E; +- real request/token/cache capture. + +Reason: this artifact runtime has no Bun installation and external DNS is blocked, so the full repository could not be cloned. Source and patches were inspected through the connected GitHub integration. + +## Required next validation + +Run `scripts/run-repo-validation.sh` inside a clean OpenCodex `dev` worktree after applying the patch. Then perform the scenarios in `040`–`049` on machines with real client/plugin credentials. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/apply-draft-synthetic-test.txt b/devlog/_plan/260813_routed_tool_discovery_profiles/results/apply-draft-synthetic-test.txt new file mode 100644 index 000000000..9844efb71 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/apply-draft-synthetic-test.txt @@ -0,0 +1,21 @@ +CHECK src/codex/catalog/tool-discovery.ts +CHECK tests/codex-tool-discovery-mode.test.ts +CHECK src/types.ts +CHECK src/codex/catalog/parsing.ts +CHECK src/codex/catalog/provider-fetch.ts +CHECK src/codex/catalog/aggregation.ts +CHECK src/codex/catalog/sync.ts +CHECK src/config.ts +CHECK src/codex/catalog.ts +draft seams: OK +ADD src/codex/catalog/tool-discovery.ts +ADD tests/codex-tool-discovery-mode.test.ts +EDIT src/types.ts +EDIT src/codex/catalog/parsing.ts +EDIT src/codex/catalog/provider-fetch.ts +EDIT src/codex/catalog/aggregation.ts +EDIT src/codex/catalog/sync.ts +EDIT src/config.ts +EDIT src/codex/catalog.ts +draft applied; review git diff before testing +synthetic seam application: PASS diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/benchmark-summary.md b/devlog/_plan/260813_routed_tool_discovery_profiles/results/benchmark-summary.md new file mode 100644 index 000000000..235772068 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/benchmark-summary.md @@ -0,0 +1,18 @@ +# Executed benchmark summary + +The included benchmark generated synthetic MCP functions with nested schemas and compared UTF-8 +payload structures. This is a structural scaling test, not a tokenizer or live Codex request capture. + +| Tools | Eager full schemas | Code Mode name/description index | Fixed meta-tools | +|---:|---:|---:|---:| +| 250 | 179,218 B | 52,393 B | 744 B | +| 1,000 | 716,969 B | 209,894 B | 744 B | + +At 250 tools, eager full declarations were 3.421× the name/description index. At 1,000 tools, +the per-tool sizes remained approximately linear while the fixed meta-tool declaration stayed +constant. + +Raw outputs: + +- `payload-benchmark-250.json` +- `payload-benchmark-1000.json` diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/current-dev-head.json b/devlog/_plan/260813_routed_tool_discovery_profiles/results/current-dev-head.json new file mode 100644 index 000000000..0113837de --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/current-dev-head.json @@ -0,0 +1,12 @@ +{ + "checkedAtKst": "2026-08-13", + "branch": "dev", + "packagingTimeHead": "2cdbf66a23f9fd8f2f38dcc702ccd3f2e60ac535", + "message": "Merge pull request #1600 ... scale timing watchdogs and per-test timeout for loaded CI runners", + "directParents": [ + "5703473041a9f4f415743652de5d86d51fd66db5", + "12c4fe603c607064dcf83d652ec6fefe913a18ec" + ], + "toolDiscoverySemanticBase": "5703473041a9f4f415743652de5d86d51fd66db5", + "note": "Relevant catalog/config source seams were fetched from dev after the head advanced. PR #1600 is a CI timeout/watchdog merge; PR #1596 remains the direct parent carrying the discovery policy change." +} diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/patch-0001-numstat.txt b/devlog/_plan/260813_routed_tool_discovery_profiles/results/patch-0001-numstat.txt new file mode 100644 index 000000000..ccc120d32 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/patch-0001-numstat.txt @@ -0,0 +1 @@ +65 0 src/codex/catalog/tool-discovery.ts diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/patch-0002-numstat.txt b/devlog/_plan/260813_routed_tool_discovery_profiles/results/patch-0002-numstat.txt new file mode 100644 index 000000000..e93cc8358 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/patch-0002-numstat.txt @@ -0,0 +1 @@ +72 0 tests/codex-tool-discovery-mode.test.ts diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/payload-benchmark-1000.json b/devlog/_plan/260813_routed_tool_discovery_profiles/results/payload-benchmark-1000.json new file mode 100644 index 000000000..041cdc3d1 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/payload-benchmark-1000.json @@ -0,0 +1,20 @@ +{ + "generatedAt": "2026-08-13T10:49:31.303Z", + "unit": "UTF-8 bytes", + "note": "Synthetic structural benchmark; it is not a tokenizer or a capture of a live Codex request.", + "toolCount": 1000, + "modes": { + "eagerFullSchemas": 716969, + "codeModeNameDescriptionIndex": 209894, + "threeMetaTools": 744 + }, + "ratios": { + "eagerToCodeModeIndex": 3.416, + "eagerToMetaTools": 963.668, + "codeModeIndexToMetaTools": 282.116 + }, + "perToolBytes": { + "eagerFullSchemas": 716.97, + "codeModeNameDescriptionIndex": 209.89 + } +} diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/payload-benchmark-250.json b/devlog/_plan/260813_routed_tool_discovery_profiles/results/payload-benchmark-250.json new file mode 100644 index 000000000..e8b0d5698 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/payload-benchmark-250.json @@ -0,0 +1,20 @@ +{ + "generatedAt": "2026-08-13T10:49:31.275Z", + "unit": "UTF-8 bytes", + "note": "Synthetic structural benchmark; it is not a tokenizer or a capture of a live Codex request.", + "toolCount": 250, + "modes": { + "eagerFullSchemas": 179218, + "codeModeNameDescriptionIndex": 52393, + "threeMetaTools": 744 + }, + "ratios": { + "eagerToCodeModeIndex": 3.421, + "eagerToMetaTools": 240.884, + "codeModeIndexToMetaTools": 70.421 + }, + "perToolBytes": { + "eagerFullSchemas": 716.87, + "codeModeNameDescriptionIndex": 209.57 + } +} diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/prototype-summary.json b/devlog/_plan/260813_routed_tool_discovery_profiles/results/prototype-summary.json new file mode 100644 index 000000000..30d85cce8 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/prototype-summary.json @@ -0,0 +1,13 @@ +{ + "suite": "dependency-free routed tool discovery policy prototype", + "runtime": "Node.js v22.16.0", + "tests": 17, + "passed": 17, + "failed": 0, + "fullRepositorySuiteExecuted": false, + "limitations": [ + "No complete OpenCodex checkout was available in the artifact runtime", + "Bun typecheck and repository test suite were not executed", + "Live Codex CLI/App and Browser plugin scenarios require external clients and credentials" + ] +} diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/prototype-test.txt b/devlog/_plan/260813_routed_tool_discovery_profiles/results/prototype-test.txt new file mode 100644 index 000000000..6f97658fe --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/prototype-test.txt @@ -0,0 +1,112 @@ +TAP version 13 +# Subtest: non-Cursor default preserves deferred discovery +ok 1 - non-Cursor default preserves deferred discovery + --- + duration_ms: 0.840758 + type: 'test' + ... +# Subtest: provider auto resolves to deferred +ok 2 - provider auto resolves to deferred + --- + duration_ms: 0.112347 + type: 'test' + ... +# Subtest: provider deferred resolves to deferred +ok 3 - provider deferred resolves to deferred + --- + duration_ms: 0.119187 + type: 'test' + ... +# Subtest: provider direct resolves to direct and emits payload warning +ok 4 - provider direct resolves to direct and emits payload warning + --- + duration_ms: 0.287757 + type: 'test' + ... +# Subtest: model direct wins over provider deferred +ok 5 - model direct wins over provider deferred + --- + duration_ms: 0.117104 + type: 'test' + ... +# Subtest: model deferred wins over provider direct +ok 6 - model deferred wins over provider direct + --- + duration_ms: 0.171975 + type: 'test' + ... +# Subtest: Cursor provider name is hard-fenced to direct +ok 7 - Cursor provider name is hard-fenced to direct + --- + duration_ms: 0.096844 + type: 'test' + ... +# Subtest: custom Cursor adapter is hard-fenced even under another provider name +ok 8 - custom Cursor adapter is hard-fenced even under another provider name + --- + duration_ms: 0.117234 + type: 'test' + ... +# Subtest: non-Cursor direct mode keeps hosted search independent +ok 9 - non-Cursor direct mode keeps hosted search independent + --- + duration_ms: 0.306916 + type: 'test' + ... +# Subtest: non-Cursor deferred mode pins code mode and search together +ok 10 - non-Cursor deferred mode pins code mode and search together + --- + duration_ms: 0.453894 + type: 'test' + ... +# Subtest: Cursor catalog row never advertises hosted or deferred search +ok 11 - Cursor catalog row never advertises hosted or deferred search + --- + duration_ms: 0.20838 + type: 'test' + ... +# Subtest: invalid override fails loudly in the prototype +ok 12 - invalid override fails loudly in the prototype + --- + duration_ms: 0.32988 + type: 'test' + ... +# Subtest: combo direct mode wins over deferred members +ok 13 - combo direct mode wins over deferred members + --- + duration_ms: 0.121831 + type: 'test' + ... +# Subtest: future resolver prefers local Code Mode +ok 14 - future resolver prefers local Code Mode + --- + duration_ms: 0.184104 + type: 'test' + ... +# Subtest: future resolver chooses verified native tool search only with complete round trip +ok 15 - future resolver chooses verified native tool search only with complete round trip + --- + duration_ms: 0.10801 + type: 'test' + ... +# Subtest: future resolver falls back to proxy meta-tools +ok 16 - future resolver falls back to proxy meta-tools + --- + duration_ms: 0.06755 + type: 'test' + ... +# Subtest: future resolver finally falls back to bounded direct mode +ok 17 - future resolver finally falls back to bounded direct mode + --- + duration_ms: 0.052869 + type: 'test' + ... +1..17 +# tests 17 +# suites 0 +# pass 17 +# fail 0 +# cancelled 0 +# skipped 0 +# todo 0 +# duration_ms 60.962763 diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/results/repository-clone-attempt.txt b/devlog/_plan/260813_routed_tool_discovery_profiles/results/repository-clone-attempt.txt new file mode 100644 index 000000000..4b8e1ba7e --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/results/repository-clone-attempt.txt @@ -0,0 +1,9 @@ +Command attempted: + git clone --depth 1 --branch dev https://github.com/lidge-jun/opencodex.git /mnt/data/opencodex-dev-verify + +Result: + fatal: unable to access 'https://github.com/lidge-jun/opencodex.git/': Could not resolve host: github.com + +Disposition: + No full-repository build or Bun test result is claimed. GitHub source inspection, + dependency-free prototypes, and worktree validation scripts are kept separate. diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/inspect-generated-catalog.mjs b/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/inspect-generated-catalog.mjs new file mode 100755 index 000000000..d83edaeb9 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/inspect-generated-catalog.mjs @@ -0,0 +1,23 @@ +#!/usr/bin/env node +import fs from "node:fs"; + +const [path, selector] = process.argv.slice(2); +if (!path || !selector) { + console.error("usage: inspect-generated-catalog.mjs "); + process.exit(2); +} +const parsed = JSON.parse(fs.readFileSync(path, "utf8")); +const row = parsed.models?.find(model => model?.slug === selector); +if (!row) { + console.error(`model not found: ${selector}`); + process.exit(1); +} +const output = { + slug: row.slug, + tool_mode: row.tool_mode, + supports_search_tool: row.supports_search_tool, + web_search_tool_type: row.web_search_tool_type, + supports_parallel_tool_calls: row.supports_parallel_tool_calls, + context_window: row.context_window, +}; +console.log(JSON.stringify(output, null, 2)); diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/run-prototype-tests.sh b/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/run-prototype-tests.sh new file mode 100755 index 000000000..7458bdecd --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/run-prototype-tests.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +set -euo pipefail +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +node --test "$ROOT/prototype/tool-discovery-profile.test.mjs" +node "$ROOT/prototype/payload-benchmark.mjs" 250 +node "$ROOT/prototype/payload-benchmark.mjs" 1000 diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/run-repo-validation.sh b/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/run-repo-validation.sh new file mode 100755 index 000000000..f7df3f1eb --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/run-repo-validation.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Run from the root of a clean OpenCodex checkout after applying the draft patch. +if [[ ! -f package.json || ! -d src/codex ]]; then + echo "error: run this script from the OpenCodex repository root" >&2 + exit 2 +fi + +bun run typecheck + +bun test \ + tests/codex-tool-discovery-mode.test.ts \ + tests/catalog-cursor-search.test.ts \ + tests/codex-catalog.test.ts \ + tests/config.test.ts \ + tests/config-user-edits.test.ts \ + tests/e2e-style/phase100-native-parity.test.ts + +# Full regression and privacy gates used by the repository. +bun run test +bun run privacy:scan diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/validate-bundle.sh b/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/validate-bundle.sh new file mode 100755 index 000000000..249a13783 --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/scripts/validate-bundle.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +set -euo pipefail +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +missing=0 +for n in 000 001 002 003 004 005 006 007 008 009 010 020 030 040 050 060 070 080 090; do + if ! compgen -G "$ROOT/${n}_*.md" >/dev/null; then + echo "missing numbered document group: $n" >&2 + missing=1 + fi +done +[[ $missing -eq 0 ]] + +find "$ROOT" -type f -size 0 -print -quit | grep -q . && { + echo "zero-length file found" >&2 + exit 1 +} || true + +node --test "$ROOT/prototype/tool-discovery-profile.test.mjs" >/dev/null +node "$ROOT/prototype/payload-benchmark.mjs" 10 >/dev/null + +echo "bundle validation: OK" diff --git a/devlog/_plan/260813_routed_tool_discovery_profiles/sources/SOURCE_INDEX.md b/devlog/_plan/260813_routed_tool_discovery_profiles/sources/SOURCE_INDEX.md new file mode 100644 index 000000000..5bba3c93f --- /dev/null +++ b/devlog/_plan/260813_routed_tool_discovery_profiles/sources/SOURCE_INDEX.md @@ -0,0 +1,42 @@ +# Source index + +Verified on 2026-08-13 unless noted. + +## OpenCodex + +- Packaging-time `dev` head: https://github.com/lidge-jun/opencodex/commit/2cdbf66a23f9fd8f2f38dcc702ccd3f2e60ac535 +- Tool-discovery base / PR #1596 merge: https://github.com/lidge-jun/opencodex/commit/5703473041a9f4f415743652de5d86d51fd66db5 +- Issue #1522: https://github.com/lidge-jun/opencodex/issues/1522 +- PR #1529: https://github.com/lidge-jun/opencodex/pull/1529 +- PR #1596: https://github.com/lidge-jun/opencodex/pull/1596 +- Current catalog parser: https://github.com/lidge-jun/opencodex/blob/dev/src/codex/catalog/parsing.ts +- Current catalog sync: https://github.com/lidge-jun/opencodex/blob/dev/src/codex/catalog/sync.ts +- Provider catalog hints: https://github.com/lidge-jun/opencodex/blob/dev/src/codex/catalog/provider-fetch.ts +- Combo aggregation: https://github.com/lidge-jun/opencodex/blob/dev/src/codex/catalog/aggregation.ts +- Config/type surfaces: https://github.com/lidge-jun/opencodex/blob/dev/src/config.ts and https://github.com/lidge-jun/opencodex/blob/dev/src/types.ts + +## OpenAI Codex + +- Code Mode globals: https://github.com/openai/codex/blob/main/codex-rs/code-mode-runtime/src/runtime/globals.rs +- Code Mode description/protocol: https://github.com/openai/codex/blob/main/codex-rs/code-mode-protocol/src/description.rs +- Code Mode tests: https://github.com/openai/codex/blob/main/codex-rs/core/tests/suite/code_mode.rs + +## Comparator implementations + +- CLIProxyAPI tool-search round trip: https://github.com/router-for-me/CLIProxyAPI/issues/3361 +- CLIProxyAPI Responses Lite `additional_tools`: https://github.com/router-for-me/CLIProxyAPI/issues/4798 +- LiteLLM MCP virtual search/call PR: https://github.com/BerriAI/litellm/pull/31777 +- LiteLLM top-k issue: https://github.com/BerriAI/litellm/issues/33440 +- Cloudflare Code Mode MCP: https://github.com/cloudflare/mcp + +## Lifecycle/cache issue evidence + +- Claude Code tool-array cache mutation: https://github.com/anthropics/claude-code/issues/81967 +- Deferred acquisition reliability: https://github.com/anthropics/claude-code/issues/84312 +- `tools/list_changed` stale index: https://github.com/anthropics/claude-code/issues/66084 +- transient tool reference poisoning: https://github.com/anthropics/claude-code/issues/79970 +- large ToolSearch batch cache rebuild: https://github.com/anthropics/claude-code/issues/83756 + +## Evidence policy + +Comparator issue reports are used as design and test evidence, not as proof that OpenCodex has the identical defect. OpenCodex conclusions are scoped separately.