feat(models): discover the harness's own model list instead of hardcoding it by davidliuk · Pull Request #213 · OpenLAIR/dr-claw

davidliuk · 2026-08-05T18:56:50Z

希望可以支持自动识别目前选的 harness 里面可选的模型（比如 codex、claude code 等，他们的模型列表可能会更新新模型，我们希望可以自动用他的列表，不需要我们反复去适配）

The drift is already real

	built-in list	what the tool actually serves
Codex (codex-cli 0.145.0)	`gpt-5.6`, `gpt-5.3-codex`, `gpt-5.2-codex`, `gpt-5.2`, `gpt-5.1-codex-max`, `o3`, `o4-mini`	`gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.3-codex-spark`
OpenRouter	~60 pinned	342 live, including `anthropic/claude-opus-5`

Note gpt-5.6 vs gpt-5.6-sol — the pinned default is a model the CLI no longer serves.

How it discovers

Provider	Source
Codex	`codex app-server` line-delimited JSON-RPC → `initialize`, then `model/list` with `nextCursor` pagination. This is the same catalogue the Codex CLI's own picker reads. Honours `CODEX_CLI_PATH`.
OpenRouter	`GET https://openrouter.ai/api/v1/models` (public, no key)
Claude / Cursor / Gemini / Nano	Built-in list — these CLIs expose no model-listing command today
Local GPU	Ollama `/api/tags`, unchanged

Adding a harness later means adding one function to the DISCOVERERS map.

It can never make things worse

This repo just spent a PR fixing hangs, so the failure modes got the most attention:

Never blocks. Any failure — CLI missing, old, logged out, unresponsive — falls back to the built-in list. GET /api/models/:provider never rejects, so the picker always renders.
Hard timeout. 15s per probe, and SIGTERM escalates to SIGKILL after 2s so a CLI that traps signals can't outlive the probe holding its pipes open. The kill timer is unref'd.
Bounded pagination. Capped at 10 pages so a harness handing back an endless cursor can't spin.
Self-healing cache. Success cached 10 min; failure re-tried after 1 min, so the picker recovers on its own once a CLI is installed or logged in.
No stale writes. Concurrent callers collapse onto one probe. Each probe carries a generation stamp, so a slow superseded probe can't land last and undo the refresh that replaced it.
No stranded preferences. Built-in models the harness no longer serves stay in the picker, marked deprecated. But if the selected model is one of them, the client moves to the harness's own default rather than silently submitting a model the harness will reject.

API

Endpoint	Description
`GET /api/models/:provider`	`source` is `discovered` or `static`. `?refresh=1` bypasses the cache.
`POST /api/models/:provider/refresh`	Drop cache and re-probe — for right after a CLI upgrade or login.
`GET /api/models/providers`	Providers this build can probe.

The /model slash command now reads from the same source, so the two lists can't disagree.

Testing

Against the real Codex CLI, end to end through the Express route:

GET /api/models/codex      -> 200 in 824ms   source=discovered  default=gpt-5.6-sol  count=14
GET /api/models/openrouter -> 200 in 555ms   source=discovered  count=342
cached codex route call: 3ms

26 tests drive the real discovery code against a fake harness that speaks the same JSON-RPC over stdio — no CLI install required in CI. Covered: multi-page pagination, endless-cursor bounding, JSON-RPC errors, non-JSON banner output, silent hangs, hidden-model filtering, multi-byte UTF-8 split across stdout chunks, TTL behaviour, concurrency collapse, forced refresh vs. in-flight stale probe, and unknown providers.

Full suite 133 passed; npm run typecheck and npm run build clean.

Review: Codex reviewed this independently and reported 7 issues. Six were real and are fixed here with tests: the discovered default being ignored by the UI, SIGTERM never escalating, forced refresh joining a stale in-flight probe, pagination stopping after two pages, the hook keeping the previous provider's options during a switch, and UTF-8 corruption from per-chunk toString(). The seventh — a claim that discovery could never work without an initialized notification — is contradicted by the live run above, which succeeded without it; the notification is sent anyway, since the documented lifecycle expects it and other versions may enforce it.

Docs

docs/configuration.md gains a Model Discovery section covering sources, caching, timeouts, and the API.

…ding it The model lists in shared/modelConstants.js are hand-maintained, so every time a CLI ships a new model somebody has to notice and send a patch — and until they do the picker offers models that no longer exist while hiding the ones that do. That drift is already real. codex-cli 0.145.0 serves gpt-5.6-sol, gpt-5.4-mini and gpt-5.3-codex-spark; the pinned list still offered gpt-5.6, o3 and o4-mini. OpenRouter's live catalogue is 342 models including claude-opus-5, against ~60 pinned. This asks the tool instead: - Codex via `codex app-server` line-delimited JSON-RPC (initialize, then model/list with nextCursor pagination) — the same catalogue the Codex CLI's own picker reads. - OpenRouter via its public /api/v1/models endpoint. - Everything else keeps its built-in list; those CLIs expose no equivalent. Discovery is strictly additive and can never make things worse: - Any failure — CLI missing, old, logged out, unresponsive — falls back to the built-in list. The endpoint never rejects, so the picker always renders. - 15s hard timeout per probe, with SIGTERM escalating to SIGKILL so a CLI that traps signals cannot outlive the probe holding its pipes. - Results cached 10 minutes; failures re-tried after 1 minute so the picker recovers on its own once a CLI is installed or logged in. - Concurrent callers collapse onto one probe. Probes carry a generation stamp so a slow superseded probe cannot land last and undo a refresh. - Models in the built-in list that the harness no longer serves stay in the picker, marked deprecated, so a saved preference is never stranded — but if the *selected* model is one of them, the client moves to the harness default rather than submitting a model the harness will reject. New: GET /api/models/:provider (?refresh=1), POST /api/models/:provider/refresh, GET /api/models/providers. The /model slash command now uses the same source. Verified against the real codex CLI end to end: 824ms cold, 3ms cached, correct live list and default. 26 tests drive the real code against a fake harness covering pagination, endless cursors, JSON-RPC errors, non-JSON banner output, silent hangs, multi-byte UTF-8 split across stdout chunks, cache/TTL behaviour, concurrency collapse, and stale-probe ordering. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR adds server-side model discovery so the UI and /model command can prefer each harness’s live model catalogue (Codex via JSON-RPC; OpenRouter via HTTP) instead of relying solely on compiled-in lists, while keeping safe fallbacks, caching, and timeouts.

Changes:

Introduces a discovery subsystem (harnessModelDiscovery) with caching, timeouts, pagination bounding, and fallback-to-static behavior.
Adds protected /api/models/* endpoints and updates the /model slash command to read from the same source.
Updates the chat model picker to consume discovered model options via a new useHarnessModels hook, and documents the behavior.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/components/chat/view/subcomponents/ChatComposer.tsx	Uses `useHarnessModels` to prefer discovered options and rescues retired stored selections to a harness default.
src/components/chat/hooks/useHarnessModels.ts	New client hook to fetch `/api/models/:provider` with caching/refresh behavior.
server/utils/harnessModelDiscovery.js	New discovery implementation for Codex/OpenRouter with merge + caching + concurrency collapse.
server/utils/tests/harnessModelDiscovery.test.js	Unit/integration-style tests for discovery transport, pagination, timeouts, cache semantics, and edge cases.
server/routes/models.js	New Express routes for model discovery, refresh, and provider listing.
server/routes/commands.js	Updates `/model` command to use `getModelsForProvider` instead of constants.
server/index.js	Mounts the new `/api/models` routes under auth.
server/tests/models-route.test.mjs	HTTP-level tests for `/api/models/*` route behavior.
docs/configuration.md	Adds a “Model Discovery” section describing sources, caching/timeouts, and API.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

bbsngg

Read through the whole diff and ran the two test files locally with vitest (CI only runs typecheck + build): 26/26 pass. The fallback discipline (hard timeouts, SIGKILL escalation, generation-stamped cache writes, static fallback on every failure path) is exactly right, so this can't break the picker when a CLI is missing or hangs. Approving.

Non-blocking follow-ups, none of which need to hold up the merge:

The three Copilot notes are all valid but minor: mergeModelOptions drops description; /model awaits Codex discovery with the full 15s timeout (only bites when Codex is installed but unresponsive, since a missing binary fails instantly with ENOENT); refreshToken > 0 makes ?refresh=1 sticky for the rest of the mount.
One extra edge case in the ChatComposer "rescue" effect: for a provider with ALLOWS_CUSTOM (OpenRouter), a user who typed a custom model id that isn't in the openrouter.ai catalogue (e.g. a relay-only model, cf. #124) will be silently switched back to the default. Suggest skipping the rescue when modelConfig.ALLOWS_CUSTOM is true, or only rescuing when the current value is marked deprecated rather than merely absent.

Note for merge order: #216 currently contains this PR's commit plus one Opus 5 commit, so merging this first and rebasing #216 keeps history clean.

- mergeModelOptions keeps discoverer metadata (description, isDefault) instead of stripping it from the API response - /model slash command caps the Codex probe at 3s so a hung CLI cannot make an interactive command feel frozen - useHarnessModels: refresh() forces exactly one uncached request; later provider switches use the cache again - ChatComposer: never auto-rescue the selected model on providers that accept free-form ids (OpenRouter), where absence from the catalogue is not evidence the harness rejects it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x

bbsngg · 2026-09-03T18:04:45Z

Pushed 15b4f53 on this branch addressing the follow-ups from my review, so nothing is left for a later PR:

mergeModelOptions now spreads the discovered option, keeping description / isDefault in the API payload (new test covers it).
/model slash command probes Codex with a 3s cap instead of the default 15s.
useHarnessModels.refresh() forces exactly one uncached request; subsequent provider switches use the cache again.
ChatComposer skips the auto-rescue on providers with ALLOWS_CUSTOM (OpenRouter), so a relay-only model id is never silently replaced.

Local: vitest 27/27 on the two discovery test files, npm run typecheck clean. @davidliuk feel free to amend if you'd rather structure any of it differently.

bbsngg

Re-approving after the follow-up commit (15b4f53): CI green, vitest 27/27 and typecheck clean locally.

With #213 merged, Codex and OpenRouter ask their harness for a live model list, but Claude still shipped a hand-maintained table, which is why Opus 5 needed a patch at all. Add a Claude discoverer that opens an Agent SDK session in streaming-input mode and reads supportedModels() from the control channel: the list then tracks whatever the bundled CLI serves (the 0.3.226 CLI in this PR lists Opus 5 and Fable 5.1). The probe consumes no turn, disables settings sources so no hooks fire, and closes the session on every path including timeout. Claude Code accepts model ids it does not advertise, so the provider is flagged acceptsUnlisted: built-in entries are kept without a deprecated marker and the configured default stands, which also keeps the picker's auto-rescue from firing on Claude. Also treat any `[1m]` alias as a 1M-context model in both token-usage tables, since the discovered menu is made of such aliases, and add the Fable 5.1 entries. The /model command caps the Claude probe at 3s like Codex. Docs updated; tests cover the happy path, timeout, and SDK errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x

…list (#216) * feat(claude): add Opus 5 support * feat(models): discover the Claude CLI's model list through the Agent SDK With #213 merged, Codex and OpenRouter ask their harness for a live model list, but Claude still shipped a hand-maintained table, which is why Opus 5 needed a patch at all. Add a Claude discoverer that opens an Agent SDK session in streaming-input mode and reads supportedModels() from the control channel: the list then tracks whatever the bundled CLI serves (the 0.3.226 CLI in this PR lists Opus 5 and Fable 5.1). The probe consumes no turn, disables settings sources so no hooks fire, and closes the session on every path including timeout. Claude Code accepts model ids it does not advertise, so the provider is flagged acceptsUnlisted: built-in entries are kept without a deprecated marker and the configured default stands, which also keeps the picker's auto-rescue from firing on Claude. Also treat any `[1m]` alias as a 1M-context model in both token-usage tables, since the discovered menu is made of such aliases, and add the Fable 5.1 entries. The /model command caps the Claude probe at 3s like Codex. Docs updated; tests cover the happy path, timeout, and SDK errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x --------- Co-authored-by: zhangsiqi044 <zhangsiqi044@ke.com> Co-authored-by: bbsngg <bbsngg@outlook.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…er (#218) * chore(repo): remove stray node_modules symlink and ignore the symlink form A node_modules symlink pointing at a developer's local path was committed in #213 (2ef73b0). It slipped past .gitignore because `node_modules/` with a trailing slash only matches directories, not symlinks. Drop the tracked symlink and add the slash-less pattern so it cannot recur. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x * fix(skills): quote the autoresearch description so its frontmatter parses The unquoted description contains `: `, which js-yaml reads as a nested mapping and rejects. gray-matter therefore threw on this file, which made scripts/export-skills-catalog-v2.mjs abort on main and left autoresearch out of any catalog regeneration. With the value quoted, all 173 SKILL.md files parse and the export script runs to completion. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* fix(models): keep discovery per-provider, surface Fable 5.1, and drive Codex effort from the CLI Three problems reported after #213/#216 landed, all in the model picker. 1. Switching provider could write one provider's model into another's slot. useHarnessModels updated its state asynchronously, so for one render after a switch it still held the previous provider's list while `provider` already named the new one. ChatComposer's rescue effect then saw a Claude model "missing" from Codex's list and replaced it with Codex's default, persisting `claude-model` = `gpt-5.6-sol`. The hook now tags every answer with the provider it was fetched for and only ever exposes an answer for the current provider. useChatProviderState also discards a stored model that cannot belong to its slot, so users already hit by this get their default back instead of a failing session. 2. Fable 5.1 was not selectable. The Claude probe disables settings sources (so no hooks fire), which also hid the model the user configured in ~/.claude/settings.json; that is where the CLI's own /model menu gets Fable 5.1 from. Discovery now reads `model` from the user settings file (honouring CLAUDE_CONFIG_DIR) and ANTHROPIC_MODEL and offers it, and Fable 5.1 is in the built-in list too so it is there even when discovery is unavailable. Duplicate CLI display names ("Fable" for both Fable 5 and Fable 5.1) are disambiguated from the description. 3. No reasoning-effort selector for discovered Codex models. Support was a table keyed by exact model name, so `gpt-5.6-sol` and friends fell to "default only". Codex's model/list reports supportedReasoningEfforts and defaultReasoningEffort per model; discovery now carries them through and the selector, its gating, and the saved-effort normalisation prefer that list over the table. The `ultra` effort the CLI offers is added to the UI and locales. useHarnessModels moved up to ChatInterface so composer state and the picker share one answer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x * fix(models): stop the picker showing one model under two names - Codex built-in list uses the ids codex-cli actually serves: the retired `gpt-5.6` (labelled "GPT-5.6 (Sol)") is replaced by `gpt-5.6-sol`, and `gpt-5.4-mini` / `gpt-5.3-codex-spark` are added; default follows. - The picker no longer lists a built-in entry the harness has retired unless it is the selected value. The API still returns such entries flagged deprecated, and the picker labels a selected one "not in CLI list", so a saved preference is neither hidden nor mistaken for a live model. - Claude menu labels are expanded from the CLI's description so they stand apart from the built-in entries: "Fable" becomes "Fable 5 [1M]", "Sonnet" becomes "Sonnet 5", "Default (recommended)" becomes "Default (Opus 5 with 1M context)". Descriptions show as tooltips. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x * fix(models): fold built-in entries behind a toggle once the harness has answered With the CLI menu and the compiled-in table shown side by side, the same Claude model appeared twice ("Fable 5 [1M]" from the CLI next to "Fable 5" from the table, likewise Fable 5.1). Mark table entries builtIn in the discovery payload and have the picker show, like Claude Code's own /model menu, only what the harness reports plus the configured and selected models; the rest sit behind a "show N more built-in models" row so an explicit version such as Opus 4.6 is still one click away. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x * fix(models): show only the harness's list once it has answered The built-in table is a fallback for when discovery is unavailable, not a second list to browse: drop the "show more built-in models" toggle and list only what the harness reported, plus the currently selected value so a saved preference never vanishes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x * fix(models): close the model picker on Escape An open picker that ignores Escape stays in the way of the next click, which is how a stray selection can land on the wrong row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NBqmTvJRhyysMvMvL6cp7x --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI lite review requested due to automatic review settings August 5, 2026 18:56

Copilot started reviewing on behalf of davidliuk August 5, 2026 18:57 View session

Copilot AI reviewed Aug 5, 2026

View reviewed changes

Comment thread server/utils/harnessModelDiscovery.js Outdated

Comment thread server/routes/commands.js

Comment thread src/components/chat/hooks/useHarnessModels.ts

davidliuk mentioned this pull request Aug 5, 2026

feat(pi): add the Pi coding agent (pi.dev) as a provider #214

Open

bbsngg previously approved these changes Sep 3, 2026

View reviewed changes

bbsngg dismissed their stale review via 15b4f53 September 3, 2026 18:04

bbsngg approved these changes Sep 3, 2026

View reviewed changes

bbsngg merged commit 2ef73b0 into main Sep 3, 2026
3 checks passed

bbsngg mentioned this pull request Sep 3, 2026

Feat/opus5 support #216

Merged

bbsngg mentioned this pull request Sep 3, 2026

fix: drop stray node_modules symlink and quote autoresearch frontmatter #218

Merged

bbsngg mentioned this pull request Sep 4, 2026

fix(models): per-provider discovery state, Fable 5.1 in the picker, Codex effort from the CLI #219

Merged

Navigation Menu

Uh oh!

feat(models): discover the harness's own model list instead of hardcoding it - #213

feat(models): discover the harness's own model list instead of hardcoding it#213
bbsngg merged 2 commits into
mainfrom
feat/harness-model-discovery

davidliuk commented Aug 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bbsngg left a comment

Uh oh!

bbsngg commented Sep 3, 2026

Uh oh!

bbsngg left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

davidliuk commented Aug 5, 2026

The drift is already real

How it discovers

It can never make things worse

API

Testing

Docs

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bbsngg left a comment

Choose a reason for hiding this comment

Uh oh!

bbsngg commented Sep 3, 2026

Uh oh!

bbsngg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants