feat(coach): local / self-hosted LLM provider with an optional API key - #51
Merged
Conversation
Adds a LOCAL_OPENAI_COMPAT coach provider pointing at any OpenAI-Chat-Completions-compatible server the user runs themselves — Ollama, llama.cpp (llama-server), vLLM, SGLang, LM Studio. Research and the full rationale are in docs/local-llm-coach.md. Key differences from the hosted providers, each forced by real local behavior: - The API key is OPTIONAL. All of these run unauthenticated by default, so requiring a key would leave the coach permanently off for the normal setup. The readiness sentinel that gates CoachFeatureFlags.coachEnabled becomes the base URL instead. - `developer` is folded to `system` and every system turn is merged into one leading message. SGLang validates roles against a pydantic Literal and raises (HTTP 400) outside it; vLLM accepts the role but hands it to a chat template with no branch for it. Many local templates also require the system turn to be first and singular. - Tool calling and structured output are user-declared switches, not assumptions: vLLM 400s on `tools` without --enable-auto-tool-choice, and LM Studio has no json_object mode. Defaults are the combination that works everywhere (tools on, response_format off + prompt-injected schema). - Cleartext HTTP is enabled via network_security_config.xml, with LocalEndpoint.validate enforcing the private/loopback-only rule that Network Security Config has no CIDR syntax for. - Read timeout is configurable (default 180 s); ResponsesHttp gains a per-call override and a GET for /v1/models model discovery. Local turns are priced at $0 regardless of the model name, so a slug like `qwen3:8b` can't prefix-match into a cloud rate. Also fixes the Settings hub trailing label, which showed the OpenAI model slug for MiniMax. 46 unit tests across LocalEndpoint, LocalOpenAICompatClient, LocalModelCatalog and the resolver.
…ities One press of "Detect server & configure" now sets up the local provider from just a base URL. The two settings most likely to fail a turn — `tools` and `response_format` — are gated by the server's LAUNCH FLAGS, which no metadata endpoint exposes (`/v1/models` describes the model, not the request surface). So LocalCapabilityProbe sends the fields and reads the answer: 1. GET /v1/models — reachability, model list, sole-model shortcut. Only fatal step. 2. Engine identity, best-effort, from each engine's own info route: /version (vLLM), /api/version (Ollama), /props (llama.cpp), /get_server_info (SGLang), /api/v0/models (LM Studio). Deliberately not `owned_by`, which every proxy rewrites. Cosmetic only. 3. A chat request carrying one throwaway tool. 4. A chat request carrying a minimal json_schema; json_object only if that is refused. 4xx = the server refused the field (vLLM answers 400 for a disabled tool parser and 422 for an unknown field, so the status carries no extra meaning). 5xx or transport failure = inconclusive, and the setting is LEFT AT ITS DEFAULT rather than switched off, with a note saying why. Tool calling only ever turns off on an explicit refusal — an inconclusive probe must not silently strip the coach of its ability to read the user's data. The probe also never picks a model when several are served and none matches the current one; guessing would move a working setup. Corrects docs/local-llm-coach.md §2 on vLLM. Verified against a live vLLM 0.27.1 server: `role: developer` returns HTTP 422 "unknown role: developer". The earlier claim that vLLM tolerates the role came from v0.11.0's pydantic models (extra="allow"); current vLLM deserializes strictly. Two of the five engines now hard-fail without the fold, so the developer -> system fold is load-bearing rather than defensive. Also: a reasoning model truncated mid-thought (content null, finish_reason "length") now reports the token limit and points at the Max tokens field instead of a bare "model returned no output". vLLM 0.27 puts the chain of thought in `message.reasoning`, older builds and SGLang in `reasoning_content`; the adapter reads neither.
…xt window
Auto-detect now reads the context window each engine advertises and fills in Max tokens from it:
vLLM `max_model_len`, llama.cpp `n_ctx`/`n_ctx_train`, LM Studio `loaded_context_length`/
`max_context_length`, Ollama `model_info["<arch>.context_length"]` via POST /api/show, SGLang
/get_model_info. The summary line now ends with e.g. "262k ctx".
It is a DERIVATION, not a copy. A context window is prompt + completion, and a server checks
`max_tokens` against what is left after the prompt — so copying 262144 across would have the
request rejected outright on any server where the prompt matters. Instead:
headroom = context - 6144 (measured coach prompt 3.1-3.3k, doubled for tool results
and replayed history)
suggested = min(headroom, 32768) (a coach_response needs far less; the cap stops a 262k
context becoming a runaway generation budget)
headroom < 512 -> leave blank and warn
The warning is the more valuable half. Ollama's default num_ctx is 2048 — smaller than the coach's
own prompt — so without detection the prompt is silently truncated and the model gets blamed. The
note points at the server-side fix (num_ctx / -c / --max-model-len), not at an app setting, because
no app setting can help there.
13 new tests: the derivation (cap, reserve, prompt+budget always fits, Ollama-2048 flagged rather
than budgeted, unreported context stays blank) and per-engine field parsing.
Detect server & configure is also how you refresh the model list, so it gets pressed on setups that already work. It was overwriting settings the user had chosen by hand: - Tool calling and Response format are now only written when the probe reached a verdict. `suggested*`'s safe defaults (tools ON, structured OFF) are right for a first run and wrong for a re-detect — a user who turned tools off for a vLLM server without --enable-auto-tool-choice had them switched back on, and every turn 400'd. Probes are skipped entirely when pickModel returns blank on a multi-model server. - Max tokens is no longer cleared when the server reports no context window. 0 means "not detected", not "discard what the user typed". - The capability probes are now measured against a baseline chat request carrying no optional fields. Without it any whole-request rejection — an unloadable model id, an auth-gated chat route, a broken chat template — read as "tools: not supported" and persisted toolCalling = false, costing the coach all data access while blaming the wrong thing. A refused baseline skips the probes and changes nothing. Also: - stripThinking handles a leading unmatched </think>. R1-style distills on llama.cpp/Ollama get the opening tag from the chat template, so the completion starts mid-thought; the whole chain of thought was reaching CoachResponseParser and burning maxFinalAttempts repair generations at up to 180 s each before the turn failed. - Local readiness is validate(baseUrl) == null, not isNotBlank(). The field persists per keystroke, so one character flipped the coach to "Active" while every turn failed with the URL error already shown inline in Settings. - An unnormalizable URL raises Decoding(MALFORMED), not MissingAPIKey — the key is optional on this provider. - isPrivateHost accepts local-only names: single-label hosts, .lan, .home, .internal, .home.arpa, .ts.net. Addressing a box by the name the router or mDNS hands out is ordinary, and it was rejected with "must be on your local network" — where it was. - The Timeout field reconciles with the store on focus loss; it clamps to 10..1800 on write and skips blank, so the text could show a value that was never stored, or an empty box over a live setting. - The local provider sends with redirects disabled. validate() vets the typed URL, not where the request lands, and cleartext is permitted app-wide — a 307 off the LAN would resend the health-context body in the clear. Cloud providers keep redirects. docs/local-llm-coach.md §3 and §5a updated to match. 1099 tests, 0 failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
LOCAL_OPENAI_COMPATcoach provider pointing at any OpenAI-Chat-Completions-compatible server the user runs themselves — Ollama, llama.cpp (llama-server), vLLM, SGLang, LM Studio. The API key is optional.Research and full rationale:
docs/local-llm-coach.md.Why Chat Completions, not Responses
Every engine in scope converged on
POST {base}/v1/chat/completions+GET {base}/v1/models. None implements the Responses API this app speaks natively in a usable/universal form, so this is a translating adapter — structurally the same as the existingMiniMaxClient/OpenRouterClient.toolsresponse_formattool_choice, no image URLs--api-key--jinjajson_object+json_schema--api-key--enable-auto-tool-choice--api-keyregex/ebnfjson_schemaonlyThe four things a local backend does that a hosted one doesn't
Optional API key. All of these run unauthenticated by default, so requiring a key would leave the coach permanently disabled for the normal setup. The readiness sentinel that gates
CoachFeatureFlags.coachEnabledbecomes the base URL instead of a key; a blank key omits theAuthorizationheader entirely.role: developeris unsafe. SGLang validates roles against a pydanticLiteralwith a validator that raises (→ HTTP 400) outside it —developerwas added to that list only recently, and released versions in the wild reject it. vLLM accepts the role (extra="allow") and hands it straight to the model's Jinja chat template, which usually has no branch for it. So the adapter foldsdeveloper→systemunconditionally. Related: many local templates require the system turn to be first and singular, so all system turns are merged into one leading message (MiniMaxClientappends the schema instruction as a trailing system message, which would break on those).Capabilities are user-declared, not assumed. Tool calling and structured output are switches. Defaults are the combination that works everywhere: tools on,
response_formatoff + prompt-injected schema (with the orchestrator's JSON-repair loop as the backstop).json_schema/json_objectare opt-in. Noreasoning,cache_control, or provider-routing block is ever sent.Cleartext + slow inference. The app shipped no
network_security_config.xml, so every LAN request would have failed withCleartextNotPermitted. Added one — NSC has no CIDR syntax, so the "private hosts only" restriction is enforced inLocalEndpoint.validate(loopback / RFC1918 / CGNAT / link-local /*.local; public hosts must behttps://).ResponsesHttp's hardcoded 60 s read timeout gains a per-call override (local default 180 s, configurable).Also in here
GET /v1/modelsdiscovery behind a "Test connection & load models" button in Settings.argumentsaccepted as a JSON string or an object — several local tool-call parsers emit the latter.qwen3:8bcan't prefix-match into a cloud rate.versionCode37 → 38.Testing
46 unit tests across
LocalEndpoint,LocalOpenAICompatClient,LocalModelCatalogand the resolver.assembleDebuggreen.Not yet hardware-tested — no local server available in the dev environment. Worth a smoke test against Ollama and one vLLM/SGLang box before merging.