From 0e0ce28568e7eb2e6c17420070cafd48fd58105c Mon Sep 17 00:00:00 2001 From: Khoa Truong Date: Sat, 22 Aug 2026 10:19:41 -0700 Subject: [PATCH] feat(coach): local / self-hosted LLM provider with an optional API key MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a `localOpenAICompat` provider pointing at any OpenAI-Chat-Completions server the user runs themselves — Ollama, llama.cpp, vLLM, SGLang, LM Studio — so a coach conversation can happen without health data leaving the network. Ported from PulseLoopAndroid #51 including the fixes from its review. Structurally the client is MiniMaxClient — same Responses→Chat translation, same accumulate-across-send statefulness, same fresh-client-per-turn contract — with four differences, each forced by something local backends do: - The API key is optional. Every engine in scope runs unauthenticated by default, so a blank key omits the Authorization header and readiness is a base URL that passes LocalEndpoint.validate. Not "non-empty": Settings persists as the user types, so a blank-check would flip the coach to "Active" on the first character and fail every turn afterwards. - `developer` folds into `system`, and all system turns merge into one leading message. SGLang 400s on a role outside its pydantic Literal; many local chat templates need the system turn first and singular. - Capabilities are declared, not assumed. vLLM 400s on `tools` without --enable-auto-tool-choice; LM Studio has no json_object mode. Detect probes a plain baseline request first, then tools and response_format, so a 4xx means "this field is refused" rather than "this request is refused" — and only a conclusive verdict overwrites a setting the user chose by hand. - A long, configurable timeout, and no redirects. Privacy note, per CONTRIBUTING: this changes what leaves the device only by making it possible for nothing to. Cleartext is three layers. NSAllowsLocalNetworking re-permits cleartext for local destinations only (never NSAllowsArbitraryLoads), LocalEndpoint.validate explains a bad host in the Settings field instead of failing opaquely at request time, and LocalHTTP refuses redirects — validate() vets the URL typed, not where the request lands, and a 307 off the LAN would resend the health-context body in the clear. New under Coach/Local/: LocalEndpoint, LocalHTTP, LocalModelCatalog, LocalCapabilityProbe, LocalOpenAICompatClient, LocalLLMKeychainStore, and LocalServerSettingsSection (its own view — the probe state should die with the provider selection, and CoachSettingsSection was at SwiftLint's type_body_length ceiling). The target uses a file-system-synchronized group, so no project.pbxproj change. Docs: design and the per-engine support matrix in docs/local-llm-coach.md (added to the mkdocs nav); README, docs/platforms/ios-vs-android.md and docs/project/privacy.md updated to list the provider. swiftlint: 0 errors. 1108 tests, 0 failures (43 new). Not yet verified against a real server. --- .../Coach/Config/CoachClientResolver.swift | 23 + .../Coach/Config/CoachFeatureFlags.swift | 12 + PulseLoop/Coach/Config/CoachSettings.swift | 70 +++ .../Coach/Config/CoachSettingsSection.swift | 63 ++- .../Coach/Local/LocalCapabilityProbe.swift | 460 ++++++++++++++++++ PulseLoop/Coach/Local/LocalEndpoint.swift | 142 ++++++ PulseLoop/Coach/Local/LocalHTTP.swift | 89 ++++ .../Coach/Local/LocalLLMKeychainStore.swift | 70 +++ PulseLoop/Coach/Local/LocalModelCatalog.swift | 120 +++++ .../Coach/Local/LocalOpenAICompatClient.swift | 389 +++++++++++++++ .../Local/LocalServerSettingsSection.swift | 270 ++++++++++ PulseLoop/Info.plist | 26 + PulseLoop/Views/SettingsView.swift | 4 + PulseLoopTests/LocalLLMTests.swift | 455 +++++++++++++++++ README.md | 6 +- docs/local-llm-coach.md | 262 ++++++++++ docs/platforms/ios-vs-android.md | 6 + docs/project/privacy.md | 6 + mkdocs.yml | 1 + 19 files changed, 2471 insertions(+), 3 deletions(-) create mode 100644 PulseLoop/Coach/Local/LocalCapabilityProbe.swift create mode 100644 PulseLoop/Coach/Local/LocalEndpoint.swift create mode 100644 PulseLoop/Coach/Local/LocalHTTP.swift create mode 100644 PulseLoop/Coach/Local/LocalLLMKeychainStore.swift create mode 100644 PulseLoop/Coach/Local/LocalModelCatalog.swift create mode 100644 PulseLoop/Coach/Local/LocalOpenAICompatClient.swift create mode 100644 PulseLoop/Coach/Local/LocalServerSettingsSection.swift create mode 100644 PulseLoopTests/LocalLLMTests.swift create mode 100644 docs/local-llm-coach.md diff --git a/PulseLoop/Coach/Config/CoachClientResolver.swift b/PulseLoop/Coach/Config/CoachClientResolver.swift index b4e95d6a..cee135ff 100644 --- a/PulseLoop/Coach/Config/CoachClientResolver.swift +++ b/PulseLoop/Coach/Config/CoachClientResolver.swift @@ -15,6 +15,9 @@ enum CoachClientResolver { geminiKeyStore: APIKeyStore, openRouterKeyStore: APIKeyStore, minimaxKeyStore: APIKeyStore, + // Defaulted so the four existing call sites don't each have to learn about a provider + // whose key is optional anyway. + localKeyStore: APIKeyStore = LocalLLMKeychainStore(), openAIClientFactory: (String) -> ResponsesClient = { OpenAIResponsesClient(apiKey: $0) } ) -> (key: String?, client: ResponsesClient) { switch settings.providerMode { @@ -31,6 +34,7 @@ enum CoachClientResolver { settings.providerMode, settings: settings, openAIKeyStore: openAIKeyStore, geminiKeyStore: geminiKeyStore, openRouterKeyStore: openRouterKeyStore, minimaxKeyStore: minimaxKeyStore, + localKeyStore: localKeyStore, openAIClientFactory: openAIClientFactory ) } @@ -46,6 +50,7 @@ enum CoachClientResolver { geminiKeyStore: APIKeyStore, openRouterKeyStore: APIKeyStore, minimaxKeyStore: APIKeyStore, + localKeyStore: APIKeyStore, openAIClientFactory: (String) -> ResponsesClient ) -> (key: String?, client: ResponsesClient) { switch mode { @@ -62,6 +67,24 @@ enum CoachClientResolver { case .userMiniMaxKey: let key = (try? minimaxKeyStore.readKey()) ?? nil return (key, MiniMaxClient(apiKey: key ?? "", model: settings.minimaxModel)) + case .localOpenAICompat: + // Readiness is a base URL that would actually work — `validate`, not "non-empty". + // Settings persists the field as the user types, so a non-empty check would flip the + // coach to "Active" on the first character and every turn would then fail inside + // `send()` with the same URL error the Settings field is already showing inline. + // The key may legitimately be absent and is passed through as nil so the client omits + // the Authorization header entirely. + let baseURL = settings.resolvedLocalBaseURL + let key = (try? localKeyStore.readKey()) ?? nil + let ready = LocalEndpoint.validate(baseURL) == nil ? baseURL : nil + return (ready, LocalOpenAICompatClient( + baseURL: baseURL, + model: settings.resolvedLocalModel, + apiKey: key, + toolCallingEnabled: settings.localToolCalling, + structuredOutput: settings.localStructuredOutput, + maxOutputTokens: settings.localMaxTokens > 0 ? settings.localMaxTokens : nil, + readTimeoutSeconds: settings.localTimeoutSeconds)) default: // userOpenAIKey / offlineStub / backendProxy (and appleOnDevice never // reaches here) all use the OpenAI key + factory. diff --git a/PulseLoop/Coach/Config/CoachFeatureFlags.swift b/PulseLoop/Coach/Config/CoachFeatureFlags.swift index 5f732305..c4265b2b 100644 --- a/PulseLoop/Coach/Config/CoachFeatureFlags.swift +++ b/PulseLoop/Coach/Config/CoachFeatureFlags.swift @@ -30,6 +30,10 @@ struct CoachFeatureFlags { return AppleOnDeviceAvailability.current.isAvailable case .userOpenAIKey, .userGeminiKey, .userOpenRouterKey, .userMiniMaxKey: return hasAPIKey + case .localOpenAICompat: + // `hasAPIKey` carries the local provider's readiness sentinel, which is a *usable base + // URL* rather than a key — the key is optional on every engine in scope. + return hasAPIKey case .backendProxy: return false // not implemented in v1 } @@ -60,6 +64,10 @@ struct CoachFeatureFlags { case .offlineStub: return "offline-stub" case .userOpenRouterKey: return settings.openRouterModel case .userMiniMaxKey: return settings.minimaxModel + // Blank is legitimate: llama.cpp ignores the field unless started with --alias. Label it + // rather than leave the usage row empty. + case .localOpenAICompat: + return settings.resolvedLocalModel.isEmpty ? "local-model" : settings.resolvedLocalModel case .userOpenAIKey, .userGeminiKey, .backendProxy: return settings.model } } @@ -80,6 +88,10 @@ struct CoachFeatureFlags { return hasAPIKey ? "Ready · \(settings.openRouterModel)" : "Add an OpenRouter key to enable." case .userMiniMaxKey: return hasAPIKey ? "Ready · \(settings.minimaxModel)" : "Add a MiniMax key to enable." + case .localOpenAICompat: + guard hasAPIKey else { return "Add your server's address to enable." } + let model = settings.resolvedLocalModel + return model.isEmpty ? "Ready · self-hosted" : "Ready · \(model)" case .backendProxy: return "Backend proxy not available yet." } diff --git a/PulseLoop/Coach/Config/CoachSettings.swift b/PulseLoop/Coach/Config/CoachSettings.swift index dcd0766b..64f3df5d 100644 --- a/PulseLoop/Coach/Config/CoachSettings.swift +++ b/PulseLoop/Coach/Config/CoachSettings.swift @@ -10,6 +10,10 @@ enum CoachProviderMode: String, Codable, CaseIterable, Identifiable { case userGeminiKey case userOpenRouterKey case userMiniMaxKey + /// Any OpenAI-Chat-Completions-compatible server the user runs themselves: Ollama, llama.cpp, + /// vLLM, SGLang, LM Studio. Declared after the keyed providers so inserting it doesn't disturb + /// anyone's stored `rawValue`. + case localOpenAICompat case backendProxy var id: String { rawValue } @@ -22,11 +26,43 @@ enum CoachProviderMode: String, Codable, CaseIterable, Identifiable { case .userGeminiKey: return "Gemini" case .userOpenRouterKey: return "OpenRouter" case .userMiniMaxKey: return "MiniMax" + case .localOpenAICompat: return "Local / self-hosted" case .backendProxy: return "Backend proxy" } } } +/// How hard to constrain a local model's output shape — see `docs/local-llm-coach.md` §5. +/// +/// `.off` is the default because it's the only mode implemented by every backend: the coach's +/// shape is carried by `CoachResponseSchema.promptInstruction` in the system message, with the +/// orchestrator's JSON-repair loop as the backstop. The other two are opt-in because the support +/// matrix is genuinely uneven — LM Studio implements `json_schema` but not `json_object`, and some +/// llama.cpp builds error when `json_schema` collides with a server-side `grammar`. +enum LocalStructuredOutput: String, Codable, CaseIterable, Identifiable, Sendable { + case off + case jsonObject + case jsonSchema + + var id: String { rawValue } + + var label: String { + switch self { + case .off: return "Prompt only" + case .jsonObject: return "JSON mode" + case .jsonSchema: return "Strict schema" + } + } + + var blurb: String { + switch self { + case .off: return "Works everywhere (default)" + case .jsonObject: return "response_format: json_object — not on LM Studio" + case .jsonSchema: return "response_format: json_schema — best when supported" + } + } +} + /// Preset MiniMax model choices surfaced in Settings. MiniMax's catalog is small /// and fixed (unlike OpenRouter), so these are exact API model names rather than a /// free-text field. The stored `CoachSettings.model` stays a string, so a new @@ -189,6 +225,34 @@ struct CoachSettings: Codable, Equatable { /// can ground practical advice (outdoor vs indoor, hydration, rain). City-level /// only — never the precise location. Off by default. var enableEnvironmentContext: Bool = false + /// Local-only: base URL of the self-hosted server, as typed (`http://192.168.1.50:11434`). + /// A URL that `LocalEndpoint.validate` accepts — not the key — is what gates readiness. + var localBaseURL: String = "" + /// Local-only: model name the server expects. Free-form; `/v1/models` populates the picker. + var localModel: String = "" + /// Local-only: send `tools`. Off for a server started without tool-call support (vLLM without + /// `--enable-auto-tool-choice` returns HTTP 400) or a model that can't call them. + var localToolCalling: Bool = true + /// Local-only: how hard to constrain the output shape. Default off — the only mode every + /// backend supports. + var localStructuredOutput: LocalStructuredOutput = .off + /// Local-only: `max_tokens`; 0 = omit and let the server decide. + var localMaxTokens: Int = 0 + /// Local-only: read timeout in seconds. Long, because CPU inference is slow. + var localTimeoutSeconds: Int = LocalOpenAICompatClient.defaultReadTimeoutSeconds + + /// The local base URL with surrounding whitespace gone; blank when unconfigured. There's no + /// default to fall back to — every engine listens on a different port. + var resolvedLocalBaseURL: String { + localBaseURL.trimmingCharacters(in: .whitespacesAndNewlines) + } + + /// The local model name. Blank is sent as-is rather than substituted: llama.cpp ignores the + /// field entirely, so an empty value is legitimate there, and inventing a slug would turn a + /// working setup into a 404 on the servers that do read it. + var resolvedLocalModel: String { + localModel.trimmingCharacters(in: .whitespacesAndNewlines) + } /// The OpenRouter model slug to use. Free-form (the user may type any slug); /// falls back to the default only when the stored `model` is blank. @@ -231,6 +295,12 @@ struct CoachSettings: Codable, Equatable { eveningHour = try c.decodeIfPresent(Int.self, forKey: .eveningHour) ?? d.eveningHour proactiveAlertsEnabled = try c.decodeIfPresent(Bool.self, forKey: .proactiveAlertsEnabled) ?? d.proactiveAlertsEnabled enableEnvironmentContext = try c.decodeIfPresent(Bool.self, forKey: .enableEnvironmentContext) ?? d.enableEnvironmentContext + localBaseURL = try c.decodeIfPresent(String.self, forKey: .localBaseURL) ?? d.localBaseURL + localModel = try c.decodeIfPresent(String.self, forKey: .localModel) ?? d.localModel + localToolCalling = try c.decodeIfPresent(Bool.self, forKey: .localToolCalling) ?? d.localToolCalling + localStructuredOutput = try c.decodeIfPresent(LocalStructuredOutput.self, forKey: .localStructuredOutput) ?? d.localStructuredOutput + localMaxTokens = try c.decodeIfPresent(Int.self, forKey: .localMaxTokens) ?? d.localMaxTokens + localTimeoutSeconds = try c.decodeIfPresent(Int.self, forKey: .localTimeoutSeconds) ?? d.localTimeoutSeconds } } diff --git a/PulseLoop/Coach/Config/CoachSettingsSection.swift b/PulseLoop/Coach/Config/CoachSettingsSection.swift index 9a7aec43..d93e9c2e 100644 --- a/PulseLoop/Coach/Config/CoachSettingsSection.swift +++ b/PulseLoop/Coach/Config/CoachSettingsSection.swift @@ -26,6 +26,7 @@ struct CoachSettingsSection: View { private let geminiKeyStore = GeminiKeychainStore() private let openRouterKeyStore = OpenRouterKeychainStore() private let minimaxKeyStore = MiniMaxKeychainStore() + private let localKeyStore = LocalLLMKeychainStore() /// Picker tag that selects the free-text "Custom" OpenRouter model entry. private let customModelTag = "__custom__" @@ -41,6 +42,8 @@ struct CoachSettingsSection: View { return selected == customModelTag ? "Custom…" : (OpenRouterModel(rawValue: selected)?.label ?? selected) case .userMiniMaxKey: return MiniMaxModel(rawValue: selected)?.label ?? selected + case .localOpenAICompat: + return store.settings.resolvedLocalModel.isEmpty ? "Not set" : store.settings.resolvedLocalModel default: return CoachModel(rawValue: selected)?.label ?? selected } @@ -70,12 +73,22 @@ struct CoachSettingsSection: View { @State private var showMiniMaxKey: Bool = false @State private var minimaxKeyError: String? + // Local / self-hosted state. The key is optional here, so it has the same shape as the others + // but never gates anything. + @State private var localKeyDraft: String = "" + @State private var hasLocalKey: Bool = false + @State private var showLocalKey: Bool = false + @State private var localKeyError: String? + private var flags: CoachFeatureFlags { let hasKey: Bool switch store.settings.providerMode { case .userGeminiKey: hasKey = hasGeminiKey case .userOpenRouterKey: hasKey = hasOpenRouterKey case .userMiniMaxKey: hasKey = hasMiniMaxKey + // The local provider's key is optional; a usable base URL is what makes it ready. Mirrors + // the readiness sentinel in `CoachClientResolver`, which is what actually gates a turn. + case .localOpenAICompat: hasKey = LocalEndpoint.validate(store.settings.localBaseURL) == nil default: hasKey = hasSavedKey } return CoachFeatureFlags(settings: store.settings, hasAPIKey: hasKey) @@ -141,6 +154,10 @@ struct CoachSettingsSection: View { // model picker. if store.settings.providerMode == .appleOnDevice { appleOnDeviceCard + } else if store.settings.providerMode == .localOpenAICompat { + // The model list comes from the user's own server, so it belongs with the rest + // of the server setup below rather than in a fixed picker here. + EmptyView() } else { FormMenuRow(title: "Model", value: currentModelLabel) { Picker("Model", selection: modelPickerBinding) { @@ -220,6 +237,23 @@ struct CoachSettingsSection: View { ) } + if store.settings.providerMode == .localOpenAICompat { + LocalServerSettingsSection() + // The key field stays here so it looks and behaves exactly like every other + // provider's — it is simply optional for this one. + apiKeyField( + placeholder: "Optional API key", + hint: "Only needed if you started your server with --api-key. Ollama ignores it " + + "entirely. Stored in your device Keychain.", + draft: $localKeyDraft, + showRaw: $showLocalKey, + hasSaved: hasLocalKey, + error: localKeyError, + onSave: saveLocalKey, + onRemove: removeLocalKey + ) + } + // OpenRouter-only routing controls. OpenRouter exposes a unified // reasoning-effort hint plus provider-level privacy and sort options // the native OpenAI/Gemini clients don't, so they only appear here. @@ -256,7 +290,11 @@ struct CoachSettingsSection: View { // and the on-device model is tool-less — so the toggle is only offered // for providers that can actually search. if store.settings.providerMode != .appleOnDevice, - store.settings.providerMode != .userMiniMaxKey { + store.settings.providerMode != .userMiniMaxKey, + // No local engine hosts a web-search tool; `LocalOpenAICompatClient` drops the + // spec defensively, and offering the switch would only promise something the + // server can't do. + store.settings.providerMode != .localOpenAICompat { FormToggleRow(title: "Web search", isOn: webSearchBinding) } @@ -394,6 +432,28 @@ struct CoachSettingsSection: View { // MARK: - Custom OpenRouter model field + private func saveLocalKey() { + do { + try localKeyStore.saveKey(localKeyDraft) + hasLocalKey = true + localKeyDraft = "" + localKeyError = nil + } catch { + localKeyError = error.localizedDescription + } + } + + private func removeLocalKey() { + do { + try localKeyStore.deleteKey() + hasLocalKey = false + localKeyDraft = "" + localKeyError = nil + } catch { + localKeyError = error.localizedDescription + } + } + private var customModelField: some View { FormField { VStack(alignment: .leading, spacing: 8) { @@ -543,6 +603,7 @@ struct CoachSettingsSection: View { hasGeminiKey = ((try? geminiKeyStore.readKey()) ?? nil) != nil hasOpenRouterKey = ((try? openRouterKeyStore.readKey()) ?? nil) != nil hasMiniMaxKey = ((try? minimaxKeyStore.readKey()) ?? nil) != nil + hasLocalKey = ((try? localKeyStore.readKey()) ?? nil) != nil } private func saveOpenAIKey() { diff --git a/PulseLoop/Coach/Local/LocalCapabilityProbe.swift b/PulseLoop/Coach/Local/LocalCapabilityProbe.swift new file mode 100644 index 00000000..43641553 --- /dev/null +++ b/PulseLoop/Coach/Local/LocalCapabilityProbe.swift @@ -0,0 +1,460 @@ +import Foundation + +/// Self-configuration for the local provider: given only a base URL, work out which engine is +/// behind it, what model it serves, and which optional request fields it will actually accept — +/// then hand back settings the user doesn't have to reason about. +/// +/// **Why capabilities have to be probed rather than looked up.** `/v1/models` describes the model, +/// not the server's request surface, and the two things most likely to fail a coach turn are +/// decided at *launch time* by flags that endpoint never mentions: vLLM rejects `tools` unless it +/// was started with `--enable-auto-tool-choice --tool-call-parser`, and structured-output support +/// varies by backend and build (LM Studio implements `json_schema` but not `json_object`; some +/// llama.cpp builds error when `json_schema` meets a server-side `grammar`). The only honest test +/// is to send the field and see whether the server takes it. +/// +/// Cost: three generations of at most a few tokens — a plain baseline request first, then the two +/// carrying the fields under test. The baseline is what makes a 4xx readable as "this field is +/// refused" rather than "this request is refused". On a server that has to page the model in first +/// (Ollama, LM Studio) the first of them can take tens of seconds — hence ``probeTimeout``. +/// +/// Failure is never destructive. A probe that errors for an unrelated reason leaves that capability +/// at its safe default rather than switching it off, and the report says the probe was inconclusive +/// so the UI can say so too — and so Settings knows not to overwrite a value the user set by hand. +enum LocalCapabilityProbe { + + /// Which server is behind the URL. Cosmetic — it drives the summary line and the hints, not + /// the request body; every real decision comes from ``Report``'s probed capabilities. + enum Engine: String, Sendable { + case ollama, llamaCPP, vllm, sglang, lmStudio, unknown + + var label: String { + switch self { + case .ollama: return "Ollama" + case .llamaCPP: return "llama.cpp" + case .vllm: return "vLLM" + case .sglang: return "SGLang" + case .lmStudio: return "LM Studio" + case .unknown: return "OpenAI-compatible server" + } + } + } + + /// A probed capability. ``unknown`` means the probe itself failed, so don't change the setting. + enum Support: Sendable { case yes, no, unknown } + + struct Report: Sendable { + var engine: Engine + /// Engine version when it advertises one. + var version: String? = nil + var models: [String] = [] + /// The model to use: the sole served model, else the previously-chosen one if the server + /// still lists it, else blank for the user to pick. + var suggestedModel: String = "" + var toolCalling: Support = .unknown + var jsonSchema: Support = .unknown + var jsonObject: Support = .unknown + /// The server's **context window** for the chosen model (prompt + completion), when it + /// reports one. This is NOT an output budget — see ``suggestedMaxTokens``. + var contextWindow: Int? = nil + /// Per-probe detail, shown under the summary so an inconclusive result is explainable. + var notes: [String] = [] + + /// The structured-output mode to store: the strongest one the server accepted. Falls back + /// to `.off`, which needs nothing from the server. + var suggestedStructuredOutput: LocalStructuredOutput { + if jsonSchema == .yes { return .jsonSchema } + if jsonObject == .yes { return .jsonObject } + return .off + } + + /// Tool calling stays ON unless the server actively refused it — an inconclusive probe + /// must not silently strip the coach of its ability to read the user's data. + var suggestedToolCalling: Bool { toolCalling != .no } + + /// Whether the probes actually reached a verdict, so a suggestion may **overwrite a + /// setting the user chose by hand**. + /// + /// ``suggestedToolCalling`` and ``suggestedStructuredOutput`` both have a safe default for + /// the inconclusive case, which is right for a first-time setup and wrong for a re-detect: + /// a user who turned tools off for a vLLM server without `--enable-auto-tool-choice`, then + /// pressed Detect to refresh the model list, would have them switched back on and every + /// turn would 400. Probes are skipped entirely when the model comes back blank + /// (``pickModel(_:currentModel:)`` on a multi-model server) or when the baseline request + /// fails — neither says anything about capabilities. + var toolCallingConclusive: Bool { toolCalling != .unknown } + + /// As ``toolCallingConclusive``. One conclusive probe is enough: a `yes` on the strict + /// schema deliberately leaves the weaker JSON mode untested. + var structuredOutputConclusive: Bool { jsonSchema != .unknown || jsonObject != .unknown } + + /// The value to store in **Max tokens**, derived from ``contextWindow``; 0 means "leave + /// blank and let the server decide", i.e. *not detected* — never a reason to clear a value + /// the user typed. + /// + /// A context window is not an output budget, and copying it across would be actively + /// harmful: `max_tokens` is checked against what's *left* after the prompt, so a request + /// with `prompt + max_tokens > context` is rejected outright. We therefore reserve + /// ``promptReserveTokens`` for the coach's own prompt — measured at ~3.3k for a plain turn, + /// doubled to cover tool results and replayed history — and cap the remainder at + /// ``maxSuggestedTokens``, well past what a coach_response needs, so a huge context doesn't + /// turn into a runaway generation budget. + var suggestedMaxTokens: Int { + guard let ctx = contextWindow else { return 0 } + let headroom = ctx - promptReserveTokens + if headroom < minUsefulOutputTokens { return 0 } + return min(headroom, maxSuggestedTokens) + } + + /// True when the reported context can't comfortably hold the coach's prompt, so the user + /// needs to raise it on the server (Ollama `num_ctx`, llama.cpp `-c`, vLLM + /// `--max-model-len`) rather than tune anything in the app. + var contextTooSmall: Bool { + guard let ctx = contextWindow else { return false } + return ctx - promptReserveTokens < minUsefulOutputTokens + } + + /// One line for the Settings summary. + var summary: String { + var out = engine.label + if let version { out += " \(version)" } + out += " · " + (suggestedModel.isEmpty ? "\(models.count) model(s)" : suggestedModel) + out += " · tools " + switch toolCalling { + case .yes: out += "yes" + case .no: out += "no" + case .unknown: out += "unknown" + } + out += " · " + switch suggestedStructuredOutput { + case .jsonSchema: out += "strict schema" + case .jsonObject: out += "JSON mode" + case .off: out += "prompt-only" + } + if let ctx = contextWindow { out += " · \(formatTokens(ctx)) ctx" } + return out + } + } + + /// Raised when the server can't be reached or isn't OpenAI-compatible — ``run`` 's only hard + /// failure. Everything after model discovery degrades to ``Support/unknown`` instead. + struct Unreachable: Error, LocalizedError { + let reason: String + var errorDescription: String? { reason } + } + + /// Discovers everything about [baseURL] in one pass. [currentModel] is preserved when the + /// server still lists it, so re-probing doesn't silently move a working setup to another model. + static func run( + baseURL: String, + apiKey: String? = nil, + currentModel: String = "" + ) async throws -> Report { + if let problem = LocalEndpoint.validate(baseURL) { + throw Unreachable(reason: LocalEndpoint.message(problem)) + } + var headers: [String: String] = [:] + if let key = apiKey?.trimmingCharacters(in: .whitespacesAndNewlines), !key.isEmpty { + headers["Authorization"] = "Bearer \(key)" + } + + // 1. Models — also the reachability check, so its failure is the one hard failure. + let entries: [LocalModelCatalog.ModelInfo] + switch await LocalModelCatalog.fetch(baseURL: baseURL, apiKey: apiKey) { + case .success(let found): entries = found + case .failure(let message): throw Unreachable(reason: message) + } + let models = entries.map(\.id) + let model = pickModel(models, currentModel: currentModel) + + // 2. Engine identity — best-effort, from the engine-specific info routes. Never fatal. + let (engine, version) = await identify(baseURL: baseURL, headers: headers) + + // Context window: from the listing when the engine puts it there (vLLM, llama.cpp, + // LM Studio), else from that engine's own route. + // `??` takes an autoclosure, which can't be async — so the fallback is spelled out. + var context = entries.first(where: { $0.id == model })?.contextWindow + if context == nil { + context = await contextFromEngine( + baseURL: baseURL, headers: headers, engine: engine, model: model) + } + + var notes: [String] = [] + if let context, context - promptReserveTokens < minUsefulOutputTokens { + notes.append( + "Context is only \(formatTokens(context)) — the coach's prompt alone is around " + + "\(formatTokens(promptReserveTokens / 2)). Raise it on the server " + + "(Ollama `num_ctx`, llama.cpp `-c`, vLLM `--max-model-len`) or replies will be " + + "truncated.") + } + if model.isEmpty { + // Capability probes need a model name on every engine except llama.cpp, and without + // one a 400 would be indistinguishable from "capability unsupported". + notes.append("Pick a model, then run this again to detect tools and response format.") + return Report(engine: engine, version: version, models: models, suggestedModel: model, + contextWindow: context, notes: notes) + } + + // 3. Baseline. A plain chat request with no optional fields at all, so the 4xx-means-no + // reading below is about the probed field rather than about the request as a whole. + // Skipping this was how an unloadable model id or an auth-gated chat route turned into + // "tools: not supported" and a persisted `toolCalling = false`. + switch await send(baseURL: baseURL, headers: headers, model: model, extra: baselineProbe) { + case .accepted: + break + case .refused(let status, let body): + notes.append( + "The server refused a plain chat request for `\(model)` (HTTP \(status)) — " + + "\(shorten(body)). Tools and response format couldn't be tested, so both are " + + "left unchanged. Check the model can actually load and that the chat route " + + "accepts the same key as /v1/models.") + return Report(engine: engine, version: version, models: models, suggestedModel: model, + contextWindow: context, notes: notes) + case .inconclusive(let reason): + notes.append( + "Couldn't complete a plain chat request (\(reason)) — tools and response format " + + "are left unchanged.") + return Report(engine: engine, version: version, models: models, suggestedModel: model, + contextWindow: context, notes: notes) + } + + // 4. Capability probes. + let tools = await probe(baseURL: baseURL, headers: headers, model: model, + extra: toolProbe, label: "Tool calling", notes: ¬es) + let schema = await probe(baseURL: baseURL, headers: headers, model: model, + extra: schemaProbe, label: "Strict schema", notes: ¬es) + // Only worth asking about the weaker mode when the stronger one was refused. + let object: Support = schema == .yes + ? .unknown + : await probe(baseURL: baseURL, headers: headers, model: model, + extra: jsonObjectProbe, label: "JSON mode", notes: ¬es) + + return Report(engine: engine, version: version, models: models, suggestedModel: model, + toolCalling: tools, jsonSchema: schema, jsonObject: object, + contextWindow: context, notes: notes) + } + + /// Sole model → use it. Otherwise keep the user's current pick when the server still has it; + /// else blank, because guessing among several would silently switch a working setup. + static func pickModel(_ models: [String], currentModel: String) -> String { + if !currentModel.isEmpty, models.contains(currentModel) { return currentModel } + if models.count == 1 { return models[0] } + return "" + } + + // MARK: - Engine identity + + /// Asks each engine's own info route in turn and stops at the first that answers. These are + /// distinct paths rather than a single field because `owned_by` in `/v1/models` is unreliable + /// (vLLM says "vllm", but Ollama says "library" and LM Studio says "organization_owner", and a + /// proxy rewrites all of them). Every call is best-effort — an engine we can't name still works. + private static func identify(baseURL: String, headers: [String: String]) async -> (Engine, String?) { + guard let base = LocalEndpoint.normalize(baseURL) else { return (.unknown, nil) } + // vLLM: GET /version -> {"version":"0.27.1"} + if let body = await get("\(base)/version", headers), let v = versionField(body) { return (.vllm, v) } + // Ollama: GET /api/version -> {"version":"0.x.y"} + if let body = await get("\(base)/api/version", headers), let v = versionField(body) { return (.ollama, v) } + // llama.cpp: GET /props -> build_info / default_generation_settings + if let body = await get("\(base)/props", headers), let root = jsonObject(body), + root["build_info"] != nil || root["default_generation_settings"] != nil { + return (.llamaCPP, root["build_info"] as? String) + } + // SGLang: GET /get_server_info -> model_path / version + if let body = await get("\(base)/get_server_info", headers), let root = jsonObject(body), + root["model_path"] != nil || root["version"] != nil { + return (.sglang, root["version"] as? String) + } + // LM Studio: its richer native listing, absent everywhere else. + if await get("\(base)/api/v0/models", headers) != nil { return (.lmStudio, nil) } + return (.unknown, nil) + } + + /// The context window from an engine's own route, for the ones that don't put it in + /// `/v1/models`. Best-effort: a nil here just means the app won't suggest a budget. + /// + /// Ollama is the one that matters. Its `/v1/models` carries no context at all, and its default + /// `num_ctx` is **2048** — smaller than the coach's own prompt — so without this a user would + /// get silently truncated context and blame the model. + private static func contextFromEngine( + baseURL: String, headers: [String: String], engine: Engine, model: String + ) async -> Int? { + guard let base = LocalEndpoint.normalize(baseURL) else { return nil } + switch engine { + case .ollama: + guard !model.isEmpty, let url = URL(string: "\(base)/api/show"), + let body = try? JSONSerialization.data(withJSONObject: ["model": model]) else { return nil } + guard let data = try? await LocalHTTP.post(url: url, body: body, headers: headers, + timeout: identifyTimeout), + let text = String(data: data, encoding: .utf8), + let info = jsonObject(text)?["model_info"] as? [String: Any] else { return nil } + // `model_info` is keyed by architecture, e.g. "qwen3.context_length", so match on the + // suffix rather than guessing the family. + for (key, value) in info where key.hasSuffix(".context_length") { + if let n = value as? Int, n > 0 { return n } + if let n = (value as? NSNumber)?.intValue, n > 0 { return n } + } + return nil + case .sglang: + guard let text = await get("\(base)/get_model_info", headers), + let root = jsonObject(text) else { return nil } + return LocalModelCatalog.contextWindow(of: root) + case .llamaCPP: + guard let text = await get("\(base)/props", headers), let root = jsonObject(text) else { return nil } + return LocalModelCatalog.contextWindow(of: root) + ?? (root["default_generation_settings"] as? [String: Any]) + .flatMap { LocalModelCatalog.contextWindow(of: $0) } + default: + return nil + } + } + + /// "262144" → "262k"; small values stay exact so a 2048 warning reads literally. + static func formatTokens(_ tokens: Int) -> String { + tokens >= 10_000 ? "\(tokens / 1000)k" : "\(tokens)" + } + + private static func get(_ url: String, _ headers: [String: String]) async -> String? { + guard let url = URL(string: url) else { return nil } + guard let data = try? await LocalHTTP.get(url: url, headers: headers, timeout: identifyTimeout) + else { return nil } // A 404 here just means "not this engine". + return String(data: data, encoding: .utf8) + } + + private static func jsonObject(_ body: String) -> [String: Any]? { + guard let data = body.data(using: .utf8) else { return nil } + return (try? JSONSerialization.jsonObject(with: data)) as? [String: Any] + } + + private static func versionField(_ body: String) -> String? { jsonObject(body)?["version"] as? String } + + // MARK: - Capability probes + + /// What one probe request actually got back, before it is read as a capability verdict. + private enum Outcome { + case accepted + /// The server answered 4xx — it read the request and refused it. + case refused(status: Int, body: String) + /// 5xx, transport failure, or an unusable URL: says nothing either way. + case inconclusive(String) + } + + /// Sends a one-token chat request carrying [extra] and reports what came back. + private static func send( + baseURL: String, headers: [String: String], model: String, extra: [String: Any] + ) async -> Outcome { + guard let url = LocalEndpoint.chatCompletionsURL(baseURL) else { + return .inconclusive("the server address couldn't be parsed") + } + var payload: [String: Any] = [ + "model": model, + "messages": [["role": "user", "content": "hi"]], + "max_tokens": probeMaxTokens, + ] + for (key, value) in extra { payload[key] = value } + guard let body = try? JSONSerialization.data(withJSONObject: payload) else { + return .inconclusive("couldn't encode the probe request") + } + do { + _ = try await LocalHTTP.post(url: url, body: body, headers: headers, timeout: probeTimeout) + return .accepted + } catch ResponsesError.http(let status, let body) { + return (400...499).contains(status) + ? .refused(status: status, body: body) + : .inconclusive("HTTP \(status)") + } catch { + return .inconclusive(error.localizedDescription) + } + } + + /// Sends [extra] alongside a one-token chat request and classifies the answer. + /// + /// A 4xx is the server telling us it won't take the field — that's ``Support/no``, and the + /// exact status doesn't matter (vLLM answers 400 for a disabled tool parser and 422 for a field + /// its deserializer doesn't know). A 5xx or a transport failure says nothing about the + /// capability, so it stays ``Support/unknown`` and the caller keeps its default. + /// + /// Reading a 4xx as "this field is unsupported" is only sound because ``run`` has already + /// established with ``baselineProbe`` that a request carrying *no* optional fields succeeds. + /// Without that, every whole-request rejection — a model id `/v1/models` lists but can't load, + /// a chat route that wants auth when the listing didn't, a broken chat template — would come + /// back as "tools: no" and silently persist tool calling off, which costs the coach all access + /// to the user's data. + private static func probe( + baseURL: String, headers: [String: String], model: String, + extra: [String: Any], label: String, notes: inout [String] + ) async -> Support { + switch await send(baseURL: baseURL, headers: headers, model: model, extra: extra) { + case .accepted: + return .yes + case .refused(let status, let body): + notes.append("\(label): not supported (HTTP \(status)) — \(shorten(body))") + return .no + case .inconclusive(let reason): + notes.append("\(label): couldn't tell (\(reason)) — left unchanged.") + return .unknown + } + } + + /// Server error bodies are verbose; the first line is the part worth showing. + private static func shorten(_ body: String) -> String { + let message = jsonObject(body) + .flatMap { ($0["error"] as? [String: Any])?["message"] as? String } ?? body + return String(message.trimmingCharacters(in: .whitespacesAndNewlines) + .split(separator: "\n", omittingEmptySubsequences: false).first ?? "").prefix(160).description + } + + /// Nothing optional at all — the control the capability probes are measured against. + private static let baselineProbe: [String: Any] = [:] + + /// A throwaway tool. Named so it can't collide with a real coach tool in a server-side log. + private static let toolProbe: [String: Any] = [ + "tools": [[ + "type": "function", + "function": [ + "name": "pulseloop_probe", + "description": "Capability probe. Do not call.", + "parameters": ["type": "object", "properties": [String: Any]()], + ], + ]], + ] + + /// A minimal schema, not the coach's: we're testing whether the *field* is accepted, and a + /// large schema risks a rejection about the schema itself rather than the capability. + private static let schemaProbe: [String: Any] = [ + "response_format": [ + "type": "json_schema", + "json_schema": [ + "name": "pulseloop_probe", + "strict": true, + "schema": [ + "type": "object", + "properties": ["ok": ["type": "string"]], + "required": ["ok"], + "additionalProperties": false, + ] as [String: Any], + ] as [String: Any], + ] as [String: Any], + ] + + private static let jsonObjectProbe: [String: Any] = [ + "response_format": ["type": "json_object"], + ] + + /// Long enough for a cold model to page in on Ollama/LM Studio. + static let probeTimeout: TimeInterval = 120 + /// Short: these routes either exist or 404 immediately. + static let identifyTimeout: TimeInterval = 10 + /// Enough that a grammar-constrained probe emits something, small enough to stay cheap. + static let probeMaxTokens = 8 + + /// Context reserved for input before any of it is offered as output budget. A plain coach turn + /// measured 3.1–3.3k input tokens on a real device; this doubles that so a turn that replays + /// history and feeds back tool results still fits. + static let promptReserveTokens = 6144 + + /// Below this much headroom, suggesting a budget is worse than saying the context is too small. + static let minUsefulOutputTokens = 512 + + /// A coach_response plus reasoning needs far less than this; the cap stops a 262k context from + /// becoming a licence for a runaway generation. + static let maxSuggestedTokens = 32_768 +} diff --git a/PulseLoop/Coach/Local/LocalEndpoint.swift b/PulseLoop/Coach/Local/LocalEndpoint.swift new file mode 100644 index 00000000..d44756b9 --- /dev/null +++ b/PulseLoop/Coach/Local/LocalEndpoint.swift @@ -0,0 +1,142 @@ +import Foundation + +/// URL handling for the self-hosted ("local") coach provider — see `docs/local-llm-coach.md` §3. +/// +/// The user types a *base* URL (`http://192.168.1.50:11434`, `http://localhost:1234/v1`, +/// `https://llm.example.com`), not a full endpoint path, because every engine serves the same +/// OpenAI-compatible routes under a `/v1` prefix: Ollama on 11434, llama.cpp on 8080, vLLM on +/// 8000, SGLang on 30000, LM Studio on 1234. This type turns whatever they typed into the two +/// concrete URLs the app calls, and enforces the plaintext-host rule. +/// +/// On iOS that rule is layered differently than on Android. ATS already blocks cleartext, and +/// `NSAllowsLocalNetworking` (Info.plist) re-permits it *only* for local-network destinations — +/// so unlike Android's `network_security_config.xml`, the platform is doing real work here. This +/// check still exists because it runs at type-time, in the Settings field, where it can say what +/// is wrong instead of letting the first request fail opaquely. +enum LocalEndpoint { + + /// Why a base URL can't be used. `nil` from ``validate(_:)`` means it's fine. + enum Problem { case blank, malformed, unsupportedScheme, publicCleartext } + + /// Normalizes a user-typed base URL to its scheme+authority+path root, with any trailing `/` + /// and any trailing `/v1` (or `/v1/chat/completions`, if they pasted the full endpoint) + /// stripped — so ``chatCompletionsURL(_:)`` and ``modelsURL(_:)`` can append the canonical + /// suffix without producing `/v1/v1`. A bare `host:port` with no scheme is assumed to be + /// `http://` (the overwhelmingly common local case; a public host would be rejected by + /// ``validate(_:)`` anyway). + /// + /// Returns nil when the input can't be parsed into a scheme + host. + static func normalize(_ raw: String) -> String? { + var text = raw.trimmingCharacters(in: .whitespacesAndNewlines) + if text.isEmpty { return nil } + if !text.contains("://") { text = "http://\(text)" } + guard let components = URLComponents(string: text), + let scheme = components.scheme?.lowercased(), + let host = components.host, !host.isEmpty else { return nil } + + var path = components.path + while path.hasSuffix("/") { path.removeLast() } + // Tolerate a pasted full endpoint or an explicit /v1 — both are re-appended by callers. + for suffix in ["/v1/chat/completions", "/chat/completions", "/v1"] where path.hasSuffix(suffix) { + path.removeLast(suffix.count) + break + } + while path.hasSuffix("/") { path.removeLast() } + + let port = components.port.map { ":\($0)" } ?? "" + // An IPv6 literal must stay bracketed in the reassembled URL; URLComponents.host strips them. + let authority = host.contains(":") && !host.hasPrefix("[") ? "[\(host)]" : host + return "\(scheme)://\(authority)\(port)\(path)" + } + + /// `POST` target for a chat turn. + static func chatCompletionsURL(_ base: String) -> URL? { + normalize(base).flatMap { URL(string: "\($0)/v1/chat/completions") } + } + + /// `GET` target that lists the models the server currently has loaded/available. + static func modelsURL(_ base: String) -> URL? { + normalize(base).flatMap { URL(string: "\($0)/v1/models") } + } + + /// The reason [raw] can't be used, or nil if it's usable. + /// + /// `https://` is unrestricted — a self-hosted box with a real certificate is the user's call. + /// Plaintext `http://` is confined to hosts that can't be on the public internet: loopback, + /// RFC1918 / CGNAT / link-local addresses, and local-only names. + static func validate(_ raw: String) -> Problem? { + if raw.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { return .blank } + guard let normalized = normalize(raw), + let components = URLComponents(string: normalized), + let scheme = components.scheme?.lowercased(), + let host = components.host?.lowercased() else { return .malformed } + switch scheme { + case "https": return nil + case "http": return isPrivateHost(host) ? nil : .publicCleartext + default: return .unsupportedScheme + } + } + + /// A short, user-facing explanation for a ``Problem``, for the Settings field. + static func message(_ problem: Problem) -> String { + switch problem { + case .blank: + return "Enter your server's address, e.g. http://192.168.1.50:11434" + case .malformed: + return "That doesn't look like a URL — use host:port or http://host:port" + case .unsupportedScheme: + return "Only http:// and https:// are supported." + case .publicCleartext: + return "Plain http:// is only allowed for a server on this device or your local " + + "network (an IP address, a plain hostname, or a .local / .lan / .ts.net name). " + + "Use https:// to reach one over the internet." + } + } + + /// True when [host] can't be routed off the local network: loopback, RFC1918 (`10/8`, + /// `172.16/12`, `192.168/16`), CGNAT `100.64/10` (Tailscale), link-local `169.254/16`, IPv6 + /// loopback/ULA/link-local, mDNS `.local` names, and the name forms that only resolve on a + /// local network. + /// + /// That last group is why this isn't purely an address test. Addressing an inference box by + /// the name its router or mDNS hands out — `http://nas:11434`, `http://ollama.lan:8080`, a + /// Tailscale MagicDNS `http://box.tail1234.ts.net:11434` — is an ordinary setup, and + /// rejecting it would tell the user their server had to be on their local network, which is + /// exactly where it is. A single-label host has no public TLD and cannot be resolved off-LAN; + /// the suffixes in ``localSuffixes`` are the reserved/local-scope ones (RFC 8375 `.home.arpa`, + /// RFC 6762 `.local`, the `.lan`/`.home`/`.internal` conventions, Tailscale's `.ts.net`). + static func isPrivateHost(_ host: String) -> Bool { + let h = host.trimmingCharacters(in: CharacterSet(charactersIn: "[]")).lowercased() + if h.isEmpty { return false } + if h == "localhost" || h.hasSuffix(".localhost") { return true } + if localSuffixes.contains(where: { h.hasSuffix($0) }) { return true } + // A bare hostname with no dot at all: resolvable only via a DNS search domain, mDNS or + // NetBIOS, i.e. on-link. The IPv6 check below still catches a bracket-less literal. + if !h.contains(".") && !h.contains(":") { return true } + if h.contains(":") { // IPv6 + return h == "::1" || h.hasPrefix("fc") || h.hasPrefix("fd") || h.hasPrefix("fe80:") + } + let octets = h.split(separator: ".", omittingEmptySubsequences: false) + guard octets.count == 4 else { return false } + let nums = octets.compactMap { Int($0) } + guard nums.count == 4, nums.allSatisfy({ (0...255).contains($0) }) else { return false } + let (a, b) = (nums[0], nums[1]) + switch true { + case a == 127: return true // loopback + case a == 10: return true // RFC1918 + case a == 192 && b == 168: return true // RFC1918 + case a == 172 && (16...31).contains(b): return true // RFC1918 + case a == 169 && b == 254: return true // link-local + case a == 100 && (64...127).contains(b): return true // CGNAT / Tailscale + default: return false + } + } + + /// Suffixes that are reserved for, or conventionally used on, a local network only. + private static let localSuffixes = [ + ".local", // RFC 6762 mDNS + ".home.arpa", // RFC 8375 + ".lan", ".home", ".internal", // common router defaults + ".ts.net", // Tailscale MagicDNS + ] +} diff --git a/PulseLoop/Coach/Local/LocalHTTP.swift b/PulseLoop/Coach/Local/LocalHTTP.swift new file mode 100644 index 00000000..9178fb2e --- /dev/null +++ b/PulseLoop/Coach/Local/LocalHTTP.swift @@ -0,0 +1,89 @@ +import Foundation + +/// The HTTP the local provider uses, kept apart from the cloud clients' plain `URLSession.shared` +/// for one reason: **it must not follow redirects.** +/// +/// `LocalEndpoint.validate` vets the URL the *user typed*. It cannot vet where a redirect lands, +/// and `NSAllowsLocalNetworking` re-permits cleartext for local destinations — so a `307` from the +/// validated LAN host to a public `http://` one would resend the coach's health-context POST body +/// in the clear, past every check the app makes. Cloud providers keep the default session; their +/// hosts are `https://` constants. +/// +/// The delegate is also where the long read timeout lives: a 30B model on CPU can spend minutes on +/// one round, which is nothing like a hosted API's latency profile. +enum LocalHTTP { + + /// Refuses every redirect. `nil` to the completion handler makes URLSession return the + /// redirect response itself instead of chasing it. + private final class NoRedirectDelegate: NSObject, URLSessionTaskDelegate, Sendable { + func urlSession( + _ session: URLSession, + task: URLSessionTask, + willPerformHTTPRedirection response: HTTPURLResponse, + newRequest request: URLRequest, + completionHandler: @escaping (URLRequest?) -> Void + ) { + completionHandler(nil) + } + } + + private static let delegate = NoRedirectDelegate() + + private static func session(timeout: TimeInterval) -> URLSession { + let config = URLSessionConfiguration.ephemeral + config.timeoutIntervalForRequest = timeout + config.timeoutIntervalForResource = timeout + // Belt and braces: the delegate is the real guard, but a session that never caches also + // never keeps a copy of a health-context response on disk. + config.urlCache = nil + config.requestCachePolicy = .reloadIgnoringLocalCacheData + return URLSession(configuration: config, delegate: delegate, delegateQueue: nil) + } + + /// `POST` a JSON body and return the raw response data. + /// + /// Throws `ResponsesError.transport` on a network failure and `ResponsesError.http` (with the + /// body) on a non-2xx status — including a 3xx, which reaches here precisely because the + /// redirect was refused. + static func post( + url: URL, + body: Data, + headers: [String: String] = [:], + timeout: TimeInterval + ) async throws -> Data { + var request = URLRequest(url: url) + request.httpMethod = "POST" + request.setValue("application/json", forHTTPHeaderField: "Content-Type") + for (name, value) in headers { request.setValue(value, forHTTPHeaderField: name) } + request.httpBody = body + request.timeoutInterval = timeout + return try await perform(request, timeout: timeout) + } + + /// A one-off `GET`, for model discovery and the engine-identity routes. + static func get( + url: URL, + headers: [String: String] = [:], + timeout: TimeInterval + ) async throws -> Data { + var request = URLRequest(url: url) + request.httpMethod = "GET" + for (name, value) in headers { request.setValue(value, forHTTPHeaderField: name) } + request.timeoutInterval = timeout + return try await perform(request, timeout: timeout) + } + + private static func perform(_ request: URLRequest, timeout: TimeInterval) async throws -> Data { + let data: Data + let response: URLResponse + do { + (data, response) = try await session(timeout: timeout).data(for: request) + } catch { + throw ResponsesError.transport(error) + } + if let http = response as? HTTPURLResponse, !(200...299).contains(http.statusCode) { + throw ResponsesError.http(status: http.statusCode, body: String(data: data, encoding: .utf8) ?? "") + } + return data + } +} diff --git a/PulseLoop/Coach/Local/LocalLLMKeychainStore.swift b/PulseLoop/Coach/Local/LocalLLMKeychainStore.swift new file mode 100644 index 00000000..a1f1adbd --- /dev/null +++ b/PulseLoop/Coach/Local/LocalLLMKeychainStore.swift @@ -0,0 +1,70 @@ +import Foundation +import Security + +/// Stores the **optional** API key for the user's self-hosted server in the iOS Keychain +/// (generic password). Mirrors the other provider stores and uses its own service + account so all +/// provider keys can coexist. +/// +/// Optional is the operative word: llama.cpp/vLLM/SGLang only want a key when started with +/// `--api-key`, and Ollama ignores the field entirely. A missing key here is the normal case, not +/// an unconfigured provider — readiness is the base URL (see `CoachClientResolver`). +struct LocalLLMKeychainStore: APIKeyStore { + private let service: String + private let account: String + + init(service: String = "com.pulseloop.coach.local", account: String = "local_api_key") { + self.service = service + self.account = account + } + + private var baseQuery: [String: Any] { + [ + kSecClass as String: kSecClassGenericPassword, + kSecAttrService as String: service, + kSecAttrAccount as String: account, + ] + } + + func readKey() throws -> String? { + var query = baseQuery + query[kSecReturnData as String] = true + query[kSecMatchLimit as String] = kSecMatchLimitOne + + var item: CFTypeRef? + let status = SecItemCopyMatching(query as CFDictionary, &item) + if status == errSecItemNotFound { return nil } + guard status == errSecSuccess else { throw KeychainError.unexpectedStatus(status) } + guard let data = item as? Data, let key = String(data: data, encoding: .utf8) else { + return nil + } + let trimmed = key.trimmingCharacters(in: .whitespacesAndNewlines) + return trimmed.isEmpty ? nil : trimmed + } + + func saveKey(_ key: String) throws { + let trimmed = key.trimmingCharacters(in: .whitespacesAndNewlines) + guard let data = trimmed.data(using: .utf8) else { throw KeychainError.dataEncoding } + + let attributes: [String: Any] = [kSecValueData as String: data] + let updateStatus = SecItemUpdate(baseQuery as CFDictionary, attributes as CFDictionary) + if updateStatus == errSecSuccess { return } + if updateStatus == errSecItemNotFound { + var insert = baseQuery + insert[kSecValueData as String] = data + insert[kSecAttrAccessible as String] = kSecAttrAccessibleAfterFirstUnlock + let addStatus = SecItemAdd(insert as CFDictionary, nil) + guard addStatus == errSecSuccess else { throw KeychainError.unexpectedStatus(addStatus) } + return + } + throw KeychainError.unexpectedStatus(updateStatus) + } + + func deleteKey() throws { + let status = SecItemDelete(baseQuery as CFDictionary) + guard status == errSecSuccess || status == errSecItemNotFound else { + throw KeychainError.unexpectedStatus(status) + } + } + + var hasKey: Bool { ((try? readKey()) ?? nil) != nil } +} diff --git a/PulseLoop/Coach/Local/LocalModelCatalog.swift b/PulseLoop/Coach/Local/LocalModelCatalog.swift new file mode 100644 index 00000000..55af730f --- /dev/null +++ b/PulseLoop/Coach/Local/LocalModelCatalog.swift @@ -0,0 +1,120 @@ +import Foundation + +/// `GET {base}/v1/models` against a self-hosted server, so Settings can offer a real model picker +/// instead of making the user type `qwen3:8b` from memory. +/// +/// Every engine in scope serves this route (it's how the OpenAI SDK enumerates models), and every +/// one returns the same envelope: `{"object":"list","data":[{"id":"…"},…]}`. Ollama lists pulled +/// models, LM Studio lists loaded ones, vLLM/SGLang list the single served model, and llama.cpp +/// lists the loaded model under its `--alias`. The list is advisory — the stored model stays a +/// free string, because a router in front of any of these can serve names the endpoint doesn't +/// enumerate. +enum LocalModelCatalog { + + /// One entry from the listing. ``contextWindow`` is the model's **context window** (prompt + + /// completion) when the server volunteers it — NOT an output budget; see + /// ``LocalCapabilityProbe`` for the derivation. Nil when the engine doesn't report it here. + struct ModelInfo: Sendable, Equatable { + let id: String + var contextWindow: Int? = nil + } + + /// The outcome of a refresh, kept as data so Settings can show the failure inline. + enum Result: Sendable { + case success([ModelInfo]) + /// Already user-facing. + case failure(String) + + var models: [String] { + if case .success(let entries) = self { return entries.map(\.id) } + return [] + } + } + + /// Short: this is a list lookup behind a button, not a generation. + static let refreshTimeout: TimeInterval = 15 + + static func fetch( + baseURL: String, + apiKey: String? = nil, + timeout: TimeInterval = refreshTimeout + ) async -> Result { + if let problem = LocalEndpoint.validate(baseURL) { + return .failure(LocalEndpoint.message(problem)) + } + guard let url = LocalEndpoint.modelsURL(baseURL) else { + return .failure(LocalEndpoint.message(.malformed)) + } + var headers: [String: String] = [:] + if let key = apiKey?.trimmingCharacters(in: .whitespacesAndNewlines), !key.isEmpty { + headers["Authorization"] = "Bearer \(key)" + } + do { + let data = try await LocalHTTP.get(url: url, headers: headers, timeout: timeout) + return .success(try parseEntries(data)) + } catch ResponsesError.http(let status, _) { + return .failure("The server answered HTTP \(status) for /v1/models.") + } catch ResponsesError.transport(let underlying) { + return .failure("Couldn't reach \(url.absoluteString) — \(underlying.localizedDescription)") + } catch { + return .failure(error.localizedDescription) + } + } + + /// Pulls the `id`s out of the OpenAI list envelope, sorted and de-duplicated. Falls back to a + /// bare top-level array, which a couple of thin proxies return instead of the envelope. + static func parse(_ data: Data) throws -> [String] { try parseEntries(data).map(\.id) } + + /// As ``parse(_:)``, but keeps each entry's context window when the server reports one + /// alongside the id. The field name differs per engine and none of them is the OpenAI spec — + /// vLLM writes `max_model_len`, llama.cpp `n_ctx` (with `n_ctx_train` as the model's trained + /// maximum), and LM Studio `loaded_context_length` / `max_context_length` in its own richer + /// listing. We take the first present, preferring what's actually *loaded* over what the model + /// could support, because the loaded value is the one a request is measured against. + static func parseEntries(_ data: Data) throws -> [ModelInfo] { + let root = try? JSONSerialization.jsonObject(with: data) + let rows: [Any] + if let object = root as? [String: Any], let list = object["data"] as? [Any] { + rows = list + } else if let list = root as? [Any] { + rows = list + } else { + throw ResponsesError.decoding("No `data` array in the /v1/models response.") + } + + var seen = Set() + var entries: [ModelInfo] = [] + for row in rows { + let entry: ModelInfo? + if let dict = row as? [String: Any] { + entry = (dict["id"] as? String) + .flatMap { $0.isEmpty ? nil : ModelInfo(id: $0, contextWindow: contextWindow(of: dict)) } + } else if let id = row as? String, !id.isEmpty { + entry = ModelInfo(id: id) + } else { + entry = nil + } + if let entry, seen.insert(entry.id).inserted { entries.append(entry) } + } + return entries.sorted { $0.id < $1.id } + } + + /// The context window an entry advertises, under whichever name its engine uses. + static func contextWindow(of entry: [String: Any]) -> Int? { + for key in contextKeys { + if let value = entry[key] as? Int, value > 0 { return value } + if let value = (entry[key] as? NSNumber)?.intValue, value > 0 { return value } + } + return nil + } + + /// Ordered by preference: loaded-context first, then configured, then trained maximum. + private static let contextKeys = [ + "loaded_context_length", // LM Studio (actually loaded) + "max_model_len", // vLLM + "n_ctx", // llama.cpp (as served) + "max_context_length", // LM Studio (model ceiling) + "context_length", // generic / proxies + "n_ctx_train", // llama.cpp (model ceiling) + ] +} diff --git a/PulseLoop/Coach/Local/LocalOpenAICompatClient.swift b/PulseLoop/Coach/Local/LocalOpenAICompatClient.swift new file mode 100644 index 00000000..a1ba7951 --- /dev/null +++ b/PulseLoop/Coach/Local/LocalOpenAICompatClient.swift @@ -0,0 +1,389 @@ +import Foundation + +/// The self-hosted / local coach client — see `docs/local-llm-coach.md`. +/// +/// Adapts the app's `ResponsesClient` protocol to the OpenAI **Chat Completions** API +/// (`POST {base}/v1/chat/completions`) as implemented by Ollama, llama.cpp's `llama-server`, +/// vLLM, SGLang, LM Studio and friends. Structurally this is `MiniMaxClient` — same +/// Responses→Chat translation, same accumulate-messages-across-`send` statefulness, same +/// fresh-client-per-turn contract from the factory — with four deliberate differences, each forced +/// by something a local backend does that a hosted one doesn't: +/// +/// 1. **The API key is optional.** Every one of these servers runs unauthenticated by default +/// (`--api-key` is opt-in on llama.cpp/vLLM/SGLang; Ollama ignores the field entirely). A blank +/// key omits the `Authorization` header rather than throwing — the readiness sentinel is the +/// **base URL** instead (see `CoachClientResolver`). +/// 2. **`developer` is folded into `system`, and all system turns are merged into one leading +/// message.** SGLang validates roles against a pydantic `Literal` and raises (→ HTTP 400) for a +/// role outside it; vLLM accepts `developer` and hands it to a Jinja chat template that usually +/// has no branch for it. Many local templates additionally require the system turn to be first +/// and singular. Folding + merging is lossless and works on all of them. +/// 3. **Capabilities are user-declared, not assumed.** vLLM 400s on `tools` unless the server was +/// started with `--enable-auto-tool-choice`; LM Studio has no `json_object` mode. Tool calling +/// and structured output are therefore switches, defaulting to the combination that works +/// everywhere (tools on, `response_format` off + prompt-injected schema). +/// 4. **A long, configurable read timeout, and no redirects.** A 30B model on CPU can spend +/// minutes on one round; and see `LocalHTTP` for why a redirect must never be followed here. +/// +/// Nothing else is sent: no `reasoning`, no `cache_control`, no provider-routing block. Only Ollama +/// documents `reasoning_effort`, and vLLM/SGLang would warn or reject the rest. +final class LocalOpenAICompatClient: ResponsesClient, @unchecked Sendable { + /// Generous by cloud standards, ordinary for a quantized model on consumer hardware. + static let defaultReadTimeoutSeconds = 180 + + /// Base URL as the user typed it; normalized via `LocalEndpoint`. + private let baseURL: String + private let model: String + /// Optional — blank means send no `Authorization` header at all. + private let apiKey: String? + private let toolCallingEnabled: Bool + private let structuredOutput: LocalStructuredOutput + /// `nil`/0 = omit `max_tokens` and let the server decide. + private let maxOutputTokens: Int? + private let readTimeoutSeconds: Int + + /// Accumulated Chat Completions messages for this turn, minus the system block. + private var messages: [[String: Any]] = [] + /// The merged leading system message (instructions + per-turn context + schema instruction). + private var systemPrompt: String = "" + /// Maps generated response IDs → the assistant message (content + tool_calls) so a + /// continuation turn can re-insert it before the matching tool results. + private var storedAssistantMessage: [String: [String: Any]] = [:] + + init( + baseURL: String, + model: String, + apiKey: String? = nil, + toolCallingEnabled: Bool = true, + structuredOutput: LocalStructuredOutput = .off, + maxOutputTokens: Int? = nil, + readTimeoutSeconds: Int = LocalOpenAICompatClient.defaultReadTimeoutSeconds + ) { + self.baseURL = baseURL + self.model = model + self.apiKey = apiKey + self.toolCallingEnabled = toolCallingEnabled + self.structuredOutput = structuredOutput + self.maxOutputTokens = maxOutputTokens + self.readTimeoutSeconds = readTimeoutSeconds + } + + func send(requestBody: Data) async throws -> OpenAIResponse { + if let problem = LocalEndpoint.validate(baseURL) { + throw ResponsesError.decoding(LocalEndpoint.message(problem)) + } + // Not `.missingAPIKey` — the key is optional on this provider, so blaming it would send the + // user to the one field that is allowed to be empty. + guard let endpoint = LocalEndpoint.chatCompletionsURL(baseURL) else { + throw ResponsesError.decoding(LocalEndpoint.message(.malformed)) + } + guard let req = try? JSONSerialization.jsonObject(with: requestBody) as? [String: Any] else { + throw ResponsesError.decoding("LocalOpenAICompatClient: invalid request body") + } + + let body = buildRequestBody(req) + let bodyData = try JSONSerialization.data(withJSONObject: body, options: [.withoutEscapingSlashes]) + + var headers: [String: String] = [:] + if let key = apiKey?.trimmingCharacters(in: .whitespacesAndNewlines), !key.isEmpty { + headers["Authorization"] = "Bearer \(key)" + } + + let data = try await LocalHTTP.post( + url: endpoint, body: bodyData, headers: headers, + timeout: TimeInterval(readTimeoutSeconds)) + + guard let root = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { + throw ResponsesError.decoding( + "The server at \(endpoint.absoluteString) did not return JSON — is it an " + + "OpenAI-compatible endpoint?") + } + return try ingestResponse(root) + } + + // MARK: - Request assembly (internal for unit tests) + + func buildRequestBody(_ req: [String: Any]) -> [String: Any] { + let input = req["input"] as? [[String: Any]] ?? [] + let tools = req["tools"] as? [[String: Any]] ?? [] + let previousResponseId = req["previous_response_id"] as? String + + if let previousResponseId { + appendContinuation(previousId: previousResponseId, input: input) + } else { + setupConversation(from: input) + } + return buildChatBody(tools: toolCallingEnabled ? convertTools(tools) : []) + } + + // MARK: - Conversation setup + + /// First turn. Every `system`/`developer` item is merged, in order, into a single leading + /// system message; `user`/`assistant` items keep their order after it. The schema instruction + /// joins the system block rather than trailing the conversation (where MiniMax puts it) because + /// a system turn after a user turn raises in several local chat templates. + private func setupConversation(from input: [[String: Any]]) { + messages = [] + storedAssistantMessage = [:] + + var systemParts: [String] = [] + var conversation: [[String: Any]] = [] + for item in input { + guard let role = item["role"] as? String, item["content"] != nil else { continue } + if role == "system" || role == "developer" { + // A system turn is always plain instruction text; flatten any content parts. + systemParts.append(flattenText(item)) + } else { + conversation.append(["role": role, "content": chatContent(from: item)]) + } + } + // Only the prompt tells an unconstrained local model what shape to answer in. Even with + // `response_format` on, this stays — it's what the orchestrator's JSON-repair loop leans on + // when a small model ignores the grammar. + systemParts.append(CoachResponseSchema.promptInstruction) + systemPrompt = systemParts.filter { !$0.isEmpty }.joined(separator: "\n\n") + messages = conversation + } + + /// Subsequent turns: replay the stored assistant message for [previousId] (Chat Completions + /// requires the assistant `tool_calls` message to precede the `tool` results answering them), + /// then append the new tool results / messages. A stray system/developer item here is folded + /// into the leading system block rather than appended mid-conversation. + private func appendContinuation(previousId: String, input: [[String: Any]]) { + if let assistant = storedAssistantMessage[previousId] { messages.append(assistant) } + for item in input { + if (item["type"] as? String) == "function_call_output", + let callId = item["call_id"] as? String, + let output = item["output"] as? String { + messages.append(["role": "tool", "tool_call_id": callId, "content": output]) + continue + } + guard let role = item["role"] as? String, item["content"] != nil else { continue } + if role == "system" || role == "developer" { + systemPrompt = [systemPrompt, flattenText(item)] + .filter { !$0.isEmpty }.joined(separator: "\n\n") + } else { + messages.append(["role": role, "content": chatContent(from: item)]) + } + } + } + + /// All text in a message item, whether `content` is a string or a content-part array. + private func flattenText(_ item: [String: Any]) -> String { + if let text = item["content"] as? String { return text } + guard let parts = item["content"] as? [[String: Any]] else { return "" } + return parts.compactMap { $0["text"] as? String }.joined(separator: "\n") + } + + /// Converts a Responses-API message item's `content` into Chat Completions `content`. Text + /// stays a plain string; images map to `{type:image_url, image_url:{url}}` parts. Local vision + /// backends take base64 `data:` URLs (Ollama explicitly rejects remote image URLs), which is + /// exactly what `CoachImagePayload.dataURL` produces. + private func chatContent(from item: [String: Any]) -> Any { + if let text = item["content"] as? String { return text } + guard let parts = item["content"] as? [[String: Any]] else { return "" } + var out: [[String: Any]] = [] + for part in parts { + switch part["type"] as? String { + case "input_text", "text": + if let text = part["text"] as? String { out.append(["type": "text", "text": text]) } + case "input_image": + if let url = part["image_url"] as? String { + out.append(["type": "image_url", "image_url": ["url": url]]) + } + default: + break + } + } + return out + } + + // MARK: - Tool conversion (Responses flat → Chat Completions nested) + + /// Flat Responses function specs → Chat Completions' nested `{type:function, function:{…}}`. + /// The hosted `web_search` tool is dropped: no local engine has one. `strict` is dropped too — + /// it's an OpenAI structured-outputs extension that vLLM/SGLang don't act on and some stricter + /// proxies reject inside a function spec. + private func convertTools(_ tools: [[String: Any]]) -> [[String: Any]] { + tools.compactMap { tool -> [String: Any]? in + let type = tool["type"] as? String + if type == "web_search" || type == "web_search_preview" { return nil } + guard type == "function", let name = tool["name"] as? String else { return nil } + var fn: [String: Any] = ["name": name] + if let desc = tool["description"] as? String { fn["description"] = desc } + if let params = tool["parameters"] as? [String: Any] { fn["parameters"] = params } + return ["type": "function", "function": fn] + } + } + + // MARK: - Build request body + + func buildChatBody(tools: [[String: Any]]) -> [String: Any] { + var allMessages: [[String: Any]] = [] + if !systemPrompt.isEmpty { + allMessages.append(["role": "system", "content": systemPrompt]) + } + allMessages.append(contentsOf: messages) + + var body: [String: Any] = [ + // llama.cpp ignores `model` unless started with --alias; everyone else requires it. + // Sending it unconditionally is correct for both. + "model": model, + "messages": allMessages, + ] + if !tools.isEmpty { body["tools"] = tools } + if let format = responseFormat() { body["response_format"] = format } + if let maxOutputTokens, maxOutputTokens > 0 { body["max_tokens"] = maxOutputTokens } + return body + } + + /// The `response_format` block, or nil when the user left structured output off (the default, + /// and the only setting that works on every backend). `.jsonSchema` uses the nested OpenAI + /// shape — `{type:"json_schema", json_schema:{name, strict, schema}}` — which vLLM, SGLang, + /// LM Studio and recent llama.cpp all accept. `.jsonObject` is the older, weaker mode; LM + /// Studio doesn't implement it, hence the choice. + func responseFormat() -> [String: Any]? { + switch structuredOutput { + case .off: + return nil + case .jsonObject: + return ["type": "json_object"] + case .jsonSchema: + return [ + "type": "json_schema", + "json_schema": [ + "name": CoachResponseSchema.name, + "strict": true, + "schema": CoachResponseSchema.jsonSchema, + ] as [String: Any], + ] + } + } + + // MARK: - Parse Chat Completions response → OpenAIResponse (internal for tests) + + func ingestResponse(_ root: [String: Any]) throws -> OpenAIResponse { + // Some servers (and most reverse proxies in front of them) report errors in the body on an + // HTTP 200. `error` may be an object or, on llama.cpp, a bare string. + if let err = root["error"] as? [String: Any] { + throw ResponsesError.decoding("Server error: \(err["message"] as? String ?? "\(err)")") + } + if let err = root["error"] as? String { + throw ResponsesError.decoding("Server error: \(err)") + } + + guard let choices = root["choices"] as? [[String: Any]], + let first = choices.first, + let message = first["message"] as? [String: Any] else { + throw ResponsesError.decoding( + "No `choices` in the response — the server may not be OpenAI-compatible. " + + "Got: \(String("\(root)".prefix(300)))") + } + + let responseId = (root["id"] as? String).flatMap { $0.isEmpty ? nil : $0 } ?? UUID().uuidString + var outputItems: [ResponseOutputItem] = [] + var assistantMessage: [String: Any] = ["role": "assistant"] + + // Open reasoning models emit their chain of thought either as an inline `…` + // block (llama.cpp/Ollama without a reasoning parser) or split into a separate field — + // `reasoning` on vLLM 0.27+, `reasoning_content` on older builds and SGLang. Neither + // belongs in the coach_response JSON: the first is stripped, the others aren't read. + let content = (message["content"] as? String).map(stripThinking) + if let content, !content.isEmpty { + outputItems.append(.message(text: content)) + assistantMessage["content"] = content + } else { + assistantMessage["content"] = NSNull() + } + + if let toolCalls = message["tool_calls"] as? [[String: Any]] { + var storedCalls: [[String: Any]] = [] + for call in toolCalls { + guard let fn = call["function"] as? [String: Any], + let name = fn["name"] as? String else { continue } + let callId = (call["id"] as? String).flatMap { $0.isEmpty ? nil : $0 } + ?? "local_call_\(UUID().uuidString.replacingOccurrences(of: "-", with: "").prefix(12))" + // `arguments` is a JSON *string* per the spec, but several local tool-call parsers + // emit a JSON object instead. Re-encode that so the orchestrator's parse succeeds + // instead of failing the round on a well-formed-but-differently-typed field. + let args: String + if let raw = fn["arguments"] as? String { + args = raw + } else if let object = fn["arguments"], + let data = try? JSONSerialization.data(withJSONObject: object), + let text = String(data: data, encoding: .utf8) { + args = text + } else { + args = "{}" + } + outputItems.append(.functionCall( + ResponseFunctionCall(name: name, callID: callId, arguments: args))) + storedCalls.append([ + "id": callId, + "type": "function", + "function": ["name": name, "arguments": args], + ]) + } + if !storedCalls.isEmpty { assistantMessage["tool_calls"] = storedCalls } + } + + if outputItems.isEmpty { + // A reasoning model that ran out of budget mid-thought returns null content, no tool + // calls, and finish_reason "length". Bare "the model returned no output" sends the + // user looking for the wrong problem — the fix is the Max tokens field. + if (first["finish_reason"] as? String) == "length" { + let spentReasoning = message["reasoning"] != nil || message["reasoning_content"] != nil + throw ResponsesError.decoding( + "The model hit its token limit before producing an answer" + + (spentReasoning ? " (it spent the budget reasoning)" : "") + + ". Raise Max tokens in Settings → AI Coach, or leave it blank.") + } + throw ResponsesError.emptyOutput + } + + storedAssistantMessage[responseId] = assistantMessage + return OpenAIResponse(id: responseId, outputItems: outputItems, usage: usage(from: root)) + } + + /// Maps the `usage` block when present. Local servers all report the OpenAI split; a server + /// that omits it leaves usage nil, and the coach shows no token counts rather than zeros. + private func usage(from root: [String: Any]) -> CoachTokenUsage? { + guard let usage = root["usage"] as? [String: Any], + let input = usage["prompt_tokens"] as? Int, + let output = usage["completion_tokens"] as? Int else { return nil } + return CoachTokenUsage(inputTokens: input, outputTokens: output) + } + + /// Removes `…` reasoning blocks. Tolerant at both ends: an unterminated trailing + /// `` (truncated output) drops its remainder, and a leading unmatched `` drops + /// everything before it. + /// + /// That second case is the common one, not an edge case. R1-style distills served by llama.cpp + /// and Ollama have the *opening* tag injected into the prompt by the chat template, so the + /// completion starts mid-thought and the only tag in `content` is a bare closing one. Matching + /// pairs only, the whole chain of thought reaches `CoachResponseParser`, which then burns the + /// orchestrator's repair budget — each attempt up to the 180 s read timeout — before the turn + /// ends in a parse failure. + func stripThinking(_ text: String) -> String { + var body = text + if let firstClose = body.range(of: "") { + let firstOpen = body.range(of: "") + if firstOpen == nil || firstClose.lowerBound < firstOpen!.lowerBound { + body = String(body[firstClose.upperBound...]) + } + } + var out = "" + var scanIndex = body.startIndex + while let openRange = body.range(of: "", range: scanIndex..", range: openRange.upperBound.. + ) -> some View { + VStack(alignment: .leading, spacing: 6) { + Text(title).font(.caption).foregroundStyle(PulseColors.textMuted) + TextField(placeholder, text: binding) + .keyboardType(.numberPad) + .font(PulseFont.subheadline.weight(.regular).monospaced()) + .foregroundStyle(PulseColors.textPrimary) + .padding(.horizontal, 14).padding(.vertical, 10) + .pulseGlass(Capsule()) + } + .frame(maxWidth: .infinity, alignment: .leading) + } + + /// Runs the probe and applies what it actually established. + /// + /// Only a conclusive verdict may overwrite a stored setting. Detect is also how you refresh the + /// model list, so it gets pressed on setups that already work — and the safe defaults behind + /// `suggested*` (tools ON, structured OFF) are right for a first run and wrong for a re-detect: + /// a user who turned tools off for a vLLM server without `--enable-auto-tool-choice` would + /// otherwise have them switched back on by a press meant to do something else. Likewise a + /// `suggestedMaxTokens` of 0 means "the server reported no context window", not "clear what the + /// user typed". + private func detectLocalServer() { + localProbeBusy = true + localProbeResult = nil + localProbeNotes = [] + let baseURL = store.settings.resolvedLocalBaseURL + let key = (try? keyStore.readKey()) ?? nil + let currentModel = store.settings.resolvedLocalModel + Task { @MainActor in + defer { localProbeBusy = false } + do { + let report = try await LocalCapabilityProbe.run( + baseURL: baseURL, apiKey: key, currentModel: currentModel) + localDiscovered = report.models + if !report.suggestedModel.isEmpty { store.settings.localModel = report.suggestedModel } + if report.toolCallingConclusive { + store.settings.localToolCalling = report.suggestedToolCalling + } + if report.structuredOutputConclusive { + store.settings.localStructuredOutput = report.suggestedStructuredOutput + } + // Derived from the context window minus a prompt reserve — never a straight copy, + // or `prompt + max_tokens` would exceed the context and the server would reject + // the request outright. + if report.suggestedMaxTokens > 0 { + store.settings.localMaxTokens = report.suggestedMaxTokens + } + localProbeOK = true + localProbeResult = report.summary + localProbeNotes = report.notes + } catch { + localProbeOK = false + localProbeResult = (error as? LocalCapabilityProbe.Unreachable)?.reason + ?? error.localizedDescription + } + } + } + + // MARK: - Bindings + + private var localBaseURLBinding: Binding { + Binding( + get: { store.settings.localBaseURL }, + set: { newValue in + store.settings.localBaseURL = newValue + // A new address invalidates everything the last probe learned — the model list + // especially, since it belongs to whatever server used to be at the old URL. + localDiscovered = [] + localProbeResult = nil + localProbeOK = false + localProbeNotes = [] + } + ) + } + + private var localModelBinding: Binding { + Binding(get: { store.settings.localModel }, set: { store.settings.localModel = $0 }) + } + + private var localToolCallingBinding: Binding { + Binding(get: { store.settings.localToolCalling }, set: { store.settings.localToolCalling = $0 }) + } + + private var localStructuredOutputBinding: Binding { + Binding( + get: { store.settings.localStructuredOutput }, + set: { store.settings.localStructuredOutput = $0 } + ) + } + + /// 0 means "omit `max_tokens` and let the server decide", which is what an empty field shows. + private var localMaxTokensBinding: Binding { + Binding( + get: { store.settings.localMaxTokens > 0 ? "\(store.settings.localMaxTokens)" : "" }, + set: { store.settings.localMaxTokens = Int($0.filter(\.isNumber)) ?? 0 } + ) + } + + /// Clamped on write, since a 2-second timeout would fail every local generation and a + /// multi-hour one would hang the coach with no way back. + private var localTimeoutBinding: Binding { + Binding( + get: { "\(store.settings.localTimeoutSeconds)" }, + set: { raw in + guard let value = Int(raw.filter(\.isNumber)) else { return } + store.settings.localTimeoutSeconds = min(max(value, 10), 1800) + } + ) + } +} diff --git a/PulseLoop/Info.plist b/PulseLoop/Info.plist index 1be94ee3..5a8675b7 100644 --- a/PulseLoop/Info.plist +++ b/PulseLoop/Info.plist @@ -6,6 +6,32 @@ NSCameraUsageDescription Attach a photo to ask the AI Coach about it. + + NSAppTransportSecurity + + NSAllowsLocalNetworking + + + NSLocalNetworkUsageDescription + PulseLoop connects to the AI model server you run on your own network, so coaching can happen without sending your health data to a cloud provider. UIBackgroundModes location diff --git a/PulseLoop/Views/SettingsView.swift b/PulseLoop/Views/SettingsView.swift index 18ad1071..fca2041f 100644 --- a/PulseLoop/Views/SettingsView.swift +++ b/PulseLoop/Views/SettingsView.swift @@ -35,6 +35,10 @@ struct SettingsView: View { return OpenRouterModel(rawValue: settings.model)?.label ?? settings.openRouterModel case .userMiniMaxKey: return MiniMaxModel(rawValue: settings.model)?.label ?? settings.minimaxModel + case .localOpenAICompat: + // The model name is whatever the user's own server calls it, so there's no preset to + // map. Blank is legitimate (llama.cpp ignores the field), hence the fallback. + return settings.resolvedLocalModel.isEmpty ? "Self-hosted" : settings.resolvedLocalModel case .backendProxy: return "Backend proxy" } diff --git a/PulseLoopTests/LocalLLMTests.swift b/PulseLoopTests/LocalLLMTests.swift new file mode 100644 index 00000000..9f6135f2 --- /dev/null +++ b/PulseLoopTests/LocalLLMTests.swift @@ -0,0 +1,455 @@ +import XCTest +@testable import PulseLoop + +/// The local / self-hosted coach provider: URL handling, the model listing, the Responses→Chat +/// translation, and the rules that decide when a probe may overwrite a setting the user chose. +/// +/// Everything here is pure — no server is contacted. The parts that genuinely need one (a real +/// `/v1/models`, a real capability probe against a running engine) are runtime verification, not +/// unit tests. +final class LocalEndpointTests: XCTestCase { + + func testNormalizesBareHostAndPortToHTTP() { + XCTAssertEqual(LocalEndpoint.normalize("192.168.1.50:11434"), "http://192.168.1.50:11434") + XCTAssertEqual(LocalEndpoint.chatCompletionsURL("192.168.1.50:11434")?.absoluteString, + "http://192.168.1.50:11434/v1/chat/completions") + } + + func testStripsTrailingSlashV1AndAPastedFullEndpoint() { + // All four spellings a user might paste must land on the same base. + for input in [ + "http://localhost:11434", + "http://localhost:11434/", + "http://localhost:11434/v1", + "http://localhost:11434/v1/chat/completions", + ] { + XCTAssertEqual(LocalEndpoint.normalize(input), "http://localhost:11434", input) + } + } + + func testKeepsAReverseProxyPathPrefix() { + XCTAssertEqual(LocalEndpoint.normalize("https://box.example.com/llm/v1"), + "https://box.example.com/llm") + XCTAssertEqual(LocalEndpoint.modelsURL("https://box.example.com/llm/v1")?.absoluteString, + "https://box.example.com/llm/v1/models") + } + + func testAcceptsCleartextOnlyForPrivateHosts() { + for host in [ + "http://localhost:11434", "http://127.0.0.1:8080", "http://192.168.1.50:11434", + "http://10.1.2.3:8000", "http://172.16.0.9:30000", "http://100.64.1.2:11434", + "http://mac-studio.local:1234", "http://[::1]:8080", + // Name forms that only resolve on a local network. Rejecting these told the user their + // server had to be on their LAN, which is exactly where it was. + "http://nas:11434", "http://ollama.lan:8080", "http://box.tail1234.ts.net:11434", + "http://pi.home:8000", "http://llm.internal:1234", "http://srv.home.arpa:11434", + ] { + XCTAssertNil(LocalEndpoint.validate(host), host) + } + for host in ["http://example.com:11434", "http://8.8.8.8:8080", "http://172.32.0.1:80"] { + XCTAssertEqual(LocalEndpoint.validate(host), .publicCleartext, host) + } + } + + func testAPublicDottedHostnameIsStillRejectedOverCleartext() { + // The single-label allowance must not leak into ordinary registered domains. + for host in ["example.com", "llm.example.com", "ollama.io", "notlocal.localdomain"] { + XCTAssertFalse(LocalEndpoint.isPrivateHost(host), host) + } + } + + func testHTTPSIsUnrestrictedAndOtherSchemesRejected() { + XCTAssertNil(LocalEndpoint.validate("https://llm.example.com")) + XCTAssertEqual(LocalEndpoint.validate("ftp://box/llm"), .unsupportedScheme) + } + + func testBlankAndMalformedAreDistinguished() { + XCTAssertEqual(LocalEndpoint.validate(" "), .blank) + XCTAssertEqual(LocalEndpoint.validate("http://"), .malformed) + } + + func test172PrivateRangeBoundaries() { + XCTAssertTrue(LocalEndpoint.isPrivateHost("172.16.0.1")) + XCTAssertTrue(LocalEndpoint.isPrivateHost("172.31.255.254")) + XCTAssertFalse(LocalEndpoint.isPrivateHost("172.15.0.1")) + XCTAssertFalse(LocalEndpoint.isPrivateHost("172.32.0.1")) + } +} + +final class LocalModelCatalogTests: XCTestCase { + + private func data(_ json: String) -> Data { json.data(using: .utf8)! } + + func testParsesTheOpenAIListEnvelope() throws { + let entries = try LocalModelCatalog.parseEntries(data(""" + {"object":"list","data":[{"id":"qwen3:8b"},{"id":"llama3.1:70b"}]} + """)) + XCTAssertEqual(entries.map(\.id), ["llama3.1:70b", "qwen3:8b"]) // sorted + } + + func testFallsBackToABareArrayAndDeduplicates() throws { + let entries = try LocalModelCatalog.parseEntries(data(""" + ["a","b","a"] + """)) + XCTAssertEqual(entries.map(\.id), ["a", "b"]) + } + + func testReadsTheContextWindowUnderEachEnginesOwnName() throws { + // Every engine spells it differently and none of them is the OpenAI spec. + let entries = try LocalModelCatalog.parseEntries(data(""" + {"data":[ + {"id":"vllm","max_model_len":32768}, + {"id":"llamacpp","n_ctx":8192,"n_ctx_train":131072}, + {"id":"lmstudio","loaded_context_length":4096,"max_context_length":65536} + ]} + """)) + let byID = Dictionary(uniqueKeysWithValues: entries.map { ($0.id, $0.contextWindow) }) + XCTAssertEqual(byID["vllm"], 32768) + XCTAssertEqual(byID["llamacpp"], 8192) // as served, not the trained ceiling + XCTAssertEqual(byID["lmstudio"], 4096) // actually loaded, not the model ceiling + } + + func testAMissingDataArrayIsAnError() { + XCTAssertThrowsError(try LocalModelCatalog.parseEntries(data(#"{"object":"list"}"#))) + } +} + +final class LocalCapabilityProbeTests: XCTestCase { + + private func report( + tools: LocalCapabilityProbe.Support = .unknown, + schema: LocalCapabilityProbe.Support = .unknown, + object: LocalCapabilityProbe.Support = .unknown, + contextWindow: Int? = nil + ) -> LocalCapabilityProbe.Report { + LocalCapabilityProbe.Report( + engine: .vllm, version: "0.27.1", models: ["qwen3-8b"], suggestedModel: "qwen3-8b", + toolCalling: tools, jsonSchema: schema, jsonObject: object, contextWindow: contextWindow) + } + + // MARK: Model choice + + func testASoleServedModelIsChosenAutomatically() { + XCTAssertEqual(LocalCapabilityProbe.pickModel(["only"], currentModel: ""), "only") + } + + func testAnExistingChoiceIsKeptWhenTheServerStillListsIt() { + XCTAssertEqual(LocalCapabilityProbe.pickModel(["a", "b"], currentModel: "b"), "b") + } + + func testSeveralModelsAndNoValidCurrentPickLeavesTheChoiceToTheUser() { + // Guessing would silently move a working setup onto a different model. + XCTAssertEqual(LocalCapabilityProbe.pickModel(["a", "b"], currentModel: "gone"), "") + XCTAssertEqual(LocalCapabilityProbe.pickModel([], currentModel: "x"), "") + } + + // MARK: Verdicts + + func testToolCallingTurnsOffOnlyOnAnExplicitRefusal() { + XCTAssertTrue(report(tools: .unknown).suggestedToolCalling) + XCTAssertTrue(report(tools: .yes).suggestedToolCalling) + XCTAssertFalse(report(tools: .no).suggestedToolCalling) + } + + func testTheStrongestAcceptedResponseFormatWins() { + XCTAssertEqual(report(schema: .yes).suggestedStructuredOutput, .jsonSchema) + XCTAssertEqual(report(schema: .no, object: .yes).suggestedStructuredOutput, .jsonObject) + XCTAssertEqual(report(schema: .no, object: .no).suggestedStructuredOutput, .off) + } + + // MARK: Whether a suggestion may overwrite a hand-set value + + func testAnUnrunProbeIsNotConclusiveSoDetectLeavesTheSettingAlone() { + // The state after a blank model pick or a failed baseline request. `suggestedToolCalling` + // is still true here — that default is for a first-time setup, not for a re-detect over a + // user who deliberately turned tools off for a vLLM without --enable-auto-tool-choice. + let r = report() + XCTAssertTrue(r.suggestedToolCalling) + XCTAssertFalse(r.toolCallingConclusive) + XCTAssertEqual(r.suggestedStructuredOutput, .off) + XCTAssertFalse(r.structuredOutputConclusive) + } + + func testARefusalIsConclusive() { + let r = report(tools: .no, schema: .no, object: .no) + XCTAssertTrue(r.toolCallingConclusive) + XCTAssertTrue(r.structuredOutputConclusive) + } + + func testAStrictSchemaYesIsConclusiveEvenThoughJSONModeGoesUntested() { + let r = report(schema: .yes) + XCTAssertTrue(r.structuredOutputConclusive) + XCTAssertEqual(r.suggestedStructuredOutput, .jsonSchema) + } + + // MARK: Max tokens + + func testAContextWindowTheServerNeverReportedSuggestsNothing() { + // 0 means "not detected", which must not clear a Max tokens value the user typed. + XCTAssertEqual(report().suggestedMaxTokens, 0) + } + + func testMaxTokensIsTheContextMinusAPromptReserveCappedAtTheCeiling() { + // A context window is not an output budget: `prompt + max_tokens > context` is rejected. + let modest = report(contextWindow: 16384) + XCTAssertEqual(modest.suggestedMaxTokens, + 16384 - LocalCapabilityProbe.promptReserveTokens) + let huge = report(contextWindow: 262144) + XCTAssertEqual(huge.suggestedMaxTokens, LocalCapabilityProbe.maxSuggestedTokens) + } + + func testAContextTooSmallForThePromptSuggestsNothingAndSaysSo() { + // Ollama ships a 2048-token num_ctx default, smaller than the coach's own prompt. + let tiny = report(contextWindow: 2048) + XCTAssertEqual(tiny.suggestedMaxTokens, 0) + XCTAssertTrue(tiny.contextTooSmall) + } + + func testTheSummaryNamesTheEngineModelAndBothCapabilities() { + let line = report(tools: .yes, schema: .yes, contextWindow: 32768).summary + XCTAssertTrue(line.contains("vLLM")) + XCTAssertTrue(line.contains("qwen3-8b")) + XCTAssertTrue(line.contains("tools yes")) + XCTAssertTrue(line.contains("strict schema")) + XCTAssertTrue(line.contains("32k ctx")) + } + + func testTheSummarySaysUnknownRatherThanImplyingANegative() { + XCTAssertTrue(report().summary.contains("tools unknown")) + } +} + +final class LocalOpenAICompatClientTests: XCTestCase { + + private func client( + toolCalling: Bool = true, + structured: LocalStructuredOutput = .off, + maxTokens: Int? = nil + ) -> LocalOpenAICompatClient { + LocalOpenAICompatClient( + baseURL: "http://localhost:11434", model: "qwen3:8b", + toolCallingEnabled: toolCalling, structuredOutput: structured, + maxOutputTokens: maxTokens) + } + + private func request( + input: [[String: Any]], tools: [[String: Any]] = [], previousResponseId: String? = nil + ) -> [String: Any] { + var req: [String: Any] = ["input": input, "tools": tools] + if let previousResponseId { req["previous_response_id"] = previousResponseId } + return req + } + + private func messages(_ body: [String: Any]) -> [[String: Any]] { + body["messages"] as? [[String: Any]] ?? [] + } + + // MARK: Request shape + + func testNoMessageEverCarriesTheDeveloperRole() { + // SGLang validates roles against a pydantic Literal and 400s on anything outside it. + let body = client().buildRequestBody(request(input: [ + ["role": "developer", "content": "instructions"], + ["role": "user", "content": "hi"], + ])) + XCTAssertFalse(messages(body).contains { ($0["role"] as? String) == "developer" }) + } + + func testSystemAndDeveloperTurnsMergeIntoOneLeadingSystemMessage() { + // Many local chat templates require the system turn to be first and singular. + let body = client().buildRequestBody(request(input: [ + ["role": "system", "content": "A"], + ["role": "developer", "content": "B"], + ["role": "user", "content": "hi"], + ])) + let msgs = messages(body) + XCTAssertEqual(msgs.count, 2) + XCTAssertEqual(msgs[0]["role"] as? String, "system") + let system = msgs[0]["content"] as? String ?? "" + XCTAssertTrue(system.contains("A")) + XCTAssertTrue(system.contains("B")) + // The schema instruction rides in the system block, not after the user turn. + XCTAssertTrue(system.contains("coach_response")) + XCTAssertEqual(msgs[1]["role"] as? String, "user") + } + + func testToolsAreConvertedToTheNestedChatShapeWithoutStrict() { + let body = client().buildRequestBody(request( + input: [["role": "user", "content": "hi"]], + tools: [["type": "function", "name": "get_steps", "description": "d", + "parameters": ["type": "object"], "strict": true]])) + let tools = body["tools"] as? [[String: Any]] ?? [] + XCTAssertEqual(tools.count, 1) + XCTAssertEqual(tools[0]["type"] as? String, "function") + let fn = tools[0]["function"] as? [String: Any] + XCTAssertEqual(fn?["name"] as? String, "get_steps") + // `strict` is an OpenAI structured-outputs extension; stricter proxies reject it here. + XCTAssertNil(fn?["strict"]) + } + + func testToolCallingOffOmitsToolsEntirely() { + // vLLM without --enable-auto-tool-choice returns HTTP 400 for the field's mere presence. + let body = client(toolCalling: false).buildRequestBody(request( + input: [["role": "user", "content": "hi"]], + tools: [["type": "function", "name": "get_steps"]])) + XCTAssertNil(body["tools"]) + } + + func testWebSearchIsDroppedSinceNoLocalEngineHostsOne() { + let body = client().buildRequestBody(request( + input: [["role": "user", "content": "hi"]], + tools: [["type": "web_search"]])) + XCTAssertNil(body["tools"]) + } + + func testResponseFormatFollowsTheStructuredOutputSetting() { + XCTAssertNil(client(structured: .off).responseFormat()) + XCTAssertEqual(client(structured: .jsonObject).responseFormat()?["type"] as? String, "json_object") + let schema = client(structured: .jsonSchema).responseFormat() + XCTAssertEqual(schema?["type"] as? String, "json_schema") + let nested = schema?["json_schema"] as? [String: Any] + XCTAssertEqual(nested?["name"] as? String, CoachResponseSchema.name) + XCTAssertEqual(nested?["strict"] as? Bool, true) + } + + func testMaxTokensIsOmittedUnlessPositive() { + XCTAssertNil(client(maxTokens: nil).buildRequestBody( + request(input: [["role": "user", "content": "hi"]]))["max_tokens"]) + XCTAssertNil(client(maxTokens: 0).buildRequestBody( + request(input: [["role": "user", "content": "hi"]]))["max_tokens"]) + XCTAssertEqual(client(maxTokens: 4096).buildRequestBody( + request(input: [["role": "user", "content": "hi"]]))["max_tokens"] as? Int, 4096) + } + + func testNoReasoningCacheControlOrProviderBlockIsEverSent() { + let body = client().buildRequestBody(request(input: [["role": "user", "content": "hi"]])) + for key in ["reasoning", "reasoning_effort", "cache_control", "provider", "usage"] { + XCTAssertNil(body[key], key) + } + } + + // MARK: Response parsing + + /// Decodes a Chat Completions response body for `ingestResponse`. Throwing rather than + /// force-casting so a malformed literal in a test fails as that test, not as a crash that + /// takes the whole suite with it. + private func root(_ json: String, file: StaticString = #filePath, line: UInt = #line) throws -> [String: Any] { + let data = Data(json.utf8) + guard let object = try JSONSerialization.jsonObject(with: data) as? [String: Any] else { + XCTFail("test fixture is not a JSON object", file: file, line: line) + return [:] + } + return object + } + + func testParsesContentAndStripsThinkBlocks() throws { + let r = try client().ingestResponse(try root(""" + {"id":"chatcmpl-1","choices":[{"message":{"role":"assistant", + "content":"hmm{\\"title\\":\\"ok\\"}"}}], + "usage":{"prompt_tokens":10,"completion_tokens":4}} + """)) + XCTAssertEqual(r.id, "chatcmpl-1") + XCTAssertEqual(r.outputText, "{\"title\":\"ok\"}") + XCTAssertEqual(r.usage?.inputTokens, 10) + XCTAssertEqual(r.usage?.outputTokens, 4) + } + + func testAnUnmatchedLeadingCloseThinkIsStripped() throws { + // R1-style distills on llama.cpp/Ollama get the OPENING tag from the chat template, so the + // completion starts mid-thought and content carries only the closing tag. Left in, the + // whole chain of thought reached the parser and burned the repair budget. + let r = try client().ingestResponse(try root(""" + {"id":"c","choices":[{"message":{"role":"assistant", + "content":"the user wants a plan. let me think.{\\"title\\":\\"ok\\"}"}}]} + """)) + XCTAssertEqual(r.outputText, "{\"title\":\"ok\"}") + } + + func testAnUnterminatedTrailingOpenThinkStillDropsItsRemainder() throws { + let r = try client().ingestResponse(try root(""" + {"id":"c","choices":[{"message":{"role":"assistant", + "content":"{\\"title\\":\\"ok\\"}and then I would"}}]} + """)) + XCTAssertEqual(r.outputText, "{\"title\":\"ok\"}") + } + + func testTextWithNoThinkTagsAtAllIsUntouched() throws { + let r = try client().ingestResponse(try root(""" + {"id":"c","choices":[{"message":{"role":"assistant","content":"{\\"title\\":\\"ok\\"}"}}]} + """)) + XCTAssertEqual(r.outputText, "{\"title\":\"ok\"}") + } + + func testToolCallArgumentsSurviveBothTheStringAndObjectEncodings() throws { + // The spec says `arguments` is a JSON string; several local parsers emit an object. + let asString = try client().ingestResponse(try root(""" + {"id":"a","choices":[{"message":{"role":"assistant","content":null,"tool_calls":[ + {"id":"call_1","type":"function","function":{"name":"get_steps","arguments":"{\\"days\\":7}"}}]}}]} + """)) + XCTAssertEqual(asString.functionCalls.first?.arguments, "{\"days\":7}") + + let asObject = try client().ingestResponse(try root(""" + {"id":"b","choices":[{"message":{"role":"assistant","content":null,"tool_calls":[ + {"id":"call_2","type":"function","function":{"name":"get_steps","arguments":{"days":7}}}]}}]} + """)) + XCTAssertEqual(asObject.functionCalls.first?.name, "get_steps") + XCTAssertTrue(asObject.functionCalls.first?.arguments.contains("days") ?? false) + } + + func testABodyLevelErrorIsSurfacedAsADecodingFailure() { + // Reverse proxies routinely report errors in the body on an HTTP 200. + XCTAssertThrowsError(try client().ingestResponse(try root(#"{"error":{"message":"no model"}}"#))) + // llama.cpp sometimes makes `error` a bare string. + XCTAssertThrowsError(try client().ingestResponse(try root(#"{"error":"no model"}"#))) + } + + func testANonOpenAIResponseSaysSoInsteadOfThrowingABareParseError() { + XCTAssertThrowsError(try client().ingestResponse(try root(#"{"hello":"world"}"#))) { error in + guard case ResponsesError.decoding(let message) = error else { + return XCTFail("expected .decoding, got \(error)") + } + XCTAssertTrue(message.contains("OpenAI-compatible")) + } + } + + func testAReasoningModelTruncatedMidThoughtReportsTheTokenLimitNotEmptyOutput() { + // "The model returned no output" would send the user after the wrong problem; the fix is + // the Max tokens field. + XCTAssertThrowsError(try client().ingestResponse(try root(""" + {"id":"c","choices":[{"finish_reason":"length","message":{"role":"assistant", + "content":null,"reasoning":"..."}}]} + """))) { error in + guard case ResponsesError.decoding(let message) = error else { + return XCTFail("expected .decoding, got \(error)") + } + XCTAssertTrue(message.contains("Max tokens")) + } + } + + func testUsageIsNilRatherThanZeroWhenTheServerOmitsTheBlock() throws { + let r = try client().ingestResponse(try root(""" + {"id":"c","choices":[{"message":{"role":"assistant","content":"{}"}}]} + """)) + XCTAssertNil(r.usage) + } + + func testAContinuationReplaysTheAssistantToolCallsBeforeTheToolResults() throws { + // Chat Completions requires the assistant `tool_calls` message to precede the `tool` + // results answering them. + let c = client() + _ = c.buildRequestBody(request(input: [["role": "user", "content": "hi"]])) + _ = try c.ingestResponse(try root(""" + {"id":"resp-1","choices":[{"message":{"role":"assistant","content":null,"tool_calls":[ + {"id":"call_1","type":"function","function":{"name":"get_steps","arguments":"{}"}}]}}]} + """)) + let body = c.buildRequestBody(request( + input: [["type": "function_call_output", "call_id": "call_1", "output": "{\"steps\":900}"]], + previousResponseId: "resp-1")) + let msgs = messages(body) + let assistantIndex = msgs.firstIndex { $0["tool_calls"] != nil } + let toolIndex = msgs.firstIndex { ($0["role"] as? String) == "tool" } + XCTAssertNotNil(assistantIndex) + XCTAssertNotNil(toolIndex) + XCTAssertLessThan(assistantIndex!, toolIndex!) + } +} diff --git a/README.md b/README.md index 0e336ee5..06296b76 100644 --- a/README.md +++ b/README.md @@ -69,8 +69,10 @@ your sleep, heart rate, activity, and recovery. (heart rate, SpO₂, steps, distance, calories, sleep stages, raw packets). - **Today / Vitals / Sleep / Activity** dashboards built natively in SwiftUI, backed by SwiftData for local persistence. -- **AI Coach** — an agentic loop (OpenAI, Google Gemini, OpenRouter, or Apple's -on-device Foundation Models) with tools for data retrieval, on-the-fly analysis, +- **AI Coach** — an agentic loop (OpenAI, Google Gemini, OpenRouter, MiniMax, +Apple's on-device Foundation Models, or **any OpenAI-compatible server you run +yourself** — Ollama, llama.cpp, vLLM, SGLang, LM Studio) with tools for data +retrieval, on-the-fly analysis, chart generation, long-term memory, and web search. It also takes image input, so you can send a photo of a meal or a label. Every answer is grounded in your actual ring data. diff --git a/docs/local-llm-coach.md b/docs/local-llm-coach.md new file mode 100644 index 00000000..25776e37 --- /dev/null +++ b/docs/local-llm-coach.md @@ -0,0 +1,262 @@ +# Local / self-hosted LLM support for the AI Coach + +Branch: `feat/local-llm-coach`. Adds a `localOpenAICompat` coach provider that points at any +OpenAI-**Chat-Completions**-compatible server the user runs themselves — Ollama, llama.cpp +(`llama-server`), vLLM, SGLang, LM Studio, and anything else speaking the same wire format — +with the API key **optional**. + +Ported from the Android implementation (PulseLoopAndroid #51), including the review fixes that +landed on top of it. §§1, 2, 5, 5a and 5b are protocol facts and carry over unchanged; §3 +(cleartext) and §4 (timeouts) are genuinely different on iOS and are rewritten here. Where the +two platforms diverge, this file is authoritative for iOS. + +## 1. What the engines actually implement + +Every popular local engine converged on the same de-facto standard: OpenAI's **Chat Completions** +(`POST {base}/v1/chat/completions`) plus `GET {base}/v1/models`. None of them implement the +OpenAI **Responses** API in the form this app speaks natively (Ollama and llama.cpp expose a +`/v1/responses` shim, but it is non-stateful and not universal), so the adapter targets Chat +Completions — exactly like `MiniMaxClient` and `OpenRouterClient` already do. + +| Engine | Default base | Auth | `tools` | `response_format` | Notes | +|---|---|---|---|---|---| +| **Ollama** | `http://localhost:11434` | none — key field "required but ignored", dummy `ollama` | yes | yes (JSON mode / schema) | `tool_choice`, `n`, `user`, `logit_bias`, image **URLs** unsupported (base64 images only) | +| **llama.cpp** `llama-server` | `http://127.0.0.1:8080` | none unless `--api-key` | yes, best with `--jinja` | `json_object` **and** `json_schema`; can't combine with `grammar` | `model` field ignored unless `--alias`/router mode | +| **vLLM** | `http://localhost:8000` | none unless `--api-key` / `VLLM_API_KEY` | only with `--enable-auto-tool-choice --tool-call-parser

` | yes (xgrammar/guided decoding) | pydantic `extra="allow"` → unknown top-level fields are **warned, not rejected** | +| **SGLang** | `http://localhost:30000` | none unless `--api-key` | yes (`tools`, `tool_choice`, `parallel_tool_calls`) | yes, plus `regex` / `ebnf` | message roles are a strict `Literal` — see §2 | +| **LM Studio** | `http://localhost:1234` | none | yes | `json_schema` only (**no** `json_object`) | also has `/v1/responses` | + +Consequence: **assume Chat Completions, assume nothing else.** Everything beyond +`model` / `messages` / `tools` / `response_format` / `max_tokens` has to be opt-in. + +## 2. The `role: developer` trap + +OpenAI's Responses API (what `CoachOrchestrator` builds) puts the per-turn context in a +`developer` message. Chat Completions predates that role, and the local engines disagree: + +- **SGLang** validates roles against a pydantic `Literal`. Current `main` has + `_GenericMessageRole = Literal["system","assistant","tool","function","developer","latest_reminder"]` + with a `_normalize_role` validator that **raises** (→ HTTP 400) for anything else. `developer` + was added later; released versions in the wild reject it outright. +- **vLLM** *rejects* it. Verified against a live vLLM **0.27.1** server: + `{"role":"developer"}` returns **HTTP 422** — `Failed to deserialize the JSON body into the + target type: messages[0]: unknown role: developer`. (Older vLLM parsed requests with pydantic + models set to `extra="allow"` and did accept the role, passing it to the Jinja chat template, + which usually has no `developer` branch either. Don't rely on the old behaviour.) +- **Ollama / llama.cpp / LM Studio** document only `system` / `user` / `assistant` / `tool`. + +Even where the server accepts the role, the *chat template* usually can't render it. So the adapter +**always folds `developer` → `system`**, unconditionally, for every local backend. This is lossless +(the content is instructions either way) and is already what `MiniMaxClient.chatRole` does. Two of +the five engines are confirmed to hard-fail without it, so this is load-bearing, not defensive. + +Second, related hazard: many local chat templates require the system message to be **first and +singular** and raise on a system turn after a user turn. `MiniMaxClient` appends +`CoachResponseSchema.promptInstruction` as a *trailing* system message; the local adapter instead +**merges all system messages into one leading system message**, so strict templates render. + +## 3. Cleartext HTTP — three layers, and iOS does more of the work + +A LAN box at `http://192.168.1.50:11434` has no TLS. App Transport Security blocks cleartext by +default, so every local request would fail before it left the device. + +Unlike Android — whose Network Security Config can't express a CIDR allowlist, forcing an +app-wide `cleartextTrafficPermitted="true"` — iOS has an exception that means exactly what we +want. The restriction is three layers: + +1. **`NSAllowsLocalNetworking`** in `PulseLoop/Info.plist`. It re-permits cleartext **only for + local-network destinations**, so the platform keeps enforcing HTTPS for everything else. This + is deliberately *not* `NSAllowsArbitraryLoads`. Every cloud provider endpoint in the app is a + hardcoded `https://` constant and is unaffected. `NSLocalNetworkUsageDescription` is set + alongside it, since talking to a LAN device triggers the iOS Local Network permission prompt. +2. **`LocalEndpoint.validate`** refuses a plaintext `http://` URL whose host could be on the + public internet. ATS would refuse it too, but at request time, opaquely — this runs in the + Settings field and can say *why*. Accepted: loopback, RFC1918, CGNAT `100.64/10`, link-local, + and local-only names (`*.local`, `*.lan`, `*.home`, `*.internal`, `*.home.arpa`, Tailscale's + `*.ts.net`, and single-label hosts like `http://nas:11434` — a box addressed by the name its + router or mDNS hands out is an ordinary setup). `https://` hosts are unrestricted. +3. **`LocalHTTP` sends with redirects disabled** (`URLSessionTaskDelegate` returning `nil` from + `willPerformHTTPRedirection`). Layer 2 vets the URL the user *typed*, not the one a request + ends up on: a `307` from the validated LAN host to a public `http://` one would resend the + coach's health-context POST body in the clear. Cloud providers keep the default session. + +## 4. Timeouts + +The cloud clients use `URLSession.shared` with a 60 s request timeout. A 30B model on CPU can +spend minutes on one tool-loop round, so `LocalHTTP` builds its own ephemeral session with a +user-configurable timeout (default 180 s, clamped to 10…1800 in Settings). Ephemeral rather than +shared for a second reason: it never writes a health-context response to a disk cache. + +There is no retry. A slow local generation that times out has still *run* on the user's hardware, +and re-sending it would queue a second one behind the first. + +## 5. Capability toggles, because local ≠ uniform + +Three switches in Settings, because the same request body is fatal on one setup and required on +another: + +- **Tool calling** (default on). vLLM 400s on `tools` without `--enable-auto-tool-choice`; small + models hallucinate calls. Off ⇒ the adapter drops `tools` entirely and the coach answers from + the prompt context alone. +- **Structured output**: `off` (default) / `json_object` / `json_schema`. Off relies on the + injected `promptInstruction` plus the orchestrator's JSON-repair loop — the same path MiniMax + uses, and the only one that works everywhere. `json_schema` sends + `{type:"json_schema", json_schema:{name, strict, schema}}` from `CoachResponseSchema.schema`. + LM Studio has no `json_object`; some llama.cpp builds error when `json_schema` meets `grammar`. +- **Max output tokens** (blank = omit). Local defaults vary from unlimited to a few hundred. + Auto-detect fills this in from the server's reported **context window** — see §5b, and note it is + a derivation, never a copy. + +`reasoning` / `reasoning_effort` are **not** sent: only Ollama documents them, and vLLM/SGLang +would warn or 400. Anthropic `cache_control` and OpenRouter's `provider` block are likewise absent. + +## 5a. Self-discovery — why the toggles are probed, not looked up + +Asking the user to know whether their vLLM was started with `--enable-auto-tool-choice` is a bad +deal, and no metadata endpoint answers it: `/v1/models` describes the *model*, while the two +fields most likely to fail a turn (`tools`, `response_format`) are gated by *launch flags*. So +`LocalCapabilityProbe` sends the fields and reads the answer. + +One press of **Detect server & configure** runs: + +1. `GET /v1/models` — reachability, the model list, and the sole-model shortcut. This is the only + step whose failure is fatal; nothing after it can be trusted if the server isn't there. +2. Engine identity, best-effort, from each engine's own info route (first hit wins): + `GET /version` (vLLM), `/api/version` (Ollama), `/props` (llama.cpp), `/get_server_info` + (SGLang), `/api/v0/models` (LM Studio). Deliberately *not* `owned_by` from `/v1/models` — + vLLM says `vllm`, but Ollama says `library` and LM Studio says `organization_owner`, and any + proxy rewrites all three. Cosmetic only: it drives the summary line, never the request body. +3. A **baseline** chat request carrying no optional fields at all. +4. A chat request carrying one throwaway tool. +5. A chat request carrying a minimal `response_format: json_schema`; only if that's refused is + `json_object` tried. + +Step 3 is what makes steps 4 and 5 readable. Without it, every rejection *of the request as a +whole* — a model id `/v1/models` lists but can't actually load (LM Studio with JIT loading off, a +model pulled between the two calls), a chat route that wants auth when the listing didn't, a broken +chat template — comes back as "tools: not supported" and persists `toolCalling = false`, costing +the coach all access to the user's data while blaming the wrong thing. If the baseline is refused +or inconclusive, steps 4 and 5 are skipped and both settings are left exactly as they were. + +Classification rule: **4xx means the server refused the field** (vLLM answers `400` for a disabled +tool parser and `422` for a field its deserializer doesn't know, so the status itself carries no +extra meaning) → `NO`. A 5xx or a transport failure says nothing about the capability → `UNKNOWN`, +and the setting is **left at its default rather than switched off**, with a note explaining why. +Tool calling in particular only ever turns off on an explicit refusal — an inconclusive probe must +not silently strip the coach of its ability to read the user's data. + +**A suggestion only overwrites a stored setting when the probe reached a verdict.** Detect is also +how you refresh the model list, so it gets pressed on setups that already work, and the safe +defaults above (tools ON, structured OFF) are right for a first run and wrong for a re-detect: +a user who turned tools off by hand for a vLLM server without `--enable-auto-tool-choice` would +otherwise have them switched back on by a press meant to do something else, and every turn would +`400`. Same for Max tokens — a server that reports no context window yields `0`, which means "not +detected", not "clear what the user typed". + +Probes 3–5 use `max_tokens: 8` and a two-character prompt, and a minimal schema rather than the +coach's own (a large schema risks a rejection *about the schema* being read as "unsupported"). The +timeout is 120 s because on Ollama/LM Studio the first probe also pays for paging the model in. + +The probe never picks a model when several are served and none matches the current setting — +guessing would silently move a working setup onto a different model. + +### 5b. Max tokens is derived from the context window, never copied from it + +Every engine reports the model's context window, under its own name: + +| Engine | Route | Field | +|---|---|---| +| vLLM | `/v1/models` | `max_model_len` (262144 on the reference server) | +| llama.cpp | `/v1/models`, `/props` | `n_ctx` as served, `n_ctx_train` as the model ceiling | +| LM Studio | `/api/v0/models` | `loaded_context_length`, `max_context_length` | +| Ollama | `POST /api/show` | `model_info[".context_length"]` | +| SGLang | `/get_model_info` | context length | + +A context window is **prompt + completion**, so writing it straight into `max_tokens` is wrong in +a way that fails closed: the server checks `max_tokens` against what is *left* after the prompt and +rejects a request where the two overflow. The derivation instead reserves room for the prompt: + +``` +headroom = context − PROMPT_RESERVE_TOKENS (6144) +suggested = min(headroom, MAX_SUGGESTED_TOKENS (32768)) +headroom < 512 ⇒ leave Max tokens blank and warn +``` + +6144 is the measured coach prompt (3.1–3.3k input tokens for a plain turn on-device) doubled, so a +turn replaying history and feeding back tool results still fits. The 32768 cap keeps a 262k context +from becoming a licence for a runaway generation — a `coach_response` needs far less. + +**The warning is the more valuable half.** Ollama ships a default `num_ctx` of **2048**, smaller +than the coach's own prompt: without detection the prompt is silently truncated and the model gets +blamed. Detecting context lets Settings say so, and point at the server-side fix (`num_ctx`, +llama.cpp `-c`, vLLM `--max-model-len`) rather than at a setting in the app. + +### Measured on a real server (vLLM 0.27.1, Qwen3.8-27B-INT8) + +| Probe | Result | +|---|---| +| `GET /v1/models` | `qwen3.8-27b-int8-w8a16-mtp`, `max_model_len` 262144 | +| `GET /version` | `{"version":"0.27.1"}` → engine identified | +| `tools` | HTTP 200 → supported | +| `response_format: json_schema` | HTTP 200 → supported | +| `max_model_len` | 262144 → Max tokens suggested as 32768 (capped) | +| `role: developer` | **HTTP 422, `unknown role: developer`** | + +Also observed: with a reasoning parser enabled, vLLM returns the chain of thought in +`message.reasoning` (older builds: `reasoning_content`) and leaves `content` null until reasoning +finishes. The adapter reads neither field, so this is inert — but it means a `max_tokens` low +enough to truncate mid-reasoning yields no content at all, which the client reports as an +out-of-tokens error rather than a bare "no output". + +## 6. Changes in this repo + +New, all under `PulseLoop/Coach/Local/` (the target uses a file-system-synchronized group, so no +`project.pbxproj` edit is needed): + +| File | What it is | +|---|---| +| `LocalEndpoint.swift` | URL normalize/validate + the private-host rule (§3 layer 2) | +| `LocalHTTP.swift` | The no-redirect, long-timeout session (§3 layer 3, §4) | +| `LocalModelCatalog.swift` | `GET /v1/models` → the model picker, plus per-engine context windows | +| `LocalCapabilityProbe.swift` | Engine identity + baseline/tools/response_format probes (§5a) | +| `LocalOpenAICompatClient.swift` | The `ResponsesClient` adapter — Responses → Chat Completions | +| `LocalLLMKeychainStore.swift` | The **optional** API key | + +Modified: + +| File | Change | +|---|---| +| `Coach/Config/CoachSettings.swift` | `.localOpenAICompat` mode, `LocalStructuredOutput`, six `local*` fields + tolerant decode | +| `Coach/Config/CoachClientResolver.swift` | Builds the client; readiness is `validate(baseURL) == nil` | +| `Coach/Config/CoachFeatureFlags.swift` | `coachEnabled` / `effectiveModel` / `statusLine` for the new mode | +| `Coach/Config/CoachSettingsSection.swift` | The Local server + Request options groups, and Detect | +| `Views/SettingsView.swift` | Provider summary row | +| `Info.plist` | `NSAllowsLocalNetworking` + `NSLocalNetworkUsageDescription` (§3 layer 1) | + +`PulseLoopTests/LocalLLMTests.swift` covers all four pure units — 43 tests. + +### Readiness gate + +Every other provider is ready when its key exists. This one is ready when +`LocalEndpoint.validate(baseURL) == nil` — the key is optional on every engine in scope, and the +Settings field persists as the user types, so a non-empty check would flip the coach to "Active" +on the first character typed and then fail every turn with the URL error already shown inline. + +`CoachFeatureFlags.hasAPIKey` carries that sentinel for the local mode; it is not a key. + +### What is not verified + +No hardware/server run yet. The honest test is: start Ollama or llama.cpp on the LAN, point the +app at it, press **Detect server & configure**, and hold a real coach conversation with tools on. +Everything in this port is unit-tested at the pure level and compiles clean, which is not the +same thing. + +## Sources + +- [Ollama — OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility) +- [llama.cpp — server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) +- [vLLM — Tool Calling](https://docs.vllm.ai/en/stable/features/tool_calling/) +- [vLLM — `entrypoints/openai/protocol.py`](https://github.com/vllm-project/vllm/blob/v0.11.0/vllm/entrypoints/openai/protocol.py) (`OpenAIBaseModel`, `extra="allow"`) +- [SGLang — `entrypoints/openai/protocol.py`](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/openai/protocol.py) (`_GenericMessageRole`, `_normalize_role`) +- [SGLang — OpenAI APIs: Completions](https://docs.sglang.io/docs/basic_usage/openai_api_completions) +- [LM Studio — OpenAI compatibility endpoints](https://lmstudio.ai/docs/developer/openai-compat) diff --git a/docs/platforms/ios-vs-android.md b/docs/platforms/ios-vs-android.md index ffd41949..babafc96 100644 --- a/docs/platforms/ios-vs-android.md +++ b/docs/platforms/ios-vs-android.md @@ -31,6 +31,12 @@ On iOS you can choose how the coach runs: - **Apple on-device Foundation Models.** Runs the model locally on supported devices with no API key and no data leaving the phone. Availability is gated to hardware that supports it. +- **Local / self-hosted.** Point the coach at any OpenAI-compatible server you + run yourself — Ollama, llama.cpp, vLLM, SGLang, LM Studio. The API key is + optional, since most of them run unauthenticated on a LAN, and nothing leaves + your network. Unlike the Apple on-device option this works on any device, and + the model is whatever you chose to run. See + [Local / self-hosted LLM coach](../local-llm-coach.md). - **Offline** scripted fallback when no provider is configured. Beyond provider choice, the iOS coach is: diff --git a/docs/project/privacy.md b/docs/project/privacy.md index 4f341fc0..6637dc89 100644 --- a/docs/project/privacy.md +++ b/docs/project/privacy.md @@ -16,6 +16,12 @@ PulseLoop is built **local-first**, which is the whole point. - **Nothing leaves the device** except a coach question you explicitly ask — that single request goes to whichever LLM provider you configured, with the API key you supply. +- **The coach can be made to leave nothing at all.** On iOS, choose Apple's + on-device model, or point the coach at your own OpenAI-compatible server + (Ollama, llama.cpp, vLLM, SGLang, LM Studio) — the request then goes to your + hardware and no third party sees it. Plain `http://` is accepted only for a + server that cannot be on the public internet, and those requests refuse + redirects, so a question cannot be bounced off your network in the clear. - **API keys are stored securely on-device** — the iOS Keychain, or Android's `EncryptedSharedPreferences`. diff --git a/mkdocs.yml b/mkdocs.yml index 1e191572..9d4a8c9b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -121,6 +121,7 @@ nav: - Project: - Roadmap: project/roadmap.md - Architecture: project/architecture.md + - Local / self-hosted LLM coach: local-llm-coach.md - Contributing: project/contributing.md - Contributors: project/contributors.md - Privacy: project/privacy.md