Host Claude Opus through the tool proxy

.env.example

@@ -18,3 +18,16 @@
 
 # VITE_AGENT_API_BASE=same-origin
 # VITE_AGENT_API_BASE=http://localhost:8787
+
+# Hosted Claude Opus (or another model) on the tool proxy. The key stays on the
+# server — never in the frontend bundle. Visitors then chat without installing
+# 448 MB or having a GPU. Get a key at https://platform.claude.com
+#
+# ANTHROPIC_API_KEY=sk-ant-...
+# ANTHROPIC_MODEL=claude-opus-5
+#
+# OpenAI-compatible fallback (OpenRouter, Groq, …) if you are not using Anthropic:
+# OPENAI_API_KEY=sk-...
+# OPENAI_BASE_URL=https://openrouter.ai/api/v1
+# OPENAI_MODEL=anthropic/claude-opus-5
+# Set either key to `mock` to exercise the wiring without a provider.

Dockerfile

@@ -1,9 +1,9 @@
-# Optional tool proxy only. The Pages site stays static; inference stays in the tab.
-# Railway (or any host) runs this image and generates a public URL for Tools → Tool proxy URL.
+# Optional tool proxy, and hosted Claude Opus when ANTHROPIC_API_KEY is set.
+# The Pages site stays static. Railway (or any host) runs this image.
 FROM node:22-alpine
 
 WORKDIR /app
-COPY tools/agent-api.ts tools/agent-api-listen.ts ./tools/
+COPY tools/agent-api.ts tools/agent-api-listen.ts tools/agent-chat.ts ./tools/
 
 ENV NODE_ENV=production
 EXPOSE 8787

README.md

@@ -8,7 +8,7 @@ It does need a connection to answer, which is a deliberate limit rather than a m
 
 The agent can search, read pages, calculate exactly, remember things you tell it, and call any MCP server you connect. Because a 0.8B model needs the help, common requests are routed through [skills](#skills) that show it a worked example rather than telling it what to do.
 
-The published site has no backend. `pnpm build` produces a directory of static files that GitHub Pages can host, and every tool ships there — that is why the deployed site above has the full tool set rather than a reduced one. An optional [tool proxy](#optional-tool-proxy) lives in `tools/` for DuckDuckGo search and page reads without CORS: `pnpm dev` uses it locally, `pnpm proxy` runs it on its own. Inference stays in the tab either way.
+The published site has no backend. `pnpm build` produces a directory of static files that GitHub Pages can host, and every tool ships there — that is why the deployed site above has the full tool set rather than a reduced one. An optional [tool proxy](#optional-tool-proxy) lives in `tools/` for DuckDuckGo search and page reads without CORS. Put an `ANTHROPIC_API_KEY` on that same process and it also hosts [Claude Opus](#hosted-model-claude-opus): visitors then chat without a GPU or a 448 MB download. Leave the key unset and inference stays in the tab, which is how the public demo still works.
 
 ## How it works
 
@@ -302,23 +302,37 @@ GitHub Pages cannot host a process, so the published site stays browser-direct.
 
 - **`pnpm dev`** — the Vite plugin serves `POST /api/search` and `POST /api/fetch` on the dev server. `.env.development` sets `VITE_AGENT_API_BASE=same-origin`, so DuckDuckGo search and non-Wikipedia page reads go there automatically.
 - **`pnpm proxy`** — the same handlers on http://localhost:8787, for a static build or the hosted site. Paste that origin into **Tools → Tool proxy URL**, or build with `VITE_AGENT_API_BASE=http://localhost:8787`.
-- **Railway (or any host)** — the `Dockerfile` in the repo root runs only this process. Create a project, connect `devbadya/Jarvis`, set `PROXY_ORIGINS` to `https://devbadya.github.io`, wait until the deploy is live, then **Settings → Networking → Generate domain**. That `https://….up.railway.app` origin is the URL: paste it into **Tools → Tool proxy URL** on the hosted site. There is no URL until a domain exists; the Hobby plan alone does not create one.
+- **Railway (or any host)** — the `Dockerfile` in the repo root runs only this process. Create a project, connect `devbadya/Jarvis`, set `PROXY_ORIGINS` to `https://devbadya.github.io`, add `ANTHROPIC_API_KEY` if you want [hosted Opus](#hosted-model-claude-opus), wait until the deploy is live, then **Settings → Networking → Generate domain**. That `https://….up.railway.app` origin is the URL: paste it into **Tools → Tool proxy URL** on the hosted site. There is no URL until a domain exists; the Hobby plan alone does not create one.
 
 To give every visitor that proxy without asking them to paste anything, set the repository variable **`AGENT_API_BASE`** to its origin: `deploy.yml` passes it to the build as `VITE_AGENT_API_BASE`. Leave it unset and the hosted site stays browser-direct, which is what a fork with no proxy of its own needs — an empty value is not a proxy, so a workflow forwarding a variable nobody set cannot aim the build at an `/api` the host does not serve.
 
 **A proxy failure is not a failed turn.** Both tools try the proxy first and fall back to calling the provider from the page, so an outage, a spent budget or an allowlist that has not caught up costs a slower search rather than the answer. That fallback is what makes it safe to point every visitor at one process.
 
 The proxy scrapes DuckDuckGo HTML itself and fetches pages itself. It does not spend the Jina reader budget, and it is not limited to CORS-friendly endpoints. Wikipedia, LangSearch and Jina still leave the tab directly — they already send the headers, and their keys must not travel through this process.
 
-A fetch-on-behalf proxy is still a confused deputy. Every target is resolved and refused if it lands on loopback, link-local or RFC1918, and redirects are re-checked. Do not bind `pnpm proxy` to the public internet without setting `PROXY_ORIGINS` to the pages that may call it (for example `https://devbadya.github.io`). Inference never goes through it.
+A fetch-on-behalf proxy is still a confused deputy. Every target is resolved and refused if it lands on loopback, link-local or RFC1918, and redirects are re-checked. Do not bind `pnpm proxy` to the public internet without setting `PROXY_ORIGINS` to the pages that may call it (for example `https://devbadya.github.io`). Search and fetch never need a model key; chat does, and that key stays on this process.
 
-The allowlist says who may call, not how often, and an allowed page is exactly what a scraper would forge. `pnpm proxy` therefore allows **30 requests a minute per caller** and answers `429` beyond that, counted from the forwarded address rather than the socket, since every edge terminates the connection itself. `PROXY_RATE_LIMIT` changes the number; `0` switches it off. Health checks are exempt.
+The allowlist says who may call, not how often, and an allowed page is exactly what a scraper would forge. `pnpm proxy` therefore allows **30 requests a minute per caller** and answers `429` beyond that, counted from the forwarded address rather than the socket, since every edge terminates the connection itself. `PROXY_RATE_LIMIT` changes the number; `0` switches it off. Health checks are exempt. When `PROXY_ORIGINS` is set, `POST /api/chat` also requires a matching `Origin` header, so a curl from the public internet cannot spend the model key.
+
+### Hosted model (Claude Opus)
+
+The 0.8B on-device model will not become ChatGPT. To give visitors a frontier model without a GPU or a 448 MB install, put Anthropic's key on the same proxy:
+
+1. Create an API key at **[platform.claude.com](https://platform.claude.com)** (Console → API keys). Add prepaid credit under Billing. That is the official Opus API; do not buy keys from resellers.
+2. On Railway (or `pnpm proxy`), set `ANTHROPIC_API_KEY`. The default model is `claude-opus-5`. `ANTHROPIC_MODEL` overrides it. Opus is billed per token — currently $5 / million input tokens and $25 / million output, and **you** pay for every visitor.
+3. Point the site at the proxy: paste the origin into **Tool proxy URL**, or set the GitHub variable `AGENT_API_BASE` so every visitor uses it.
+
+`GET /api/health` then reports `{ ok: true, chat: { model: "claude-opus-5", provider: "anthropic" } }`. The landing page switches to **Start chatting** and skips the download. Tools still run in the tab; only generation goes to Anthropic.
+
+An OpenAI-compatible host still works if `ANTHROPIC_API_KEY` is unset: `OPENAI_API_KEY` plus optional `OPENAI_BASE_URL` / `OPENAI_MODEL` (Groq, OpenRouter, and so on). OpenRouter can serve Opus too (`OPENAI_BASE_URL=https://openrouter.ai/api/v1`, `OPENAI_MODEL=anthropic/claude-opus-5`) but you then have a middleman on top of Anthropic. Prefer the official Console.
+
+Set either key to `mock` to exercise the wiring without a provider.
 
 ### What leaves the browser
 
-Inference does not: prompts, reasoning, and replies never leave the GPU, and neither do [memories](#memory), which are written to IndexedDB in this browser and read back into a prompt that goes no further than the GPU either. Tools are the exception, and always were. A `web_search` call sends the query to the chosen provider, a `read_page` call sends the URL to the reader, a `weather` call sends the place name to Open-Meteo's geocoder and its coordinates to the two forecast services, and a `current_time` call with a place sends the name to the same geocoder.
+On the on-device path, inference does not leave: prompts, reasoning, and replies stay on the GPU, and neither do [memories](#memory), which are written to IndexedDB in this browser. Tools are the exception, and always were. A `web_search` call sends the query to the chosen provider, a `read_page` call sends the URL to the reader, a `weather` call sends the place name to Open-Meteo's geocoder and its coordinates to the two forecast services, and a `current_time` call with a place sends the name to the same geocoder.
 
-On the hosted site those go direct, with no server of ours in the path to log them. With the optional proxy, DuckDuckGo search and non-Wikipedia page reads go to that process first — one more party than a search API, and the one you run.
+With hosted Opus, the conversation itself goes to Anthropic through your proxy. Memories and MCP keys still stay in this browser. On the hosted site without a proxy, searches go direct, with no server of ours in the path to log them. With the optional proxy, DuckDuckGo search and non-Wikipedia page reads go to that process first — one more party than a search API, and the one you run.
 
 Worth being precise about on the default provider without a proxy: a search sends the query to `r.jina.ai`, which then sends it to DuckDuckGo. That is one more party than a search API like LangSearch or Jina involves.
 

src/App.test.tsx

@@ -0,0 +1,49 @@
+import { render, screen } from '@testing-library/react'
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import App from './App'
+import { EMPTY_STORAGE_STATUS } from '@/lib/storage'
+import { useChatStore } from '@/store/chat'
+
+function stubAdapter(): void {
+  Object.defineProperty(navigator, 'gpu', {
+    configurable: true,
+    value: {
+      requestAdapter: async () => ({
+        info: { vendor: 'test', architecture: 'gpu' },
+        limits: { maxBufferSize: 1024 * 1024 * 1024 },
+      }),
+    },
+  })
+}
+
+beforeEach(() => {
+  stubAdapter()
+  vi.spyOn(globalThis, 'fetch').mockRejectedValue(new Error('no proxy'))
+  vi.spyOn(useChatStore.getState(), 'refreshStorage').mockResolvedValue()
+  vi.spyOn(useChatStore.getState(), 'probeHosted').mockResolvedValue()
+  useChatStore.setState({
+    status: 'idle',
+    error: null,
+    storage: EMPTY_STORAGE_STATUS,
+    hostedChat: null,
+  })
+})
+
+afterEach(() => {
+  vi.restoreAllMocks()
+  useChatStore.setState({ status: 'idle', storage: EMPTY_STORAGE_STATUS, hostedChat: null })
+})
+
+describe('App header', () => {
+  it('names the on-device model until a hosted one is advertised', () => {
+    render(<App />)
+    expect(screen.getByText('Qwen3.5-0.8B · on-device')).toBeInTheDocument()
+  })
+
+  it('names the hosted model once the proxy advertises one', () => {
+    useChatStore.setState({ hostedChat: { base: 'https://proxy.example', model: 'claude-opus-5' } })
+    render(<App />)
+    expect(screen.getByText('claude-opus-5 · hosted')).toBeInTheDocument()
+    expect(screen.queryByText('Qwen3.5-0.8B · on-device')).not.toBeInTheDocument()
+  })
+})

src/App.tsx

@@ -23,15 +23,19 @@ const EVAL_MODE = new URLSearchParams(window.location.search).has('eval')
  */
 function BrandMark() {
   const busy = useChatStore((state) => state.busy)
+  const hostedChat = useChatStore((state) => state.hostedChat)
+  const label = EVAL_MODE
+    ? 'eval harness'
+    : hostedChat
+      ? `${hostedChat.model} · hosted`
+      : 'Qwen3.5-0.8B · on-device'
 
   return (
     <div className="flex items-center gap-2.5">
       <Orb active={busy} />
       <div className="flex items-baseline gap-2">
         <h1 className="font-semibold tracking-tight">Jarvis</h1>
-        <p className="hidden text-xs text-muted sm:block">
-          {EVAL_MODE ? 'eval harness' : 'Qwen3.5-0.8B · on-device'}
-        </p>
+        <p className="hidden text-xs text-muted sm:block">{label}</p>
       </div>
     </div>
   )

src/agent/loop.test.ts

@@ -413,3 +413,42 @@ describe('checking the answer before returning it', () => {
     expect(result.content).toBe('Ama Osei has led the airline since 2023.')
   })
 })
+
+describe('hosted native tool calls', () => {
+  function nativeClient(
+    rounds: {
+      text: string
+      toolCalls?: { id: string; name: string; arguments: Record<string, unknown> }[]
+    }[],
+  ): LlmClient {
+    let round = 0
+    return {
+      generate: vi.fn(async () => {
+        const output = rounds[round] ?? { text: '' }
+        round += 1
+        return { text: output.text, tokens: 10, thinkTokens: 0, durationMs: 50, toolCalls: output.toolCalls }
+      }),
+    } as unknown as LlmClient
+  }
+
+  it('executes structured calls and pairs the result with the provider id', async () => {
+    const execute = vi.fn(async () => '4')
+    const calculator = defineTool('calculator', 'maths', { type: 'object', properties: {} }, execute)
+    const client = nativeClient([
+      { text: '', toolCalls: [{ id: 'call_1', name: 'calculator', arguments: { expression: '2+2' } }] },
+      { text: '2 + 2 = 4' },
+    ])
+
+    const result = await runAgent(client, turns, [calculator], callbacks())
+
+    expect(execute).toHaveBeenCalledWith({ expression: '2+2' })
+    expect(result.content).toBe('2 + 2 = 4')
+    const conversation = vi.mocked(client.generate).mock.calls[1]?.[0] ?? []
+    expect(conversation).toContainEqual({
+      role: 'assistant',
+      content: '',
+      toolCalls: [{ id: 'call_1', name: 'calculator', arguments: { expression: '2+2' } }],
+    })
+    expect(conversation).toContainEqual({ role: 'tool', content: '4', toolCallId: 'call_1' })
+  })
+})

src/agent/loop.ts

@@ -1,4 +1,4 @@
-import type { LlmClient } from '@/llm/client'
+import type { InferenceClient } from '@/llm/client'
 import type { ChatTurn } from '@/llm/protocol'
 import {
   DEFAULT_STRATEGY,
@@ -119,7 +119,7 @@ function settle(latest: AgentResult, remaining: ReviewCheck[], draft: Draft | nu
  * them.
  */
 export async function runAgent(
-  client: LlmClient,
+  client: InferenceClient,
   turns: ChatTurn[],
   tools: Tool[],
   callbacks: AgentCallbacks,
@@ -172,11 +172,18 @@ export async function runAgent(
       durationMs: generation.durationMs,
       tokensPerSecond: generation.durationMs > 0 ? (generation.tokens / generation.durationMs) * 1000 : 0,
     }
+    // Hosted models return native tool calls; the on-device path parses XML.
+    const nativeCalls = generation.toolCalls ?? []
+    const parsedCalls = nativeCalls.length > 0 ? nativeCalls : parsed.toolCalls
     // A model that asks for a tool anyway once they are gone is asking for
     // something nothing can run, so the request is dropped and the round counts
     // as the answer it was meant to be.
-    const toolCalls = windDown ? [] : parsed.toolCalls
-    const outcome = { content: parsed.content, reasoning: parsed.reasoning, stats }
+    const toolCalls = windDown ? [] : parsedCalls
+    const outcome = {
+      content: nativeCalls.length > 0 ? generation.text || parsed.content : parsed.content,
+      reasoning: parsed.reasoning,
+      stats,
+    }
     // Only when the turn is over: before a tool call, reasoning is just reasoning.
     last = toolCalls.length === 0 ? promoteReasoningIfEmpty(outcome) : outcome
     callbacks.onRoundEnd(last)
@@ -207,18 +214,32 @@ export async function runAgent(
     }
 
     // Echo the assistant's tool request back so the model sees its own decision.
-    conversation.push({ role: 'assistant', content: generation.text || raw })
+    // Hosted APIs need the structured calls and the matching tool ids; the
+    // on-device worker needs the raw XML the chat template already emitted.
+    const pending = toolCalls.map((call) => ({
+      ...call,
+      id: call.id ?? crypto.randomUUID(),
+    }))
+    if (nativeCalls.length > 0) {
+      conversation.push({
+        role: 'assistant',
+        content: generation.text || parsed.content || '',
+        toolCalls: pending.map((call) => ({ id: call.id, name: call.name, arguments: call.arguments })),
+      })
+    } else {
+      conversation.push({ role: 'assistant', content: generation.text || raw })
+    }
 
-    for (const call of toolCalls) {
-      const id = crypto.randomUUID()
-      callbacks.onToolStart({ ...call, id })
+    for (const call of pending) {
+      const { id } = call
+      callbacks.onToolStart(call)
       const startedAt = performance.now()
       const tool = byName.get(call.name)
 
       if (!tool) {
         const error = `Unknown tool "${call.name}". Available tools: ${[...byName.keys()].join(', ')}`
         callbacks.onToolEnd(id, { error, durationMs: performance.now() - startedAt })
-        conversation.push({ role: 'tool', content: error })
+        conversation.push({ role: 'tool', content: error, toolCallId: id })
         continue
       }
 
@@ -231,23 +252,23 @@ export async function runAgent(
       if (earlier !== undefined) {
         const note = repeatedCallNote(call.name, earlier)
         callbacks.onToolEnd(id, { result: note, durationMs: performance.now() - startedAt })
-        conversation.push({ role: 'tool', content: note })
+        conversation.push({ role: 'tool', content: note, toolCallId: id })
         continue
       }
 
       try {
         const result = await tool.execute(call.arguments)
         executed.set(fingerprint, result)
         callbacks.onToolEnd(id, { result, durationMs: performance.now() - startedAt })
-        conversation.push({ role: 'tool', content: result })
+        conversation.push({ role: 'tool', content: result, toolCallId: id })
         // Only what a tool actually returned is evidence. A failure has nothing
         // to check an answer against, and demanding a citation for a page that
         // never loaded would be worse than saying nothing.
         evidence.toolResults.push({ tool: call.name, result })
       } catch (error) {
         const message = error instanceof Error ? error.message : String(error)
         callbacks.onToolEnd(id, { error: message, durationMs: performance.now() - startedAt })
-        conversation.push({ role: 'tool', content: `Tool "${call.name}" failed: ${message}` })
+        conversation.push({ role: 'tool', content: `Tool "${call.name}" failed: ${message}`, toolCallId: id })
       }
     }
 

src/agent/parse.ts

@@ -1,6 +1,8 @@
 export interface ParsedToolCall {
   name: string
   arguments: Record<string, unknown>
+  /** Provider id, when the call came from a hosted API rather than XML. */
+  id?: string
 }
 
 export interface ParsedOutput {