diff --git a/.env.example b/.env.example index 073c320..f290e89 100644 --- a/.env.example +++ b/.env.example @@ -18,3 +18,16 @@ # VITE_AGENT_API_BASE=same-origin # VITE_AGENT_API_BASE=http://localhost:8787 + +# Hosted Claude Opus (or another model) on the tool proxy. The key stays on the +# server — never in the frontend bundle. Visitors then chat without installing +# 448 MB or having a GPU. Get a key at https://platform.claude.com +# +# ANTHROPIC_API_KEY=sk-ant-... +# ANTHROPIC_MODEL=claude-opus-5 +# +# OpenAI-compatible fallback (OpenRouter, Groq, …) if you are not using Anthropic: +# OPENAI_API_KEY=sk-... +# OPENAI_BASE_URL=https://openrouter.ai/api/v1 +# OPENAI_MODEL=anthropic/claude-opus-5 +# Set either key to `mock` to exercise the wiring without a provider. diff --git a/Dockerfile b/Dockerfile index 42ecad1..e3b20e0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,9 +1,9 @@ -# Optional tool proxy only. The Pages site stays static; inference stays in the tab. -# Railway (or any host) runs this image and generates a public URL for Tools → Tool proxy URL. +# Optional tool proxy, and hosted Claude Opus when ANTHROPIC_API_KEY is set. +# The Pages site stays static. Railway (or any host) runs this image. FROM node:22-alpine WORKDIR /app -COPY tools/agent-api.ts tools/agent-api-listen.ts ./tools/ +COPY tools/agent-api.ts tools/agent-api-listen.ts tools/agent-chat.ts ./tools/ ENV NODE_ENV=production EXPOSE 8787 diff --git a/README.md b/README.md index 9d010de..61f6b77 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ It does need a connection to answer, which is a deliberate limit rather than a m The agent can search, read pages, calculate exactly, remember things you tell it, and call any MCP server you connect. Because a 0.8B model needs the help, common requests are routed through [skills](#skills) that show it a worked example rather than telling it what to do. -The published site has no backend. `pnpm build` produces a directory of static files that GitHub Pages can host, and every tool ships there — that is why the deployed site above has the full tool set rather than a reduced one. An optional [tool proxy](#optional-tool-proxy) lives in `tools/` for DuckDuckGo search and page reads without CORS: `pnpm dev` uses it locally, `pnpm proxy` runs it on its own. Inference stays in the tab either way. +The published site has no backend. `pnpm build` produces a directory of static files that GitHub Pages can host, and every tool ships there — that is why the deployed site above has the full tool set rather than a reduced one. An optional [tool proxy](#optional-tool-proxy) lives in `tools/` for DuckDuckGo search and page reads without CORS. Put an `ANTHROPIC_API_KEY` on that same process and it also hosts [Claude Opus](#hosted-model-claude-opus): visitors then chat without a GPU or a 448 MB download. Leave the key unset and inference stays in the tab, which is how the public demo still works. ## How it works @@ -302,7 +302,7 @@ GitHub Pages cannot host a process, so the published site stays browser-direct. - **`pnpm dev`** — the Vite plugin serves `POST /api/search` and `POST /api/fetch` on the dev server. `.env.development` sets `VITE_AGENT_API_BASE=same-origin`, so DuckDuckGo search and non-Wikipedia page reads go there automatically. - **`pnpm proxy`** — the same handlers on http://localhost:8787, for a static build or the hosted site. Paste that origin into **Tools → Tool proxy URL**, or build with `VITE_AGENT_API_BASE=http://localhost:8787`. -- **Railway (or any host)** — the `Dockerfile` in the repo root runs only this process. Create a project, connect `devbadya/Jarvis`, set `PROXY_ORIGINS` to `https://devbadya.github.io`, wait until the deploy is live, then **Settings → Networking → Generate domain**. That `https://….up.railway.app` origin is the URL: paste it into **Tools → Tool proxy URL** on the hosted site. There is no URL until a domain exists; the Hobby plan alone does not create one. +- **Railway (or any host)** — the `Dockerfile` in the repo root runs only this process. Create a project, connect `devbadya/Jarvis`, set `PROXY_ORIGINS` to `https://devbadya.github.io`, add `ANTHROPIC_API_KEY` if you want [hosted Opus](#hosted-model-claude-opus), wait until the deploy is live, then **Settings → Networking → Generate domain**. That `https://….up.railway.app` origin is the URL: paste it into **Tools → Tool proxy URL** on the hosted site. There is no URL until a domain exists; the Hobby plan alone does not create one. To give every visitor that proxy without asking them to paste anything, set the repository variable **`AGENT_API_BASE`** to its origin: `deploy.yml` passes it to the build as `VITE_AGENT_API_BASE`. Leave it unset and the hosted site stays browser-direct, which is what a fork with no proxy of its own needs — an empty value is not a proxy, so a workflow forwarding a variable nobody set cannot aim the build at an `/api` the host does not serve. @@ -310,15 +310,29 @@ To give every visitor that proxy without asking them to paste anything, set the The proxy scrapes DuckDuckGo HTML itself and fetches pages itself. It does not spend the Jina reader budget, and it is not limited to CORS-friendly endpoints. Wikipedia, LangSearch and Jina still leave the tab directly — they already send the headers, and their keys must not travel through this process. -A fetch-on-behalf proxy is still a confused deputy. Every target is resolved and refused if it lands on loopback, link-local or RFC1918, and redirects are re-checked. Do not bind `pnpm proxy` to the public internet without setting `PROXY_ORIGINS` to the pages that may call it (for example `https://devbadya.github.io`). Inference never goes through it. +A fetch-on-behalf proxy is still a confused deputy. Every target is resolved and refused if it lands on loopback, link-local or RFC1918, and redirects are re-checked. Do not bind `pnpm proxy` to the public internet without setting `PROXY_ORIGINS` to the pages that may call it (for example `https://devbadya.github.io`). Search and fetch never need a model key; chat does, and that key stays on this process. -The allowlist says who may call, not how often, and an allowed page is exactly what a scraper would forge. `pnpm proxy` therefore allows **30 requests a minute per caller** and answers `429` beyond that, counted from the forwarded address rather than the socket, since every edge terminates the connection itself. `PROXY_RATE_LIMIT` changes the number; `0` switches it off. Health checks are exempt. +The allowlist says who may call, not how often, and an allowed page is exactly what a scraper would forge. `pnpm proxy` therefore allows **30 requests a minute per caller** and answers `429` beyond that, counted from the forwarded address rather than the socket, since every edge terminates the connection itself. `PROXY_RATE_LIMIT` changes the number; `0` switches it off. Health checks are exempt. When `PROXY_ORIGINS` is set, `POST /api/chat` also requires a matching `Origin` header, so a curl from the public internet cannot spend the model key. + +### Hosted model (Claude Opus) + +The 0.8B on-device model will not become ChatGPT. To give visitors a frontier model without a GPU or a 448 MB install, put Anthropic's key on the same proxy: + +1. Create an API key at **[platform.claude.com](https://platform.claude.com)** (Console → API keys). Add prepaid credit under Billing. That is the official Opus API; do not buy keys from resellers. +2. On Railway (or `pnpm proxy`), set `ANTHROPIC_API_KEY`. The default model is `claude-opus-5`. `ANTHROPIC_MODEL` overrides it. Opus is billed per token — currently $5 / million input tokens and $25 / million output, and **you** pay for every visitor. +3. Point the site at the proxy: paste the origin into **Tool proxy URL**, or set the GitHub variable `AGENT_API_BASE` so every visitor uses it. + +`GET /api/health` then reports `{ ok: true, chat: { model: "claude-opus-5", provider: "anthropic" } }`. The landing page switches to **Start chatting** and skips the download. Tools still run in the tab; only generation goes to Anthropic. + +An OpenAI-compatible host still works if `ANTHROPIC_API_KEY` is unset: `OPENAI_API_KEY` plus optional `OPENAI_BASE_URL` / `OPENAI_MODEL` (Groq, OpenRouter, and so on). OpenRouter can serve Opus too (`OPENAI_BASE_URL=https://openrouter.ai/api/v1`, `OPENAI_MODEL=anthropic/claude-opus-5`) but you then have a middleman on top of Anthropic. Prefer the official Console. + +Set either key to `mock` to exercise the wiring without a provider. ### What leaves the browser -Inference does not: prompts, reasoning, and replies never leave the GPU, and neither do [memories](#memory), which are written to IndexedDB in this browser and read back into a prompt that goes no further than the GPU either. Tools are the exception, and always were. A `web_search` call sends the query to the chosen provider, a `read_page` call sends the URL to the reader, a `weather` call sends the place name to Open-Meteo's geocoder and its coordinates to the two forecast services, and a `current_time` call with a place sends the name to the same geocoder. +On the on-device path, inference does not leave: prompts, reasoning, and replies stay on the GPU, and neither do [memories](#memory), which are written to IndexedDB in this browser. Tools are the exception, and always were. A `web_search` call sends the query to the chosen provider, a `read_page` call sends the URL to the reader, a `weather` call sends the place name to Open-Meteo's geocoder and its coordinates to the two forecast services, and a `current_time` call with a place sends the name to the same geocoder. -On the hosted site those go direct, with no server of ours in the path to log them. With the optional proxy, DuckDuckGo search and non-Wikipedia page reads go to that process first — one more party than a search API, and the one you run. +With hosted Opus, the conversation itself goes to Anthropic through your proxy. Memories and MCP keys still stay in this browser. On the hosted site without a proxy, searches go direct, with no server of ours in the path to log them. With the optional proxy, DuckDuckGo search and non-Wikipedia page reads go to that process first — one more party than a search API, and the one you run. Worth being precise about on the default provider without a proxy: a search sends the query to `r.jina.ai`, which then sends it to DuckDuckGo. That is one more party than a search API like LangSearch or Jina involves. diff --git a/src/App.test.tsx b/src/App.test.tsx new file mode 100644 index 0000000..a71b0ae --- /dev/null +++ b/src/App.test.tsx @@ -0,0 +1,49 @@ +import { render, screen } from '@testing-library/react' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import App from './App' +import { EMPTY_STORAGE_STATUS } from '@/lib/storage' +import { useChatStore } from '@/store/chat' + +function stubAdapter(): void { + Object.defineProperty(navigator, 'gpu', { + configurable: true, + value: { + requestAdapter: async () => ({ + info: { vendor: 'test', architecture: 'gpu' }, + limits: { maxBufferSize: 1024 * 1024 * 1024 }, + }), + }, + }) +} + +beforeEach(() => { + stubAdapter() + vi.spyOn(globalThis, 'fetch').mockRejectedValue(new Error('no proxy')) + vi.spyOn(useChatStore.getState(), 'refreshStorage').mockResolvedValue() + vi.spyOn(useChatStore.getState(), 'probeHosted').mockResolvedValue() + useChatStore.setState({ + status: 'idle', + error: null, + storage: EMPTY_STORAGE_STATUS, + hostedChat: null, + }) +}) + +afterEach(() => { + vi.restoreAllMocks() + useChatStore.setState({ status: 'idle', storage: EMPTY_STORAGE_STATUS, hostedChat: null }) +}) + +describe('App header', () => { + it('names the on-device model until a hosted one is advertised', () => { + render() + expect(screen.getByText('Qwen3.5-0.8B · on-device')).toBeInTheDocument() + }) + + it('names the hosted model once the proxy advertises one', () => { + useChatStore.setState({ hostedChat: { base: 'https://proxy.example', model: 'claude-opus-5' } }) + render() + expect(screen.getByText('claude-opus-5 · hosted')).toBeInTheDocument() + expect(screen.queryByText('Qwen3.5-0.8B · on-device')).not.toBeInTheDocument() + }) +}) diff --git a/src/App.tsx b/src/App.tsx index d7ad789..557e456 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -23,15 +23,19 @@ const EVAL_MODE = new URLSearchParams(window.location.search).has('eval') */ function BrandMark() { const busy = useChatStore((state) => state.busy) + const hostedChat = useChatStore((state) => state.hostedChat) + const label = EVAL_MODE + ? 'eval harness' + : hostedChat + ? `${hostedChat.model} · hosted` + : 'Qwen3.5-0.8B · on-device' return (

Jarvis

-

- {EVAL_MODE ? 'eval harness' : 'Qwen3.5-0.8B · on-device'} -

+

{label}

) diff --git a/src/agent/loop.test.ts b/src/agent/loop.test.ts index 957ce41..fc0e9ac 100644 --- a/src/agent/loop.test.ts +++ b/src/agent/loop.test.ts @@ -413,3 +413,42 @@ describe('checking the answer before returning it', () => { expect(result.content).toBe('Ama Osei has led the airline since 2023.') }) }) + +describe('hosted native tool calls', () => { + function nativeClient( + rounds: { + text: string + toolCalls?: { id: string; name: string; arguments: Record }[] + }[], + ): LlmClient { + let round = 0 + return { + generate: vi.fn(async () => { + const output = rounds[round] ?? { text: '' } + round += 1 + return { text: output.text, tokens: 10, thinkTokens: 0, durationMs: 50, toolCalls: output.toolCalls } + }), + } as unknown as LlmClient + } + + it('executes structured calls and pairs the result with the provider id', async () => { + const execute = vi.fn(async () => '4') + const calculator = defineTool('calculator', 'maths', { type: 'object', properties: {} }, execute) + const client = nativeClient([ + { text: '', toolCalls: [{ id: 'call_1', name: 'calculator', arguments: { expression: '2+2' } }] }, + { text: '2 + 2 = 4' }, + ]) + + const result = await runAgent(client, turns, [calculator], callbacks()) + + expect(execute).toHaveBeenCalledWith({ expression: '2+2' }) + expect(result.content).toBe('2 + 2 = 4') + const conversation = vi.mocked(client.generate).mock.calls[1]?.[0] ?? [] + expect(conversation).toContainEqual({ + role: 'assistant', + content: '', + toolCalls: [{ id: 'call_1', name: 'calculator', arguments: { expression: '2+2' } }], + }) + expect(conversation).toContainEqual({ role: 'tool', content: '4', toolCallId: 'call_1' }) + }) +}) diff --git a/src/agent/loop.ts b/src/agent/loop.ts index 30e8616..2f84749 100644 --- a/src/agent/loop.ts +++ b/src/agent/loop.ts @@ -1,4 +1,4 @@ -import type { LlmClient } from '@/llm/client' +import type { InferenceClient } from '@/llm/client' import type { ChatTurn } from '@/llm/protocol' import { DEFAULT_STRATEGY, @@ -119,7 +119,7 @@ function settle(latest: AgentResult, remaining: ReviewCheck[], draft: Draft | nu * them. */ export async function runAgent( - client: LlmClient, + client: InferenceClient, turns: ChatTurn[], tools: Tool[], callbacks: AgentCallbacks, @@ -172,11 +172,18 @@ export async function runAgent( durationMs: generation.durationMs, tokensPerSecond: generation.durationMs > 0 ? (generation.tokens / generation.durationMs) * 1000 : 0, } + // Hosted models return native tool calls; the on-device path parses XML. + const nativeCalls = generation.toolCalls ?? [] + const parsedCalls = nativeCalls.length > 0 ? nativeCalls : parsed.toolCalls // A model that asks for a tool anyway once they are gone is asking for // something nothing can run, so the request is dropped and the round counts // as the answer it was meant to be. - const toolCalls = windDown ? [] : parsed.toolCalls - const outcome = { content: parsed.content, reasoning: parsed.reasoning, stats } + const toolCalls = windDown ? [] : parsedCalls + const outcome = { + content: nativeCalls.length > 0 ? generation.text || parsed.content : parsed.content, + reasoning: parsed.reasoning, + stats, + } // Only when the turn is over: before a tool call, reasoning is just reasoning. last = toolCalls.length === 0 ? promoteReasoningIfEmpty(outcome) : outcome callbacks.onRoundEnd(last) @@ -207,18 +214,32 @@ export async function runAgent( } // Echo the assistant's tool request back so the model sees its own decision. - conversation.push({ role: 'assistant', content: generation.text || raw }) + // Hosted APIs need the structured calls and the matching tool ids; the + // on-device worker needs the raw XML the chat template already emitted. + const pending = toolCalls.map((call) => ({ + ...call, + id: call.id ?? crypto.randomUUID(), + })) + if (nativeCalls.length > 0) { + conversation.push({ + role: 'assistant', + content: generation.text || parsed.content || '', + toolCalls: pending.map((call) => ({ id: call.id, name: call.name, arguments: call.arguments })), + }) + } else { + conversation.push({ role: 'assistant', content: generation.text || raw }) + } - for (const call of toolCalls) { - const id = crypto.randomUUID() - callbacks.onToolStart({ ...call, id }) + for (const call of pending) { + const { id } = call + callbacks.onToolStart(call) const startedAt = performance.now() const tool = byName.get(call.name) if (!tool) { const error = `Unknown tool "${call.name}". Available tools: ${[...byName.keys()].join(', ')}` callbacks.onToolEnd(id, { error, durationMs: performance.now() - startedAt }) - conversation.push({ role: 'tool', content: error }) + conversation.push({ role: 'tool', content: error, toolCallId: id }) continue } @@ -231,7 +252,7 @@ export async function runAgent( if (earlier !== undefined) { const note = repeatedCallNote(call.name, earlier) callbacks.onToolEnd(id, { result: note, durationMs: performance.now() - startedAt }) - conversation.push({ role: 'tool', content: note }) + conversation.push({ role: 'tool', content: note, toolCallId: id }) continue } @@ -239,7 +260,7 @@ export async function runAgent( const result = await tool.execute(call.arguments) executed.set(fingerprint, result) callbacks.onToolEnd(id, { result, durationMs: performance.now() - startedAt }) - conversation.push({ role: 'tool', content: result }) + conversation.push({ role: 'tool', content: result, toolCallId: id }) // Only what a tool actually returned is evidence. A failure has nothing // to check an answer against, and demanding a citation for a page that // never loaded would be worse than saying nothing. @@ -247,7 +268,7 @@ export async function runAgent( } catch (error) { const message = error instanceof Error ? error.message : String(error) callbacks.onToolEnd(id, { error: message, durationMs: performance.now() - startedAt }) - conversation.push({ role: 'tool', content: `Tool "${call.name}" failed: ${message}` }) + conversation.push({ role: 'tool', content: `Tool "${call.name}" failed: ${message}`, toolCallId: id }) } } diff --git a/src/agent/parse.ts b/src/agent/parse.ts index e3481f7..0efe830 100644 --- a/src/agent/parse.ts +++ b/src/agent/parse.ts @@ -1,6 +1,8 @@ export interface ParsedToolCall { name: string arguments: Record + /** Provider id, when the call came from a hosted API rather than XML. */ + id?: string } export interface ParsedOutput { diff --git a/src/components/InstallPanel.tsx b/src/components/InstallPanel.tsx index 8ab31d6..dcd081a 100644 --- a/src/components/InstallPanel.tsx +++ b/src/components/InstallPanel.tsx @@ -2,14 +2,20 @@ import { useEffect, useState, type ReactNode } from 'react' import { Alert } from '@heroui/react/alert' import { Button } from '@heroui/react/button' import { Chip } from '@heroui/react/chip' +import { Description } from '@heroui/react/description' +import { FieldError } from '@heroui/react/field-error' +import { Input } from '@heroui/react/input' +import { Label } from '@heroui/react/label' import { Link } from '@heroui/react/link' import { Meter } from '@heroui/react/meter' import { ProgressBar } from '@heroui/react/progress-bar' import { Spinner } from '@heroui/react/spinner' +import { TextField } from '@heroui/react/textfield' import { MODEL_DOWNLOAD_BYTES, MODEL_ID } from '@/llm/config' import { detectWebGpu, type GpuCapability } from '@/lib/webgpu' import { formatBytes } from '@/lib/format' import { hasRoomFor } from '@/lib/storage' +import { isHttpUrl } from '@/tools/mcp' import { useChatStore } from '@/store/chat' /** One row of the specification list, so the labels stay in one column. */ @@ -23,22 +29,32 @@ function Row({ children, label }: { children: ReactNode; label: string }) { } /** - * Everything the one-time download needs to be legible: whether this browser can - * run the model at all, what is already on disk, whether there is room for the - * rest, and the button that starts it. - * - * It is the landing page's call to action, which is why it carries no heading of - * its own — the hero above it already said what this is. + * The landing page's call to action. Hosted chat (Claude Opus on the tool + * proxy) starts with no download. The on-device path still shows the GPU + * check, the storage figures, and the 448 MB install. */ export function InstallPanel() { const [gpu, setGpu] = useState(null) - const { status, loadMessage, loadProgress, error, storage, initialize, refreshStorage, removeModel } = - useChatStore() + const { + status, + loadMessage, + loadProgress, + error, + storage, + hostedChat, + initialize, + refreshStorage, + removeModel, + probeHosted, + webAccess, + setWebAccess, + } = useChatStore() useEffect(() => { void detectWebGpu().then(setGpu) void refreshStorage() - }, [refreshStorage]) + void probeHosted() + }, [refreshStorage, probeHosted]) const loaded = loadProgress.reduce((sum, file) => sum + file.loaded, 0) const total = loadProgress.reduce((sum, file) => sum + file.total, 0) @@ -51,6 +67,9 @@ export function InstallPanel() { // continues from it, so the gate offers to resume rather than to start again. const resumeBytes = installed ? 0 : storage.partialBytes const remainingBytes = Math.max(MODEL_DOWNLOAD_BYTES - resumeBytes, 0) + const typedProxy = webAccess.proxyUrl ?? '' + const badProxy = typedProxy.trim().length > 0 && !isHttpUrl(typedProxy.trim()) + const hosted = hostedChat !== null return (
@@ -62,29 +81,55 @@ export function InstallPanel() { />
- {gpu === null && ( + {gpu === null && !hosted && (

Checking GPU support…

)} - {gpu?.supported === false && ( + {gpu?.supported === false && !hosted && ( WebGPU is unavailable - {gpu.reason} Generation has no CPU fallback, so the chat cannot start here.{' '} + {gpu.reason} On-device generation has no CPU fallback.{' '} Which browsers support WebGPU + . Paste a tool proxy with Claude Opus below to chat without a GPU. )} - {gpu?.supported && status !== 'loading' && ( + {hosted && status !== 'loading' && ( + <> +
+ +
+ {status === 'error' && ( + + + + Could not reach the hosted model + {error} + + + )} +
+ + {hostedChat.model} + + Hosted via the tool proxy — nothing downloads to this browser +
+ + )} + + {!hosted && gpu?.supported && status !== 'loading' && ( <>