Llm proxy

llm_proxy/README.md

@@ -0,0 +1,93 @@
+# llm_proxy
+
+A local, single-machine pass-through proxy for **OpenAI**, **Anthropic**, and
+**Bedrock**. Callers keep their exact SDK calling convention — the only change is
+one base-URL env var per provider. Every call flows through one function
+(`core.proxy_request`) where token/metrics hooks fire.
+
+**Scope:** request/response ("call and return") only. Streaming is intentionally
+not implemented yet.
+
+## How it works
+
+```
+your app (unchanged)         localhost:8080                 real upstream
+  openai SDK   ─/openai/... ─┐
+  anthropic SDK ─/anthropic/ ─┼─▶ proxy_request(ctx) ─▶ provider ─▶ api.openai.com
+  boto3 bedrock ─/bedrock/... ┘    (metrics hooks)      adapter     api.anthropic.com
+                                                                    bedrock-runtime.<region>.amazonaws.com
+```
+
+- **OpenAI / Anthropic** — straight HTTP reverse-proxy: rewrite host, swap in the
+  real key, forward with `requests`, return the response.
+- **Bedrock** — re-issued through the proxy's own `boto3` client (handles SigV4
+  signing + URL-encoding correctly). Only `invoke` is wired up.
+
+## Run
+
+```bash
+pip install -r llm_proxy/requirements.txt
+
+# real upstream credentials live here; callers can use dummy keys
+export OPENAI_API_KEY=sk-...
+export ANTHROPIC_API_KEY=sk-ant-...
+export AWS_REGION=us-east-1          # + normal AWS creds (env / ~/.aws / role)
+
+python -m llm_proxy                  # listens on 127.0.0.1:8080
+```
+
+## Point your SDKs at it
+
+No code changes — just env vars:
+
+```bash
+export OPENAI_BASE_URL=http://localhost:8080/openai/v1
+export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
+export AWS_ENDPOINT_URL_BEDROCK_RUNTIME=http://localhost:8080/bedrock
+```
+
+Then your existing code works unchanged:
+
+```python
+from openai import OpenAI
+OpenAI().chat.completions.create(model="gpt-4o-mini",
+                                 messages=[{"role": "user", "content": "hi"}])
+
+from anthropic import Anthropic
+Anthropic().messages.create(model="claude-3-5-sonnet-20241022", max_tokens=64,
+                            messages=[{"role": "user", "content": "hi"}])
+
+import boto3, json
+boto3.client("bedrock-runtime").invoke_model(
+    modelId="anthropic.claude-3-5-sonnet-20240620-v1:0",
+    body=json.dumps({"anthropic_version": "bedrock-2023-05-31",
+                     "max_tokens": 64,
+                     "messages": [{"role": "user", "content": "hi"}]}))
+```
+
+## Configuration (env vars)
+
+| Var | Default | Purpose |
+|---|---|---|
+| `PROXY_HOST` / `PROXY_PORT` | `127.0.0.1` / `8080` | where the proxy listens |
+| `PROXY_CONNECT_TIMEOUT` / `PROXY_READ_TIMEOUT` | `10` / `600` | upstream timeouts (s) |
+| `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` | — | real upstream keys the proxy injects |
+| `OPENAI_UPSTREAM_BASE` / `ANTHROPIC_UPSTREAM_BASE` | official APIs | override upstream (e.g. Azure/gateway) |
+| `BEDROCK_REGION` (or `AWS_REGION`) | `us-east-1` | Bedrock region |
+| `BEDROCK_UPSTREAM_HOST` | `bedrock-runtime.<region>.amazonaws.com` | override Bedrock host |
+
+## The metrics seam
+
+`hooks.py` defines `on_request` / `on_response`; today they only log. Because the
+whole response is buffered, token accounting is a one-liner later —
+`resp.json().get("usage")` for OpenAI/Anthropic, or the per-model field in
+Bedrock's response body.
+
+## Known limitations (current scope)
+
+- **No streaming.** `stream=True` / `invoke-with-response-stream` are not handled.
+- **Bedrock error bodies are reconstructed**, not passed through byte-for-byte
+  (boto3 raises on 4xx/5xx; we rebuild a JSON body with the real status +
+  message). OpenAI/Anthropic errors pass through unchanged.
+- **Dev server.** Runs on Flask's built-in server — fine for a local proxy, not
+  meant for production traffic.

llm_proxy/__init__.py

@@ -0,0 +1,14 @@
+"""Local LLM proxy.
+
+A transparent, single-machine pass-through for OpenAI, Anthropic, and Bedrock.
+Point each provider's SDK at this service via its base-URL env var and calls flow
+through one choke point (``llm_proxy.core.proxy_request``) where request/response
+metrics hooks fire.
+
+Scope: request/response ("call and return") only. Streaming is intentionally
+not implemented yet.
+"""
+
+__all__ = ["__version__"]
+
+__version__ = "0.1.0"

llm_proxy/__main__.py

@@ -0,0 +1,29 @@
+"""Entry point: ``python -m llm_proxy``."""
+
+from __future__ import annotations
+
+import logging
+
+from llm_proxy.app import create_app
+from llm_proxy.config import Config
+
+
+def main() -> None:
+    logging.basicConfig(
+        level=logging.INFO,
+        format="%(asctime)s %(levelname)s %(name)s: %(message)s",
+    )
+    cfg = Config.from_env()
+    app = create_app(cfg)
+    logging.getLogger("llm_proxy").info(
+        "llm_proxy on http://%s:%d  (openai=%s, anthropic=%s, bedrock=%s [%s])",
+        cfg.host, cfg.port, cfg.openai.upstream_base, cfg.anthropic.upstream_base,
+        cfg.bedrock_upstream_host, cfg.bedrock_region,
+    )
+    # threaded so concurrent callers don't serialize; dev server is fine for a
+    # local proxy.
+    app.run(host=cfg.host, port=cfg.port, threaded=True)
+
+
+if __name__ == "__main__":
+    main()

llm_proxy/app.py

@@ -0,0 +1,42 @@
+"""Flask app: one catch-all route per provider prefix, all funneled through
+``proxy_request``."""
+
+from __future__ import annotations
+
+import logging
+
+from flask import Flask, jsonify, request
+
+from llm_proxy.config import Config
+from llm_proxy.core import proxy_request
+from llm_proxy.providers import build_registry
+
+log = logging.getLogger("llm_proxy")
+
+ALL_METHODS = ["GET", "POST", "PUT", "PATCH", "DELETE"]
+
+
+def create_app(cfg: Config = None) -> Flask:
+    cfg = cfg or Config.from_env()
+    app = Flask(__name__)
+    registry = build_registry(cfg)
+
+    @app.route("/healthz", methods=["GET"])
+    def healthz():
+        return jsonify(status="ok", providers=sorted(registry.keys()))
+
+    @app.route("/<provider>/<path:subpath>", methods=ALL_METHODS)
+    def dispatch(provider, subpath):
+        prov = registry.get(provider)
+        if prov is None:
+            return (
+                jsonify(error=f"unknown provider '{provider}'", known=sorted(registry.keys())),
+                404,
+            )
+        try:
+            return proxy_request(prov, subpath, request)
+        except Exception as exc:  # surface upstream/adapter errors as 502
+            log.exception("proxy error for %s/%s", provider, subpath)
+            return jsonify(error="proxy_error", detail=str(exc)), 502
+
+    return app

llm_proxy/config.py

@@ -0,0 +1,61 @@
+"""Configuration, read once from the environment at startup.
+
+The proxy holds the *real* upstream credentials; callers can send dummy keys.
+"""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+from typing import Optional
+
+
+@dataclass
+class ProviderConfig:
+    upstream_base: str
+    api_key: Optional[str] = None
+
+
+@dataclass
+class Config:
+    host: str
+    port: int
+    connect_timeout: float
+    read_timeout: float
+
+    openai: ProviderConfig
+    anthropic: ProviderConfig
+
+    bedrock_region: str
+    bedrock_upstream_host: str
+
+    @classmethod
+    def from_env(cls) -> "Config":
+        region = (
+            os.getenv("BEDROCK_REGION")
+            or os.getenv("AWS_REGION")
+            or os.getenv("AWS_DEFAULT_REGION")
+            or "us-east-1"
+        )
+        return cls(
+            host=os.getenv("PROXY_HOST", "127.0.0.1"),
+            port=int(os.getenv("PROXY_PORT", "8080")),
+            connect_timeout=float(os.getenv("PROXY_CONNECT_TIMEOUT", "10")),
+            read_timeout=float(os.getenv("PROXY_READ_TIMEOUT", "600")),
+            openai=ProviderConfig(
+                upstream_base=os.getenv(
+                    "OPENAI_UPSTREAM_BASE", "https://api.openai.com"
+                ).rstrip("/"),
+                api_key=os.getenv("OPENAI_API_KEY"),
+            ),
+            anthropic=ProviderConfig(
+                upstream_base=os.getenv(
+                    "ANTHROPIC_UPSTREAM_BASE", "https://api.anthropic.com"
+                ).rstrip("/"),
+                api_key=os.getenv("ANTHROPIC_API_KEY"),
+            ),
+            bedrock_region=region,
+            bedrock_upstream_host=os.getenv(
+                "BEDROCK_UPSTREAM_HOST", f"bedrock-runtime.{region}.amazonaws.com"
+            ),
+        )

llm_proxy/core.py

@@ -0,0 +1,43 @@
+"""The single choke point every proxied call flows through."""
+
+from __future__ import annotations
+
+import json
+import time
+from typing import Optional
+
+from flask import Response
+
+from llm_proxy.hooks import Ctx, hooks
+
+
+def _guess_model(body: bytes) -> Optional[str]:
+    """Best-effort model name from the JSON body, for logging/metrics.
+
+    Never raises. Returns None for requests whose model isn't in the body
+    (e.g. Bedrock, where it's in the path and already shown via the subpath).
+    """
+    try:
+        model = json.loads(body).get("model")
+        return model if isinstance(model, str) else None
+    except Exception:
+        return None
+
+
+def proxy_request(provider, subpath, flask_request):
+    body = flask_request.get_data()
+    ctx = Ctx(
+        provider=provider.name,
+        method=flask_request.method,
+        subpath=subpath,
+        body=body,
+        headers=dict(flask_request.headers),
+        t0=time.monotonic(),
+        model=_guess_model(body),
+    )
+    hooks.on_request(ctx)
+
+    pr = provider.forward(flask_request, subpath, body)
+
+    hooks.on_response(ctx, pr)
+    return Response(pr.content, status=pr.status, headers=pr.headers)

llm_proxy/hooks.py

@@ -0,0 +1,49 @@
+"""The metrics seam.
+
+Every proxied call passes through ``on_request`` / ``on_response``. Today these
+only log. Token accounting lands here later: because the whole response is
+buffered, usage extraction is a one-liner, e.g. ``resp.json().get("usage")`` for
+OpenAI/Anthropic (Bedrock's usage lives in its per-model response body).
+"""
+
+from __future__ import annotations
+
+import logging
+import time
+from dataclasses import dataclass
+from typing import Any, Dict, Optional
+
+log = logging.getLogger("llm_proxy")
+
+
+@dataclass
+class Ctx:
+    provider: str
+    method: str
+    subpath: str
+    body: bytes
+    headers: Dict[str, str]
+    t0: float
+    model: Optional[str] = None
+
+    def elapsed_ms(self) -> float:
+        return (time.monotonic() - self.t0) * 1000.0
+
+
+class Hooks:
+    def on_request(self, ctx: Ctx) -> None:
+        log.info(
+            "→ %s %s /%s model=%s (%d bytes)",
+            ctx.provider, ctx.method, ctx.subpath, ctx.model, len(ctx.body),
+        )
+
+    def on_response(self, ctx: Ctx, resp: Any) -> None:
+        log.info(
+            "← %s %s /%s -> %s in %.0fms",
+            ctx.provider, ctx.method, ctx.subpath,
+            getattr(resp, "status", "?"), ctx.elapsed_ms(),
+        )
+        # TODO(metrics): pull token usage off `resp` and emit it.
+
+
+hooks = Hooks()

llm_proxy/providers/__init__.py

@@ -0,0 +1,14 @@
+from __future__ import annotations
+
+from llm_proxy.providers.anthropic import AnthropicProvider
+from llm_proxy.providers.bedrock import BedrockProvider
+from llm_proxy.providers.openai import OpenAIProvider
+
+
+def build_registry(cfg):
+    """Map the URL prefix -> provider instance."""
+    return {
+        "openai": OpenAIProvider(cfg),
+        "anthropic": AnthropicProvider(cfg),
+        "bedrock": BedrockProvider(cfg),
+    }

llm_proxy/providers/anthropic.py

@@ -0,0 +1,19 @@
+from __future__ import annotations
+
+from llm_proxy.providers.base import HttpProvider, UpstreamRequest, client_headers
+
+
+class AnthropicProvider(HttpProvider):
+    name = "anthropic"
+
+    def target(self, req, subpath, body):
+        headers = client_headers(req, drop=["x-api-key", "authorization"])
+        if self.cfg.anthropic.api_key:
+            headers["x-api-key"] = self.cfg.anthropic.api_key
+        # `anthropic-version` is supplied by the SDK and passes through untouched.
+        return UpstreamRequest(
+            method=req.method,
+            url=f"{self.cfg.anthropic.upstream_base}/{subpath}",
+            headers=headers,
+            params=req.args.to_dict(flat=True),
+        )