runner-gateway

A simple gateway for interacting with model runner APIs (i.e. Ollama) that provides TLS and OAuth2.0 authentication with PASETO token validation.

Features

• Authentication: Secure access to the Ollama API using PASETO tokens.
• Configuration: Easy configuration through a YAML file.
• Logging: Detailed logging for monitoring and debugging.
• SSL/TLS Support: SSL/TLS support for secure communication.

How It Works

The runner-gateway acts as a bridge between your application and the Ollama API. It handles authentication, logging, and SSL/TLS encryption to ensure secure communication. There are tools within the code base that can be used to generate PASETO tokens and validate them, but the server can also be configured to validate public or internal v4 PASETO tokens generated by other services.

PASETO Token Validation

For more information on PASETO token validation and usage, refer to the PASETO documentation.

By default, the service is configured to validate v4 PASETO tokens. The service can support v2 and v4 PASETO tokens generated by other services with an adjustment to the settings.

Installation

Clone the repository and run the following commands to install dependencies and build the project:

git clone https://github.com/jtlabsio/runner-gateway.git
cd runner-gateway

Next, configure the service...

Configuration

PASETO Token Generation and Validation

Generate a PASETO v4 asymmetric key pair:

go run tools/paseto -action=asymmetric

This will generate paseto.key and paseto.pub files in the ./settings directory (this is what the server is configured to look at by default).

Note: If you have an existing private / public key pair, you can use your existing public key for validation instead by adjusting the configuration accordingly. The private key is only used for generating tokens via the tools provided, but it is not required for runtime gateway operation.

Create a Custom Settings File

Within the ./settings folder, create a custom settings file with the suffix of .yaml. For example, my-domain.yaml:

logging:
  # override the logging level if desired (trace, debug, info, warn, error)
  level: info 
paseto:
  # if using the server to generate long living access tokens, the expiration can be overridden
  expiration: 8766h # 1 year
  # set your secret key here (optional if the service should validate v4.local tokens)
  secretKey: Xthisshouldbeasecretkeythatis64bytestosupportpasetov4standardsXX
runners:
  - host: host.docker.internal:11434 # note the host name - this is for Docker
    name: ollama
    path: /
    scheme: http
server:
  # paths to your TLS certificate and keys (optional if TLS is desired)
  certificatePath: .acme.sh/my-domain.com_ecc/fullchain.cer
  keyPath: .acme.sh/my-domain.com_ecc/my-domain.com.key

note: A simple way to generate TLS certificates and keys is to use the acme.sh script. This script can be used to automate the creation and renewal of Let's Encrypt TLS certificates for your domain: https://github.com/acmesh-official/acme.sh

Model Runners

The service can be configured to run multiple model runners. Each model runner is defined in the runners section of the configuration file, and the path element of the configuration can be used to route requests to the appropriate model runner. Here is a simple example of a configuration with two different model runners:

runners:
  - host: 127.0.0.1:11434
    name: ollama
    path: /local-ollama
    scheme: http
  - host: other.ollama.host:11434
    name: ollama
    path: /corp-ollama
    scheme: http

With the above configuration, inbound requests to the gateway with a prefix of /local-ollama (for example, GET /local-ollama/api/tags) would be sent to the downstream host as follows: GET http://127.0.0.1:11434/api/tags). Similarly, requests to the gateway with a prefix of /corp-ollama would be sent to http://other.ollama.host:11434/api/tags.

Build

Once the configuration is set, the service can be built as a docker image:

make docker

Running

Replace the GO_ENV environment variable below with the name of the configuration file created above.

docker run -d \
  -p 8443:8443 \
  --add-host=host.docker.internal:host-gateway \
  --restart unless-stopped \
  --name runner-gateway \
  -e GO_ENV=my-domain \
  -v ./settings:/opt/gateway/settings \
  -v ${HOME}/.acme.sh:/opt/gateway/.acme.sh \
  runner-gateway

The volume maps above are specified to allow the service to access the configuration overrides and TLS certificates, respectively... adjust these values as necessary to your environment. The docker build will also copy the settings to the image at time of build, but by using a volume mount, the docker image will not need to be rebuilt each time the configuration or certificates change (it will only need to be restarted).

Tools

The following tools are available for generating and working with PASETO tokens:

Generate Asymmetric (Private/Public) Key Pair

GO_ENV=my-domain go run tools/paseto -action=asymmetric

Generate Symmetric Key

GO_ENV=my-domain go run tools/paseto -action=symmetric

Generate Private PASETO Token Signed by Public Key

GO_ENV=my-domain go run tools/paseto -action=private

Generate Public PASETO Token Signed by Public Key

GO_ENV=my-domain go run tools/paseto -action=public