Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ jobs:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2.9.1
- name: Build the workspace
run: cargo build --locked --workspace --all-features
- name: Build the production binary for conformance testing
run: cargo build --locked -p contextforge-data-plane --features plugins

- name: Strip the conformance binary
run: strip target/debug/contextforge-data-plane
Expand Down
9 changes: 2 additions & 7 deletions .github/workflows/conformance.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ jobs:
with:
tool: cargo-binstall

- name: Install cf-integration 0.3.1 binary
run: cargo binstall cf-integration@0.3.1 --no-confirm
- name: Install cf-integration 0.3.2 binary
run: cargo binstall cf-integration@0.3.2 --no-confirm

- name: Prepare the exact data-plane image
env:
Expand All @@ -74,11 +74,6 @@ jobs:
--binary contextforge-data-plane
--image contextforge-data-plane:conformance

- name: Install Node.js 22
uses: actions/setup-node@v6.5.0
with:
node-version: 22

- name: Run strict modern conformance
id: conformance
env:
Expand Down
22 changes: 7 additions & 15 deletions .secrets.baseline

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ DETECT_SECRETS_EXCLUDE := '(?x)(Cargo\.lock$$|\.lock$$)|^\.secrets\.baseline$$'
help: ## Show available commands
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-22s\033[0m %s\n", $$1, $$2}'

docker-prod: ## Build production Docker image (contextforge-data-plane:latest) from docker/Dockerfile
docker-prod: ## Build production Docker image with plugins and without testing-only with_tools
docker build -t $(IMAGE_NAME) -f docker/Dockerfile .

compose-up: ## Launch stack: nginx, control plane, redis, postgres, pgbouncer, dataplane, fast_time_server
Expand Down
22 changes: 15 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,16 @@ Build the production image and start the supported control-plane + data-plane
test stack:

```bash
export CONTEXTFORGE_DATA_PLANE_JWKS_URL=https://your-issuer.example/.well-known/jwks.json
make docker-prod
make compose-up
```

Replace the example JWKS URL with the HTTPS endpoint for the issuer of your
control-plane tokens. The production image includes the plugin factories and
`/contextforge-rs/health`; it does not include `with_tools`. That feature is for
testing only and enables unauthenticated token, JWKS, and config helpers.

The stack uses the current `fast_time_server` backend and exercises config
publication through the external ContextForge control plane. See
[getting-started.md](_context/wiki/getting-started.md) for the complete smoke
Expand Down Expand Up @@ -72,29 +78,31 @@ cargo run --release \
--address 0.0.0.0:8001 \
--redis-port 6379 \
--redis-address 127.0.0.1 \
--token-verification-public-key assets/jwt.key.pub \
--token-verification-private-key assets/jwt.key \
--jwks-url "$CONTEXTFORGE_DATA_PLANE_JWKS_URL" \
--number-of-cpus 16 \
--redis-mode=plain-text \
--upstream-connection-mode=plain-text-or-tls \
--runtime-plugins-enabled true
```

### Testing-Only Bootstrap Helpers

For local testing, explicitly enable `with_tools` and use its local JWKS
endpoint. Never enable this feature or use `--all-features` in production.

```bash
cargo run --features contextforge-data-plane-lib/with_tools \
cargo run -p contextforge-data-plane --features with_tools \
-- \
--address 0.0.0.0:8080 \
--address 127.0.0.1:8080 \
--redis-address 127.0.0.1 \
--redis-port 6379 \
--redis-mode plain-text \
--token-verification-private-key ./assets/jwt.key \
--token-verification-public-key ./assets/jwt.key.pub \
--jwks-url http://127.0.0.1:8080/contextforge-rs/admin/.well-known/jwks.json \
--upstream-connection-mode plain-text-or-tls \
--tls-address 0.0.0.0:8443 \
--tls-address 127.0.0.1:8443 \
--server-private-key ./assets/tls_key.pem \
--server-certificate ./assets/tls_certificate.pem
--runtime-plugins-enabled true
```

## Tracing and Metrics
Expand Down
6 changes: 5 additions & 1 deletion _context/wiki/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,11 @@ Order is invariant: auth/config before backend selection; request plugins before
| `gateway/session_store/` | Local and Redis user session storage |
| `user_config_store/` | `UserConfigStore` trait, Redis-backed store |
| `transports/` | Downstream TCP and TLS listener setup |
| `tools.rs` | Local bootstrap helpers (`with_tools` feature only) |
| `tools.rs` | Testing-only token, JWKS, and config helpers (`with_tools` feature) |

`Gateway::into_router` registers `/contextforge-rs/health` in every build,
outside the MCP authentication and config layers. Health remains covered by the
outer HTTP tracing and metrics layers.

## State Ownership

Expand Down
21 changes: 12 additions & 9 deletions _context/wiki/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
--redis-address --redis-port --redis-mode
```

Plus at least: `--address` or `--tls-address`, `--token-verification-public-key` or `--token-verification-secret`.
Plus `--jwks-url` and at least one listener: `--address` or `--tls-address`.

## Complete CLI and Environment Reference

Expand All @@ -30,9 +30,9 @@ Origin and Host settings retain the explicitly configured
| `--tls-address <host:port>` | `CONTEXTFORGE_DATA_PLANE_TLS_ADDRESS` | Optional | TLS listener; requires server certificate and key. |
| `--server-certificate <path>` | `CONTEXTFORGE_DATA_PLANE_TLS_SERVER_CERTIFICATE` | With `--tls-address` | PEM certificate chain for downstream TLS. |
| `--server-private-key <path>` | `CONTEXTFORGE_DATA_PLANE_TLS_SERVER_PRIVATE_KEY` | With `--tls-address` | PEM private key for downstream TLS. |
| `--token-verification-public-key <path>` | `CONTEXTFORGE_DATA_PLANE_TOKEN_VERIFICATION_PUBLIC_KEY` | For RSA tokens | Verifies `RS256`, `RS384`, and `RS512` tokens. |
| `--token-verification-secret <secret>` | `CONTEXTFORGE_DATA_PLANE_TOKEN_SECRET` | For HMAC tokens | Verifies `HS256`, `HS384`, and `HS512` tokens. |
| `--token-verification-private-key <path>` | `CONTEXTFORGE_DATA_PLANE_TOKEN_VERIFICATION_PRIVATE_KEY` | Required when built with `with_tools` | Signs tokens for the optional local bootstrap helper. |
| `--jwks-url <url>` | `CONTEXTFORGE_DATA_PLANE_JWKS_URL` | Required | Token issuer's HTTPS JWKS endpoint; HTTP is allowed only on loopback for testing. |
| `--jwks-ca-cert-path <path>` | `CONTEXTFORGE_DATA_PLANE_JWKS_CA_PATH` | Optional | PEM trust bundle for the JWKS endpoint. |
| `--token-verification-private-key <path>` | None (CLI only) | Required when built with `with_tools` | Signs tokens for testing-only bootstrap helpers. Never use this feature in production. |

### MCP request validation

Expand Down Expand Up @@ -250,17 +250,18 @@ docker compose -f docker/docker-compose-local.yaml exec -T redis \
}'
```

Build and run with demo factories and runtime execution enabled:
For local testing only, build and run with demo factories, testing-only
`with_tools` helpers, and runtime execution enabled:

```bash
cargo run -p contextforge-data-plane \
--features 'contextforge-data-plane-lib/with_tools,test-plugins' \
--features 'with_tools,test-plugins' \
--bin contextforge-data-plane -- \
--address 127.0.0.1:8001 \
--redis-address 127.0.0.1 \
--redis-port 6379 \
--redis-mode plain-text \
--token-verification-public-key assets/jwt.key.pub \
--jwks-url http://127.0.0.1:8001/contextforge-rs/admin/.well-known/jwks.json \
--token-verification-private-key assets/jwt.key \
--upstream-connection-mode plain-text-or-tls \
--runtime-plugins-enabled true
Expand Down Expand Up @@ -387,13 +388,15 @@ docker compose \
up -d
```

Run the gateway with export enabled (RUST_TRACE_LOG=debug required for trace export):
Run the gateway with export enabled (RUST_TRACE_LOG=debug required for trace
export). Set `CONTEXTFORGE_DATA_PLANE_JWKS_URL` to your token issuer's HTTPS JWKS
endpoint; this command does not enable testing helpers.
```bash
RUST_TRACE_LOG=debug \
cargo run --release --bin contextforge-data-plane -- \
--address 0.0.0.0:8001 \
--redis-port 6379 --redis-address 127.0.0.1 --redis-mode=plain-text \
--token-verification-public-key assets/jwt.key.pub \
--jwks-url "$CONTEXTFORGE_DATA_PLANE_JWKS_URL" \
--number-of-cpus 4 \
--upstream-connection-mode=plain-text-or-tls \
--enable-open-telemetry true \
Expand Down
28 changes: 25 additions & 3 deletions _context/wiki/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,34 @@
3. Redis reachable; TLS/mTLS across trust zones; write access restricted to the control plane; `DATAPLANE_PUBLISHER=true` on the control plane.
4. Upstream connection mode matches backend URL schemes.
5. One replica per `Mcp-session-id` (single replica or sticky routing).
6. `with_tools` feature **disabled** in the production build.
6. `with_tools` feature **disabled** in the production build; it is for testing only.
7. Telemetry export pointed at the collector.
8. System limits raised: `nofile 65535`, TCP tuning (`tcp_fin_timeout=15`, widened local port range).

## Health Endpoint

**`/contextforge-rs/health` is a `with_tools` bootstrap helper only.** Production builds compile it out. Use TCP-level liveness checks or the exported metrics until a real health endpoint exists.
`GET /contextforge-rs/health` is available in every build, including production
builds without `with_tools`. It returns HTTP `200` with
`{"status": "healthy"}` and does not require authentication. The reference nginx
configuration also exposes it at `/health`.

Use it for HTTP liveness checks. It reports that the HTTP server is responding;
it does not check Redis, JWKS availability, or backend readiness.

## Production Builds

```bash
make docker-prod
# Equivalent native build:
cargo build --locked --release -p contextforge-data-plane --features plugins
```

`make docker-prod`, direct builds of `docker/Dockerfile`, and the image publishing
workflow all compile the production plugin factories without `with_tools`.
Do not use `--all-features` for production artifacts: it also enables testing
helpers and demo plugins. Configure `--jwks-url` (or
`CONTEXTFORGE_DATA_PLANE_JWKS_URL`) with the HTTPS JWKS endpoint for the token
issuer; the dataplane does not need a token-signing private key.

## nginx Front-Door Routing

Expand Down Expand Up @@ -72,7 +93,8 @@ Both default to ~60s. For functional tests, shorten the publisher interval and d
| --- | --- |
| JWT revocation | None. A leaked token is valid until `exp`. Rotate the key and restart to invalidate. |
| CORS / Origin | CORS response headers are permissive. `mcp_origin_layer` validates Origin before authentication, and RMCP validates Host at the MCP service boundary. Configure both `--mcp-allowed-hosts` and `--mcp-allowed-origins` for production. |
| Local bootstrap routes | `/contextforge-rs/admin/tokens/{user}`, `/admin/userconfigs/{user}`, `/health` are **outside auth middleware — unauthenticated by design.** Only exist with `with_tools`. Production builds must not enable `with_tools`. |
| Testing helpers | The token, JWKS, and user-config routes under `/contextforge-rs/admin/` are unauthenticated and exist only with `with_tools`. The feature is for testing only and must never be enabled in production. |
| Health | `/contextforge-rs/health` is unauthenticated and available in every build. It checks HTTP liveness only. |
| Redis trust | Whoever can write Redis controls routing (arbitrary backend URLs receive caller traffic) AND which registered plugin hooks execute on payloads. Protect with TLS/mTLS and restrict write access to the control plane. |
| Downstream TLS | Optional. Plain HTTP is acceptable only behind a trusted front door on a private network. Identity is always the bearer JWT, not mTLS. |
| Plugin code | Fully trusted, in-process. Redis config activates compiled-in factories only — it cannot inject new Rust code. |
33 changes: 24 additions & 9 deletions _context/wiki/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,16 @@
## Full Docker Stack

```bash
make docker-prod # build contextforge-data-plane:latest from docker/Dockerfile
export CONTEXTFORGE_DATA_PLANE_JWKS_URL=https://your-issuer.example/.well-known/jwks.json
make docker-prod # production plugins and health endpoint; no testing helpers
make compose-up # start nginx, Python control/built-in components, Redis, Postgres, external dataplane, fast_time_server
```

Replace the example JWKS URL with the HTTPS endpoint serving the signing keys
for the tokens issued by your control plane. Compose passes it to the dataplane
through `CONTEXTFORGE_DATA_PLANE_JWKS_URL`. Production images do not contain the
testing-only `with_tools` helpers or need token-signing private keys.

Wait for `register_fast_time` to finish, then allow ~60s config propagation:

```bash
Expand All @@ -17,7 +23,8 @@ docker compose -f docker/docker-compose.yml logs -f register_fast_time
| Resource | URL |
| --- | --- |
| MCP endpoint | `http://localhost:8080/contextforge-rs/servers/{virtual_host_id}/mcp` |
| Bearer token | `GET http://localhost:8080/contextforge-rs/admin/tokens/admin@example.com` |
| Bearer token | Issue through the control plane; the production dataplane has no token helper. |
| Health | `http://localhost:8080/health` (also `/contextforge-rs/health`) |
| fast_time_server virtual host id | `b8e3f1a2c4d5e6f7a1b2c3d4e5f6a7b8` |

> **Critical**: `/contextforge-rs` prefix → ContextForge external dataplane.
Expand All @@ -28,15 +35,20 @@ Teardown: `make compose-down` (stops containers; volumes kept).

## cf-integration Conformance

Install the same harness version used by CI. Official conformance packages and their Node/npm
runtime are installed only inside Docker images.

```bash
cargo binstall cf-integration@0.1.0 --no-confirm
cargo binstall cf-integration@0.3.2 --no-confirm
make conformance
```

This runs the modern client and modern server eras through the committed
external-dataplane `HEAD`, including fixture-direct server comparison and the
scoped client suite. Use `make conformance-bless` to replace all selected
baselines transactionally after a fully successful run. Generated checkouts,
scoped client suite. It uses the production build without `with_tools`; the
harness owns test JWT signing, loopback JWKS, and Redis fixture publication.
Use `make conformance-bless` to
replace all selected baselines transactionally after a fully successful run. Generated checkouts,
results, reports, and logs stay under `.integration/`.

## Local Cargo Dev Workflow
Expand All @@ -54,17 +66,19 @@ docker compose -f docker/docker-compose-local.yaml ps redis gateway-one gateway-
| `gateway-one` | `http://127.0.0.1:5555/mcp` | MCP Rust SDK counter fixture. |
| `gateway-two` | `http://127.0.0.1:5556/mcp` | MCP Rust SDK conformance fixture. |

Run the binary with bootstrap helpers:
Run the binary with bootstrap helpers **for local testing only**. `with_tools`
enables unauthenticated token, JWKS, and config routes. It must never be enabled
in production, and it is not needed for `/contextforge-rs/health`.

```bash
cargo run -p contextforge-data-plane \
--features contextforge-data-plane-lib/with_tools \
--features with_tools \
--bin contextforge-data-plane -- \
--address 127.0.0.1:8001 \
--redis-address 127.0.0.1 \
--redis-port 6379 \
--redis-mode plain-text \
--token-verification-public-key assets/jwt.key.pub \
--jwks-url http://127.0.0.1:8001/contextforge-rs/admin/.well-known/jwks.json \
--token-verification-private-key assets/jwt.key \
--upstream-connection-mode plain-text-or-tls \
--number-of-cpus 4
Expand All @@ -76,8 +90,9 @@ The client-facing route is `http://127.0.0.1:8001/contextforge-rs/servers/{virtu

```bash
USER_ID=11111111-1111-1111-1111-111111111111
TENANT_ID=team_awesome
TOKEN=$(curl --silent --show-error \
--url "http://127.0.0.1:8001/contextforge-rs/admin/tokens/${USER_ID}?email=admin@example.com")
--url "http://127.0.0.1:8001/contextforge-rs/admin/tokens/${TENANT_ID}/${USER_ID}?email=admin@example.com")
```

### Seed runtime configuration
Expand Down
2 changes: 1 addition & 1 deletion _context/wiki/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ then follow only the links that are relevant.
| [mcp-capability-allocation.md](mcp-capability-allocation.md) | Tentative ContextForge 2.0 target topology, ownership, state model, Phase 1-4 roadmap, and Phase 3 flows |
| [failure-modes.md](failure-modes.md) | HTTP/MCP/routing/backend/plugin failure table — exact HTTP codes and JSON-RPC errors |
| [config.md](config.md) | Key CLI flags, JWT claims, UserConfig shape, plugin config, telemetry debugging, startup validation, local observability stack |
| [deployment.md](deployment.md) | External-dataplane deployment checklist, health endpoint caveat, nginx routing, TLS choices, session affinity, Redis availability, image pinning |
| [deployment.md](deployment.md) | External-dataplane production builds, health endpoint, deployment checklist, nginx routing, TLS choices, session affinity, Redis availability, image pinning |
| [security.md](security.md) | Trust boundaries among the control plane, built-in dataplane, and external dataplane; Origin/Host validation; transport security; secrets handling |
| [performance.md](performance.md) | Control-plane Locust load runs, benchmark settings, and built-in-dataplane baseline |
| [testing.md](testing.md) | Workspace checks, in-repo integration tests, full-stack harness lanes, settings, and control-plane baseline |
Expand Down
Loading