Standalone single-deploy Modal packages for each stack

.gitignore

@@ -0,0 +1,6 @@
+__pycache__/
+*.py[cod]
+.env
+license.jwt
+*.jwt
+.ruff_cache/

README.md

@@ -14,21 +14,30 @@ and, within each service, by **model**.
 
 - **Streaming** transcribes a live audio stream over a WebSocket connection. One
   stack serves multiple models; the client selects the model per session. See
-  [`streaming/README.md`](streaming/README.md).
+  [`streaming/docker/README.md`](streaming/docker/README.md).
 - **Sync** transcribes a complete file in a single HTTP request/response (audio
   ≤ 120 s by default). It is self-contained — a single GPU container plus the
-  license-and-usage-proxy, no load balancer. See [`sync/README.md`](sync/README.md).
+  license-and-usage-proxy, no load balancer. See [`sync/docker/README.md`](sync/docker/README.md).
 
-Each service directory is self-contained: its compose file(s), `.env.example`,
-example client, and `README.md` live together. Run commands from inside the
-service directory.
+Each stack is self-contained under `<service>/docker/`: its compose file(s),
+`.env.example`, example client, and `README.md` live together. Run compose
+commands from inside that directory.
+
+Each stack can also run on serverless GPUs instead of hardware you manage. Every
+stack is a self-contained Modal App deployed with a single `modal deploy` — see
+[`sync/modal/`](sync/modal/) and
+[`streaming/modal/`](streaming/modal/).
 
 ## Repository layout
 
 ```
 .
-├── streaming/   # WebSocket streaming ASR (Universal English/Multilingual, Universal-3.5 Pro)
-└── sync/        # Synchronous full-file HTTP transcription (Universal-3.5 Pro)
+├── streaming/
+│   ├── docker/  # compose stack (Universal English/Multilingual, Universal-3.5 Pro)
+│   └── modal/   # serverless-GPU packages
+└── sync/
+    ├── docker/  # compose stack (Universal-3.5 Pro)
+    └── modal/   # serverless-GPU package
 ```
 
 ## Prerequisites (all services)
@@ -60,9 +69,9 @@ aws ecr get-login-password --region us-west-2 \
 
 ### License file
 
-Place your AssemblyAI `license.jwt` in the directory of the service you are
-running (`streaming/` or `sync/`), or point the `LICENSE_FILE_PATH` environment
-variable in that service's compose file at your license file's location.
+Place your AssemblyAI `license.jwt` in the directory of the stack you are
+running (`streaming/docker/` or `sync/docker/`), or point the `LICENSE_FILE_PATH`
+environment variable in that stack's compose file at your license file's location.
 
 ## Shared component: license-and-usage-proxy
 
@@ -143,7 +152,7 @@ This release introduces the **Sync self-hosted service**
 model. It transcribes a complete audio file (≤ 120 s) in a single
 `POST /transcribe` request/response — a single GPU container plus the
 license-and-usage-proxy, no load balancer. It exposes `GET /readyz` (200 once
-the model is warm) for readiness probes. See [`sync/README.md`](sync/README.md).
+the model is warm) for readiness probes. See [`sync/docker/README.md`](sync/docker/README.md).
 
 #### Streaming — U3 Pro replaced by Universal-3.5 Pro (BREAKING)
 
@@ -165,9 +174,10 @@ upgrading from the v0.6.0 U3 Pro stack:
 
 #### Images
 
-`release-v1.0.0` is published for `self-hosted-streaming-api`,
-`self-hosted-streaming-license-and-usage-proxy`,
-`self-hosted-streaming-asr-universal-3-5-pro`, and `self-hosted-sync-asr-u3-pro`.
+`release-v1.0.1` is published for `self-hosted-streaming-api` (adds the
+peer-aborted-handshake logging fix) and `self-hosted-streaming-asr-universal-3-5-pro`;
+`release-v1.0.0` for `self-hosted-streaming-license-and-usage-proxy` and
+`self-hosted-sync-asr-u3-pro`.
 The English and Multilingual ASR images are unchanged since v0.6.0 — keep
 `STREAMING_ASR_ENGLISH_IMAGE` and `STREAMING_ASR_MULTILANG_IMAGE` at
 `release-v0.6.0` (see `streaming/.env.example`).

streaming/.env.example → streaming/docker/.env.example

@@ -1,5 +1,5 @@
 # Required for every streaming stack:
-STREAMING_API_IMAGE=344839248844.dkr.ecr.us-west-2.amazonaws.com/self-hosted-streaming-api:release-v1.0.0
+STREAMING_API_IMAGE=344839248844.dkr.ecr.us-west-2.amazonaws.com/self-hosted-streaming-api:release-v1.0.1
 LICENSE_AND_USAGE_PROXY_IMAGE=344839248844.dkr.ecr.us-west-2.amazonaws.com/self-hosted-streaming-license-and-usage-proxy:release-v1.0.0
 USAGE_TRACKING_API_KEY=<your_usage_tracking_api_key_here>
 # Required only when running the Universal stack (docker-compose.english-multilang.yml):

streaming/README.md → streaming/docker/README.md

@@ -5,7 +5,7 @@ Real-time transcription over a WebSocket connection. Run all commands from this
 
 > Prerequisites (license, Docker, GPU runtime, ECR auth) and the shared
 > license-and-usage-proxy (usage reporting, license status endpoint, proxy
-> production recommendations) are documented in the [top-level README](../README.md).
+> production recommendations) are documented in the [top-level README](../../README.md).
 
 ## Choosing a stack
 
@@ -24,7 +24,7 @@ To switch between stacks, run `docker compose -f <file> down` before starting th
 Both stacks include:
 - **streaming-api**: Gateway API service handling WebSocket connections.
 - **streaming-asr-lb**: nginx load balancer for ASR services with header-based routing.
-- **license-and-usage-proxy**: License validation and usage reporting (see [top-level README](../README.md#shared-component-license-and-usage-proxy)).
+- **license-and-usage-proxy**: License validation and usage reporting (see [top-level README](../../README.md#shared-component-license-and-usage-proxy)).
 
 ASR backends differ by stack:
 - Universal stack (`docker-compose.english-multilang.yml`): `streaming-asr-english` and `streaming-asr-multilang`.
@@ -64,7 +64,7 @@ to an available backend.
 
 ## Setup
 
-Complete the [shared prerequisites](../README.md#prerequisites-all-services)
+Complete the [shared prerequisites](../../README.md#prerequisites-all-services)
 (GPU runtime, ECR authentication, license file) first.
 
 Copy the env reference and set the image variables for the stack you plan to run:
@@ -188,7 +188,7 @@ python example_with_prerecorded_audio_file.py --help
 ### Usage reporting
 
 The license-and-usage-proxy's billing modes and behavior are documented in the
-[top-level README](../README.md#usage-reporting).
+[top-level README](../../README.md#usage-reporting).
 
 ## Monitoring & debugging
 
@@ -215,9 +215,14 @@ docker compose -f docker-compose.english-multilang.yml restart streaming-asr-mul
 docker compose -f docker-compose.universal-3-5-pro.yml restart streaming-asr-universal-3-5-pro
 ```
 
+## Deploying on Modal (serverless GPU)
+
+Both streaming stacks also run on Modal's serverless GPUs as self-contained,
+single-`modal deploy` Modal Apps. See [`../modal/`](../modal/).
+
 ## Production deployment recommendations
 
-See the [top-level README](../README.md#production-recommendations-license-and-usage-proxy)
+See the [top-level README](../../README.md#production-recommendations-license-and-usage-proxy)
 for the license-and-usage-proxy. Streaming-specific services follow.
 
 ### streaming-api service

streaming/modal/README.md

@@ -0,0 +1,114 @@
+# Streaming stacks on Modal (serverless GPU)
+
+Each streaming stack runs on [Modal](https://modal.com) as a self-contained
+Modal App: one `modal deploy` brings up every service and wires them together,
+with no dependency on any other deployment. Compose equivalents live in
+[`../docker/`](../docker/).
+
+| Stack | File | Servers |
+|---|---|---|
+| Universal-3.5 Pro | `modal_app_universal_3_5_pro.py` | `StreamingApi` (CPU), `Asr` (L40S), `LicenseProxy` (CPU) |
+| English + Multilingual | `modal_app_english_multilang.py` | `StreamingApi` (CPU), `Lb` (CPU nginx), `AsrEnglish` (L40S), `AsrMultilang` (L40S), `LicenseProxy` (CPU) |
+
+`StreamingApi` resolves its backend and proxy URLs from the same App at startup,
+so there is no manual wiring or two-phase deploy. The Universal-3.5 Pro stack
+serves one model and needs no router, so nginx is dropped. The
+English + Multilingual stack serves two models, so it keeps an nginx `Lb` that
+routes the `x-model-version` gRPC metadata (from the client's `speech_model`) to
+the matching backend, exactly as `streaming-asr-lb` does in compose.
+
+## Prerequisites and secrets
+
+Identical to the [sync stack](../../sync/modal/README.md#store-credentials-as-modal-secrets):
+create the `aai-ecr-credentials` and `aai-license` Modal secrets once; all three
+stacks share them.
+
+## Deploy
+
+```bash
+modal deploy modal_app_universal_3_5_pro.py     # or modal_app_english_multilang.py
+```
+
+Each GPU backend keeps one L40S warm (`min_containers=1`) and gates readiness on
+`grpc_health_probe`, so the first deploy takes a few minutes to warm the model;
+Modal then autoscales on concurrent sessions, with `target_concurrency` set to
+each stack's `MAX_OPEN_STREAMS` (Universal-3.5 Pro 32, English + Multilingual 48,
+matching compose). The endpoint URLs are printed, of the form
+`https://<workspace>--<app>-streamingapi.<region>.modal.direct`.
+
+## Verify
+
+The `streamingapi` endpoint is behind Modal proxy auth by default, so send a
+proxy-auth token (`--modal-key` / `--modal-secret`, or `MODAL_KEY` /
+`MODAL_SECRET`); a `licenseproxy` `/v1/status` check needs none. To probe
+without a token, deploy the endpoint with `AAI_REQUIRE_MODAL_AUTH=0`.
+
+```bash
+curl -fsS https://<workspace>--aai-streaming-u3pro-licenseproxy.<region>.modal.direct/v1/status
+
+# Stream with the bundled sample client (it forwards Modal proxy-auth headers;
+# the repo's example_with_prerecorded_audio_file.py does not, so it only works
+# against an AAI_REQUIRE_MODAL_AUTH=0 endpoint):
+python sample_streaming.py \
+  --endpoint wss://<workspace>--aai-streaming-u3pro-streamingapi.<region>.modal.direct \
+  --audio ../docker/example/example_audio_file.wav \
+  --speech-model universal-3-5-pro \
+  --modal-key "$MODAL_KEY" --modal-secret "$MODAL_SECRET"
+```
+
+For the English + Multilingual stack use `--speech-model universal-streaming-english`
+or `universal-streaming-multilingual`; the API maps these to the `en-default` /
+`ml-default` routing keys and the `Lb` sends each to its backend. Or use the
+[sample script](#sample-requests).
+
+## Authentication and security
+
+`StreamingApi` requires a Modal proxy-auth token by default
+(`unauthenticated=False`); Modal enforces it on the WebSocket upgrade, so a
+guessed URL alone gets `401`. Send the token as `Modal-Key` / `Modal-Secret`
+headers, or deploy with `AAI_REQUIRE_MODAL_AUTH=0` for a throwaway public test
+endpoint (any non-empty `Authorization` then connects, as behind your own
+gateway).
+
+The internal hops (`StreamingApi` → `Asr`/`Lb`, and → `LicenseProxy`) cross
+Modal's TLS edge, **not** a private bridge network as in compose: Modal has no
+private inter-container network by default, so these `.modal.direct` endpoints
+are public. The gRPC hop is encrypted — `h2_enabled` advertises ALPN h2 so the
+API's default-TLS gRPC client connects with `AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True`
+— but the backends and proxy are `unauthenticated=True`, because the API dials
+them server-side and cannot attach Modal auth headers. Their URLs are
+unguessable but reachable by anyone who learns them; a determined operator can
+close that gap by co-locating the API and ASR in one container (localhost hop)
+or by putting the backends on Modal's `i6pn` private network (an address
+handshake via `modal.Dict`, same region). Treat the shipped topology as suitable
+for evaluation, not untrusted public exposure of the backends.
+
+## Sample requests
+
+`sample_streaming.py` streams the audio at real time and prints turns live
+(partial `…`, finalized `✓`); `--speech-model` picks the model and `--load N`
+opens N concurrent sessions.
+
+```bash
+pip install websockets
+python sample_streaming.py \
+  --endpoint wss://<workspace>--aai-streaming-u3pro-streamingapi.<region>.modal.direct \
+  --audio ../docker/example/example_audio_file.wav \
+  --speech-model universal-3-5-pro
+```
+
+If the stack was deployed with the default proxy auth, pass `--modal-key` /
+`--modal-secret` (or set `MODAL_KEY` / `MODAL_SECRET`).
+
+## Cost and teardown
+
+Each GPU backend holds an L40S while up (Modal bills it), scaling to at most
+`max_containers` and down after `scaledown_window`. Tear a stack down when done:
+
+```bash
+modal app stop aai-streaming-u3pro                 # or aai-streaming-english-multilang
+```
+
+Audio is processed on Modal's multi-tenant cloud in the configured region
+(default `us-east`); pin `routing_region`/`compute_region` near your callers,
+and note the data-residency difference from a self-hosted deployment.