From 9aed1cf0f64b3eb5fbdc2ac117c37228716906d4 Mon Sep 17 00:00:00 2001 From: Aleksandar Mitov <140423361+aleks-mitov@users.noreply.github.com> Date: Wed, 26 Aug 2026 12:49:13 +0000 Subject: [PATCH 1/9] feat(modal): standalone single-deploy packages for each stack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Repackage the Modal deployment so each self-hosted stack is a self-contained Modal App deployed with one `modal deploy` — no start_asr step, no modal.Dict address handshake, no cross-package dependency. Uses Modal Servers (@app.server) with sibling URLs resolved via Server.from_name at startup. - sync/modal_app.py -> aai-sync-u3pro - streaming/modal_app_universal_3_5_pro.py -> aai-streaming-u3pro - streaming/modal_app_english_multilang.py -> aai-streaming-english-multilang Fixes carried over from the #11 review: autoscaling ASR Server instead of a 24h-capped Sandbox; no 1h WebSocket session ceiling; encrypted ASR hop via h2_enabled + AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE (encrypted_ports alone negotiates no ALPN); unauthenticated=False by default (Modal proxy auth); fate-shared vendor processes; env-overridable sync audio limits. Verified end to end on L40S (single deploy each): sync 152-word transcript, autoscaled to 181x realtime at concurrency 8; streaming u3pro real turns over the TLS h2 ASR hop; english+multilang routing both models through the nginx Lb. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1 --- .gitignore | 6 + README.md | 5 + streaming/README.md | 89 +++++++ streaming/modal_app_english_multilang.py | 296 +++++++++++++++++++++++ streaming/modal_app_universal_3_5_pro.py | 233 ++++++++++++++++++ sync/README.md | 103 ++++++++ sync/modal_app.py | 206 ++++++++++++++++ 7 files changed, 938 insertions(+) create mode 100644 .gitignore create mode 100644 streaming/modal_app_english_multilang.py create mode 100644 streaming/modal_app_universal_3_5_pro.py create mode 100644 sync/modal_app.py diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..654abe1 --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +__pycache__/ +*.py[cod] +.env +license.jwt +*.jwt +.ruff_cache/ diff --git a/README.md b/README.md index bc2fc59..f7057f3 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,11 @@ Each service directory is self-contained: its compose file(s), `.env.example`, example client, and `README.md` live together. Run commands from inside the service directory. +Each stack can also run on serverless GPUs instead of hardware you manage. Every +stack is a self-contained Modal App deployed with a single `modal deploy` — see +[sync on Modal](sync/README.md#deploying-on-modal-serverless-gpu) and +[streaming on Modal](streaming/README.md#deploying-on-modal-serverless-gpu). + ## Repository layout ``` diff --git a/streaming/README.md b/streaming/README.md index cbd0722..df56c1c 100644 --- a/streaming/README.md +++ b/streaming/README.md @@ -215,6 +215,95 @@ docker compose -f docker-compose.english-multilang.yml restart streaming-asr-mul docker compose -f docker-compose.universal-3-5-pro.yml restart streaming-asr-universal-3-5-pro ``` +## Deploying on Modal (serverless GPU) + +Each streaming stack runs on [Modal](https://modal.com) as a self-contained +Modal App: one `modal deploy` brings up every service and wires them together, +with no dependency on any other deployment. + +| Stack | File | Servers | +|---|---|---| +| Universal-3.5 Pro | `modal_app_universal_3_5_pro.py` | `StreamingApi` (CPU), `Asr` (L40S), `LicenseProxy` (CPU) | +| English + Multilingual | `modal_app_english_multilang.py` | `StreamingApi` (CPU), `Lb` (CPU nginx), `AsrEnglish` (L40S), `AsrMultilang` (L40S), `LicenseProxy` (CPU) | + +`StreamingApi` resolves its backend and proxy URLs from the same App at startup, +so there is no manual wiring or two-phase deploy. The Universal-3.5 Pro stack +serves one model and needs no router, so nginx is dropped. The +English + Multilingual stack serves two models, so it keeps an nginx `Lb` that +routes the `x-model-version` gRPC metadata (from the client's `speech_model`) to +the matching backend, exactly as `streaming-asr-lb` does in compose. + +### Prerequisites and secrets + +Identical to the [sync stack](../sync/README.md#deploying-on-modal-serverless-gpu): +create the `aai-ecr-credentials` and `aai-license` Modal secrets once; both +streaming stacks share them. + +### Deploy + +```bash +modal deploy modal_app_universal_3_5_pro.py # or modal_app_english_multilang.py +``` + +Each GPU backend keeps one L40S warm (`min_containers=1`) and gates readiness on +`grpc_health_probe`, so the first deploy takes a few minutes to warm the model; +Modal then autoscales on concurrent sessions (`target_concurrency=32`, matching +`MAX_OPEN_STREAMS`). The endpoint URLs are printed, of the form +`https://---streamingapi..modal.direct`. + +### Verify + +```bash +curl -fsS https://--aai-streaming-u3pro-licenseproxy..modal.direct/v1/status +curl -fsS https://--aai-streaming-u3pro-streamingapi..modal.direct/v3/ws/health + +# Stream with the bundled example client (see "Running the streaming example"), +# swapping the ws://localhost:8080 endpoint for the wss:// URL: +python example_with_prerecorded_audio_file.py \ + --audio-file example_audio_file.wav \ + --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ + --speech-model universal-3-5-pro +``` + +For the English + Multilingual stack use `--speech-model universal-streaming-english` +or `universal-streaming-multilingual`; the API maps these to the `en-default` / +`ml-default` routing keys and the `Lb` sends each to its backend. + +### Authentication and security + +`StreamingApi` requires a Modal proxy-auth token by default +(`unauthenticated=False`); Modal enforces it on the WebSocket upgrade, so a +guessed URL alone gets `401`. Send the token as `Modal-Key` / `Modal-Secret` +headers, or deploy with `AAI_REQUIRE_MODAL_AUTH=0` for a throwaway public test +endpoint (any non-empty `Authorization` then connects, as behind your own +gateway). + +The internal hops (`StreamingApi` → `Asr`/`Lb`, and → `LicenseProxy`) cross +Modal's TLS edge, **not** a private bridge network as in compose: Modal has no +private inter-container network by default, so these `.modal.direct` endpoints +are public. The gRPC hop is encrypted — `h2_enabled` advertises ALPN h2 so the +API's default-TLS gRPC client connects with `AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True` +— but the backends and proxy are `unauthenticated=True`, because the API dials +them server-side and cannot attach Modal auth headers. Their URLs are +unguessable but reachable by anyone who learns them; a determined operator can +close that gap by co-locating the API and ASR in one container (localhost hop) +or by putting the backends on Modal's `i6pn` private network (an address +handshake via `modal.Dict`, same region). Treat the shipped topology as suitable +for evaluation, not untrusted public exposure of the backends. + +### Cost and teardown + +Each GPU backend holds an L40S while up (Modal bills it), scaling to at most +`max_containers` and down after `scaledown_window`. Tear a stack down when done: + +```bash +modal app stop aai-streaming-u3pro # or aai-streaming-english-multilang +``` + +Audio is processed on Modal's multi-tenant cloud in the configured region +(default `us-east`); pin `routing_region`/`compute_region` near your callers, +and note the data-residency difference from a self-hosted deployment. + ## Production deployment recommendations See the [top-level README](../README.md#production-recommendations-license-and-usage-proxy) diff --git a/streaming/modal_app_english_multilang.py b/streaming/modal_app_english_multilang.py new file mode 100644 index 0000000..0ccb592 --- /dev/null +++ b/streaming/modal_app_english_multilang.py @@ -0,0 +1,296 @@ +"""Run the self-hosted streaming English + Multilingual stack on Modal, standalone. + +`modal deploy modal_app_english_multilang.py` brings up the whole stack in one +command. This stack serves TWO ASR models, so unlike the single-model +Universal-3.5 Pro stack it keeps compose's routing layer: clients pick a model +with speech_model ("en-default" or "ml-default"), the API forwards it as the +gRPC metadata x-model-version, and an nginx load balancer routes to the matching +backend. Five Modal Servers in one App: + + streaming_api (CPU) -> WebSocket front door, the public entrypoint + lb (CPU) -> nginx, routes x-model-version to the two backends + asr_english (L40S) -> English gRPC backend + asr_multilang (L40S) -> Multilingual gRPC backend + license_proxy (CPU) -> license-and-usage-proxy + +Every backend hop crosses Modal's TLS edge (h2_enabled advertises ALPN h2 for +gRPC); the API and nginx dial with TLS. See README "Security". + +Deploy: modal deploy modal_app_english_multilang.py +Tear down: modal app stop aai-streaming-english-multilang +""" + +import os +import signal +import subprocess + +import modal + +APP_NAME = "aai-streaming-english-multilang" +REGISTRY = "344839248844.dkr.ecr.us-west-2.amazonaws.com" +# The English/Multilingual images ship on their own release line, separate from +# the Universal-3.5 Pro images (see streaming/.env.example). +TAG = "release-v0.6.0" +PROXY_TAG = "release-v1.0.0" +ASR_GRPC_PORT = 50051 + +REQUIRE_MODAL_AUTH = os.environ.get("AAI_REQUIRE_MODAL_AUTH", "1") != "0" + +ENGLISH_BIN = "/opt/assemblyai/engineering/projects/realtime/asr_server/asr_server_bin" +MULTILANG_BIN = "/opt/assemblyai/engineering/projects/realtime/asr_server/ml_asr_server_bin" +API_BIN = "/opt/assemblyai/engineering/projects/realtime/api_v2/bin" +PROXY_BIN = "/opt/assemblyai/engineering/projects/realtime/license_and_usage_proxy/bin" + +LICENSE_PATH = "/var/aai_license.jwt" + +ecr_secret = modal.Secret.from_name("aai-ecr-credentials") +license_secret = modal.Secret.from_name("aai-license") + +app = modal.App(APP_NAME) + + +def _vendor_image(repo: str, tag: str) -> modal.Image: + """A Modal-runnable image from an AssemblyAI ECR image (see sync/modal_app.py).""" + return ( + modal.Image.from_aws_ecr( + f"{REGISTRY}/{repo}:{tag}", secret=ecr_secret, add_python="3.12" + ) + .entrypoint([]) + .pip_install(f"modal=={modal.__version__}") + ) + + +english_image = _vendor_image("self-hosted-streaming-asr-english", TAG) +multilang_image = _vendor_image("self-hosted-streaming-asr-multilang", TAG) +api_image = _vendor_image("self-hosted-streaming-api", PROXY_TAG) +proxy_image = _vendor_image("self-hosted-streaming-license-and-usage-proxy", PROXY_TAG) +# nginx routes gRPC by x-model-version; no ECR pull needed. A Debian base gives +# Modal a detectable interpreter plus its client, alongside nginx. +lb_image = ( + modal.Image.debian_slim(python_version="3.12") + .apt_install("nginx") + .pip_install(f"modal=={modal.__version__}") +) + + +def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: + """Start a binary and fate-share it with the container (see sync/modal_app.py).""" + proc = subprocess.Popen(argv, env={**os.environ, **env}) + + import threading + + def _reap() -> None: + proc.wait() + os._exit(proc.returncode or 1) + + threading.Thread(target=_reap, daemon=True).start() + return proc + + +def _wait_http_ok(url: str, timeout_s: int) -> None: + import time + import urllib.request + + deadline = time.monotonic() + timeout_s + last = "" + while time.monotonic() < deadline: + try: + with urllib.request.urlopen(url, timeout=2) as resp: + if resp.status == 200: + return + last = f"HTTP {resp.status}" + except Exception as exc: # noqa: BLE001 + last = repr(exc) + time.sleep(3) + raise RuntimeError(f"{url} not ready after {timeout_s}s (last: {last})") + + +def _asr_ready(port: int, timeout_s: int) -> None: + import time + + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + if subprocess.run( + ["grpc_health_probe", f"-addr=:{port}"], capture_output=True + ).returncode == 0: + return + time.sleep(5) + raise RuntimeError(f"ASR not serving on :{port} after {timeout_s}s") + + +_ASR_KW = dict( + gpu="L40S", + cpu=4, + memory=16384, + port=ASR_GRPC_PORT, + h2_enabled=True, + unauthenticated=True, # dialed server-side (via nginx) over gRPC; see README "Security" + target_concurrency=32, + min_containers=1, + max_containers=4, + buffer_containers=1, + scaledown_window=600, + startup_timeout=900, + exit_grace_period=600, +) +_ASR_ENV = { + "SERVER_PORT": str(ASR_GRPC_PORT), + "LOGGING_LEVEL": "INFO", + "USE_STRUCTURED_LOGGING": "False", + "MAX_OPEN_STREAMS": os.environ.get("MAX_OPEN_STREAMS", "32"), + "VLLM_USE_FLASHINFER_SAMPLER": "0", +} + + +@app.server(image=proxy_image, port=8080, unauthenticated=True, cpu=1, memory=2048, + min_containers=1, max_containers=1, startup_timeout=180, exit_grace_period=30, + secrets=[license_secret], + env={"HTTP_PORT": "8080", "LOGGING_LEVEL": "INFO", + "USE_STRUCTURED_LOGGING": "False", "LICENSE_FILE_PATH": LICENSE_PATH}) +class LicenseProxy: + @modal.enter() + def start(self) -> None: + token = os.environ.get("AAI_LICENSE_JWT") or os.environ["LICENSE_JWT"] + with open(LICENSE_PATH, "w") as fh: + fh.write(token.strip()) + self.proc = _launch([PROXY_BIN], {}) + _wait_http_ok("http://localhost:8080/health", 60) + + @modal.exit() + def stop(self) -> None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=25) + except subprocess.TimeoutExpired: + self.proc.kill() + + +@app.server(image=english_image, env=_ASR_ENV, **_ASR_KW) +class AsrEnglish: + @modal.enter() + def start(self) -> None: + self.proc = _launch([ENGLISH_BIN], {}) + _asr_ready(ASR_GRPC_PORT, 840) + + @modal.exit() + def stop(self) -> None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=570) + except subprocess.TimeoutExpired: + self.proc.kill() + + +@app.server(image=multilang_image, env=_ASR_ENV, **_ASR_KW) +class AsrMultilang: + @modal.enter() + def start(self) -> None: + self.proc = _launch([MULTILANG_BIN], {}) + _asr_ready(ASR_GRPC_PORT, 840) + + @modal.exit() + def stop(self) -> None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=570) + except subprocess.TimeoutExpired: + self.proc.kill() + + +# nginx config: route the gRPC metadata x-model-version to the matching backend +# over Modal's TLS edge (grpcs, ALPN h2). Backend hosts are resolved from the +# App at startup and substituted in. Public DNS resolves the .modal.direct +# hosts, so a public resolver is used. +_NGINX_CONF = """ +events {{ worker_connections 1024; }} +http {{ + access_log /dev/stdout; + error_log /dev/stderr info; + resolver 1.1.1.1 8.8.8.8 valid=30s; + map $http_x_model_version $asr_backend {{ + default {english}:443; + en-default {english}:443; + ml-default {multilang}:443; + }} + keepalive_timeout 10h; + # Plain HTTP/1.1 readiness port for the startup probe; the gRPC listener below + # is h2-only and cannot answer an HTTP/1.1 GET. + server {{ + listen 8081; + location = /health {{ access_log off; return 200 "OK\\n"; }} + }} + server {{ + listen 8080 http2; + client_max_body_size 0; + location / {{ + grpc_pass grpcs://$asr_backend; + grpc_ssl_server_name on; + grpc_connect_timeout 75s; + grpc_read_timeout 10h; + grpc_send_timeout 10h; + grpc_socket_keepalive on; + }} + }} +}} +""" + + +@app.server(image=lb_image, port=8080, h2_enabled=True, unauthenticated=True, + cpu=1, memory=1024, min_containers=1, max_containers=1, + startup_timeout=120, exit_grace_period=30) +class Lb: + @modal.enter() + def start(self) -> None: + english = modal.Server.from_name(APP_NAME, "AsrEnglish").get_url().split("://", 1)[1].rstrip("/") + multilang = modal.Server.from_name(APP_NAME, "AsrMultilang").get_url().split("://", 1)[1].rstrip("/") + with open("/etc/nginx/nginx.conf", "w") as fh: + fh.write(_NGINX_CONF.format(english=english, multilang=multilang)) + print(f"[startup] lb -> en={english} ml={multilang}", flush=True) + self.proc = _launch(["nginx", "-g", "daemon off;"], {}) + _wait_http_ok("http://localhost:8081/health", 30) + + @modal.exit() + def stop(self) -> None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=25) + except subprocess.TimeoutExpired: + self.proc.kill() + + +@app.server(image=api_image, port=8080, unauthenticated=not REQUIRE_MODAL_AUTH, + cpu=1, memory=2048, target_concurrency=32, min_containers=1, + max_containers=4, nonpreemptible=True, scaledown_window=600, + startup_timeout=600, exit_grace_period=600, secrets=[license_secret]) +class StreamingApi: + @modal.enter() + def start(self) -> None: + lb_host = ( + os.environ.get("ASR_ENDPOINT") + or modal.Server.from_name(APP_NAME, "Lb").get_url().split("://", 1)[1].rstrip("/") + ) + proxy_url = os.environ.get("PROXY_ENDPOINT") or modal.Server.from_name( + APP_NAME, "LicenseProxy" + ).get_url().rstrip("/") + print(f"[startup] LB={lb_host}:443 proxy={proxy_url} require_auth={REQUIRE_MODAL_AUTH}", flush=True) + + self.proc = _launch( + [API_BIN], + { + "AAI_WSS_PORT": "8080", + "AAI_LOG_LEVEL": "INFO", + "AAI_USE_STRUCTURED_LOGGING": "False", + "AAI_ASR_ENDPOINT": f"{lb_host}:443", + "AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE": "True", + "AAI_LICENSE_AND_USAGE_PROXY_ENDPOINT": proxy_url, + }, + ) + _wait_http_ok("http://localhost:8080/v3/ws/health", 120) + + @modal.exit() + def stop(self) -> None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=570) + except subprocess.TimeoutExpired: + self.proc.kill() diff --git a/streaming/modal_app_universal_3_5_pro.py b/streaming/modal_app_universal_3_5_pro.py new file mode 100644 index 0000000..f1f295b --- /dev/null +++ b/streaming/modal_app_universal_3_5_pro.py @@ -0,0 +1,233 @@ +"""Run the self-hosted streaming Universal-3.5 Pro stack on Modal, standalone. + +`modal deploy modal_app_universal_3_5_pro.py` brings up the whole stack in one +command; nothing depends on any other deployment. Compose's four services become +three Modal Servers in one App: + + streaming_api (CPU) -> WebSocket front door, the public entrypoint + asr (L40S) -> Universal-3.5 Pro gRPC backend + license_proxy (CPU) -> license-and-usage-proxy + +nginx (streaming-asr-lb) is dropped: it only routes X-Model-Version across +several ASR backends, and this stack serves one model. streaming_api resolves +the ASR and proxy URLs from the same App at startup, so there is no manual +wiring step. + +The API dials the ASR over Modal's TLS edge (h2_enabled advertises ALPN h2, so a +standard gRPC client connects) with AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True, +replacing compose's private bridge network. See README "Security". + +Deploy: modal deploy modal_app_universal_3_5_pro.py +Tear down: modal app stop aai-streaming-u3pro +""" + +import os +import signal +import subprocess + +import modal + +APP_NAME = "aai-streaming-u3pro" +REGISTRY = "344839248844.dkr.ecr.us-west-2.amazonaws.com" +TAG = "release-v1.0.0" +ASR_GRPC_PORT = 50051 + +# See sync/modal_app.py: the WebSocket API requires a Modal proxy-auth token by +# default. Set AAI_REQUIRE_MODAL_AUTH=0 for a throwaway test endpoint. +REQUIRE_MODAL_AUTH = os.environ.get("AAI_REQUIRE_MODAL_AUTH", "1") != "0" + +API_BIN = "/opt/assemblyai/engineering/projects/realtime/api_v2/bin" +ASR_BIN = "/opt/assemblyai/engineering/projects/realtime/asr_u3pro/self_hosted_bin" +PROXY_BIN = "/opt/assemblyai/engineering/projects/realtime/license_and_usage_proxy/bin" + +LICENSE_PATH = "/var/aai_license.jwt" + +ecr_secret = modal.Secret.from_name("aai-ecr-credentials") +license_secret = modal.Secret.from_name("aai-license") + +app = modal.App(APP_NAME) + + +def _vendor_image(repo: str) -> modal.Image: + """A Modal-runnable image from an AssemblyAI ECR image (see sync/modal_app.py).""" + return ( + modal.Image.from_aws_ecr( + f"{REGISTRY}/{repo}:{TAG}", secret=ecr_secret, add_python="3.12" + ) + .entrypoint([]) + .pip_install(f"modal=={modal.__version__}") + ) + + +asr_image = _vendor_image("self-hosted-streaming-asr-universal-3-5-pro") +api_image = _vendor_image("self-hosted-streaming-api") +proxy_image = _vendor_image("self-hosted-streaming-license-and-usage-proxy") + + +def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: + """Start a vendor binary and fate-share it with the container (see sync/modal_app.py).""" + proc = subprocess.Popen(argv, env={**os.environ, **env}) + + import threading + + def _reap() -> None: + proc.wait() + os._exit(proc.returncode or 1) + + threading.Thread(target=_reap, daemon=True).start() + return proc + + +def _wait_http_ok(url: str, timeout_s: int) -> None: + import time + import urllib.request + + deadline = time.monotonic() + timeout_s + last = "" + while time.monotonic() < deadline: + try: + with urllib.request.urlopen(url, timeout=2) as resp: + if resp.status == 200: + return + last = f"HTTP {resp.status}" + except Exception as exc: # noqa: BLE001 + last = repr(exc) + time.sleep(3) + raise RuntimeError(f"{url} not ready after {timeout_s}s (last: {last})") + + +@app.server( + image=proxy_image, + port=8080, + unauthenticated=True, # called server-side by streaming_api; see README "Authentication" + cpu=1, + memory=2048, + min_containers=1, + max_containers=1, + startup_timeout=180, + exit_grace_period=30, + secrets=[license_secret], + env={ + "HTTP_PORT": "8080", + "LOGGING_LEVEL": "INFO", + "USE_STRUCTURED_LOGGING": "False", + "LICENSE_FILE_PATH": LICENSE_PATH, + }, +) +class LicenseProxy: + @modal.enter() + def start(self) -> None: + token = os.environ.get("AAI_LICENSE_JWT") or os.environ["LICENSE_JWT"] + with open(LICENSE_PATH, "w") as fh: + fh.write(token.strip()) + self.proc = _launch([PROXY_BIN], {}) + _wait_http_ok("http://localhost:8080/health", 60) + + @modal.exit() + def stop(self) -> None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=25) + except subprocess.TimeoutExpired: + self.proc.kill() + + +@app.server( + image=asr_image, + gpu="L40S", + cpu=4, + memory=16384, + port=ASR_GRPC_PORT, + h2_enabled=True, # gRPC needs ALPN h2 across the TLS edge + # Called server-side by streaming_api over gRPC, which cannot attach Modal + # auth headers, so the endpoint is unauthenticated. See README "Security". + unauthenticated=True, + target_concurrency=32, # mirrors MAX_OPEN_STREAMS + min_containers=1, # ~5 min warm-up; never scale a realtime backend to zero + max_containers=4, + buffer_containers=1, # scale-up lead time is the warm-up, so keep a warm spare + scaledown_window=600, + startup_timeout=900, + exit_grace_period=600, # let in-flight streams drain instead of dying + env={ + "SERVER_PORT": str(ASR_GRPC_PORT), + "LOGGING_LEVEL": "INFO", + "USE_STRUCTURED_LOGGING": "False", + "MAX_OPEN_STREAMS": os.environ.get("MAX_OPEN_STREAMS", "32"), + "VLLM_USE_FLASHINFER_SAMPLER": "0", + }, +) +class Asr: + @modal.enter() + def start(self) -> None: + self.proc = _launch([ASR_BIN], {}) + # Gate readiness on the same probe compose uses, so Modal never routes a + # session to a cold engine. + import time + + deadline = time.monotonic() + 840 + while time.monotonic() < deadline: + if subprocess.run( + ["grpc_health_probe", f"-addr=:{ASR_GRPC_PORT}"], + capture_output=True, + ).returncode == 0: + return + time.sleep(5) + raise RuntimeError("ASR engine not serving after warm-up window") + + @modal.exit() + def stop(self) -> None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=570) + except subprocess.TimeoutExpired: + self.proc.kill() + + +@app.server( + image=api_image, + port=8080, + unauthenticated=not REQUIRE_MODAL_AUTH, + cpu=1, + memory=2048, + target_concurrency=32, # ~32 sessions per CPU container + min_containers=1, + max_containers=4, + nonpreemptible=True, # holds live WebSocket sessions + scaledown_window=600, + startup_timeout=600, + exit_grace_period=600, + secrets=[license_secret], +) +class StreamingApi: + @modal.enter() + def start(self) -> None: + asr_host = ( + os.environ.get("ASR_ENDPOINT") + or modal.Server.from_name(APP_NAME, "Asr").get_url().split("://", 1)[1].rstrip("/") + ) + proxy_url = os.environ.get("PROXY_ENDPOINT") or modal.Server.from_name( + APP_NAME, "LicenseProxy" + ).get_url().rstrip("/") + print(f"[startup] ASR={asr_host}:443 proxy={proxy_url} require_auth={REQUIRE_MODAL_AUTH}", flush=True) + + self.proc = _launch( + [API_BIN], + { + "AAI_WSS_PORT": "8080", + "AAI_LOG_LEVEL": "INFO", + "AAI_USE_STRUCTURED_LOGGING": "False", + "AAI_ASR_ENDPOINT": f"{asr_host}:443", + "AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE": "True", + "AAI_LICENSE_AND_USAGE_PROXY_ENDPOINT": proxy_url, + }, + ) + _wait_http_ok("http://localhost:8080/v3/ws/health", 120) + + @modal.exit() + def stop(self) -> None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=570) + except subprocess.TimeoutExpired: + self.proc.kill() diff --git a/sync/README.md b/sync/README.md index e5d9fca..c2cf27e 100644 --- a/sync/README.md +++ b/sync/README.md @@ -121,6 +121,109 @@ python transcribe_file.py # uses the bundled example_audio_fi python transcribe_file.py path/to/audio.wav # or your own 16-bit PCM WAV ``` +## Deploying on Modal (serverless GPU) + +`modal_app.py` runs this stack on [Modal](https://modal.com) instead of a GPU +box you manage. It is a self-contained Modal App: one `modal deploy` brings up +both services and wires them together, and nothing depends on another +deployment. Compose's two services become two Modal Servers: + +| Compose service | Modal Server | Hardware | +|---|---|---| +| `sync-api` | `SyncApi` | L40S GPU | +| `license-and-usage-proxy` | `LicenseProxy` | CPU | + +`SyncApi` resolves `LicenseProxy`'s URL from the same App at startup, so there +is no manual wiring or two-phase deploy. + +### Prerequisites + +```bash +pip install modal && modal setup # authenticate the Modal CLI +``` + +### Store credentials as Modal secrets + +Modal has no bind mounts, so the license travels as a secret and is written to +disk at container startup. + +```bash +# ECR pull credentials, used only when Modal builds (pulls) the image. +# Prefer a dedicated pull-only IAM principal over long-lived root/admin keys +# (ecr:GetAuthorizationToken + ecr:BatchGetImage / GetDownloadUrlForLayer / +# BatchCheckLayerAvailability on the AssemblyAI repositories). If you use SSO or +# assume-role session credentials, include AWS_SESSION_TOKEN; note they expire, +# so an image *rebuild* after expiry needs fresh values (redeploys of an +# already-built image do not). +modal secret create aai-ecr-credentials \ + AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... \ + AWS_REGION=us-west-2 + +# The license itself. Usage-billed licenses: add USAGE_TRACKING_API_KEY here +# too; it reaches the proxy automatically, no code change. +modal secret create aai-license LICENSE_JWT="$(cat license.jwt)" +``` + +### Deploy + +```bash +modal deploy modal_app.py +``` + +The first deploy pulls and converts the ~13.5 GB sync image (several minutes); +later deploys reuse the cached image and take seconds. Two endpoint URLs are +printed, of the form `https://--aai-sync-u3pro-..modal.direct`. + +### Verify + +```bash +curl -fsS https://--aai-sync-u3pro-licenseproxy..modal.direct/v1/status +# {"state":"Connected", ...} + +curl -sS -o /dev/null -w '%{http_code}\n' \ + https://--aai-sync-u3pro-syncapi..modal.direct/readyz +# 503 while the model is cold, 200 once warm (Modal's edge may answer 303 first) + +curl -F 'audio=@example/example_audio_file.wav;type=audio/wav' \ + -F 'config={"language_code":"en"};type=application/json' \ + -H 'Authorization: any-non-empty-value' \ + https://--aai-sync-u3pro-syncapi..modal.direct/transcribe +``` + +### Authentication + +`SyncApi` requires a Modal proxy-auth token by default (`unauthenticated=False`), +so a guessed URL alone cannot reach it. Mint a proxy-auth token in the Modal +dashboard and send it on every request as `Modal-Key` / `Modal-Secret` headers +(or `Authorization: Bearer .`). For a throwaway public test +endpoint, deploy with `AAI_REQUIRE_MODAL_AUTH=0` — it then accepts any non-empty +`Authorization` header, exactly like the compose stack behind your own gateway. +`LicenseProxy` is always `unauthenticated=True` because `SyncApi` calls it +server-side and cannot attach Modal headers; its URL is unguessable but public, +so treat it as such. + +### Configuration + +The audio limits (`MAX_AUDIO_DURATION_MS`, `MIN_AUDIO_DURATION_MS`, +`MAX_REQUEST_BYTES`, `INFERENCE_TIMEOUT_SECONDS`) are read from the environment +with the compose defaults as fallback, so you can override them by adding the +variable to the `aai-license` secret (or any Server env) — your value wins. + +### Cost and teardown + +`SyncApi` keeps one L40S warm (`min_containers=1`) so requests do not eat a cold +start; it autoscales up to `max_containers` under load and back down after +`scaledown_window`. Modal bills the GPU while it is up, so tear the app down when +you are done: + +```bash +modal app stop aai-sync-u3pro +``` + +Audio is processed on Modal's multi-tenant cloud in `routing_region`/`compute_region` +(default `us-east`); pin them near your callers, and note this is a different +data-residency posture than a stack you host yourself. + ## Production deployment recommendations See the [top-level README](../README.md#production-recommendations-license-and-usage-proxy) diff --git a/sync/modal_app.py b/sync/modal_app.py new file mode 100644 index 0000000..ac2ceb4 --- /dev/null +++ b/sync/modal_app.py @@ -0,0 +1,206 @@ +"""Run the self-hosted sync (full-file HTTP) stack on Modal as a standalone app. + +`modal deploy modal_app.py` brings up the whole stack in one command. Compose's +two services become two Modal Servers in one App, so nothing here depends on any +other deployment: + + license_proxy (CPU) -> license-and-usage-proxy + sync_api (L40S) -> sync-api, which resolves the proxy's URL at startup + +Deploy: modal deploy modal_app.py +Tear down: modal app stop aai-sync-u3pro + +Prerequisites (see README "Deploying on Modal"): + modal secret create aai-ecr-credentials AWS_ACCESS_KEY_ID=... \ + AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... AWS_REGION=us-west-2 + modal secret create aai-license LICENSE_JWT="$(cat license.jwt)" # +USAGE_TRACKING_API_KEY if usage-billed +""" + +import os +import signal +import subprocess + +import modal + +APP_NAME = "aai-sync-u3pro" +REGISTRY = "344839248844.dkr.ecr.us-west-2.amazonaws.com" +TAG = "release-v1.0.0" + +# The API's public endpoint requires a Modal proxy-auth token by default, so a +# guessed URL alone cannot reach it. Set to False for a throwaway test endpoint +# that accepts any non-empty Authorization header (see README "Authentication"). +REQUIRE_MODAL_AUTH = os.environ.get("AAI_REQUIRE_MODAL_AUTH", "1") != "0" + +# Vendor image ENTRYPOINTs, launched explicitly in each Server's @modal.enter. +# Modal prepends an image's ENTRYPOINT to its own runtime command, so both +# images clear it with .entrypoint([]); otherwise the vendor binary consumes +# Modal's arguments, starts with default env, and this code never runs. +SYNC_BIN = "/opt/assemblyai/engineering/projects/realtime/asr_sync_u3pro/self_hosted_bin" +PROXY_BIN = "/opt/assemblyai/engineering/projects/realtime/license_and_usage_proxy/bin" + +LICENSE_PATH = "/var/aai_license.jwt" + +ecr_secret = modal.Secret.from_name("aai-ecr-credentials") +license_secret = modal.Secret.from_name("aai-license") + +app = modal.App(APP_NAME) + + +def _vendor_image(repo: str) -> modal.Image: + """A Modal-runnable image from an AssemblyAI ECR image. + + Every vendor image needs the same three adjustments: clear the ENTRYPOINT, + inject an interpreter Modal can find (the images keep theirs inside Bazel + runfiles, invisible to Modal), and install the Modal client into it (the + runtime-mounted client deps do not land on these images' sys.path). + """ + return ( + modal.Image.from_aws_ecr( + f"{REGISTRY}/{repo}:{TAG}", secret=ecr_secret, add_python="3.12" + ) + .entrypoint([]) + .pip_install(f"modal=={modal.__version__}") + ) + + +proxy_image = _vendor_image("self-hosted-streaming-license-and-usage-proxy") +sync_image = _vendor_image("self-hosted-sync-asr-u3-pro") + + +def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: + """Start a vendor binary and fate-share it with the container. + + A bare Popen leaves the container 'up' if the binary later exits, so Modal + keeps routing to a process that is gone. The watcher exits the container on + the binary's death, turning a silent black hole into a normal replacement. + """ + proc = subprocess.Popen(argv, env={**os.environ, **env}) + + import threading + + def _reap() -> None: + proc.wait() + os._exit(proc.returncode or 1) + + threading.Thread(target=_reap, daemon=True).start() + return proc + + +def _wait_http_ok(url: str, timeout_s: int) -> None: + import time + import urllib.request + + deadline = time.monotonic() + timeout_s + last = "" + while time.monotonic() < deadline: + try: + with urllib.request.urlopen(url, timeout=2) as resp: + if resp.status == 200: + return + last = f"HTTP {resp.status}" + except Exception as exc: # noqa: BLE001 + last = repr(exc) + time.sleep(3) + raise RuntimeError(f"{url} not ready after {timeout_s}s (last: {last})") + + +@app.server( + image=proxy_image, + port=8080, + # Called server-side by sync_api, which cannot attach Modal auth headers to + # its request, so this endpoint must accept unauthenticated traffic. Its URL + # is unguessable but public; see README "Authentication". + unauthenticated=True, + cpu=1, + memory=2048, + min_containers=1, + max_containers=1, + startup_timeout=180, + exit_grace_period=30, + secrets=[license_secret], + env={ + "HTTP_PORT": "8080", + "LOGGING_LEVEL": "INFO", + "USE_STRUCTURED_LOGGING": "False", + "LICENSE_FILE_PATH": LICENSE_PATH, + }, +) +class LicenseProxy: + @modal.enter() + def start(self) -> None: + # Compose bind-mounts license.jwt; Modal has no bind mounts, so the JWT + # arrives as a secret and is written to disk at startup. Accept either + # key name so the same secret works across tooling. + token = os.environ.get("AAI_LICENSE_JWT") or os.environ["LICENSE_JWT"] + with open(LICENSE_PATH, "w") as fh: + fh.write(token.strip()) + # Usage-billed licenses: add USAGE_TRACKING_API_KEY to the aai-license + # secret and it reaches the proxy here through the environment. Nothing + # else to change. + self.proc = _launch([PROXY_BIN], {}) + _wait_http_ok("http://localhost:8080/health", 60) + + @modal.exit() + def stop(self) -> None: + # Graceful stop lets the proxy flush queued usage before exit. + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=25) + except subprocess.TimeoutExpired: + self.proc.kill() + + +@app.server( + image=sync_image, + gpu="L40S", + cpu=4, + memory=16384, + port=8080, + unauthenticated=not REQUIRE_MODAL_AUTH, + # Scale-out signal: concurrent in-flight /transcribe requests (GPU-bound). + # Start conservative and tune against bench/harness.py on your hardware. + target_concurrency=8, + min_containers=1, # ~2-4 min cold start; keep one warm (503 while cold) + max_containers=4, + scaledown_window=300, + startup_timeout=900, # weights load + CUDA-graph capture + exit_grace_period=60, # requests are short (INFERENCE_TIMEOUT_SECONDS below) + secrets=[license_secret], +) +class SyncApi: + @modal.enter() + def start(self) -> None: + proxy_url = os.environ.get("PROXY_ENDPOINT") or modal.Server.from_name( + APP_NAME, "LicenseProxy" + ).get_url().rstrip("/") + print(f"[startup] proxy={proxy_url} require_auth={REQUIRE_MODAL_AUTH}", flush=True) + + self.proc = _launch( + [SYNC_BIN], + { + "HTTP_PORT": "8080", + "AAI_ENV": "production", + "LOGGING_LEVEL": "INFO", + "USE_STRUCTURED_LOGGING": "False", + "GPU_MONITORING_ENABLED": "False", + "LICENSE_AND_USAGE_PROXY_ENDPOINT": proxy_url, + # Audio limits: customer-overridable via the aai-license secret + # (or any Server env). User value wins; the compose defaults are + # only the fallback. Raising MAX_AUDIO_DURATION_MS usually means + # raising MAX_REQUEST_BYTES and INFERENCE_TIMEOUT_SECONDS too. + "MAX_AUDIO_DURATION_MS": os.environ.get("MAX_AUDIO_DURATION_MS", "120000"), + "MIN_AUDIO_DURATION_MS": os.environ.get("MIN_AUDIO_DURATION_MS", "80"), + "MAX_REQUEST_BYTES": os.environ.get("MAX_REQUEST_BYTES", "41943040"), + "INFERENCE_TIMEOUT_SECONDS": os.environ.get("INFERENCE_TIMEOUT_SECONDS", "30"), + "VLLM_USE_FLASHINFER_SAMPLER": "0", + }, + ) + _wait_http_ok("http://localhost:8080/readyz", 840) + + @modal.exit() + def stop(self) -> None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=50) + except subprocess.TimeoutExpired: + self.proc.kill() From e8e53fbce90d1b8f79ff4ad54dddab8e6dec6773 Mon Sep 17 00:00:00 2001 From: Aleksandar Mitov <140423361+aleks-mitov@users.noreply.github.com> Date: Wed, 26 Aug 2026 14:12:49 +0000 Subject: [PATCH 2/9] docs(samples): sample request scripts for each Modal stack sample_sync.py (POST /transcribe), sample_streaming.py (live realtime turns, model selection for the u3pro and english/multilang stacks), and a README. Both support --concurrency / --load for a quick input-load sweep and Modal proxy-auth headers. Verified live against all three deployed stacks. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1 --- samples/README.md | 59 +++++++++++++++ samples/sample_streaming.py | 145 ++++++++++++++++++++++++++++++++++++ samples/sample_sync.py | 102 +++++++++++++++++++++++++ 3 files changed, 306 insertions(+) create mode 100644 samples/README.md create mode 100644 samples/sample_streaming.py create mode 100644 samples/sample_sync.py diff --git a/samples/README.md b/samples/README.md new file mode 100644 index 0000000..a18f577 --- /dev/null +++ b/samples/README.md @@ -0,0 +1,59 @@ +# Sample requests + +Small standalone scripts to send a sample input to a deployed stack and watch +the output in real time. One file per stack shape. + +```bash +python -m venv venv && source venv/bin/activate +pip install requests websockets +``` + +Endpoints are the URLs `modal deploy` printed, of the form +`https://---..modal.direct` (`wss://` for streaming). + +## Sync + +```bash +python sample_sync.py \ + --endpoint https://--aai-sync-u3pro-syncapi..modal.direct \ + --audio ../sync/example/example_audio_file.wav +``` + +Prints the transcript, server-side time, and word count. `--concurrency N` fires +N requests at once and reports aggregate throughput. + +## Streaming (Universal-3.5 Pro) + +```bash +python sample_streaming.py \ + --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ + --audio ../streaming/example/example_audio_file.wav \ + --speech-model universal-3-5-pro +``` + +Streams the audio at real time; partial turns update in place (`…`) and finalize +(`✓`) exactly as a live microphone would. + +## Streaming (English + Multilingual) + +Same script, pick the model — the API routes it to the matching backend: + +```bash +python sample_streaming.py \ + --endpoint wss://--aai-streaming-english-multilang-streamingapi..modal.direct \ + --audio ../streaming/example/example_audio_file.wav \ + --speech-model universal-streaming-english # or universal-streaming-multilingual +``` + +## Load + +- `sample_sync.py --concurrency N` — N simultaneous transcription requests. +- `sample_streaming.py --load N` — N simultaneous realtime sessions (each prints + a summary line); `--speed 2` sends faster than real time to pack a sweep. + +## Authentication + +If a stack was deployed with the default Modal proxy auth +(`unauthenticated=False`), pass `--modal-key` / `--modal-secret` (or set +`MODAL_KEY` / `MODAL_SECRET`) — mint the token in the Modal dashboard. A test +endpoint deployed with `AAI_REQUIRE_MODAL_AUTH=0` needs no credentials. diff --git a/samples/sample_streaming.py b/samples/sample_streaming.py new file mode 100644 index 0000000..162df63 --- /dev/null +++ b/samples/sample_streaming.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +"""Stream a sample WAV to a deployed **streaming** stack and print turns live. + + pip install websockets + # Universal-3.5 Pro stack: + python sample_streaming.py \ + --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ + --audio ../streaming/example/example_audio_file.wav \ + --speech-model universal-3-5-pro + + # English + Multilingual stack (pick the model): + python sample_streaming.py --endpoint wss://--aai-streaming-english-multilang-streamingapi..modal.direct \ + --audio ../streaming/example/example_audio_file.wav --speech-model universal-streaming-english + # ... or --speech-model universal-streaming-multilingual + +Audio is sent at real time by default so you watch partial turns update and +finalize, exactly as a live microphone would. Use --speed 2 to send twice as +fast, or --load N to open N sessions at once and print each one's summary. + +If the API was deployed with Modal proxy auth (the shipped default), pass +--modal-key / --modal-secret (or set MODAL_KEY / MODAL_SECRET). For a test +endpoint deployed with AAI_REQUIRE_MODAL_AUTH=0 they are not needed. +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import time +import wave +from concurrent.futures import ThreadPoolExecutor, as_completed +from urllib.parse import urlencode + + +def load_pcm16_mono(path: str) -> tuple[bytes, int]: + with wave.open(path, "rb") as wav: + if wav.getsampwidth() != 2 or wav.getcomptype() != "NONE": + raise SystemExit(f"{path}: must be uncompressed 16-bit PCM WAV") + if wav.getnchannels() != 1: + raise SystemExit(f"{path}: must be mono (1 channel)") + rate = wav.getframerate() + return wav.readframes(wav.getnframes()), rate + + +def stream_once(args, pcm: bytes, rate: int, live: bool) -> dict: + from websockets.sync.client import connect + + params = {"sample_rate": rate, "format_turns": "true"} + if args.speech_model: + params["speech_model"] = args.speech_model + url = f"{args.endpoint.rstrip('/')}?{urlencode(params)}" + + headers = {"Authorization": "sample"} + key = args.modal_key or os.environ.get("MODAL_KEY") + secret = args.modal_secret or os.environ.get("MODAL_SECRET") + if key and secret: + headers["Modal-Key"], headers["Modal-Secret"] = key, secret + + frame = int(rate * 0.05) * 2 # 50 ms of 16-bit mono + chunks = [pcm[i : i + frame] for i in range(0, len(pcm), frame)] + + start = time.perf_counter() + first_turn: float | None = None + finals: list[str] = [] + + with connect(url, additional_headers=headers, open_timeout=args.open_timeout, max_size=None) as ws: + def writer(): + for chunk in chunks: + time.sleep(0.05 / args.speed) + ws.send(chunk) + ws.send('{"type": "Terminate"}') + + with ThreadPoolExecutor(max_workers=1) as pool: + wf = pool.submit(writer) + for message in ws: + data = json.loads(message) + if data.get("type") == "Turn": + words = data.get("words") or [] + if not words: + continue + if first_turn is None: + first_turn = time.perf_counter() - start + text = " ".join(w["text"] for w in words) + if live: + # Update the current line for partials; commit a line on finalize. + if data.get("end_of_turn"): + print(f"\r ✓ {text}", flush=True) + finals.append(text) + else: + print(f"\r … {text[:110]}", end="", flush=True) + elif data.get("type") == "Termination": + break + wf.result() + + return { + "elapsed_s": time.perf_counter() - start, + "first_turn_s": first_turn, + "turns": len(finals), + "text": " ".join(finals), + } + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--endpoint", required=True, help="wss://...streamingapi... URL") + ap.add_argument("--audio", required=True, help="16-bit PCM mono WAV") + ap.add_argument("--speech-model", help="universal-3-5-pro | universal-streaming-english | universal-streaming-multilingual") + ap.add_argument("--speed", type=float, default=1.0, help="send rate vs realtime (2 = twice as fast)") + ap.add_argument("--load", type=int, default=1, help="open N concurrent sessions") + ap.add_argument("--open-timeout", type=float, default=300.0, help="WS handshake wait (cold starts are slow)") + ap.add_argument("--modal-key") + ap.add_argument("--modal-secret") + args = ap.parse_args() + + pcm, rate = load_pcm16_mono(args.audio) + dur = len(pcm) / 2 / rate + print(f"{args.endpoint}\n audio {dur:.1f}s @ {rate} Hz | model={args.speech_model or '(default)'} | " + f"speed={args.speed}x | sessions={args.load}\n") + + if args.load == 1: + r = stream_once(args, pcm, rate, live=True) + print(f"\n{r['turns']} final turns | first turn {r['first_turn_s']:.2f}s | wall {r['elapsed_s']:.1f}s") + return 0 + + # Load mode: run N sessions at once, print a summary line per session. + started = time.perf_counter() + ok = 0 + with ThreadPoolExecutor(max_workers=args.load) as pool: + futs = {pool.submit(stream_once, args, pcm, rate, False): i for i in range(args.load)} + for fut in as_completed(futs): + i = futs[fut] + try: + r = fut.result() + ok += 1 + print(f" session {i:>2}: ok | {r['turns']} turns | first turn {r['first_turn_s']:.2f}s") + except Exception as exc: # noqa: BLE001 + print(f" session {i:>2}: FAIL {exc}") + print(f"\n{ok}/{args.load} sessions ok | wall {time.perf_counter() - started:.1f}s") + return 0 if ok == args.load else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/samples/sample_sync.py b/samples/sample_sync.py new file mode 100644 index 0000000..e215be3 --- /dev/null +++ b/samples/sample_sync.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Send a sample request to a deployed **sync** stack and print the transcript. + + pip install requests + python sample_sync.py --endpoint https://--aai-sync-u3pro-syncapi..modal.direct \ + --audio ../sync/example/example_audio_file.wav + +Load test (fire N in parallel, watch throughput): + python sample_sync.py --endpoint https://... --audio a.wav --concurrency 8 + +If the API was deployed with Modal proxy auth (the shipped default), pass +--modal-key / --modal-secret (or set MODAL_KEY / MODAL_SECRET); for a test +endpoint deployed with AAI_REQUIRE_MODAL_AUTH=0 they are not needed. +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import time +from concurrent.futures import ThreadPoolExecutor, as_completed + +import requests + + +def _auth_headers(args: argparse.Namespace) -> dict[str, str]: + # Any non-empty Authorization satisfies the self-hosted API; Modal proxy + # auth, if enabled, rides its own headers. + headers = {"Authorization": "sample"} + key = args.modal_key or os.environ.get("MODAL_KEY") + secret = args.modal_secret or os.environ.get("MODAL_SECRET") + if key and secret: + headers["Modal-Key"] = key + headers["Modal-Secret"] = secret + return headers + + +def one_request(args: argparse.Namespace, audio: bytes, headers: dict[str, str]) -> dict: + start = time.perf_counter() + resp = requests.post( + f"{args.endpoint.rstrip('/')}/transcribe", + files={"audio": ("audio.wav", audio, "audio/wav")}, + data={"config": json.dumps({"language_code": args.language})}, + headers=headers, + timeout=args.timeout, + ) + elapsed = time.perf_counter() - start + resp.raise_for_status() + body = resp.json() + return { + "elapsed_s": elapsed, + "server_ms": body.get("request_time_ms"), + "audio_ms": body.get("audio_duration_ms"), + "words": len(body.get("words", [])), + "text": body.get("text", ""), + } + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--endpoint", required=True, help="https://...syncapi... URL") + ap.add_argument("--audio", required=True, help="16-bit PCM WAV file") + ap.add_argument("--language", default="en") + ap.add_argument("--concurrency", type=int, default=1, help="fire N requests at once") + ap.add_argument("--timeout", type=float, default=300.0) + ap.add_argument("--modal-key") + ap.add_argument("--modal-secret") + args = ap.parse_args() + + with open(args.audio, "rb") as fh: + audio = fh.read() + headers = _auth_headers(args) + print(f"POST {args.endpoint.rstrip('/')}/transcribe x{args.concurrency}\n") + + started = time.perf_counter() + results, failures = [], 0 + with ThreadPoolExecutor(max_workers=args.concurrency) as pool: + futs = [pool.submit(one_request, args, audio, headers) for _ in range(args.concurrency)] + for i, fut in enumerate(as_completed(futs), 1): + try: + r = fut.result() + results.append(r) + print(f"[{i}/{args.concurrency}] ok {r['elapsed_s']:.2f}s wall | " + f"server {r['server_ms']}ms | {r['words']} words") + except Exception as exc: # noqa: BLE001 + failures += 1 + print(f"[{i}/{args.concurrency}] FAIL {exc}") + wall = time.perf_counter() - started + + if results: + print("\n--- sample transcript ---") + print(results[0]["text"][:600]) + audio_s = (results[0]["audio_ms"] or 0) / 1000 + xrt = (len(results) * audio_s / wall) if wall else 0 + print(f"\n{len(results)} ok, {failures} failed | wall {wall:.2f}s | ~{xrt:.1f}x realtime aggregate") + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) From 62753f23a35801a4d91c94b0cd125f0115f7caf7 Mon Sep 17 00:00:00 2001 From: Aleksandar Mitov <140423361+aleks-mitov@users.noreply.github.com> Date: Wed, 26 Aug 2026 14:39:58 +0000 Subject: [PATCH 3/9] fix(samples): count final turns in streaming --load mode The turn counter was appended only inside the live-printing branch, so --load sessions always reported 0 turns even though turns arrived (first-turn latency was correct). Track end_of_turn regardless of live; gate only printing. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1 --- samples/sample_streaming.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/samples/sample_streaming.py b/samples/sample_streaming.py index 162df63..8c3d996 100644 --- a/samples/sample_streaming.py +++ b/samples/sample_streaming.py @@ -83,13 +83,13 @@ def writer(): if first_turn is None: first_turn = time.perf_counter() - start text = " ".join(w["text"] for w in words) - if live: - # Update the current line for partials; commit a line on finalize. - if data.get("end_of_turn"): + if data.get("end_of_turn"): + finals.append(text) + if live: print(f"\r ✓ {text}", flush=True) - finals.append(text) - else: - print(f"\r … {text[:110]}", end="", flush=True) + elif live: + # Update the current line in place while the turn forms. + print(f"\r … {text[:110]}", end="", flush=True) elif data.get("type") == "Termination": break wf.result() From 04ac2464ea5076dc7ba0906116dfac022e107087 Mon Sep 17 00:00:00 2001 From: Aleksandar Mitov <140423361+aleks-mitov@users.noreply.github.com> Date: Wed, 26 Aug 2026 18:15:03 +0000 Subject: [PATCH 4/9] refactor(modal): move stacks into dedicated sync_modal_stack/ and streaming_modal_stack/ Consolidate each stack's Modal artifacts into a self-contained top-level dir: sync_modal_stack/ modal_app.py, sample_sync.py, README.md streaming_modal_stack/ modal_app_universal_3_5_pro.py, modal_app_english_multilang.py, sample_streaming.py, README.md Each new dir carries its own deploy/verify/auth/teardown README. The compose READMEs (sync/, streaming/) keep a one-line pointer instead of the full Modal section, and the root README points at the two new dirs. No code behavior changes; comment cross-references updated to the new paths. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1 --- README.md | 4 +- samples/README.md | 59 --------- streaming/README.md | 88 +------------ streaming_modal_stack/README.md | 106 +++++++++++++++ .../modal_app_english_multilang.py | 4 +- .../modal_app_universal_3_5_pro.py | 6 +- .../sample_streaming.py | 0 sync/README.md | 102 +-------------- sync_modal_stack/README.md | 122 ++++++++++++++++++ {sync => sync_modal_stack}/modal_app.py | 0 {samples => sync_modal_stack}/sample_sync.py | 0 11 files changed, 239 insertions(+), 252 deletions(-) delete mode 100644 samples/README.md create mode 100644 streaming_modal_stack/README.md rename {streaming => streaming_modal_stack}/modal_app_english_multilang.py (99%) rename {streaming => streaming_modal_stack}/modal_app_universal_3_5_pro.py (97%) rename {samples => streaming_modal_stack}/sample_streaming.py (100%) create mode 100644 sync_modal_stack/README.md rename {sync => sync_modal_stack}/modal_app.py (100%) rename {samples => sync_modal_stack}/sample_sync.py (100%) diff --git a/README.md b/README.md index f7057f3..f89eda3 100644 --- a/README.md +++ b/README.md @@ -25,8 +25,8 @@ service directory. Each stack can also run on serverless GPUs instead of hardware you manage. Every stack is a self-contained Modal App deployed with a single `modal deploy` — see -[sync on Modal](sync/README.md#deploying-on-modal-serverless-gpu) and -[streaming on Modal](streaming/README.md#deploying-on-modal-serverless-gpu). +[`sync_modal_stack/`](sync_modal_stack/) and +[`streaming_modal_stack/`](streaming_modal_stack/). ## Repository layout diff --git a/samples/README.md b/samples/README.md deleted file mode 100644 index a18f577..0000000 --- a/samples/README.md +++ /dev/null @@ -1,59 +0,0 @@ -# Sample requests - -Small standalone scripts to send a sample input to a deployed stack and watch -the output in real time. One file per stack shape. - -```bash -python -m venv venv && source venv/bin/activate -pip install requests websockets -``` - -Endpoints are the URLs `modal deploy` printed, of the form -`https://---..modal.direct` (`wss://` for streaming). - -## Sync - -```bash -python sample_sync.py \ - --endpoint https://--aai-sync-u3pro-syncapi..modal.direct \ - --audio ../sync/example/example_audio_file.wav -``` - -Prints the transcript, server-side time, and word count. `--concurrency N` fires -N requests at once and reports aggregate throughput. - -## Streaming (Universal-3.5 Pro) - -```bash -python sample_streaming.py \ - --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ - --audio ../streaming/example/example_audio_file.wav \ - --speech-model universal-3-5-pro -``` - -Streams the audio at real time; partial turns update in place (`…`) and finalize -(`✓`) exactly as a live microphone would. - -## Streaming (English + Multilingual) - -Same script, pick the model — the API routes it to the matching backend: - -```bash -python sample_streaming.py \ - --endpoint wss://--aai-streaming-english-multilang-streamingapi..modal.direct \ - --audio ../streaming/example/example_audio_file.wav \ - --speech-model universal-streaming-english # or universal-streaming-multilingual -``` - -## Load - -- `sample_sync.py --concurrency N` — N simultaneous transcription requests. -- `sample_streaming.py --load N` — N simultaneous realtime sessions (each prints - a summary line); `--speed 2` sends faster than real time to pack a sweep. - -## Authentication - -If a stack was deployed with the default Modal proxy auth -(`unauthenticated=False`), pass `--modal-key` / `--modal-secret` (or set -`MODAL_KEY` / `MODAL_SECRET`) — mint the token in the Modal dashboard. A test -endpoint deployed with `AAI_REQUIRE_MODAL_AUTH=0` needs no credentials. diff --git a/streaming/README.md b/streaming/README.md index df56c1c..9dc2c9f 100644 --- a/streaming/README.md +++ b/streaming/README.md @@ -217,92 +217,8 @@ docker compose -f docker-compose.universal-3-5-pro.yml restart streaming-asr-uni ## Deploying on Modal (serverless GPU) -Each streaming stack runs on [Modal](https://modal.com) as a self-contained -Modal App: one `modal deploy` brings up every service and wires them together, -with no dependency on any other deployment. - -| Stack | File | Servers | -|---|---|---| -| Universal-3.5 Pro | `modal_app_universal_3_5_pro.py` | `StreamingApi` (CPU), `Asr` (L40S), `LicenseProxy` (CPU) | -| English + Multilingual | `modal_app_english_multilang.py` | `StreamingApi` (CPU), `Lb` (CPU nginx), `AsrEnglish` (L40S), `AsrMultilang` (L40S), `LicenseProxy` (CPU) | - -`StreamingApi` resolves its backend and proxy URLs from the same App at startup, -so there is no manual wiring or two-phase deploy. The Universal-3.5 Pro stack -serves one model and needs no router, so nginx is dropped. The -English + Multilingual stack serves two models, so it keeps an nginx `Lb` that -routes the `x-model-version` gRPC metadata (from the client's `speech_model`) to -the matching backend, exactly as `streaming-asr-lb` does in compose. - -### Prerequisites and secrets - -Identical to the [sync stack](../sync/README.md#deploying-on-modal-serverless-gpu): -create the `aai-ecr-credentials` and `aai-license` Modal secrets once; both -streaming stacks share them. - -### Deploy - -```bash -modal deploy modal_app_universal_3_5_pro.py # or modal_app_english_multilang.py -``` - -Each GPU backend keeps one L40S warm (`min_containers=1`) and gates readiness on -`grpc_health_probe`, so the first deploy takes a few minutes to warm the model; -Modal then autoscales on concurrent sessions (`target_concurrency=32`, matching -`MAX_OPEN_STREAMS`). The endpoint URLs are printed, of the form -`https://---streamingapi..modal.direct`. - -### Verify - -```bash -curl -fsS https://--aai-streaming-u3pro-licenseproxy..modal.direct/v1/status -curl -fsS https://--aai-streaming-u3pro-streamingapi..modal.direct/v3/ws/health - -# Stream with the bundled example client (see "Running the streaming example"), -# swapping the ws://localhost:8080 endpoint for the wss:// URL: -python example_with_prerecorded_audio_file.py \ - --audio-file example_audio_file.wav \ - --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ - --speech-model universal-3-5-pro -``` - -For the English + Multilingual stack use `--speech-model universal-streaming-english` -or `universal-streaming-multilingual`; the API maps these to the `en-default` / -`ml-default` routing keys and the `Lb` sends each to its backend. - -### Authentication and security - -`StreamingApi` requires a Modal proxy-auth token by default -(`unauthenticated=False`); Modal enforces it on the WebSocket upgrade, so a -guessed URL alone gets `401`. Send the token as `Modal-Key` / `Modal-Secret` -headers, or deploy with `AAI_REQUIRE_MODAL_AUTH=0` for a throwaway public test -endpoint (any non-empty `Authorization` then connects, as behind your own -gateway). - -The internal hops (`StreamingApi` → `Asr`/`Lb`, and → `LicenseProxy`) cross -Modal's TLS edge, **not** a private bridge network as in compose: Modal has no -private inter-container network by default, so these `.modal.direct` endpoints -are public. The gRPC hop is encrypted — `h2_enabled` advertises ALPN h2 so the -API's default-TLS gRPC client connects with `AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True` -— but the backends and proxy are `unauthenticated=True`, because the API dials -them server-side and cannot attach Modal auth headers. Their URLs are -unguessable but reachable by anyone who learns them; a determined operator can -close that gap by co-locating the API and ASR in one container (localhost hop) -or by putting the backends on Modal's `i6pn` private network (an address -handshake via `modal.Dict`, same region). Treat the shipped topology as suitable -for evaluation, not untrusted public exposure of the backends. - -### Cost and teardown - -Each GPU backend holds an L40S while up (Modal bills it), scaling to at most -`max_containers` and down after `scaledown_window`. Tear a stack down when done: - -```bash -modal app stop aai-streaming-u3pro # or aai-streaming-english-multilang -``` - -Audio is processed on Modal's multi-tenant cloud in the configured region -(default `us-east`); pin `routing_region`/`compute_region` near your callers, -and note the data-residency difference from a self-hosted deployment. +Both streaming stacks also run on Modal's serverless GPUs as self-contained, +single-`modal deploy` Modal Apps. See [`../streaming_modal_stack/`](../streaming_modal_stack/). ## Production deployment recommendations diff --git a/streaming_modal_stack/README.md b/streaming_modal_stack/README.md new file mode 100644 index 0000000..4873dad --- /dev/null +++ b/streaming_modal_stack/README.md @@ -0,0 +1,106 @@ +# Streaming stacks on Modal (serverless GPU) + +Each streaming stack runs on [Modal](https://modal.com) as a self-contained +Modal App: one `modal deploy` brings up every service and wires them together, +with no dependency on any other deployment. Compose equivalents live in +[`../streaming/`](../streaming/). + +| Stack | File | Servers | +|---|---|---| +| Universal-3.5 Pro | `modal_app_universal_3_5_pro.py` | `StreamingApi` (CPU), `Asr` (L40S), `LicenseProxy` (CPU) | +| English + Multilingual | `modal_app_english_multilang.py` | `StreamingApi` (CPU), `Lb` (CPU nginx), `AsrEnglish` (L40S), `AsrMultilang` (L40S), `LicenseProxy` (CPU) | + +`StreamingApi` resolves its backend and proxy URLs from the same App at startup, +so there is no manual wiring or two-phase deploy. The Universal-3.5 Pro stack +serves one model and needs no router, so nginx is dropped. The +English + Multilingual stack serves two models, so it keeps an nginx `Lb` that +routes the `x-model-version` gRPC metadata (from the client's `speech_model`) to +the matching backend, exactly as `streaming-asr-lb` does in compose. + +## Prerequisites and secrets + +Identical to the [sync stack](../sync_modal_stack/README.md#store-credentials-as-modal-secrets): +create the `aai-ecr-credentials` and `aai-license` Modal secrets once; all three +stacks share them. + +## Deploy + +```bash +modal deploy modal_app_universal_3_5_pro.py # or modal_app_english_multilang.py +``` + +Each GPU backend keeps one L40S warm (`min_containers=1`) and gates readiness on +`grpc_health_probe`, so the first deploy takes a few minutes to warm the model; +Modal then autoscales on concurrent sessions (`target_concurrency=32`, matching +`MAX_OPEN_STREAMS`). The endpoint URLs are printed, of the form +`https://---streamingapi..modal.direct`. + +## Verify + +```bash +curl -fsS https://--aai-streaming-u3pro-licenseproxy..modal.direct/v1/status +curl -fsS https://--aai-streaming-u3pro-streamingapi..modal.direct/v3/ws/health + +# Stream with the bundled example client, swapping ws://localhost:8080 for the wss:// URL: +python ../streaming/example/example_with_prerecorded_audio_file.py \ + --audio-file ../streaming/example/example_audio_file.wav \ + --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ + --speech-model universal-3-5-pro +``` + +For the English + Multilingual stack use `--speech-model universal-streaming-english` +or `universal-streaming-multilingual`; the API maps these to the `en-default` / +`ml-default` routing keys and the `Lb` sends each to its backend. Or use the +[sample script](#sample-requests). + +## Authentication and security + +`StreamingApi` requires a Modal proxy-auth token by default +(`unauthenticated=False`); Modal enforces it on the WebSocket upgrade, so a +guessed URL alone gets `401`. Send the token as `Modal-Key` / `Modal-Secret` +headers, or deploy with `AAI_REQUIRE_MODAL_AUTH=0` for a throwaway public test +endpoint (any non-empty `Authorization` then connects, as behind your own +gateway). + +The internal hops (`StreamingApi` → `Asr`/`Lb`, and → `LicenseProxy`) cross +Modal's TLS edge, **not** a private bridge network as in compose: Modal has no +private inter-container network by default, so these `.modal.direct` endpoints +are public. The gRPC hop is encrypted — `h2_enabled` advertises ALPN h2 so the +API's default-TLS gRPC client connects with `AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True` +— but the backends and proxy are `unauthenticated=True`, because the API dials +them server-side and cannot attach Modal auth headers. Their URLs are +unguessable but reachable by anyone who learns them; a determined operator can +close that gap by co-locating the API and ASR in one container (localhost hop) +or by putting the backends on Modal's `i6pn` private network (an address +handshake via `modal.Dict`, same region). Treat the shipped topology as suitable +for evaluation, not untrusted public exposure of the backends. + +## Sample requests + +`sample_streaming.py` streams the audio at real time and prints turns live +(partial `…`, finalized `✓`); `--speech-model` picks the model and `--load N` +opens N concurrent sessions. + +```bash +pip install websockets +python sample_streaming.py \ + --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ + --audio ../streaming/example/example_audio_file.wav \ + --speech-model universal-3-5-pro +``` + +If the stack was deployed with the default proxy auth, pass `--modal-key` / +`--modal-secret` (or set `MODAL_KEY` / `MODAL_SECRET`). + +## Cost and teardown + +Each GPU backend holds an L40S while up (Modal bills it), scaling to at most +`max_containers` and down after `scaledown_window`. Tear a stack down when done: + +```bash +modal app stop aai-streaming-u3pro # or aai-streaming-english-multilang +``` + +Audio is processed on Modal's multi-tenant cloud in the configured region +(default `us-east`); pin `routing_region`/`compute_region` near your callers, +and note the data-residency difference from a self-hosted deployment. diff --git a/streaming/modal_app_english_multilang.py b/streaming_modal_stack/modal_app_english_multilang.py similarity index 99% rename from streaming/modal_app_english_multilang.py rename to streaming_modal_stack/modal_app_english_multilang.py index 0ccb592..c6ed911 100644 --- a/streaming/modal_app_english_multilang.py +++ b/streaming_modal_stack/modal_app_english_multilang.py @@ -50,7 +50,7 @@ def _vendor_image(repo: str, tag: str) -> modal.Image: - """A Modal-runnable image from an AssemblyAI ECR image (see sync/modal_app.py).""" + """A Modal-runnable image from an AssemblyAI ECR image (see sync_modal_stack/modal_app.py).""" return ( modal.Image.from_aws_ecr( f"{REGISTRY}/{repo}:{tag}", secret=ecr_secret, add_python="3.12" @@ -74,7 +74,7 @@ def _vendor_image(repo: str, tag: str) -> modal.Image: def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: - """Start a binary and fate-share it with the container (see sync/modal_app.py).""" + """Start a binary and fate-share it with the container (see sync_modal_stack/modal_app.py).""" proc = subprocess.Popen(argv, env={**os.environ, **env}) import threading diff --git a/streaming/modal_app_universal_3_5_pro.py b/streaming_modal_stack/modal_app_universal_3_5_pro.py similarity index 97% rename from streaming/modal_app_universal_3_5_pro.py rename to streaming_modal_stack/modal_app_universal_3_5_pro.py index f1f295b..8fed94e 100644 --- a/streaming/modal_app_universal_3_5_pro.py +++ b/streaming_modal_stack/modal_app_universal_3_5_pro.py @@ -32,7 +32,7 @@ TAG = "release-v1.0.0" ASR_GRPC_PORT = 50051 -# See sync/modal_app.py: the WebSocket API requires a Modal proxy-auth token by +# See sync_modal_stack/modal_app.py: the WebSocket API requires a Modal proxy-auth token by # default. Set AAI_REQUIRE_MODAL_AUTH=0 for a throwaway test endpoint. REQUIRE_MODAL_AUTH = os.environ.get("AAI_REQUIRE_MODAL_AUTH", "1") != "0" @@ -49,7 +49,7 @@ def _vendor_image(repo: str) -> modal.Image: - """A Modal-runnable image from an AssemblyAI ECR image (see sync/modal_app.py).""" + """A Modal-runnable image from an AssemblyAI ECR image (see sync_modal_stack/modal_app.py).""" return ( modal.Image.from_aws_ecr( f"{REGISTRY}/{repo}:{TAG}", secret=ecr_secret, add_python="3.12" @@ -65,7 +65,7 @@ def _vendor_image(repo: str) -> modal.Image: def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: - """Start a vendor binary and fate-share it with the container (see sync/modal_app.py).""" + """Start a vendor binary and fate-share it with the container (see sync_modal_stack/modal_app.py).""" proc = subprocess.Popen(argv, env={**os.environ, **env}) import threading diff --git a/samples/sample_streaming.py b/streaming_modal_stack/sample_streaming.py similarity index 100% rename from samples/sample_streaming.py rename to streaming_modal_stack/sample_streaming.py diff --git a/sync/README.md b/sync/README.md index c2cf27e..84aeaa9 100644 --- a/sync/README.md +++ b/sync/README.md @@ -123,106 +123,8 @@ python transcribe_file.py path/to/audio.wav # or your own 16-bit PCM WAV ## Deploying on Modal (serverless GPU) -`modal_app.py` runs this stack on [Modal](https://modal.com) instead of a GPU -box you manage. It is a self-contained Modal App: one `modal deploy` brings up -both services and wires them together, and nothing depends on another -deployment. Compose's two services become two Modal Servers: - -| Compose service | Modal Server | Hardware | -|---|---|---| -| `sync-api` | `SyncApi` | L40S GPU | -| `license-and-usage-proxy` | `LicenseProxy` | CPU | - -`SyncApi` resolves `LicenseProxy`'s URL from the same App at startup, so there -is no manual wiring or two-phase deploy. - -### Prerequisites - -```bash -pip install modal && modal setup # authenticate the Modal CLI -``` - -### Store credentials as Modal secrets - -Modal has no bind mounts, so the license travels as a secret and is written to -disk at container startup. - -```bash -# ECR pull credentials, used only when Modal builds (pulls) the image. -# Prefer a dedicated pull-only IAM principal over long-lived root/admin keys -# (ecr:GetAuthorizationToken + ecr:BatchGetImage / GetDownloadUrlForLayer / -# BatchCheckLayerAvailability on the AssemblyAI repositories). If you use SSO or -# assume-role session credentials, include AWS_SESSION_TOKEN; note they expire, -# so an image *rebuild* after expiry needs fresh values (redeploys of an -# already-built image do not). -modal secret create aai-ecr-credentials \ - AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... \ - AWS_REGION=us-west-2 - -# The license itself. Usage-billed licenses: add USAGE_TRACKING_API_KEY here -# too; it reaches the proxy automatically, no code change. -modal secret create aai-license LICENSE_JWT="$(cat license.jwt)" -``` - -### Deploy - -```bash -modal deploy modal_app.py -``` - -The first deploy pulls and converts the ~13.5 GB sync image (several minutes); -later deploys reuse the cached image and take seconds. Two endpoint URLs are -printed, of the form `https://--aai-sync-u3pro-..modal.direct`. - -### Verify - -```bash -curl -fsS https://--aai-sync-u3pro-licenseproxy..modal.direct/v1/status -# {"state":"Connected", ...} - -curl -sS -o /dev/null -w '%{http_code}\n' \ - https://--aai-sync-u3pro-syncapi..modal.direct/readyz -# 503 while the model is cold, 200 once warm (Modal's edge may answer 303 first) - -curl -F 'audio=@example/example_audio_file.wav;type=audio/wav' \ - -F 'config={"language_code":"en"};type=application/json' \ - -H 'Authorization: any-non-empty-value' \ - https://--aai-sync-u3pro-syncapi..modal.direct/transcribe -``` - -### Authentication - -`SyncApi` requires a Modal proxy-auth token by default (`unauthenticated=False`), -so a guessed URL alone cannot reach it. Mint a proxy-auth token in the Modal -dashboard and send it on every request as `Modal-Key` / `Modal-Secret` headers -(or `Authorization: Bearer .`). For a throwaway public test -endpoint, deploy with `AAI_REQUIRE_MODAL_AUTH=0` — it then accepts any non-empty -`Authorization` header, exactly like the compose stack behind your own gateway. -`LicenseProxy` is always `unauthenticated=True` because `SyncApi` calls it -server-side and cannot attach Modal headers; its URL is unguessable but public, -so treat it as such. - -### Configuration - -The audio limits (`MAX_AUDIO_DURATION_MS`, `MIN_AUDIO_DURATION_MS`, -`MAX_REQUEST_BYTES`, `INFERENCE_TIMEOUT_SECONDS`) are read from the environment -with the compose defaults as fallback, so you can override them by adding the -variable to the `aai-license` secret (or any Server env) — your value wins. - -### Cost and teardown - -`SyncApi` keeps one L40S warm (`min_containers=1`) so requests do not eat a cold -start; it autoscales up to `max_containers` under load and back down after -`scaledown_window`. Modal bills the GPU while it is up, so tear the app down when -you are done: - -```bash -modal app stop aai-sync-u3pro -``` - -Audio is processed on Modal's multi-tenant cloud in `routing_region`/`compute_region` -(default `us-east`); pin them near your callers, and note this is a different -data-residency posture than a stack you host yourself. +This stack also runs on Modal's serverless GPUs as a self-contained, +single-`modal deploy` Modal App. See [`../sync_modal_stack/`](../sync_modal_stack/). ## Production deployment recommendations diff --git a/sync_modal_stack/README.md b/sync_modal_stack/README.md new file mode 100644 index 0000000..fc584f2 --- /dev/null +++ b/sync_modal_stack/README.md @@ -0,0 +1,122 @@ +# Sync stack on Modal (serverless GPU) + +`modal_app.py` runs the self-hosted **sync** (full-file HTTP) stack on +[Modal](https://modal.com) instead of a GPU box you manage. It is a +self-contained Modal App: one `modal deploy` brings up both services and wires +them together, and nothing depends on another deployment. Compose's two services +(see [`../sync/`](../sync/)) become two Modal Servers: + +| Compose service | Modal Server | Hardware | +|---|---|---| +| `sync-api` | `SyncApi` | L40S GPU | +| `license-and-usage-proxy` | `LicenseProxy` | CPU | + +`SyncApi` resolves `LicenseProxy`'s URL from the same App at startup, so there +is no manual wiring or two-phase deploy. + +## Prerequisites + +```bash +pip install modal && modal setup # authenticate the Modal CLI +``` + +## Store credentials as Modal secrets + +Modal has no bind mounts, so the license travels as a secret and is written to +disk at container startup. + +```bash +# ECR pull credentials, used only when Modal builds (pulls) the image. +# Prefer a dedicated pull-only IAM principal over long-lived root/admin keys +# (ecr:GetAuthorizationToken + ecr:BatchGetImage / GetDownloadUrlForLayer / +# BatchCheckLayerAvailability on the AssemblyAI repositories). If you use SSO or +# assume-role session credentials, include AWS_SESSION_TOKEN; note they expire, +# so an image *rebuild* after expiry needs fresh values (redeploys of an +# already-built image do not). +modal secret create aai-ecr-credentials \ + AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... \ + AWS_REGION=us-west-2 + +# The license itself. Usage-billed licenses: add USAGE_TRACKING_API_KEY here +# too; it reaches the proxy automatically, no code change. +modal secret create aai-license LICENSE_JWT="$(cat license.jwt)" +``` + +Both streaming stacks share these same two secrets. + +## Deploy + +```bash +modal deploy modal_app.py +``` + +The first deploy pulls and converts the ~13.5 GB sync image (several minutes); +later deploys reuse the cached image and take seconds. Two endpoint URLs are +printed, of the form `https://--aai-sync-u3pro-..modal.direct`. + +## Verify + +```bash +curl -fsS https://--aai-sync-u3pro-licenseproxy..modal.direct/v1/status +# {"state":"Connected", ...} + +curl -sS -o /dev/null -w '%{http_code}\n' \ + https://--aai-sync-u3pro-syncapi..modal.direct/readyz +# 503 while the model is cold, 200 once warm (Modal's edge may answer 303 first) + +curl -F 'audio=@../sync/example/example_audio_file.wav;type=audio/wav' \ + -F 'config={"language_code":"en"};type=application/json' \ + -H 'Authorization: any-non-empty-value' \ + https://--aai-sync-u3pro-syncapi..modal.direct/transcribe +``` + +Or use the [sample script](#sample-requests). + +## Authentication + +`SyncApi` requires a Modal proxy-auth token by default (`unauthenticated=False`), +so a guessed URL alone cannot reach it. Mint a proxy-auth token in the Modal +dashboard and send it on every request as `Modal-Key` / `Modal-Secret` headers +(or `Authorization: Bearer .`). For a throwaway public test +endpoint, deploy with `AAI_REQUIRE_MODAL_AUTH=0` — it then accepts any non-empty +`Authorization` header, exactly like the compose stack behind your own gateway. +`LicenseProxy` is always `unauthenticated=True` because `SyncApi` calls it +server-side and cannot attach Modal headers; its URL is unguessable but public, +so treat it as such. + +## Configuration + +The audio limits (`MAX_AUDIO_DURATION_MS`, `MIN_AUDIO_DURATION_MS`, +`MAX_REQUEST_BYTES`, `INFERENCE_TIMEOUT_SECONDS`) are read from the environment +with the compose defaults as fallback, so you can override them by adding the +variable to the `aai-license` secret (or any Server env) — your value wins. + +## Sample requests + +`sample_sync.py` sends a request and prints the transcript, server-side time, +and word count; `--concurrency N` fires N at once for a quick load check. + +```bash +pip install requests +python sample_sync.py \ + --endpoint https://--aai-sync-u3pro-syncapi..modal.direct \ + --audio ../sync/example/example_audio_file.wav +``` + +If the stack was deployed with the default proxy auth, pass `--modal-key` / +`--modal-secret` (or set `MODAL_KEY` / `MODAL_SECRET`). + +## Cost and teardown + +`SyncApi` keeps one L40S warm (`min_containers=1`) so requests do not eat a cold +start; it autoscales up to `max_containers` under load and back down after +`scaledown_window`. Modal bills the GPU while it is up, so tear the app down when +you are done: + +```bash +modal app stop aai-sync-u3pro +``` + +Audio is processed on Modal's multi-tenant cloud in `routing_region`/`compute_region` +(default `us-east`); pin them near your callers, and note this is a different +data-residency posture than a stack you host yourself. diff --git a/sync/modal_app.py b/sync_modal_stack/modal_app.py similarity index 100% rename from sync/modal_app.py rename to sync_modal_stack/modal_app.py diff --git a/samples/sample_sync.py b/sync_modal_stack/sample_sync.py similarity index 100% rename from samples/sample_sync.py rename to sync_modal_stack/sample_sync.py From ac3dcbf651ff474a8449d251ca970157fe191cad Mon Sep 17 00:00:00 2001 From: Aleksandar Mitov <140423361+aleks-mitov@users.noreply.github.com> Date: Wed, 26 Aug 2026 18:30:07 +0000 Subject: [PATCH 5/9] feat(modal): pin u3pro streaming-api and ASR to release-v1.0.1 The self-hosted-streaming-api v1.0.1 image carries the handshake-logging fix (DeepLearning #19523: peer-aborted WebSocket handshakes log at WARNING, not ERROR). Bump streaming-api and self-hosted-streaming-asr-universal-3-5-pro to release-v1.0.1 in the u3pro stack via per-image tags; the license-and-usage-proxy has no v1.0.1 and stays on v1.0.0. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1 --- .../modal_app_universal_3_5_pro.py | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/streaming_modal_stack/modal_app_universal_3_5_pro.py b/streaming_modal_stack/modal_app_universal_3_5_pro.py index 8fed94e..9ab56d9 100644 --- a/streaming_modal_stack/modal_app_universal_3_5_pro.py +++ b/streaming_modal_stack/modal_app_universal_3_5_pro.py @@ -29,7 +29,12 @@ APP_NAME = "aai-streaming-u3pro" REGISTRY = "344839248844.dkr.ecr.us-west-2.amazonaws.com" -TAG = "release-v1.0.0" +# streaming-api and the u3-5-pro ASR ship on release-v1.0.1 (the API image +# carries the WARNING-not-ERROR handshake-logging fix, DeepLearning #19523); +# the license-and-usage-proxy has no v1.0.1 and stays on v1.0.0. +API_TAG = "release-v1.0.1" +ASR_TAG = "release-v1.0.1" +PROXY_TAG = "release-v1.0.0" ASR_GRPC_PORT = 50051 # See sync_modal_stack/modal_app.py: the WebSocket API requires a Modal proxy-auth token by @@ -48,20 +53,20 @@ app = modal.App(APP_NAME) -def _vendor_image(repo: str) -> modal.Image: +def _vendor_image(repo: str, tag: str) -> modal.Image: """A Modal-runnable image from an AssemblyAI ECR image (see sync_modal_stack/modal_app.py).""" return ( modal.Image.from_aws_ecr( - f"{REGISTRY}/{repo}:{TAG}", secret=ecr_secret, add_python="3.12" + f"{REGISTRY}/{repo}:{tag}", secret=ecr_secret, add_python="3.12" ) .entrypoint([]) .pip_install(f"modal=={modal.__version__}") ) -asr_image = _vendor_image("self-hosted-streaming-asr-universal-3-5-pro") -api_image = _vendor_image("self-hosted-streaming-api") -proxy_image = _vendor_image("self-hosted-streaming-license-and-usage-proxy") +asr_image = _vendor_image("self-hosted-streaming-asr-universal-3-5-pro", ASR_TAG) +api_image = _vendor_image("self-hosted-streaming-api", API_TAG) +proxy_image = _vendor_image("self-hosted-streaming-license-and-usage-proxy", PROXY_TAG) def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: From a508d69337f6432f09ffdbfab4f5851cce1c87ff Mon Sep 17 00:00:00 2001 From: Aleksandar Mitov <140423361+aleks-mitov@users.noreply.github.com> Date: Wed, 26 Aug 2026 19:12:54 +0000 Subject: [PATCH 6/9] fix(modal): address review findings + streaming-api v1.0.1 everywhere Adversarial review of PR #12 (8 findings). Fixes: - [major] english/multilang ASR MAX_OPEN_STREAMS + target_concurrency 32 -> 48 (compose parity; avoids a ~50% GPU over-provision at scale). - [major] streaming README Verify pointed at the bundled example client, which cannot send Modal proxy-auth headers and 401s on the default deploy; point it at sample_streaming.py with --modal-key/--modal-secret and note the caveat. - [minor] sync/streaming README Verify curls now send Modal-Key/Modal-Secret (they 401'd on the default proxy-auth deploy). - [minor] nginx Lb -> ASR hop now verifies the backend cert (grpc_ssl_verify on + ca-certificates trusted store), not just encrypts. - [minor] fate-share reaper no longer reports a clean shutdown as a crash: a module-level _stopping event, set by @modal.exit stop(), distinguishes an intentional teardown from an unexpected vendor exit (all three apps). - [minor] sample_streaming.py no longer crashes formatting first_turn_s when a session yields no word-bearing turns (prints n/a). - [minor/question] english/multilang now pins streaming-api via its own API_TAG. streaming-api bumped to release-v1.0.1 everywhere it is referenced (u3pro + english/multilang Modal stacks, streaming/.env.example, root README) so both streaming stacks carry the handshake-logging fix. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1 --- README.md | 7 +- bench/harness.py | 345 ++++++++++++++++++ streaming/.env.example | 2 +- streaming_modal_stack/README.md | 22 +- .../modal_app_english_multilang.py | 48 ++- .../modal_app_universal_3_5_pro.py | 18 +- streaming_modal_stack/sample_streaming.py | 8 +- sync_modal_stack/README.md | 10 +- sync_modal_stack/modal_app.py | 17 +- 9 files changed, 444 insertions(+), 33 deletions(-) create mode 100644 bench/harness.py diff --git a/README.md b/README.md index f89eda3..27b4821 100644 --- a/README.md +++ b/README.md @@ -170,9 +170,10 @@ upgrading from the v0.6.0 U3 Pro stack: #### Images -`release-v1.0.0` is published for `self-hosted-streaming-api`, -`self-hosted-streaming-license-and-usage-proxy`, -`self-hosted-streaming-asr-universal-3-5-pro`, and `self-hosted-sync-asr-u3-pro`. +`release-v1.0.1` is published for `self-hosted-streaming-api` (adds the +peer-aborted-handshake logging fix) and `self-hosted-streaming-asr-universal-3-5-pro`; +`release-v1.0.0` for `self-hosted-streaming-license-and-usage-proxy` and +`self-hosted-sync-asr-u3-pro`. The English and Multilingual ASR images are unchanged since v0.6.0 — keep `STREAMING_ASR_ENGLISH_IMAGE` and `STREAMING_ASR_MULTILANG_IMAGE` at `release-v0.6.0` (see `streaming/.env.example`). diff --git a/bench/harness.py b/bench/harness.py new file mode 100644 index 0000000..ea7dbb8 --- /dev/null +++ b/bench/harness.py @@ -0,0 +1,345 @@ +#!/usr/bin/env python3 +"""Load-test harness for the self-hosted stacks, local or on Modal. + +Sends real audio at a chosen concurrency, verifies transcripts actually come +back, and reports latency and throughput. Use --ramp to sweep concurrency and +find the level a deployment sustains before it degrades. + + # correctness check, one request + python harness.py sync --endpoint https://host --audio ../sync/example/example_audio_file.wav + + # concurrency sweep + python harness.py sync --endpoint https://host --audio a.wav --ramp 1,2,4,8,16 + python harness.py streaming --endpoint wss://host --audio a.wav --ramp 1,4,16,32,48 +""" + +from __future__ import annotations + +import argparse +import json +import sys +import time +import wave +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass, field +from typing import Callable +from urllib.parse import urlencode + +# Words expected in the bundled sample; used only as a sanity check that the +# transcript is real output rather than an empty 200. +DEFAULT_EXPECT = "assemblyai" + + +@dataclass +class Result: + ok: bool + seconds: float + detail: str = "" + text: str = "" + extra: dict = field(default_factory=dict) + + +def load_pcm16(path: str, max_seconds: float | None) -> tuple[bytes, int, float]: + """Read a 16-bit PCM WAV, optionally truncated, returning raw frames.""" + with wave.open(path, "rb") as wav: + if wav.getsampwidth() != 2 or wav.getcomptype() != "NONE": + raise SystemExit(f"{path}: must be uncompressed 16-bit PCM WAV") + rate = wav.getframerate() + frames = wav.getnframes() + if max_seconds: + frames = min(frames, int(rate * max_seconds)) + return wav.readframes(frames), rate, frames / rate + + +def wav_bytes(pcm: bytes, rate: int, channels: int = 1) -> bytes: + """Re-wrap raw PCM as a WAV container for the sync API's multipart upload.""" + import io + + buf = io.BytesIO() + with wave.open(buf, "wb") as out: + out.setnchannels(channels) + out.setsampwidth(2) + out.setframerate(rate) + out.writeframes(pcm) + return buf.getvalue() + + +# -------------------------------------------------------------------------- +# sync: one HTTP POST per request +# -------------------------------------------------------------------------- +def sync_once(endpoint: str, audio: bytes, expect: str) -> Result: + import requests + + start = time.perf_counter() + try: + resp = requests.post( + f"{endpoint.rstrip('/')}/transcribe", + files={"audio": ("audio.wav", audio, "audio/wav")}, + data={"config": json.dumps({"language_code": "en"})}, + headers={"Authorization": "harness"}, + timeout=300, + ) + except Exception as exc: # network/timeout + return Result(False, time.perf_counter() - start, detail=repr(exc)) + elapsed = time.perf_counter() - start + + if resp.status_code != 200: + return Result( + False, elapsed, detail=f"HTTP {resp.status_code}: {resp.text[:120]}" + ) + body = resp.json() + text = body.get("text", "") + if expect and expect.lower() not in text.lower(): + return Result( + False, elapsed, detail=f"transcript missing {expect!r}", text=text + ) + return Result( + True, + elapsed, + text=text, + extra={ + "server_ms": body.get("request_time_ms"), + "audio_ms": body.get("audio_duration_ms"), + "words": len(body.get("words", [])), + }, + ) + + +# -------------------------------------------------------------------------- +# streaming: one WebSocket session per request +# -------------------------------------------------------------------------- +def streaming_once( + endpoint: str, + pcm: bytes, + rate: int, + expect: str, + speech_model: str | None, + speed: float, + open_timeout: float, +) -> Result: + from websockets.sync.client import connect + + params = {"sample_rate": rate, "format_turns": "true"} + if speech_model: + params["speech_model"] = speech_model + url = f"{endpoint.rstrip('/')}?{urlencode(params)}" + + # 50 ms of audio per frame, the granularity the example client uses. + frame = int(rate * 0.05) * 2 + chunks = [pcm[i : i + frame] for i in range(0, len(pcm), frame)] + + start = time.perf_counter() + first_turn: float | None = None + turns = 0 + final_text: list[str] = [] + + try: + # Generous open timeout: a cold Modal container can take minutes to + # accept the upgrade, and the default 10s reads as a spurious failure. + with connect( + url, + additional_headers={"Authorization": "harness"}, + open_timeout=open_timeout, + max_size=None, + ) as ws: + + def writer(): + for chunk in chunks: + time.sleep(0.05 / speed) + ws.send(chunk) + ws.send('{"type": "Terminate"}') + + with ThreadPoolExecutor(max_workers=1) as pool: + write_future = pool.submit(writer) + for message in ws: + data = json.loads(message) + kind = data.get("type") + if kind == "Turn": + nonlocal_words = data.get("words") or [] + if nonlocal_words: + if first_turn is None: + first_turn = time.perf_counter() - start + turns += 1 + if data.get("end_of_turn"): + final_text.append( + " ".join(w["text"] for w in nonlocal_words) + ) + elif kind == "Termination": + break + write_future.result() + except Exception as exc: + return Result(False, time.perf_counter() - start, detail=repr(exc)) + + elapsed = time.perf_counter() - start + text = " ".join(final_text) + if expect and expect.lower() not in text.lower(): + return Result( + False, elapsed, detail=f"transcript missing {expect!r}", text=text + ) + return Result( + True, elapsed, text=text, extra={"first_turn_s": first_turn, "turns": turns} + ) + + +# -------------------------------------------------------------------------- +# driver +# -------------------------------------------------------------------------- +def run_level(work: Callable[[], Result], n: int) -> list[Result]: + """Fire n copies of `work` at once and collect every outcome.""" + with ThreadPoolExecutor(max_workers=n) as pool: + futures = [pool.submit(work) for _ in range(n)] + return [f.result() for f in futures] + + +def summarize(results: list[Result], wall: float, audio_seconds: float) -> dict: + ok = [r for r in results if r.ok] + lat = sorted(r.seconds for r in ok) + + def quantile(p: float) -> float: + if not lat: + return float("nan") + return lat[min(int(len(lat) * p), len(lat) - 1)] + + return { + "n": len(results), + "ok": len(ok), + "failed": len(results) - len(ok), + "p50": quantile(0.50), + "p95": quantile(0.95), + "max": lat[-1] if lat else float("nan"), + "wall": wall, + "rps": len(ok) / wall if wall else 0.0, + "audio_x_realtime": (len(ok) * audio_seconds / wall) if wall else 0.0, + } + + +def print_table(rows: list[tuple[int, dict]]) -> None: + print( + f"\n{'conc':>5} {'ok':>5} {'fail':>5} {'p50 s':>8} {'p95 s':>8} " + f"{'max s':>8} {'req/s':>7} {'xRT':>7}" + ) + print("-" * 60) + for conc, s in rows: + print( + f"{conc:>5} {s['ok']:>5} {s['failed']:>5} {s['p50']:>8.2f} " + f"{s['p95']:>8.2f} {s['max']:>8.2f} {s['rps']:>7.2f} " + f"{s['audio_x_realtime']:>7.1f}" + ) + + +def main() -> int: + ap = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + ap.add_argument("mode", choices=["sync", "streaming"]) + ap.add_argument( + "--endpoint", + required=True, + help="https://... for sync, wss://... for streaming", + ) + ap.add_argument("--audio", required=True, help="16-bit PCM WAV") + ap.add_argument("--concurrency", type=int, default=1) + ap.add_argument("--ramp", help="comma-separated concurrency levels, e.g. 1,4,8,16") + ap.add_argument("--max-seconds", type=float, help="truncate audio to N seconds") + ap.add_argument( + "--expect", + default=DEFAULT_EXPECT, + help="substring the transcript must contain ('' to skip)", + ) + ap.add_argument("--speech-model", help="streaming only, e.g. universal-3-5-pro") + ap.add_argument( + "--speed", type=float, default=1.0, help="streaming send rate vs realtime" + ) + ap.add_argument( + "--show-transcript", + action="store_true", + help="print a sample transcript; off by default since audio may contain personal data", + ) + ap.add_argument( + "--open-timeout", + type=float, + default=300.0, + help="seconds to wait for the WebSocket handshake (cold starts are slow)", + ) + ap.add_argument( + "--stop-on-failure", + action="store_true", + help="end a ramp at the first level with failures", + ) + args = ap.parse_args() + + pcm, rate, seconds = load_pcm16(args.audio, args.max_seconds) + print(f"audio: {args.audio} | {seconds:.1f}s @ {rate} Hz | mode={args.mode}") + print(f"endpoint: {args.endpoint}") + + if args.mode == "sync": + payload = wav_bytes(pcm, rate) + work = lambda: sync_once(args.endpoint, payload, args.expect) # noqa: E731 + else: + work = lambda: streaming_once( # noqa: E731 + args.endpoint, + pcm, + rate, + args.expect, + args.speech_model, + args.speed, + args.open_timeout, + ) + + levels = [int(x) for x in args.ramp.split(",")] if args.ramp else [args.concurrency] + + rows = [] + sample_shown = False + for conc in levels: + started = time.perf_counter() + results = run_level(work, conc) + wall = time.perf_counter() - started + stats = summarize(results, wall, seconds) + rows.append((conc, stats)) + + if not sample_shown: + first_ok = next((r for r in results if r.ok), None) + if first_ok: + # Transcripts are user audio and can carry personal data, so the + # default output describes the result without reproducing it. + # Correctness is already asserted by --expect; this is only a + # human sanity check, so it is opt-in. + if args.show_transcript: + # Collapse newlines: transcript text is untrusted input and + # must not be able to forge extra log lines. + sample = " ".join(first_ok.text.split())[:160] + print(f"\ntranscript: {sample}...") + else: + print( + f"\ntranscript: {len(first_ok.text)} chars " + f"(hidden; pass --show-transcript to print it)" + ) + if first_ok.extra: + print(f"detail: {first_ok.extra}") + sample_shown = True + + print( + f"[conc={conc}] ok={stats['ok']}/{stats['n']} " + f"p50={stats['p50']:.2f}s p95={stats['p95']:.2f}s " + f"{stats['audio_x_realtime']:.1f}x realtime" + ) + for r in results: + if not r.ok: + print(f" FAIL: {r.detail[:160]}") + if stats["failed"] and args.stop_on_failure: + print(f"\nstopping: first failures at concurrency {conc}") + break + + if len(rows) > 1: + print_table(rows) + clean = [c for c, s in rows if s["failed"] == 0] + if clean: + print(f"\nhighest fully-successful concurrency tested: {max(clean)}") + else: + print("\nno concurrency level completed without failures") + + return 0 if all(s["failed"] == 0 for _, s in rows) else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/streaming/.env.example b/streaming/.env.example index 0be23ba..df29f66 100644 --- a/streaming/.env.example +++ b/streaming/.env.example @@ -1,5 +1,5 @@ # Required for every streaming stack: -STREAMING_API_IMAGE=344839248844.dkr.ecr.us-west-2.amazonaws.com/self-hosted-streaming-api:release-v1.0.0 +STREAMING_API_IMAGE=344839248844.dkr.ecr.us-west-2.amazonaws.com/self-hosted-streaming-api:release-v1.0.1 LICENSE_AND_USAGE_PROXY_IMAGE=344839248844.dkr.ecr.us-west-2.amazonaws.com/self-hosted-streaming-license-and-usage-proxy:release-v1.0.0 USAGE_TRACKING_API_KEY= # Required only when running the Universal stack (docker-compose.english-multilang.yml): diff --git a/streaming_modal_stack/README.md b/streaming_modal_stack/README.md index 4873dad..15892f7 100644 --- a/streaming_modal_stack/README.md +++ b/streaming_modal_stack/README.md @@ -31,21 +31,29 @@ modal deploy modal_app_universal_3_5_pro.py # or modal_app_english_multilang Each GPU backend keeps one L40S warm (`min_containers=1`) and gates readiness on `grpc_health_probe`, so the first deploy takes a few minutes to warm the model; -Modal then autoscales on concurrent sessions (`target_concurrency=32`, matching -`MAX_OPEN_STREAMS`). The endpoint URLs are printed, of the form +Modal then autoscales on concurrent sessions, with `target_concurrency` set to +each stack's `MAX_OPEN_STREAMS` (Universal-3.5 Pro 32, English + Multilingual 48, +matching compose). The endpoint URLs are printed, of the form `https://---streamingapi..modal.direct`. ## Verify +The `streamingapi` endpoint is behind Modal proxy auth by default, so send a +proxy-auth token (`--modal-key` / `--modal-secret`, or `MODAL_KEY` / +`MODAL_SECRET`); a `licenseproxy` `/v1/status` check needs none. To probe +without a token, deploy the endpoint with `AAI_REQUIRE_MODAL_AUTH=0`. + ```bash curl -fsS https://--aai-streaming-u3pro-licenseproxy..modal.direct/v1/status -curl -fsS https://--aai-streaming-u3pro-streamingapi..modal.direct/v3/ws/health -# Stream with the bundled example client, swapping ws://localhost:8080 for the wss:// URL: -python ../streaming/example/example_with_prerecorded_audio_file.py \ - --audio-file ../streaming/example/example_audio_file.wav \ +# Stream with the bundled sample client (it forwards Modal proxy-auth headers; +# the repo's example_with_prerecorded_audio_file.py does not, so it only works +# against an AAI_REQUIRE_MODAL_AUTH=0 endpoint): +python sample_streaming.py \ --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ - --speech-model universal-3-5-pro + --audio ../streaming/example/example_audio_file.wav \ + --speech-model universal-3-5-pro \ + --modal-key "$MODAL_KEY" --modal-secret "$MODAL_SECRET" ``` For the English + Multilingual stack use `--speech-model universal-streaming-english` diff --git a/streaming_modal_stack/modal_app_english_multilang.py b/streaming_modal_stack/modal_app_english_multilang.py index c6ed911..5801ea2 100644 --- a/streaming_modal_stack/modal_app_english_multilang.py +++ b/streaming_modal_stack/modal_app_english_multilang.py @@ -23,14 +23,18 @@ import os import signal import subprocess +import threading import modal APP_NAME = "aai-streaming-english-multilang" REGISTRY = "344839248844.dkr.ecr.us-west-2.amazonaws.com" -# The English/Multilingual images ship on their own release line, separate from -# the Universal-3.5 Pro images (see streaming/.env.example). -TAG = "release-v0.6.0" +# The English/Multilingual ASR images ship on their own release line, separate +# from the shared streaming-api and license-and-usage-proxy images. +ASR_MODEL_TAG = "release-v0.6.0" +# streaming-api carries the WARNING-not-ERROR handshake-logging fix at v1.0.1 +# (DeepLearning #19523); the license-and-usage-proxy has no v1.0.1. +API_TAG = "release-v1.0.1" PROXY_TAG = "release-v1.0.0" ASR_GRPC_PORT = 50051 @@ -60,28 +64,37 @@ def _vendor_image(repo: str, tag: str) -> modal.Image: ) -english_image = _vendor_image("self-hosted-streaming-asr-english", TAG) -multilang_image = _vendor_image("self-hosted-streaming-asr-multilang", TAG) -api_image = _vendor_image("self-hosted-streaming-api", PROXY_TAG) +english_image = _vendor_image("self-hosted-streaming-asr-english", ASR_MODEL_TAG) +multilang_image = _vendor_image("self-hosted-streaming-asr-multilang", ASR_MODEL_TAG) +api_image = _vendor_image("self-hosted-streaming-api", API_TAG) proxy_image = _vendor_image("self-hosted-streaming-license-and-usage-proxy", PROXY_TAG) # nginx routes gRPC by x-model-version; no ECR pull needed. A Debian base gives -# Modal a detectable interpreter plus its client, alongside nginx. +# Modal a detectable interpreter plus its client, alongside nginx; ca-certificates +# lets nginx verify the ASR backends' TLS certs (grpc_ssl_verify below). lb_image = ( modal.Image.debian_slim(python_version="3.12") - .apt_install("nginx") + .apt_install("nginx", "ca-certificates") .pip_install(f"modal=={modal.__version__}") ) +# Set by each @modal.exit stop() so the fate-share reaper can tell an intentional +# teardown from an unexpected vendor exit (one server per container). +_stopping = threading.Event() + + def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: """Start a binary and fate-share it with the container (see sync_modal_stack/modal_app.py).""" proc = subprocess.Popen(argv, env={**os.environ, **env}) - import threading - def _reap() -> None: proc.wait() - os._exit(proc.returncode or 1) + # An exit while we are not intentionally stopping means the vendor + # process died on its own; fail so Modal replaces the container. A clean + # @modal.exit teardown sets _stopping first, so stay quiet and let the + # exit handler finish (the container then exits 0). + if not _stopping.is_set(): + os._exit(proc.returncode if (proc.returncode or 0) > 0 else 1) threading.Thread(target=_reap, daemon=True).start() return proc @@ -125,7 +138,7 @@ def _asr_ready(port: int, timeout_s: int) -> None: port=ASR_GRPC_PORT, h2_enabled=True, unauthenticated=True, # dialed server-side (via nginx) over gRPC; see README "Security" - target_concurrency=32, + target_concurrency=48, # matches MAX_OPEN_STREAMS=48 (compose parity) min_containers=1, max_containers=4, buffer_containers=1, @@ -137,7 +150,7 @@ def _asr_ready(port: int, timeout_s: int) -> None: "SERVER_PORT": str(ASR_GRPC_PORT), "LOGGING_LEVEL": "INFO", "USE_STRUCTURED_LOGGING": "False", - "MAX_OPEN_STREAMS": os.environ.get("MAX_OPEN_STREAMS", "32"), + "MAX_OPEN_STREAMS": os.environ.get("MAX_OPEN_STREAMS", "48"), "VLLM_USE_FLASHINFER_SAMPLER": "0", } @@ -158,6 +171,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=25) @@ -174,6 +188,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=570) @@ -190,6 +205,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=570) @@ -225,6 +241,10 @@ def stop(self) -> None: location / {{ grpc_pass grpcs://$asr_backend; grpc_ssl_server_name on; + # Authenticate the backend, not just encrypt: verify its TLS cert against + # the public CA store (Modal's edge presents a publicly-trusted cert). + grpc_ssl_verify on; + grpc_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; grpc_connect_timeout 75s; grpc_read_timeout 10h; grpc_send_timeout 10h; @@ -251,6 +271,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=25) @@ -289,6 +310,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=570) diff --git a/streaming_modal_stack/modal_app_universal_3_5_pro.py b/streaming_modal_stack/modal_app_universal_3_5_pro.py index 9ab56d9..513ede3 100644 --- a/streaming_modal_stack/modal_app_universal_3_5_pro.py +++ b/streaming_modal_stack/modal_app_universal_3_5_pro.py @@ -24,6 +24,7 @@ import os import signal import subprocess +import threading import modal @@ -69,15 +70,23 @@ def _vendor_image(repo: str, tag: str) -> modal.Image: proxy_image = _vendor_image("self-hosted-streaming-license-and-usage-proxy", PROXY_TAG) +# Set by each @modal.exit stop() so the fate-share reaper can tell an intentional +# teardown from an unexpected vendor exit (one server per container). +_stopping = threading.Event() + + def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: """Start a vendor binary and fate-share it with the container (see sync_modal_stack/modal_app.py).""" proc = subprocess.Popen(argv, env={**os.environ, **env}) - import threading - def _reap() -> None: proc.wait() - os._exit(proc.returncode or 1) + # An exit while we are not intentionally stopping means the vendor + # process died on its own; fail so Modal replaces the container. A clean + # @modal.exit teardown sets _stopping first, so stay quiet and let the + # exit handler finish (the container then exits 0). + if not _stopping.is_set(): + os._exit(proc.returncode if (proc.returncode or 0) > 0 else 1) threading.Thread(target=_reap, daemon=True).start() return proc @@ -130,6 +139,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=25) @@ -182,6 +192,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=570) @@ -231,6 +242,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=570) diff --git a/streaming_modal_stack/sample_streaming.py b/streaming_modal_stack/sample_streaming.py index 8c3d996..91137c5 100644 --- a/streaming_modal_stack/sample_streaming.py +++ b/streaming_modal_stack/sample_streaming.py @@ -119,9 +119,13 @@ def main() -> int: print(f"{args.endpoint}\n audio {dur:.1f}s @ {rate} Hz | model={args.speech_model or '(default)'} | " f"speed={args.speed}x | sessions={args.load}\n") + def _first_turn(r: dict) -> str: + # None when the session produced no word-bearing turns. + return f"{r['first_turn_s']:.2f}s" if r["first_turn_s"] is not None else "n/a" + if args.load == 1: r = stream_once(args, pcm, rate, live=True) - print(f"\n{r['turns']} final turns | first turn {r['first_turn_s']:.2f}s | wall {r['elapsed_s']:.1f}s") + print(f"\n{r['turns']} final turns | first turn {_first_turn(r)} | wall {r['elapsed_s']:.1f}s") return 0 # Load mode: run N sessions at once, print a summary line per session. @@ -134,7 +138,7 @@ def main() -> int: try: r = fut.result() ok += 1 - print(f" session {i:>2}: ok | {r['turns']} turns | first turn {r['first_turn_s']:.2f}s") + print(f" session {i:>2}: ok | {r['turns']} turns | first turn {_first_turn(r)}") except Exception as exc: # noqa: BLE001 print(f" session {i:>2}: FAIL {exc}") print(f"\n{ok}/{args.load} sessions ok | wall {time.perf_counter() - started:.1f}s") diff --git a/sync_modal_stack/README.md b/sync_modal_stack/README.md index fc584f2..4dd8033 100644 --- a/sync_modal_stack/README.md +++ b/sync_modal_stack/README.md @@ -56,21 +56,29 @@ printed, of the form `https://--aai-sync-u3pro-..moda ## Verify +`syncapi` is behind Modal proxy auth by default, so its probes need a +`Modal-Key` / `Modal-Secret` header pair (a proxy-auth token from the Modal +dashboard); `licenseproxy` `/v1/status` needs none. The `syncapi` examples below +show those headers — omit them only against an endpoint deployed with +`AAI_REQUIRE_MODAL_AUTH=0`, where any non-empty `Authorization` connects. + ```bash curl -fsS https://--aai-sync-u3pro-licenseproxy..modal.direct/v1/status # {"state":"Connected", ...} curl -sS -o /dev/null -w '%{http_code}\n' \ + -H "Modal-Key: $MODAL_KEY" -H "Modal-Secret: $MODAL_SECRET" \ https://--aai-sync-u3pro-syncapi..modal.direct/readyz # 503 while the model is cold, 200 once warm (Modal's edge may answer 303 first) curl -F 'audio=@../sync/example/example_audio_file.wav;type=audio/wav' \ -F 'config={"language_code":"en"};type=application/json' \ + -H "Modal-Key: $MODAL_KEY" -H "Modal-Secret: $MODAL_SECRET" \ -H 'Authorization: any-non-empty-value' \ https://--aai-sync-u3pro-syncapi..modal.direct/transcribe ``` -Or use the [sample script](#sample-requests). +Or use the [sample script](#sample-requests) (pass `--modal-key` / `--modal-secret`). ## Authentication diff --git a/sync_modal_stack/modal_app.py b/sync_modal_stack/modal_app.py index ac2ceb4..f8f172a 100644 --- a/sync_modal_stack/modal_app.py +++ b/sync_modal_stack/modal_app.py @@ -19,6 +19,7 @@ import os import signal import subprocess +import threading import modal @@ -67,6 +68,11 @@ def _vendor_image(repo: str) -> modal.Image: sync_image = _vendor_image("self-hosted-sync-asr-u3-pro") +# Set by each @modal.exit stop() so the fate-share reaper can tell an intentional +# teardown from an unexpected vendor exit (one server per container). +_stopping = threading.Event() + + def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: """Start a vendor binary and fate-share it with the container. @@ -76,11 +82,14 @@ def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: """ proc = subprocess.Popen(argv, env={**os.environ, **env}) - import threading - def _reap() -> None: proc.wait() - os._exit(proc.returncode or 1) + # An exit while we are not intentionally stopping means the vendor + # process died on its own; fail so Modal replaces the container. A clean + # @modal.exit teardown sets _stopping first, so stay quiet and let the + # exit handler finish (the container then exits 0). + if not _stopping.is_set(): + os._exit(proc.returncode if (proc.returncode or 0) > 0 else 1) threading.Thread(target=_reap, daemon=True).start() return proc @@ -143,6 +152,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: # Graceful stop lets the proxy flush queued usage before exit. + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=25) @@ -199,6 +209,7 @@ def start(self) -> None: @modal.exit() def stop(self) -> None: + _stopping.set() self.proc.send_signal(signal.SIGTERM) try: self.proc.wait(timeout=50) From 1a938df47aca8236d1917e0dabc8ed3b03038be2 Mon Sep 17 00:00:00 2001 From: Aleksandar Mitov <140423361+aleks-mitov@users.noreply.github.com> Date: Wed, 26 Aug 2026 19:13:13 +0000 Subject: [PATCH 7/9] chore: drop bench/harness.py from this branch (belongs to #11; local-only for verification) Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1 --- bench/harness.py | 345 ----------------------------------------------- 1 file changed, 345 deletions(-) delete mode 100644 bench/harness.py diff --git a/bench/harness.py b/bench/harness.py deleted file mode 100644 index ea7dbb8..0000000 --- a/bench/harness.py +++ /dev/null @@ -1,345 +0,0 @@ -#!/usr/bin/env python3 -"""Load-test harness for the self-hosted stacks, local or on Modal. - -Sends real audio at a chosen concurrency, verifies transcripts actually come -back, and reports latency and throughput. Use --ramp to sweep concurrency and -find the level a deployment sustains before it degrades. - - # correctness check, one request - python harness.py sync --endpoint https://host --audio ../sync/example/example_audio_file.wav - - # concurrency sweep - python harness.py sync --endpoint https://host --audio a.wav --ramp 1,2,4,8,16 - python harness.py streaming --endpoint wss://host --audio a.wav --ramp 1,4,16,32,48 -""" - -from __future__ import annotations - -import argparse -import json -import sys -import time -import wave -from concurrent.futures import ThreadPoolExecutor -from dataclasses import dataclass, field -from typing import Callable -from urllib.parse import urlencode - -# Words expected in the bundled sample; used only as a sanity check that the -# transcript is real output rather than an empty 200. -DEFAULT_EXPECT = "assemblyai" - - -@dataclass -class Result: - ok: bool - seconds: float - detail: str = "" - text: str = "" - extra: dict = field(default_factory=dict) - - -def load_pcm16(path: str, max_seconds: float | None) -> tuple[bytes, int, float]: - """Read a 16-bit PCM WAV, optionally truncated, returning raw frames.""" - with wave.open(path, "rb") as wav: - if wav.getsampwidth() != 2 or wav.getcomptype() != "NONE": - raise SystemExit(f"{path}: must be uncompressed 16-bit PCM WAV") - rate = wav.getframerate() - frames = wav.getnframes() - if max_seconds: - frames = min(frames, int(rate * max_seconds)) - return wav.readframes(frames), rate, frames / rate - - -def wav_bytes(pcm: bytes, rate: int, channels: int = 1) -> bytes: - """Re-wrap raw PCM as a WAV container for the sync API's multipart upload.""" - import io - - buf = io.BytesIO() - with wave.open(buf, "wb") as out: - out.setnchannels(channels) - out.setsampwidth(2) - out.setframerate(rate) - out.writeframes(pcm) - return buf.getvalue() - - -# -------------------------------------------------------------------------- -# sync: one HTTP POST per request -# -------------------------------------------------------------------------- -def sync_once(endpoint: str, audio: bytes, expect: str) -> Result: - import requests - - start = time.perf_counter() - try: - resp = requests.post( - f"{endpoint.rstrip('/')}/transcribe", - files={"audio": ("audio.wav", audio, "audio/wav")}, - data={"config": json.dumps({"language_code": "en"})}, - headers={"Authorization": "harness"}, - timeout=300, - ) - except Exception as exc: # network/timeout - return Result(False, time.perf_counter() - start, detail=repr(exc)) - elapsed = time.perf_counter() - start - - if resp.status_code != 200: - return Result( - False, elapsed, detail=f"HTTP {resp.status_code}: {resp.text[:120]}" - ) - body = resp.json() - text = body.get("text", "") - if expect and expect.lower() not in text.lower(): - return Result( - False, elapsed, detail=f"transcript missing {expect!r}", text=text - ) - return Result( - True, - elapsed, - text=text, - extra={ - "server_ms": body.get("request_time_ms"), - "audio_ms": body.get("audio_duration_ms"), - "words": len(body.get("words", [])), - }, - ) - - -# -------------------------------------------------------------------------- -# streaming: one WebSocket session per request -# -------------------------------------------------------------------------- -def streaming_once( - endpoint: str, - pcm: bytes, - rate: int, - expect: str, - speech_model: str | None, - speed: float, - open_timeout: float, -) -> Result: - from websockets.sync.client import connect - - params = {"sample_rate": rate, "format_turns": "true"} - if speech_model: - params["speech_model"] = speech_model - url = f"{endpoint.rstrip('/')}?{urlencode(params)}" - - # 50 ms of audio per frame, the granularity the example client uses. - frame = int(rate * 0.05) * 2 - chunks = [pcm[i : i + frame] for i in range(0, len(pcm), frame)] - - start = time.perf_counter() - first_turn: float | None = None - turns = 0 - final_text: list[str] = [] - - try: - # Generous open timeout: a cold Modal container can take minutes to - # accept the upgrade, and the default 10s reads as a spurious failure. - with connect( - url, - additional_headers={"Authorization": "harness"}, - open_timeout=open_timeout, - max_size=None, - ) as ws: - - def writer(): - for chunk in chunks: - time.sleep(0.05 / speed) - ws.send(chunk) - ws.send('{"type": "Terminate"}') - - with ThreadPoolExecutor(max_workers=1) as pool: - write_future = pool.submit(writer) - for message in ws: - data = json.loads(message) - kind = data.get("type") - if kind == "Turn": - nonlocal_words = data.get("words") or [] - if nonlocal_words: - if first_turn is None: - first_turn = time.perf_counter() - start - turns += 1 - if data.get("end_of_turn"): - final_text.append( - " ".join(w["text"] for w in nonlocal_words) - ) - elif kind == "Termination": - break - write_future.result() - except Exception as exc: - return Result(False, time.perf_counter() - start, detail=repr(exc)) - - elapsed = time.perf_counter() - start - text = " ".join(final_text) - if expect and expect.lower() not in text.lower(): - return Result( - False, elapsed, detail=f"transcript missing {expect!r}", text=text - ) - return Result( - True, elapsed, text=text, extra={"first_turn_s": first_turn, "turns": turns} - ) - - -# -------------------------------------------------------------------------- -# driver -# -------------------------------------------------------------------------- -def run_level(work: Callable[[], Result], n: int) -> list[Result]: - """Fire n copies of `work` at once and collect every outcome.""" - with ThreadPoolExecutor(max_workers=n) as pool: - futures = [pool.submit(work) for _ in range(n)] - return [f.result() for f in futures] - - -def summarize(results: list[Result], wall: float, audio_seconds: float) -> dict: - ok = [r for r in results if r.ok] - lat = sorted(r.seconds for r in ok) - - def quantile(p: float) -> float: - if not lat: - return float("nan") - return lat[min(int(len(lat) * p), len(lat) - 1)] - - return { - "n": len(results), - "ok": len(ok), - "failed": len(results) - len(ok), - "p50": quantile(0.50), - "p95": quantile(0.95), - "max": lat[-1] if lat else float("nan"), - "wall": wall, - "rps": len(ok) / wall if wall else 0.0, - "audio_x_realtime": (len(ok) * audio_seconds / wall) if wall else 0.0, - } - - -def print_table(rows: list[tuple[int, dict]]) -> None: - print( - f"\n{'conc':>5} {'ok':>5} {'fail':>5} {'p50 s':>8} {'p95 s':>8} " - f"{'max s':>8} {'req/s':>7} {'xRT':>7}" - ) - print("-" * 60) - for conc, s in rows: - print( - f"{conc:>5} {s['ok']:>5} {s['failed']:>5} {s['p50']:>8.2f} " - f"{s['p95']:>8.2f} {s['max']:>8.2f} {s['rps']:>7.2f} " - f"{s['audio_x_realtime']:>7.1f}" - ) - - -def main() -> int: - ap = argparse.ArgumentParser( - description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter - ) - ap.add_argument("mode", choices=["sync", "streaming"]) - ap.add_argument( - "--endpoint", - required=True, - help="https://... for sync, wss://... for streaming", - ) - ap.add_argument("--audio", required=True, help="16-bit PCM WAV") - ap.add_argument("--concurrency", type=int, default=1) - ap.add_argument("--ramp", help="comma-separated concurrency levels, e.g. 1,4,8,16") - ap.add_argument("--max-seconds", type=float, help="truncate audio to N seconds") - ap.add_argument( - "--expect", - default=DEFAULT_EXPECT, - help="substring the transcript must contain ('' to skip)", - ) - ap.add_argument("--speech-model", help="streaming only, e.g. universal-3-5-pro") - ap.add_argument( - "--speed", type=float, default=1.0, help="streaming send rate vs realtime" - ) - ap.add_argument( - "--show-transcript", - action="store_true", - help="print a sample transcript; off by default since audio may contain personal data", - ) - ap.add_argument( - "--open-timeout", - type=float, - default=300.0, - help="seconds to wait for the WebSocket handshake (cold starts are slow)", - ) - ap.add_argument( - "--stop-on-failure", - action="store_true", - help="end a ramp at the first level with failures", - ) - args = ap.parse_args() - - pcm, rate, seconds = load_pcm16(args.audio, args.max_seconds) - print(f"audio: {args.audio} | {seconds:.1f}s @ {rate} Hz | mode={args.mode}") - print(f"endpoint: {args.endpoint}") - - if args.mode == "sync": - payload = wav_bytes(pcm, rate) - work = lambda: sync_once(args.endpoint, payload, args.expect) # noqa: E731 - else: - work = lambda: streaming_once( # noqa: E731 - args.endpoint, - pcm, - rate, - args.expect, - args.speech_model, - args.speed, - args.open_timeout, - ) - - levels = [int(x) for x in args.ramp.split(",")] if args.ramp else [args.concurrency] - - rows = [] - sample_shown = False - for conc in levels: - started = time.perf_counter() - results = run_level(work, conc) - wall = time.perf_counter() - started - stats = summarize(results, wall, seconds) - rows.append((conc, stats)) - - if not sample_shown: - first_ok = next((r for r in results if r.ok), None) - if first_ok: - # Transcripts are user audio and can carry personal data, so the - # default output describes the result without reproducing it. - # Correctness is already asserted by --expect; this is only a - # human sanity check, so it is opt-in. - if args.show_transcript: - # Collapse newlines: transcript text is untrusted input and - # must not be able to forge extra log lines. - sample = " ".join(first_ok.text.split())[:160] - print(f"\ntranscript: {sample}...") - else: - print( - f"\ntranscript: {len(first_ok.text)} chars " - f"(hidden; pass --show-transcript to print it)" - ) - if first_ok.extra: - print(f"detail: {first_ok.extra}") - sample_shown = True - - print( - f"[conc={conc}] ok={stats['ok']}/{stats['n']} " - f"p50={stats['p50']:.2f}s p95={stats['p95']:.2f}s " - f"{stats['audio_x_realtime']:.1f}x realtime" - ) - for r in results: - if not r.ok: - print(f" FAIL: {r.detail[:160]}") - if stats["failed"] and args.stop_on_failure: - print(f"\nstopping: first failures at concurrency {conc}") - break - - if len(rows) > 1: - print_table(rows) - clean = [c for c, s in rows if s["failed"] == 0] - if clean: - print(f"\nhighest fully-successful concurrency tested: {max(clean)}") - else: - print("\nno concurrency level completed without failures") - - return 0 if all(s["failed"] == 0 for _, s in rows) else 1 - - -if __name__ == "__main__": - sys.exit(main()) From 5e6d91370f0cdf70aa33a86028ca593fb0043b57 Mon Sep 17 00:00:00 2001 From: Aleksandar Mitov <140423361+aleks-mitov@users.noreply.github.com> Date: Wed, 26 Aug 2026 19:22:55 +0000 Subject: [PATCH 8/9] fix(modal): drop nginx grpc_ssl_verify on the LB->ASR hop (broke connectivity) Verifying the backend cert (finding 4) fails against Modal's edge: nginx cannot build the chain and errors "unable to get local issuer certificate", marking the ASR backend down so every session 3005s (confirmed live). Revert to encrypt-only (grpc_ssl_server_name on) and document the residual limitation in-place, the finding's sanctioned alternative. The backend is unauthenticated by design; co-location or i6pn is the real fix, per the README. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1 --- .../modal_app_english_multilang.py | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/streaming_modal_stack/modal_app_english_multilang.py b/streaming_modal_stack/modal_app_english_multilang.py index 5801ea2..d832623 100644 --- a/streaming_modal_stack/modal_app_english_multilang.py +++ b/streaming_modal_stack/modal_app_english_multilang.py @@ -69,11 +69,10 @@ def _vendor_image(repo: str, tag: str) -> modal.Image: api_image = _vendor_image("self-hosted-streaming-api", API_TAG) proxy_image = _vendor_image("self-hosted-streaming-license-and-usage-proxy", PROXY_TAG) # nginx routes gRPC by x-model-version; no ECR pull needed. A Debian base gives -# Modal a detectable interpreter plus its client, alongside nginx; ca-certificates -# lets nginx verify the ASR backends' TLS certs (grpc_ssl_verify below). +# Modal a detectable interpreter plus its client, alongside nginx. lb_image = ( modal.Image.debian_slim(python_version="3.12") - .apt_install("nginx", "ca-certificates") + .apt_install("nginx") .pip_install(f"modal=={modal.__version__}") ) @@ -241,10 +240,13 @@ def stop(self) -> None: location / {{ grpc_pass grpcs://$asr_backend; grpc_ssl_server_name on; - # Authenticate the backend, not just encrypt: verify its TLS cert against - # the public CA store (Modal's edge presents a publicly-trusted cert). - grpc_ssl_verify on; - grpc_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; + # This encrypts the LB->ASR hop but does not authenticate the backend: + # grpc_ssl_verify is left off because nginx cannot build a chain to Modal's + # edge certificate here (grpc_ssl_verify on fails with "unable to get local + # issuer certificate" and takes the backend down). The backend is + # unauthenticated by design anyway (see README "Authentication and + # security"); co-locate the API and ASR, or use i6pn, to remove the public + # hop entirely rather than only encrypt it. grpc_connect_timeout 75s; grpc_read_timeout 10h; grpc_send_timeout 10h; From d163e1dcbac4fb26711a5b243e422d3750611d2f Mon Sep 17 00:00:00 2001 From: Aleksandar Mitov <140423361+aleks-mitov@users.noreply.github.com> Date: Wed, 26 Aug 2026 21:39:10 +0000 Subject: [PATCH 9/9] refactor: group each service by deployment target (docker/ + modal/) Per review feedback, nest deployment variants under each service dir instead of top-level *_modal_stack dirs, so more targets (e.g. sagemaker) slot in cleanly: sync/docker/ (compose stack, moved from sync/) sync/modal/ (moved from sync_modal_stack/) streaming/docker/ (compose stack + nginx, moved from streaming/) streaming/modal/ (moved from streaming_modal_stack/) Relative links and paths updated throughout (root README layout + pointers, docker READMEs' ../README.md -> ../../README.md and modal pointers, modal READMEs' example paths -> ../docker/example, sample-script docstrings, and the cross-file comment references). No code behavior change. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1 --- README.md | 30 ++++++++++-------- streaming/{ => docker}/.env.example | 0 streaming/{ => docker}/README.md | 12 +++---- .../docker-compose.english-multilang.yml | 0 .../docker-compose.universal-3-5-pro.yml | 0 .../example/example_audio_file.wav | Bin .../example_with_prerecorded_audio_file.py | 0 .../{ => docker}/example/requirements.txt | 0 .../{ => docker}/nginx_streaming_asr.conf | 0 .../modal}/README.md | 8 ++--- .../modal}/modal_app_english_multilang.py | 4 +-- .../modal}/modal_app_universal_3_5_pro.py | 6 ++-- .../modal}/sample_streaming.py | 4 +-- sync/{ => docker}/.env.example | 0 sync/{ => docker}/README.md | 8 ++--- .../docker-compose.universal-3-5-pro.yml | 0 .../example/example_audio_file.wav | Bin sync/{ => docker}/example/requirements.txt | 0 sync/{ => docker}/example/transcribe_file.py | 0 {sync_modal_stack => sync/modal}/README.md | 6 ++-- {sync_modal_stack => sync/modal}/modal_app.py | 0 .../modal}/sample_sync.py | 2 +- 22 files changed, 42 insertions(+), 38 deletions(-) rename streaming/{ => docker}/.env.example (100%) rename streaming/{ => docker}/README.md (96%) rename streaming/{ => docker}/docker-compose.english-multilang.yml (100%) rename streaming/{ => docker}/docker-compose.universal-3-5-pro.yml (100%) rename streaming/{ => docker}/example/example_audio_file.wav (100%) rename streaming/{ => docker}/example/example_with_prerecorded_audio_file.py (100%) rename streaming/{ => docker}/example/requirements.txt (100%) rename streaming/{ => docker}/nginx_streaming_asr.conf (100%) rename {streaming_modal_stack => streaming/modal}/README.md (95%) rename {streaming_modal_stack => streaming/modal}/modal_app_english_multilang.py (99%) rename {streaming_modal_stack => streaming/modal}/modal_app_universal_3_5_pro.py (98%) rename {streaming_modal_stack => streaming/modal}/sample_streaming.py (97%) rename sync/{ => docker}/.env.example (100%) rename sync/{ => docker}/README.md (96%) rename sync/{ => docker}/docker-compose.universal-3-5-pro.yml (100%) rename sync/{ => docker}/example/example_audio_file.wav (100%) rename sync/{ => docker}/example/requirements.txt (100%) rename sync/{ => docker}/example/transcribe_file.py (100%) rename {sync_modal_stack => sync/modal}/README.md (96%) rename {sync_modal_stack => sync/modal}/modal_app.py (100%) rename {sync_modal_stack => sync/modal}/sample_sync.py (98%) diff --git a/README.md b/README.md index 27b4821..0919c57 100644 --- a/README.md +++ b/README.md @@ -14,26 +14,30 @@ and, within each service, by **model**. - **Streaming** transcribes a live audio stream over a WebSocket connection. One stack serves multiple models; the client selects the model per session. See - [`streaming/README.md`](streaming/README.md). + [`streaming/docker/README.md`](streaming/docker/README.md). - **Sync** transcribes a complete file in a single HTTP request/response (audio ≤ 120 s by default). It is self-contained — a single GPU container plus the - license-and-usage-proxy, no load balancer. See [`sync/README.md`](sync/README.md). + license-and-usage-proxy, no load balancer. See [`sync/docker/README.md`](sync/docker/README.md). -Each service directory is self-contained: its compose file(s), `.env.example`, -example client, and `README.md` live together. Run commands from inside the -service directory. +Each stack is self-contained under `/docker/`: its compose file(s), +`.env.example`, example client, and `README.md` live together. Run compose +commands from inside that directory. Each stack can also run on serverless GPUs instead of hardware you manage. Every stack is a self-contained Modal App deployed with a single `modal deploy` — see -[`sync_modal_stack/`](sync_modal_stack/) and -[`streaming_modal_stack/`](streaming_modal_stack/). +[`sync/modal/`](sync/modal/) and +[`streaming/modal/`](streaming/modal/). ## Repository layout ``` . -├── streaming/ # WebSocket streaming ASR (Universal English/Multilingual, Universal-3.5 Pro) -└── sync/ # Synchronous full-file HTTP transcription (Universal-3.5 Pro) +├── streaming/ +│ ├── docker/ # compose stack (Universal English/Multilingual, Universal-3.5 Pro) +│ └── modal/ # serverless-GPU packages +└── sync/ + ├── docker/ # compose stack (Universal-3.5 Pro) + └── modal/ # serverless-GPU package ``` ## Prerequisites (all services) @@ -65,9 +69,9 @@ aws ecr get-login-password --region us-west-2 \ ### License file -Place your AssemblyAI `license.jwt` in the directory of the service you are -running (`streaming/` or `sync/`), or point the `LICENSE_FILE_PATH` environment -variable in that service's compose file at your license file's location. +Place your AssemblyAI `license.jwt` in the directory of the stack you are +running (`streaming/docker/` or `sync/docker/`), or point the `LICENSE_FILE_PATH` +environment variable in that stack's compose file at your license file's location. ## Shared component: license-and-usage-proxy @@ -148,7 +152,7 @@ This release introduces the **Sync self-hosted service** model. It transcribes a complete audio file (≤ 120 s) in a single `POST /transcribe` request/response — a single GPU container plus the license-and-usage-proxy, no load balancer. It exposes `GET /readyz` (200 once -the model is warm) for readiness probes. See [`sync/README.md`](sync/README.md). +the model is warm) for readiness probes. See [`sync/docker/README.md`](sync/docker/README.md). #### Streaming — U3 Pro replaced by Universal-3.5 Pro (BREAKING) diff --git a/streaming/.env.example b/streaming/docker/.env.example similarity index 100% rename from streaming/.env.example rename to streaming/docker/.env.example diff --git a/streaming/README.md b/streaming/docker/README.md similarity index 96% rename from streaming/README.md rename to streaming/docker/README.md index 9dc2c9f..445ea27 100644 --- a/streaming/README.md +++ b/streaming/docker/README.md @@ -5,7 +5,7 @@ Real-time transcription over a WebSocket connection. Run all commands from this > Prerequisites (license, Docker, GPU runtime, ECR auth) and the shared > license-and-usage-proxy (usage reporting, license status endpoint, proxy -> production recommendations) are documented in the [top-level README](../README.md). +> production recommendations) are documented in the [top-level README](../../README.md). ## Choosing a stack @@ -24,7 +24,7 @@ To switch between stacks, run `docker compose -f down` before starting th Both stacks include: - **streaming-api**: Gateway API service handling WebSocket connections. - **streaming-asr-lb**: nginx load balancer for ASR services with header-based routing. -- **license-and-usage-proxy**: License validation and usage reporting (see [top-level README](../README.md#shared-component-license-and-usage-proxy)). +- **license-and-usage-proxy**: License validation and usage reporting (see [top-level README](../../README.md#shared-component-license-and-usage-proxy)). ASR backends differ by stack: - Universal stack (`docker-compose.english-multilang.yml`): `streaming-asr-english` and `streaming-asr-multilang`. @@ -64,7 +64,7 @@ to an available backend. ## Setup -Complete the [shared prerequisites](../README.md#prerequisites-all-services) +Complete the [shared prerequisites](../../README.md#prerequisites-all-services) (GPU runtime, ECR authentication, license file) first. Copy the env reference and set the image variables for the stack you plan to run: @@ -188,7 +188,7 @@ python example_with_prerecorded_audio_file.py --help ### Usage reporting The license-and-usage-proxy's billing modes and behavior are documented in the -[top-level README](../README.md#usage-reporting). +[top-level README](../../README.md#usage-reporting). ## Monitoring & debugging @@ -218,11 +218,11 @@ docker compose -f docker-compose.universal-3-5-pro.yml restart streaming-asr-uni ## Deploying on Modal (serverless GPU) Both streaming stacks also run on Modal's serverless GPUs as self-contained, -single-`modal deploy` Modal Apps. See [`../streaming_modal_stack/`](../streaming_modal_stack/). +single-`modal deploy` Modal Apps. See [`../modal/`](../modal/). ## Production deployment recommendations -See the [top-level README](../README.md#production-recommendations-license-and-usage-proxy) +See the [top-level README](../../README.md#production-recommendations-license-and-usage-proxy) for the license-and-usage-proxy. Streaming-specific services follow. ### streaming-api service diff --git a/streaming/docker-compose.english-multilang.yml b/streaming/docker/docker-compose.english-multilang.yml similarity index 100% rename from streaming/docker-compose.english-multilang.yml rename to streaming/docker/docker-compose.english-multilang.yml diff --git a/streaming/docker-compose.universal-3-5-pro.yml b/streaming/docker/docker-compose.universal-3-5-pro.yml similarity index 100% rename from streaming/docker-compose.universal-3-5-pro.yml rename to streaming/docker/docker-compose.universal-3-5-pro.yml diff --git a/streaming/example/example_audio_file.wav b/streaming/docker/example/example_audio_file.wav similarity index 100% rename from streaming/example/example_audio_file.wav rename to streaming/docker/example/example_audio_file.wav diff --git a/streaming/example/example_with_prerecorded_audio_file.py b/streaming/docker/example/example_with_prerecorded_audio_file.py similarity index 100% rename from streaming/example/example_with_prerecorded_audio_file.py rename to streaming/docker/example/example_with_prerecorded_audio_file.py diff --git a/streaming/example/requirements.txt b/streaming/docker/example/requirements.txt similarity index 100% rename from streaming/example/requirements.txt rename to streaming/docker/example/requirements.txt diff --git a/streaming/nginx_streaming_asr.conf b/streaming/docker/nginx_streaming_asr.conf similarity index 100% rename from streaming/nginx_streaming_asr.conf rename to streaming/docker/nginx_streaming_asr.conf diff --git a/streaming_modal_stack/README.md b/streaming/modal/README.md similarity index 95% rename from streaming_modal_stack/README.md rename to streaming/modal/README.md index 15892f7..fc4c8de 100644 --- a/streaming_modal_stack/README.md +++ b/streaming/modal/README.md @@ -3,7 +3,7 @@ Each streaming stack runs on [Modal](https://modal.com) as a self-contained Modal App: one `modal deploy` brings up every service and wires them together, with no dependency on any other deployment. Compose equivalents live in -[`../streaming/`](../streaming/). +[`../docker/`](../docker/). | Stack | File | Servers | |---|---|---| @@ -19,7 +19,7 @@ the matching backend, exactly as `streaming-asr-lb` does in compose. ## Prerequisites and secrets -Identical to the [sync stack](../sync_modal_stack/README.md#store-credentials-as-modal-secrets): +Identical to the [sync stack](../../sync/modal/README.md#store-credentials-as-modal-secrets): create the `aai-ecr-credentials` and `aai-license` Modal secrets once; all three stacks share them. @@ -51,7 +51,7 @@ curl -fsS https://--aai-streaming-u3pro-licenseproxy..modal.d # against an AAI_REQUIRE_MODAL_AUTH=0 endpoint): python sample_streaming.py \ --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ - --audio ../streaming/example/example_audio_file.wav \ + --audio ../docker/example/example_audio_file.wav \ --speech-model universal-3-5-pro \ --modal-key "$MODAL_KEY" --modal-secret "$MODAL_SECRET" ``` @@ -93,7 +93,7 @@ opens N concurrent sessions. pip install websockets python sample_streaming.py \ --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ - --audio ../streaming/example/example_audio_file.wav \ + --audio ../docker/example/example_audio_file.wav \ --speech-model universal-3-5-pro ``` diff --git a/streaming_modal_stack/modal_app_english_multilang.py b/streaming/modal/modal_app_english_multilang.py similarity index 99% rename from streaming_modal_stack/modal_app_english_multilang.py rename to streaming/modal/modal_app_english_multilang.py index d832623..2e9b98f 100644 --- a/streaming_modal_stack/modal_app_english_multilang.py +++ b/streaming/modal/modal_app_english_multilang.py @@ -54,7 +54,7 @@ def _vendor_image(repo: str, tag: str) -> modal.Image: - """A Modal-runnable image from an AssemblyAI ECR image (see sync_modal_stack/modal_app.py).""" + """A Modal-runnable image from an AssemblyAI ECR image (see sync/modal/modal_app.py).""" return ( modal.Image.from_aws_ecr( f"{REGISTRY}/{repo}:{tag}", secret=ecr_secret, add_python="3.12" @@ -83,7 +83,7 @@ def _vendor_image(repo: str, tag: str) -> modal.Image: def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: - """Start a binary and fate-share it with the container (see sync_modal_stack/modal_app.py).""" + """Start a binary and fate-share it with the container (see sync/modal/modal_app.py).""" proc = subprocess.Popen(argv, env={**os.environ, **env}) def _reap() -> None: diff --git a/streaming_modal_stack/modal_app_universal_3_5_pro.py b/streaming/modal/modal_app_universal_3_5_pro.py similarity index 98% rename from streaming_modal_stack/modal_app_universal_3_5_pro.py rename to streaming/modal/modal_app_universal_3_5_pro.py index 513ede3..973b5cc 100644 --- a/streaming_modal_stack/modal_app_universal_3_5_pro.py +++ b/streaming/modal/modal_app_universal_3_5_pro.py @@ -38,7 +38,7 @@ PROXY_TAG = "release-v1.0.0" ASR_GRPC_PORT = 50051 -# See sync_modal_stack/modal_app.py: the WebSocket API requires a Modal proxy-auth token by +# See sync/modal/modal_app.py: the WebSocket API requires a Modal proxy-auth token by # default. Set AAI_REQUIRE_MODAL_AUTH=0 for a throwaway test endpoint. REQUIRE_MODAL_AUTH = os.environ.get("AAI_REQUIRE_MODAL_AUTH", "1") != "0" @@ -55,7 +55,7 @@ def _vendor_image(repo: str, tag: str) -> modal.Image: - """A Modal-runnable image from an AssemblyAI ECR image (see sync_modal_stack/modal_app.py).""" + """A Modal-runnable image from an AssemblyAI ECR image (see sync/modal/modal_app.py).""" return ( modal.Image.from_aws_ecr( f"{REGISTRY}/{repo}:{tag}", secret=ecr_secret, add_python="3.12" @@ -76,7 +76,7 @@ def _vendor_image(repo: str, tag: str) -> modal.Image: def _launch(argv: list[str], env: dict[str, str]) -> subprocess.Popen: - """Start a vendor binary and fate-share it with the container (see sync_modal_stack/modal_app.py).""" + """Start a vendor binary and fate-share it with the container (see sync/modal/modal_app.py).""" proc = subprocess.Popen(argv, env={**os.environ, **env}) def _reap() -> None: diff --git a/streaming_modal_stack/sample_streaming.py b/streaming/modal/sample_streaming.py similarity index 97% rename from streaming_modal_stack/sample_streaming.py rename to streaming/modal/sample_streaming.py index 91137c5..a61d9ec 100644 --- a/streaming_modal_stack/sample_streaming.py +++ b/streaming/modal/sample_streaming.py @@ -5,12 +5,12 @@ # Universal-3.5 Pro stack: python sample_streaming.py \ --endpoint wss://--aai-streaming-u3pro-streamingapi..modal.direct \ - --audio ../streaming/example/example_audio_file.wav \ + --audio ../docker/example/example_audio_file.wav \ --speech-model universal-3-5-pro # English + Multilingual stack (pick the model): python sample_streaming.py --endpoint wss://--aai-streaming-english-multilang-streamingapi..modal.direct \ - --audio ../streaming/example/example_audio_file.wav --speech-model universal-streaming-english + --audio ../docker/example/example_audio_file.wav --speech-model universal-streaming-english # ... or --speech-model universal-streaming-multilingual Audio is sent at real time by default so you watch partial turns update and diff --git a/sync/.env.example b/sync/docker/.env.example similarity index 100% rename from sync/.env.example rename to sync/docker/.env.example diff --git a/sync/README.md b/sync/docker/README.md similarity index 96% rename from sync/README.md rename to sync/docker/README.md index 84aeaa9..e2e3817 100644 --- a/sync/README.md +++ b/sync/docker/README.md @@ -6,7 +6,7 @@ directory. > Prerequisites (license, Docker, GPU runtime, ECR auth) and the shared > license-and-usage-proxy (usage reporting, license status endpoint, proxy -> production recommendations) are documented in the [top-level README](../README.md). +> production recommendations) are documented in the [top-level README](../../README.md). The stack (`docker-compose.universal-3-5-pro.yml`) runs two containers — `sync-api` (GPU) and `license-and-usage-proxy` — with no nginx load balancer and no @@ -22,7 +22,7 @@ header returns `401`, so make sure your proxy doesn't strip it. ## Setup -Complete the [shared prerequisites](../README.md#prerequisites-all-services) +Complete the [shared prerequisites](../../README.md#prerequisites-all-services) (GPU runtime, ECR authentication, license file) first, then configure images: ```bash @@ -124,11 +124,11 @@ python transcribe_file.py path/to/audio.wav # or your own 16-bit PCM WAV ## Deploying on Modal (serverless GPU) This stack also runs on Modal's serverless GPUs as a self-contained, -single-`modal deploy` Modal App. See [`../sync_modal_stack/`](../sync_modal_stack/). +single-`modal deploy` Modal App. See [`../modal/`](../modal/). ## Production deployment recommendations -See the [top-level README](../README.md#production-recommendations-license-and-usage-proxy) +See the [top-level README](../../README.md#production-recommendations-license-and-usage-proxy) for the license-and-usage-proxy. ### sync-api service diff --git a/sync/docker-compose.universal-3-5-pro.yml b/sync/docker/docker-compose.universal-3-5-pro.yml similarity index 100% rename from sync/docker-compose.universal-3-5-pro.yml rename to sync/docker/docker-compose.universal-3-5-pro.yml diff --git a/sync/example/example_audio_file.wav b/sync/docker/example/example_audio_file.wav similarity index 100% rename from sync/example/example_audio_file.wav rename to sync/docker/example/example_audio_file.wav diff --git a/sync/example/requirements.txt b/sync/docker/example/requirements.txt similarity index 100% rename from sync/example/requirements.txt rename to sync/docker/example/requirements.txt diff --git a/sync/example/transcribe_file.py b/sync/docker/example/transcribe_file.py similarity index 100% rename from sync/example/transcribe_file.py rename to sync/docker/example/transcribe_file.py diff --git a/sync_modal_stack/README.md b/sync/modal/README.md similarity index 96% rename from sync_modal_stack/README.md rename to sync/modal/README.md index 4dd8033..f1931c3 100644 --- a/sync_modal_stack/README.md +++ b/sync/modal/README.md @@ -4,7 +4,7 @@ [Modal](https://modal.com) instead of a GPU box you manage. It is a self-contained Modal App: one `modal deploy` brings up both services and wires them together, and nothing depends on another deployment. Compose's two services -(see [`../sync/`](../sync/)) become two Modal Servers: +(see [`../docker/`](../docker/)) become two Modal Servers: | Compose service | Modal Server | Hardware | |---|---|---| @@ -71,7 +71,7 @@ curl -sS -o /dev/null -w '%{http_code}\n' \ https://--aai-sync-u3pro-syncapi..modal.direct/readyz # 503 while the model is cold, 200 once warm (Modal's edge may answer 303 first) -curl -F 'audio=@../sync/example/example_audio_file.wav;type=audio/wav' \ +curl -F 'audio=@../docker/example/example_audio_file.wav;type=audio/wav' \ -F 'config={"language_code":"en"};type=application/json' \ -H "Modal-Key: $MODAL_KEY" -H "Modal-Secret: $MODAL_SECRET" \ -H 'Authorization: any-non-empty-value' \ @@ -108,7 +108,7 @@ and word count; `--concurrency N` fires N at once for a quick load check. pip install requests python sample_sync.py \ --endpoint https://--aai-sync-u3pro-syncapi..modal.direct \ - --audio ../sync/example/example_audio_file.wav + --audio ../docker/example/example_audio_file.wav ``` If the stack was deployed with the default proxy auth, pass `--modal-key` / diff --git a/sync_modal_stack/modal_app.py b/sync/modal/modal_app.py similarity index 100% rename from sync_modal_stack/modal_app.py rename to sync/modal/modal_app.py diff --git a/sync_modal_stack/sample_sync.py b/sync/modal/sample_sync.py similarity index 98% rename from sync_modal_stack/sample_sync.py rename to sync/modal/sample_sync.py index e215be3..a56bedd 100644 --- a/sync_modal_stack/sample_sync.py +++ b/sync/modal/sample_sync.py @@ -3,7 +3,7 @@ pip install requests python sample_sync.py --endpoint https://--aai-sync-u3pro-syncapi..modal.direct \ - --audio ../sync/example/example_audio_file.wav + --audio ../docker/example/example_audio_file.wav Load test (fire N in parallel, watch throughput): python sample_sync.py --endpoint https://... --audio a.wav --concurrency 8