Standalone single-deploy Modal packages for each stack by aleks-mitov · Pull Request #12 · AssemblyAI/streaming-self-hosting-stack

aleks-mitov · 2026-08-26T12:49:21Z

Runs each self-hosted stack on Modal's serverless GPUs. Each stack is a self-contained Modal App deployed with a single modal deploy: the app defines its servers, and the API resolves its ASR and license-proxy URLs from the same App at startup, so one command brings up and wires the whole stack.

Three self-contained packages

File	App	Servers
`sync_modal_stack/modal_app.py`	`aai-sync-u3pro`	`SyncApi` (L40S), `LicenseProxy` (CPU)
`streaming_modal_stack/modal_app_universal_3_5_pro.py`	`aai-streaming-u3pro`	`StreamingApi` (CPU), `Asr` (L40S), `LicenseProxy` (CPU)
`streaming_modal_stack/modal_app_english_multilang.py`	`aai-streaming-english-multilang`	`StreamingApi` (CPU), `Lb` (nginx), `AsrEnglish` (L40S), `AsrMultilang` (L40S), `LicenseProxy` (CPU)

Each directory also ships a README and a sample client (sample_sync.py / sample_streaming.py) that prints live output.

Design

Modal Servers (@app.server). The ASR runs as an autoscaling Server (min_containers=1, target_concurrency set to MAX_OPEN_STREAMS), health-gated on grpc_health_probe; WebSocket sessions are unbounded.
Single-command deploy. modal.Server.from_name(...).get_url() resolves the ASR and proxy URLs from the same App at container start, so no address has to be wired in by hand.
Encrypted ASR hop. h2_enabled=True makes Modal's TLS edge advertise ALPN h2, so the API's default-TLS gRPC client connects with AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True. (encrypted_ports alone negotiates no ALPN and fails with missing selected ALPN property; h2_ports / h2_enabled is required.)
Secure by default. StreamingApi / SyncApi ship unauthenticated=False (Modal proxy auth, enforced on the WebSocket upgrade); AAI_REQUIRE_MODAL_AUTH=0 opts into a throwaway public test endpoint.
Supervised binaries. Each server fate-shares its vendor process with the container, so a post-startup crash triggers a Modal replacement rather than a silent black hole.
Operator-configurable audio limits. The sync caps (MAX_AUDIO_DURATION_MS, etc.) read from the environment with the compose defaults as fallback.
Model routing. The single-model u3pro stack needs no router; the english/multilang stack keeps an nginx Lb that routes the x-model-version gRPC metadata (en-default / ml-default) to the two backends.
Image tags. streaming-api and the Universal-3.5 Pro ASR are pinned to release-v1.0.1 (the API image logs peer-aborted handshakes at WARNING rather than ERROR); the license-and-usage-proxy stays on release-v1.0.0.

Security note

Modal has no private inter-container network by default, so the internal hops (StreamingApi to the ASR / Lb, and to LicenseProxy) cross Modal's public TLS edge rather than a private bridge. The gRPC hop is encrypted, but the backends and proxy are unauthenticated=True because the API dials them server-side and cannot attach Modal auth headers: their URLs are unguessable but public. Suitable for evaluation; co-locate the API and ASR in one container, or use Modal's i6pn private network, before exposing the backends to untrusted traffic.

Validated live (Modal, L40S)

Deployed from scratch and torn down:

sync: 60s WAV to 152 word-level timestamps, ~2.0s server-side.
streaming u3pro: real turns over the TLS h2 ASR hop; first turn ~1.4s warm.
streaming english + multilang: both universal-streaming-english and universal-streaming-multilingual return real turns, each routed by the nginx Lb to its own L40S backend.
License gate: both APIs log Proxy is connected and license is valid and SIGKILL on failure; an invalid license makes the proxy exit 1 at startup.
Usage tracking: every stack POSTs to usage-tracker.assemblyai.com/v1/usage with status_code=200.
Handshake logging: on the v1.0.1 API, opening handshake failed (from L4 health probes and aborted clients) logs at WARNING, not ERROR.

🤖 Generated with Claude Code

Repackage the Modal deployment so each self-hosted stack is a self-contained Modal App deployed with one `modal deploy` — no start_asr step, no modal.Dict address handshake, no cross-package dependency. Uses Modal Servers (@app.server) with sibling URLs resolved via Server.from_name at startup. - sync/modal_app.py -> aai-sync-u3pro - streaming/modal_app_universal_3_5_pro.py -> aai-streaming-u3pro - streaming/modal_app_english_multilang.py -> aai-streaming-english-multilang Fixes carried over from the #11 review: autoscaling ASR Server instead of a 24h-capped Sandbox; no 1h WebSocket session ceiling; encrypted ASR hop via h2_enabled + AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE (encrypted_ports alone negotiates no ALPN); unauthenticated=False by default (Modal proxy auth); fate-shared vendor processes; env-overridable sync audio limits. Verified end to end on L40S (single deploy each): sync 152-word transcript, autoscaled to 181x realtime at concurrency 8; streaming u3pro real turns over the TLS h2 ASR hop; english+multilang routing both models through the nginx Lb. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

sample_sync.py (POST /transcribe), sample_streaming.py (live realtime turns, model selection for the u3pro and english/multilang stacks), and a README. Both support --concurrency / --load for a quick input-load sweep and Modal proxy-auth headers. Verified live against all three deployed stacks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

The turn counter was appended only inside the live-printing branch, so --load sessions always reported 0 turns even though turns arrived (first-turn latency was correct). Track end_of_turn regardless of live; gate only printing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

…eaming_modal_stack/ Consolidate each stack's Modal artifacts into a self-contained top-level dir: sync_modal_stack/ modal_app.py, sample_sync.py, README.md streaming_modal_stack/ modal_app_universal_3_5_pro.py, modal_app_english_multilang.py, sample_streaming.py, README.md Each new dir carries its own deploy/verify/auth/teardown README. The compose READMEs (sync/, streaming/) keep a one-line pointer instead of the full Modal section, and the root README points at the two new dirs. No code behavior changes; comment cross-references updated to the new paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

The self-hosted-streaming-api v1.0.1 image carries the handshake-logging fix (DeepLearning #19523: peer-aborted WebSocket handshakes log at WARNING, not ERROR). Bump streaming-api and self-hosted-streaming-asr-universal-3-5-pro to release-v1.0.1 in the u3pro stack via per-image tags; the license-and-usage-proxy has no v1.0.1 and stays on v1.0.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

aleks-mitov · 2026-08-26T18:50:48Z

Fresh deploy + verification (v1.0.1 bump)

Tore down all three apps and redeployed from scratch off this branch in the dlange Modal workspace (L40S), then verified:

sync — 152-word transcript, ~2.0s server-side, 25.6x realtime.
streaming english/multilingual — both universal-streaming-english (8 turns) and universal-streaming-multilingual (14 turns) stream correctly, each routed by the nginx Lb to its own backend.
streaming u3pro (v1.0.1 images) — self-hosted-streaming-api + self-hosted-streaming-asr-universal-3-5-pro pulled at release-v1.0.1; streams end to end (12 turns, first turn ~1.5s).
Handshake log-noise fix confirmed in the v1.0.1 API: after firing peer-abort probes, all 42 opening handshake failed lines on the deployed streaming_api are [warning], zero [error] — i.e. DeepLearning #19523 (commit 415137e) is live in the v1.0.1 image.

Deploys used the reorganized layout (sync_modal_stack/, streaming_modal_stack/) and the relocated sample scripts. All apps left running for now.

Note: the english/multilang stack still uses self-hosted-streaming-api:release-v1.0.0, so it does not yet carry the handshake fix — bump it to v1.0.1 too if desired (only the u3pro stack was requested).

Adversarial review of PR #12 (8 findings). Fixes: - [major] english/multilang ASR MAX_OPEN_STREAMS + target_concurrency 32 -> 48 (compose parity; avoids a ~50% GPU over-provision at scale). - [major] streaming README Verify pointed at the bundled example client, which cannot send Modal proxy-auth headers and 401s on the default deploy; point it at sample_streaming.py with --modal-key/--modal-secret and note the caveat. - [minor] sync/streaming README Verify curls now send Modal-Key/Modal-Secret (they 401'd on the default proxy-auth deploy). - [minor] nginx Lb -> ASR hop now verifies the backend cert (grpc_ssl_verify on + ca-certificates trusted store), not just encrypts. - [minor] fate-share reaper no longer reports a clean shutdown as a crash: a module-level _stopping event, set by @modal.exit stop(), distinguishes an intentional teardown from an unexpected vendor exit (all three apps). - [minor] sample_streaming.py no longer crashes formatting first_turn_s when a session yields no word-bearing turns (prints n/a). - [minor/question] english/multilang now pins streaming-api via its own API_TAG. streaming-api bumped to release-v1.0.1 everywhere it is referenced (u3pro + english/multilang Modal stacks, streaming/.env.example, root README) so both streaming stacks carry the handshake-logging fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

…only for verification) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

…ectivity) Verifying the backend cert (finding 4) fails against Modal's edge: nginx cannot build the chain and errors "unable to get local issuer certificate", marking the ASR backend down so every session 3005s (confirmed live). Revert to encrypt-only (grpc_ssl_server_name on) and document the residual limitation in-place, the finding's sanctioned alternative. The backend is unauthenticated by design; co-location or i6pn is the real fix, per the README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

aleks-mitov · 2026-08-26T19:23:20Z

Adversarial review fixes + streaming-api v1.0.1 everywhere — applied & verified

Ran an adversarial review (14 raw findings → 3 refuted → 8 confirmed) and addressed all of them:

[major] english/multilang ASR concurrency raised 32 → 48 (MAX_OPEN_STREAMS + target_concurrency) to match compose; avoids a ~50% GPU over-provision.
[major] streaming README Verify now uses sample_streaming.py with --modal-key/--modal-secret (the bundled example client can't send Modal proxy auth and 401s on the default deploy).
[minor] README Verify curls now send Modal-Key/Modal-Secret.
[minor] fate-share reaper no longer reports a clean shutdown as a crash — a _stopping event set by @modal.exit distinguishes intentional teardown from an unexpected vendor exit (all three apps).
[minor] sample_streaming.py prints n/a instead of crashing when a session yields no turns.
[minor/question] english/multilang now pins streaming-api via its own API_TAG.
streaming-api → release-v1.0.1 everywhere (both Modal streaming stacks, streaming/.env.example, root README) so both stacks carry the handshake-logging fix.

[minor] LB→ASR cert verification (grpc_ssl_verify): attempted, then reverted. Turning it on broke the hop — nginx can't build a chain to Modal's edge cert (unable to get local issuer certificate), marking the ASR backend down so every session 3005d (caught in live verification). Reverted to encrypt-only and documented the residual limitation in-place (the backend is unauthenticated by design; co-location or i6pn is the real fix).

Re-verified from scratch (dlange workspace, L40S): sync (26.9x realtime), u3pro v1.0.1 (12 turns), english + multilingual with the 48-cap and encrypt-only LB (8 / 14 turns). Handshake fix confirmed on both v1.0.1 streaming APIs — u3pro 42/42 and english/multilang 33/33 opening handshake failed lines at [warning], zero [error].

bgotthold-aai · 2026-08-26T20:35:31Z

nice! From organization perspective can we add folders like streaming/modal streaming/docker streaming/sagemaker etc? I can see us adding more variations here and I dont think top level makes sense

Per review feedback, nest deployment variants under each service dir instead of top-level *_modal_stack dirs, so more targets (e.g. sagemaker) slot in cleanly: sync/docker/ (compose stack, moved from sync/) sync/modal/ (moved from sync_modal_stack/) streaming/docker/ (compose stack + nginx, moved from streaming/) streaming/modal/ (moved from streaming_modal_stack/) Relative links and paths updated throughout (root README layout + pointers, docker READMEs' ../README.md -> ../../README.md and modal pointers, modal READMEs' example paths -> ../docker/example, sample-script docstrings, and the cross-file comment references). No code behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

aleks-mitov · 2026-08-26T21:39:29Z

Good call — done. Reorganized so each service groups its deployment targets, and the top-level *_modal_stack/ dirs are gone:

sync/
  docker/   # compose stack (moved from sync/)
  modal/    # serverless-GPU package (was sync_modal_stack/)
streaming/
  docker/   # compose stack + nginx (moved from streaming/)
  modal/    # serverless-GPU packages (was streaming_modal_stack/)

I moved the existing compose stacks into docker/ too (not just the new modal/ dirs), so a future sagemaker/ etc. slots in the same way. Relative links/paths were updated throughout (root README layout + pointers, the docker READMEs' ../README.md → ../../README.md, the modal READMEs' example paths → ../docker/example, and the sample-script docstrings); READMEs' links all resolve and the stacks still deploy + verify clean.

aikido-pr-checks · 2026-08-26T21:39:41Z

+curl -F 'audio=@../docker/example/example_audio_file.wav;type=audio/wav' \
+  -F 'config={"language_code":"en"};type=application/json' \
+  -H "Modal-Key: $MODAL_KEY" -H "Modal-Secret: $MODAL_SECRET" \
+  -H 'Authorization: any-non-empty-value' \


Exposed secret in sync/modal/README.md - low severity
Discovered a potential authorization token provided in a curl command header, which could compromise the curl accessed resource.

_{Reply @AikidoSec ignore: [REASON] to ignore this issue.}
_{More Info}

bgotthold-aai

Approving this because I think it looks good overall.

Should we still add a .env for model to get the registry and tag out of code?