Run both self-hosted stacks on Modal, with a load-test harness

[alexkroman]

Adds Modal deployments for the sync and streaming stacks so they can run on
serverless GPUs instead of self-managed hardware, plus a harness that points at
either deployment with real audio and measures the concurrency it sustains.

Both were deployed and verified end to end on an L40S before this PR, then torn
down. Nothing here changes the existing compose stacks.

What's here

Path	Purpose
sync/modal_app.py	sync-api (L40S) + license-and-usage-proxy as two Modal functions
streaming/modal_app.py	streaming-api + license proxy as functions, ASR as a GPU Sandbox
bench/	load-test harness for either stack, local or Modal
README sections	deploy, verify, tear down, and the gotchas below

Verified

• sync — 60 s WAV transcribed with 152 word-level timestamps; ~2.0 s
  server-side (~30x realtime). Throughput plateaus near 33x realtime at
  concurrency 8; 401 on an empty Authorization header, as documented.
• streaming — real turns over WebSocket, first turn ~7 s. 40 concurrent
  realtime sessions with zero failures; first failures at 64, which were
  connection-level rather than GPU saturation (throughput was still climbing at
  96).

Four things needed to lift these images onto Modal

1. Clear the ENTRYPOINT. Modal prepends an image's ENTRYPOINT to its own
   runtime command, so the vendor binary otherwise swallows Modal's arguments,
   starts with default env, and the deployment code never runs. Every image uses
   .entrypoint([]).
2. Interpreter handling differs per image. The Wolfi proxy exposes python3
   and must not get add_python; the sync and streaming ASR images keep
   theirs inside Bazel runfiles and need one injected.
3. Install the Modal client into the image. Modal's runtime-mounted client
   dependencies do not resolve on these images' sys.path (symptom:
   ModuleNotFoundError: grpclib). The proxy bootstraps pip via ensurepip
   first, since Wolfi ships none.
4. The license travels as a secret written to disk at startup — Modal has no
   bind mounts.

Streaming specifics

nginx is dropped: it only routes X-Model-Version across several ASR backends,
and Modal's autoscaler covers the load-balancing half.

The ASR runs in a Sandbox rather than a Function because a Modal tunnel's
lifetime is bound to the function call — a web_server body returns as soon
as it has started its server, Modal tears the tunnel down, and the port stops
answering while the container stays up.

Please read before using streaming for real traffic

The ASR's gRPC port is exposed with unencrypted_ports, i.e. a plaintext
public TCP socket carrying audio, where compose keeps that hop on a private
bridge network. This is fine for testing and is called out in the README, but it
needs TLS or co-location first. Co-locating the API and ASR in one container is
currently blocked by colliding /opt/deps trees between the two Bazel images.

Also note the ASR Sandbox does not scale to zero — it holds an L40S until
stop_asr. The sync stack does scale to zero.

Checks

ruff check and ruff format --check pass on all five Python files, including
the two pre-existing examples. bench/README.md is markdownlint clean; the
pre-existing READMEs have 141 baseline violations (mostly MD013/MD060) that this
PR deliberately leaves alone.

🤖 Generated with Claude Code

[bgotthold-aai]

perhaps we should not include streaming here if we can not support it in a secure way.

[bgotthold-aai]

I think these already set in the .env

[aleks-mitov]

Extensive review, as discussed in Slack. Verdict up front: this is genuinely useful work. The Modal-lift mechanics are real and correctly diagnosed, the code is clean, and the docs are unusually honest about their own gaps. I do not think it should merge as-is, for three reasons: the security posture is broader than the one issue Ben flagged, the headline streaming benchmark conclusion does not survive contact with the ASR's capacity cap, and two lifecycle cliffs (a 24-hour sandbox hard-stop, a 1-hour WebSocket ceiling) are undocumented. Everything is fixable, and most of it is small.

Method note: I verified the load-bearing claims against Modal's documentation and against the services' actual behavior, and re-ran the lint checks locally, so the inline comments state what the code does rather than what it looks like it does.

Suggested merge gate (everything else can be follow-up)

1. Settle the streaming security question with data. One live test decides whether encrypted_ports + AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True closes the plaintext hop (the client side already supports it; details inline). If it passes, streaming can ship with TLS and an honest reachability caveat; if not, Ben's suggestion to hold streaming back is right.
2. Close or clearly gate the public license proxy. It is publicly reachable and unauthenticated in BOTH stacks, and its usage-recording route is state-mutating, so dropping streaming alone would not close the most consequential exposure. Co-locating the proxy in the API container is the clean fix.
3. Ship requires_proxy_auth=True on the API endpoints by default, with the README caveat as the opt-out rather than the control.
4. Fix the env-override ordering so customer configuration wins over the hardcoded literals; today it silently regresses #10.
5. Correct the benchmark conclusion (MAX_OPEN_STREAMS=32 is a hard cap masked by connect retries) and the autoscaler claim in the streaming docstring.
6. Document the two lifecycle cliffs: the sandbox's 24h stop and the 1h WebSocket session ceiling; raise the latter's timeout.
7. Fate-share the vendor processes with their containers so a post-startup crash does not black-hole traffic.
8. Harden the harness's three failure modes (stereo input, missing session deadline, writer-thread stall) before it becomes the tool customers size deployments with.

What checked out

Worth recording so review energy goes where it matters: ruff check and ruff format --check pass as claimed (re-ran locally); dropping nginx is routing-safe with a single backend; the Sandbox-not-Function tunnel rationale matches Modal's docs; .entrypoint([]) is genuinely required; every env var name used in both Modal apps is a real configuration field; the "restarting the ASR invalidates the API" gotcha is real and the documented remedy is correct; the bench README's options table matches the argparse definitions exactly; and .gitignore covers .env, license.jwt, and *.jwt.

29 inline comments carry the details, each tagged [major]/[minor]/[nit] with a concrete fix.

[aleks-mitov]

[major] The Sandbox hard-stops after 24 hours and nothing detects it.

timeout=24 * 60 * 60 is the sandbox's maximum lifetime (and Modal's platform max), after which Modal terminates it, tunnel included. Nothing cleans the modal.Dict when that happens: address and sandbox_id stay populated, so warm streaming_api containers keep dialing the dead tunnel, and cold-started ones pass the if not address guard and boot "successfully" against a corpse. A customer returning after a day sees a green deployment that fails every session (close code 3005 after the API exhausts its connect retries), with no documented cause. Note this is a different failure mode from the Missing expected server metadata keys log the README's Restarting section quotes (that one fires when the endpoint answers but is not the ASR).

Suggested fix: (1) document the 24h lifetime in Deploy/Tear down and cross-link it from the Restarting section as the most likely trigger; (2) in streaming_api, also read sandbox_id and fail fast at startup when modal.Sandbox.from_id(sandbox_id).poll() is not None, raising the same "run start_asr" error; (3) optionally, a background thread in streaming_api that exits the container on sandbox death or address change, which would also remove the manual stop-and-redeploy step the README requires after every start_asr.

[aleks-mitov]

[major] Concrete path to resolving Ben's plaintext concern, and the half TLS does not fix.

Confidentiality half: swap unencrypted_ports for encrypted_ports and set AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True. I checked the client side: when that flag is true the streaming API dials the ASR with standard TLS credentials using the default public trust roots, so no vendor change is needed if Modal's TLS tunnel presents a publicly trusted cert on the tunnel hostname. Modal documents that tunnels "terminate TLS automatically" but does not document the CA or whether the socket negotiates ALPN h2 (gRPC requires it), and the comment here says encrypted was never re-tested. One live test settles it: openssl s_client -alpn h2 against the tunnel, then a health check over a secure channel. Worth running before deciding whether streaming ships.

Reachability half: TLS does not authenticate clients. The tunnel address is on the public internet, so anyone who finds it gets free GPU inference on the customer's bill, and since MAX_OPEN_STREAMS=32 is a hard cap, 32 attacker-held streams starve every legitimate session. Complete fix is co-locating the API and ASR in one container so the hop stays on localhost (the Security section already names this; I would promote it to the recommendation), or holding the streaming recipe back until that works.

[aleks-mitov]

[major] timeout=3600 caps every WebSocket session at one hour.

Modal treats each WebSocket connection as a single function call ("WebSockets on Modal maintain a single function call per connection"), and the function-level timeout bounds each call. So a realtime session open past 1h is terminated mid-stream. The compose stack never severs sessions (nginx is configured with 10h timeouts). The load tests used 15-20s sessions, so this never surfaced.

Suggested fix: raise timeout here toward Modal's 24h max and document the resulting hard per-session ceiling in the README next to the security caveats (clients must reconnect). Also worth one line on why sync_api keeps 3600 (its requests are bounded by INFERENCE_TIMEOUT_SECONDS=30 anyway).

[aleks-mitov]

[major] All four vendor binaries run unsupervised; a post-startup crash black-holes traffic.

Every @modal.web_server body here (both license_proxy functions, streaming_api, sync_api) does a bare subprocess.Popen and returns. Modal only verifies the port opens once at container start; there is no ongoing health check. If the binary exits later (the license proxy is the sharpest case, since the APIs shut themselves down on license failure, and the GPU process can OOM), the container stays alive, keeps counting toward autoscaling capacity, and requests routed to it fail until it happens to scale down. The compose stacks have healthchecks on all four services that catch exactly this.

Suggested fix at all four launch sites: keep the handle and fate-share, e.g.

proc = subprocess.Popen(...)
threading.Thread(target=lambda: (proc.wait(), os._exit(proc.returncode or 1)), daemon=True).start()

so a dead server kills the container and Modal replaces it. This also turns crash-after-bind into a visible restart loop instead of a silent black hole.

[aleks-mitov]

[major] "Modal's autoscaler covers the second job" is not true for the component nginx actually balanced.

The ASR is a single Sandbox pinned to one L40S, and Sandboxes do not autoscale; only the CPU streaming_api/license_proxy functions scale. The architecture also cannot grow ASR replicas without reintroducing routing (the modal.Dict holds exactly one address). The X-Model-Version half checks out (the header is routing metadata a single backend safely ignores), but a reader sizing for more concurrent streams and trusting this sentence is misled.

Suggested fix: correct the docstring (autoscaling applies only to the CPU functions) and add one README sentence: this deployment is capped at a single GPU backend; for more capacity run multiple deployments or use compose with nginx.

[aleks-mitov]

[major] No read deadline: a live-but-silent server hangs the whole sweep.

for message in ws blocks without a timeout and nothing bounds a session's total duration. The websockets keepalive only unblocks a dead TCP connection; a server that accepts the upgrade and audio but never sends Termination (exactly the overload regime this harness exists to probe, and the PR itself reports connection-level failures at 64) leaves the thread waiting forever. run_level's f.result() has no timeout either, so one stuck session hangs the entire ramp: no table, no exit, at precisely the saturation point the tool is meant to find.

Suggested fix: compute a per-session deadline (e.g. audio_seconds / speed + open_timeout + margin), replace the iterator with ws.recv(timeout=remaining) returning Result(False, ..., detail="session deadline exceeded") on TimeoutError, and optionally pass a timeout to f.result() as a backstop.

[aleks-mitov]

[minor] Off-by-one rank: p95 equals max for every level with n <= 20.

Nearest-rank quantile index is ceil(n * p) - 1; int(n * p) is one rank high, so with n <= 20 the printed p95 is always exactly the max (which covers most levels in the documented ramps, and every ramp table in this PR shows it), and with n = 2 the p50 reports the slower of the two as the median. idx = max(0, math.ceil(len(lat) * p) - 1) fixes it.

[aleks-mitov]

[minor] For streaming, this metric is tautological: it measures the client's own pacing.

Streaming sessions are realtime-paced by the writer, so wall ~= audio_seconds / speed and audio_x_realtime ~= ok * speed regardless of server capacity. Meanwhile the number that does reflect server health under load, per-session first_turn_s (already collected in Result.extra), is never aggregated; only the first ok result's extra is printed once. Suggest omitting xRT in streaming mode (print "-") or replacing it with first_turn_s percentiles, and scoping the README's saturation guidance ("throughput plateauing while latency climbs") to sync.

[aleks-mitov]

[nit] --max-seconds can truncate the audio before the --expect word occurs, turning every request into a false failure that reads like a server problem (the README recommends --max-seconds 20 while --expect defaults to "assemblyai"). A note in the bench README ("pick --expect from the first N seconds when truncating"), or a startup warning when both flags are set with the default expect, avoids the trap.

[aleks-mitov]

[nit] Only true for sync. In streaming mode the summary carries first_turn_s and turn count, not words; the character count is what both modes print. (The rest of this README verified clean against the script: the options table matches the argparse definitions exactly, including defaults, and the exit-status claim matches main().)