Summary
A single misconfigured stdio MCP server can drive the host into a process-spawn meltdown. On my Mac mini this produced load average 143 with ~76 concurrent child processes and file-descriptor exhaustion (Too many open files (os error 24)).
Environment
- mcplex v0.3.0, macOS (Apple Silicon), launchd-managed (
com.mcplex.gateway.local)
- Affected backend: a
telegram stdio server (mcp-telegram start) that was never configured (missing api_id/api_hash, no session file)
Root cause
dead_server_monitor in src/main.rs respawns dead stdio servers with exponential backoff, capped at MAX_RESPAWN_ATTEMPTS = 5. That cap only protects against connect() failing repeatedly within one respawn task.
It does NOT protect against the crash-after-successful-connect loop:
StdioConnection::connect() succeeds — the child process spawns and completes the MCP handshake.- The child crashes seconds later (in our case
mcp-telegram raises RuntimeError: Client not created! inside app_lifespan because no Telegram credentials exist).
- The watchdog emits a fresh death event on
death_rx.
dead_server_monitor receives it and tokio::spawns another respawn task — with its own fresh 1..=5 counter.- Goto 1, forever.
Because each crash also produces a fresh respawn task, and connect() partially succeeds (the wrapper with-secrets.sh → age → python child has already forked) before some attempts fail, processes pile up faster than they exit → FD exhaustion → os error 24 → host meltdown.
There is no memory of "this server has already died N times in the last minute" across respawn cycles, and concurrent death events can spawn concurrent respawn tasks for the same server.
Impact
One broken/misconfigured backend takes down the whole gateway host. Severity is high because the failure mode is silent (mcplex log keeps writing Respawn attempt 1/5 cheerfully) and sustained (hours, until manually killed).
Suggested fix
Add a cross-cycle circuit breaker in dead_server_monitor:
- Track death timestamps per server (local
HashMap<String, Vec<Instant>> — the monitor while loop is single-consumer, no lock needed).
- If a server dies more than
MAX_CRASHES_IN_WINDOW times within CRASH_WINDOW, mark the circuit open: log an error and stop respawning that server entirely.
- This bounds the blast radius: a crash-looping server is abandoned after a few seconds instead of running forever.
Optionally also: ensure only one in-flight respawn task per server, and kill any leaked child before respawn.
I have a patch implementing the circuit breaker and will open a PR.
Workaround
Comment out the offending [[servers]] block in the config and restart mcplex. Load dropped from 143 to 2.
Summary
A single misconfigured stdio MCP server can drive the host into a process-spawn meltdown. On my Mac mini this produced load average 143 with ~76 concurrent child processes and file-descriptor exhaustion (
Too many open files (os error 24)).Environment
com.mcplex.gateway.local)telegramstdio server (mcp-telegram start) that was never configured (missingapi_id/api_hash, no session file)Root cause
dead_server_monitorinsrc/main.rsrespawns dead stdio servers with exponential backoff, capped atMAX_RESPAWN_ATTEMPTS = 5. That cap only protects againstconnect()failing repeatedly within one respawn task.It does NOT protect against the crash-after-successful-connect loop:
StdioConnection::connect()succeeds — the child process spawns and completes the MCP handshake.mcp-telegramraisesRuntimeError: Client not created!insideapp_lifespanbecause no Telegram credentials exist).death_rx.dead_server_monitorreceives it andtokio::spawns another respawn task — with its own fresh1..=5counter.Because each crash also produces a fresh respawn task, and
connect()partially succeeds (the wrapperwith-secrets.sh→age→ python child has already forked) before some attempts fail, processes pile up faster than they exit → FD exhaustion →os error 24→ host meltdown.There is no memory of "this server has already died N times in the last minute" across respawn cycles, and concurrent death events can spawn concurrent respawn tasks for the same server.
Impact
One broken/misconfigured backend takes down the whole gateway host. Severity is high because the failure mode is silent (mcplex log keeps writing
Respawn attempt 1/5cheerfully) and sustained (hours, until manually killed).Suggested fix
Add a cross-cycle circuit breaker in
dead_server_monitor:HashMap<String, Vec<Instant>>— the monitorwhileloop is single-consumer, no lock needed).MAX_CRASHES_IN_WINDOWtimes withinCRASH_WINDOW, mark the circuit open: log an error and stop respawning that server entirely.Optionally also: ensure only one in-flight respawn task per server, and kill any leaked child before respawn.
I have a patch implementing the circuit breaker and will open a PR.
Workaround
Comment out the offending
[[servers]]block in the config and restart mcplex. Load dropped from 143 to 2.