(fix) require authentication on the MQTT broker and stop exposing it on 0.0.0.0

[cardosofede]

Problem

The EMQX broker was accepting anonymous connections, published on 0.0.0.0.

emqx ctl conf show authentication returned authentication = []. EMQX ships that way and nothing in the compose file overrode it, so the BROKER_USERNAME / BROKER_PASSWORD pair in .env was never actually enforced — it was only ever passed by our own clients, never checked by the broker.

Anyone able to reach port 1883 could:

1. subscribe hbot/# — read every bot's logs and performance, and enumerate instance IDs
2. publish hbot/<instance_id>/stop — halt a live trading bot

No credential required at any step. Seven ports were published on 0.0.0.0 (1883, 8883, 8083, 8084, 8081, 18083, 61613), and with the Tailscale overlay enabled that means reachable from the whole tailnet. The dashboard on 18083 still had the well-known admin/public default.

Changes

Four commits, ordered so no intermediate state breaks a running deployment:

Commit	Change
1b6a1f5	Ports 7 → 2, both bound to 127.0.0.1
3f52c3a	Bot instances send broker credentials (no-op while the broker is still anonymous)
c324687	Broker requires authentication
1b2fde4	Docs

Ports. Bot containers run with network_mode: host and reach the broker at 127.0.0.1:1883; hummingbot-api reaches it in-network as emqx:1883. Neither path ever needed a routable interface. Only 1883 and the dashboard on 18083 remain, both on loopback — conf_client.yml sets mqtt_ssl: false and the API speaks plain TCP MQTT, so the SSL/WS/management/STOMP listeners were pure attack surface.

Authentication. password_based auth against EMQX's built-in database, seeded from a bootstrap file that make emqx-auth generates out of .env before every run/deploy. The file holds the password in plaintext, so it is written 0600 and gitignored.

Bot side. The credential templates under bots/credentials/ ship with an empty mqtt_username/mqtt_password, which the broker now rejects — so create_hummingbot_instance seeds each instance's conf_client.yml from the API's own broker credentials. mqtt_host is deliberately left alone: the instance container is on the host network and must use the host loopback, not the compose DNS name the API uses.

Weak defaults. The dashboard password now comes from BROKER_PASSWORD instead of admin/public, and setup.sh generates a 32-character broker password on fresh installs rather than writing the literal string password.

Verification

Run against a live broker, not inferred:

Check	Before	After
anonymous subscribe hbot/#	accepted	Connection Refused: bad user name or password
anonymous publish .../stop	accepted	refused
credentialed connect	n/a	connects; API reports mqtt_connected: true
docker port	7 × 0.0.0.0	127.0.0.1:1883, 127.0.0.1:18083
dashboard admin/public	logs in	Auth failed

Upgrade note for existing deployments

EMQX imports the bootstrap file only for accounts it does not already have, so editing BROKER_PASSWORD in .env does nothing on its own. make emqx-auth-reset drops the EMQX state volume and re-seeds. Existing deployments running the placeholder password should rotate.

EMQX_DASHBOARD__DEFAULT_PASSWORD likewise only applies on first init — an existing volume keeps admin/public until the state volume is recreated.

Any already-deployed bot instances have empty MQTT credentials baked into their conf_client.yml and will fail to reconnect after this change; they need to be redeployed to pick up credentials.

Follow-up (not in this PR)

This hardens the broker rather than questioning it. Separately worth discussing: on a single-host deployment the broker has exactly one subscriber (the API) and all publishers are containers on the same machine, and four of its eight subscribed channels land in empty handlers. Retiring EMQX in favour of an embedded broker, then the direct channels the repo already uses elsewhere, is written up separately.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DUBD1gt2FbshzLd1pevxuy

[cardosofede]

Follow-up: an incident review found this broker was actively abused

A read-only forensic review of one deployed server found the broker had been reached from the internet and exploited — ~112 MB of live bot traffic (hbot/#: orders, fills, balances, strategy parameters) exfiltrated across five sessions from a single IP, plus a rule-engine backdoor installed 2026-06-16 that makes EMQX issue GET http://hummingbot-api:8000/accounts/ with a hardcoded Authorization: Basic credential, triggered by publishing to ssrf/probe.

The original three commits hardened the broker but left three real gaps. Four more commits close them.

e98ffdd — the API and Postgres were still on 0.0.0.0

The first pass hardened emqx and walked past the two services beside it in the same file. hummingbot-api holds exchange credentials and can move funds, and it was being scanned externally. Both now default to 127.0.0.1, with API_BIND/DB_BIND as deliberate escape hatches.

API_BIND matters for Tailscale: make deploy does not run tailscale serve, so MagicDNS reaches port 8000 on the host interface directly and a hard loopback bind would break it. Set API_BIND=<tailscale-ip> there — a specific interface, not 0.0.0.0.

228a8a4 — authentication without authorization

The first pass answered who may connect and left what may they do entirely stock: no_match = allow with an acl.conf ending in {allow, all}. Stock EMQX denies a bare # but not hbot/# — which is exactly the topic that was exfiltrated. EMQX's own shipped acl.conf documents this as the thing to change before production.

Now deny-by-default, granting only hbot/# and hummingbot-api/response/#. The bootstrap account is is_superuser=false; superusers bypass authorization entirely, which would have made the ACL dead config. The stock {allow, {ipaddr, "127.0.0.1"}, all, ["#"]} rule is dropped — every client here arrives over loopback or the compose bridge, so it granted the firehose to everyone.

Verified against a running broker:

pub/sub hbot/#, hummingbot-api/response/#	allowed — API reconnects and serves normally
#, +/#, $SYS/#	denied
arbitrary topics	denied
ssrf/probe	denied — the backdoor's trigger topic no longer fires

cd350f8 — make emqx-audit

The broker was hardened without first checking whether something was already living in it. This prints listeners, published ports, auth, authorization, the ACL, and any rules/actions/connectors/bridges in full, including target URLs and Authorization headers.

Verified in both directions: a clean broker reports none; a broker with a replica of the reported backdoor prints the credential, target and trigger under a REVIEW banner. On a stock broker it also surfaces authentication = [] and the 0.0.0.0 ssl/ws listeners. EMQX_CONTAINER=<name> audits another deployment.

Worth noting how cheap the backdoor was to install in testing: three API calls, authenticated with nothing but admin/public.

Note on the firewall approach

The review recommends moving port rules from INPUT to DOCKER-USER, since published container ports traverse FORWARD/DOCKER-USER and INPUT rules silently do nothing. Correct — but this PR sidesteps it for these ports by unpublishing them rather than filtering them. Docker never creates a DNAT rule for a port it doesn't publish, so there is no chain to get wrong. Unpublish beats firewall wherever the port isn't genuinely needed.

Still outside this repo

Deleting the live backdoor (rule 4715e59c / connector hbapi_ssrf), the host firewall, CUPS, authorized_keys, and exchange key rotation are all server-side. Nothing in this PR touches the compromised host.

[greptile-apps]

Greptile Summary

The PR secures EMQX by restricting published ports, requiring broker authentication, applying deny-by-default topic authorization, and provisioning credentials for managed bots. It also separates dashboard credentials, adds reset and audit operations, strengthens fresh-install defaults, and updates deployment diagnostics and documentation.

• Binds EMQX, API, and PostgreSQL ports to loopback by default.
• Generates and imports broker credentials while configuring bot instances to authenticate.
• Adds an EMQX ACL limiting the shared client account to required Hummingbot topics.
• Adds broker credential reset and security-audit Make targets.
• Updates setup, Tailscale diagnostics, and operator documentation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; both previously reported issues are addressed by the current setup pipeline handling and bootstrap credential validation.

Important Files Changed

Filename	Overview
Makefile	Generates the validated EMQX bootstrap file and adds broker reset and audit operations; the previously reported CSV field-corruption path is addressed.
setup.sh	Generates separate alphanumeric MQTT and dashboard credentials without allowing the expected pipe SIGPIPE to abort setup; the previously reported setup failure is addressed.
docker-compose.yml	Enables EMQX authentication and authorization, mounts bootstrap and ACL files, separates dashboard credentials, and narrows published service bindings.
emqx/acl.conf	Introduces deny-by-default authorization while allowing the shared account only on required Hummingbot topic prefixes.
services/docker_service.py	Seeds managed bot MQTT configurations with the broker credentials required by the new authentication boundary.
doctor.sh	Updates bind diagnostics for the renamed API setting and recognizes direct Tailscale-IP binding.
README.md	Documents the secured broker topology, credential separation, rotation procedure, audit command, and loopback defaults.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Env[".env credentials"] --> Bootstrap["make emqx-auth"]
  Bootstrap --> CSV["EMQX bootstrap CSV"]
  CSV --> Broker["EMQX broker"]
  Env --> API["Hummingbot API"]
  API --> BotConfig["Managed bot conf_client.yml"]
  BotConfig --> Bots["Bot containers"]
  API -->|"Authenticated MQTT"| Broker
  Bots -->|"Authenticated MQTT via loopback"| Broker
  ACL["Deny-by-default ACL"] --> Broker

Loading

_{Reviews (2): Last reviewed commit: "(fix) require authentication on the MQTT..." | Re-trigger Greptile}