Skip to content

Dashboard stops xmrig-proxy when the node goes unreachable but never restarts it — a transient blip kills mining permanently #875

Description

@VijitSingh97

Found on the bench appliance immediately after a routine A/B update (develop-v2 @ 4dd3ddb). Mining was dead on arrival and stayed dead.

What happens

During the post-update boot, monerod restarts a few seconds after the other containers. The dashboard's supervisor sees the node unreachable and deliberately stops the proxy:

02:57:16 [WARNING] Required node unreachable — stopping xmrig-proxy so workers fail over to their backup pools.
02:57:17 [INFO] Container xmrig-proxy stop: ok (HTTP 204)

That stop is correct and intentional (workers fail over to their backup pools rather than hashing into a void). The bug is that nothing ever starts it again. monerod came back Up (healthy) at 100% sync within seconds; five minutes later:

xmrig-proxy   Exited (0) 3 minutes ago
[ERROR] Proxy Data Fetch Error: ... 172.28.0.29:3344 ... Max retries exceeded   (repeating)

and the local miner is in a connect-refused loop against 127.0.0.1:3333. The stack looks healthy — 7 containers up, 0 failed units, dashboard serving, node synced — while earning nothing. pithead up restores it.

Why it matters on the appliance

The stop path is designed for a transient condition, but the recovery is manual — and on a shell-less release appliance the operator has no pithead up. Any monerod restart (an update, an OOM-restart, a container recreate on config change) can silently end mining until someone intervenes. It also survives the health-gated commit: the slot commits because the dashboard serves and the revenue containers that ARE running are healthy — the stopped proxy reads as the deliberate sync-hold state.

Fix direction

Whatever stops the proxy must own restarting it: re-check reachability on the supervisor's normal cadence and start the proxy again once the required node is healthy (the same edge that stopped it, inverted). Worth auditing the sibling case — p2pool under the same supervisor — for the identical one-way transition.

Coverage: a stack-tier test that drives node-unreachable → node-healthy and asserts the proxy is started again; the KVM battery's post-update leg should assert the proxy is RUNNING (not merely that containers exist) after monerod restarts.

Note: this may predate the current wave — the stop logic is long-standing. Observed and reproduced on 4dd3ddb.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    appliancePithead OS appliance work — lands on feat/phase2-bakery-imagebugSomething isn't workingdashboardMining dashboard web UI

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions