perf(ssh): reuse one connection per Spark instead of a login per poll

[de1tydev]

The problem

Every collector poll opens a brand-new SSH connection. sshExec shells out to
ssh/sshpass per command, and nothing is reused between commands or between
ticks.

At the default cadence one remote Spark receives 217 logins per minute:

domain	interval	SSH per poll	per minute
gpu	2s	1	30
cpu	2s	1	30
ram	2s (shares POLL_INTERVAL_CPU)	1	30
network	2s	2 (main command + a second one for link speed)	60
memory / unified	2s	1	30
storage	5s	1	12
liveness	5s	2 (echo ok, then cat /proc/uptime)	24
NV_ERR journal scan	60s	1	1
total			217

Measured on a DGX Spark (Ubuntu 24.04, OpenSSH 9.6, 20 cores) monitored by a
Spark Dash instance with 10 units registered — journalctl -u ssh counted 1085
Accepted in 5 minutes, i.e. exactly 217/min, matching the table.

What that costs on the monitored Spark, sampled over 60s via cgroup CPU
accounting:

ssh.service      (preauth: KEX + the password KDF)   16.4 s/60s  = 0.27 core
user-1000.slice  (session sshd + PAM/logind + cmd)   58.4 s/60s  = 0.97 core
user@1000.service (desktop units, control)            0.34 s/60s ≈ 0
                                                     ─────────────────────
                                                      ~1.25 cores, constantly

The metric commands themselves are nowhere near that: the GPU command (three
nvidia-smi calls plus /proc/meminfo) measures 120 ms of CPU, unified memory
30 ms, the rest single-digit milliseconds — about 5.5 s/min of actual work
inside 75 s/min of cost. Roughly 90% of what the polling loop burns is
connection setup, and password auth makes it worse because the KDF runs on
every single one of those 217 logins.

The dashboard host pays too: its container measured 106 s of CPU per 30 s
(3.5 cores) with 40–50 concurrent ssh/sshpass processes. And each login
writes ~3 lines to the target's journal — ~940k lines per Spark per day.

The change

1. Multiplex. sshCommandSpec now adds ControlMaster=auto,
   ControlPath=$TMPDIR/sparkdash-%C and ControlPersist=300. The first
   command connects; the rest open a channel on a socket that is already
   authenticated. %C hashes (host, user, port) to a fixed-length name so the
   path can't exceed the ~104-byte sun_path limit. A master that died leaves
   a stale socket; ControlMaster=auto notices and replaces it.

   Opt out with SSH_MULTIPLEX=0; tune the idle lifetime with
   SSH_CONTROL_PERSIST. The -N tunnel in llmTunnel.js passes
   multiplex: false — a forward has to own its connection so that killing the
   process tears it down.

2. Fold the network link-speed read into the network round trip. It was a
   second sshExec for /sys/class/net/<primary>/speed, issued after parsing
   the route table. The command now dumps speeds for every interface alongside
   the operstate loop it already runs, and the primary is picked from that map.

3. Fold the uptime read into the liveness check. _checkOnline() ran
   sshTest() (echo ok) and then _readUptime() (cat /proc/uptime) — two
   logins to learn one thing. Reading /proc/uptime already proves the session
   came up, so it is now the liveness probe. The local-Spark path is unchanged.

Together 2 and 3 remove 42 of the 217 sessions on their own; multiplexing
removes essentially all of what remains.

Results

Same Spark, same cadence, after the change:

Same Spark, same cadence, after the change (60s cgroup samples, sshd
Accepted counted from the journal):

	before	after
logins per minute	82	0
ssh.service CPU / 60s	4.1 s	0 s
user-1000.slice CPU / 60s	15.9 s	2.2 s
total on the Spark	0.33 core	0.037 core

(The "before" column is already at a slowed-down 5s cadence; against the stock
2s cadence it is 217 logins/min and 1.25 cores, so end to end this is 1.25 →
0.037.) A second Spark measured 0.93 → 0.051 core. The dashboard host went from
3.5 cores to 0.167, and from 40–50 short-lived ssh processes to ~20 mostly
idle masters.

ssh.service dropping to a flat zero is the direct evidence: preauth and the
password KDF stop happening entirely. What is left on the Spark is close to the
cost of the metric commands themselves. Only one Accepted shows up in three
minutes — the master being established.

Snapshot fields all still populate, including the two that moved:
uptime, network.primaryInterface, network.linkSpeedMbps, network.wolMac,
and online for 8 of 10 registered units (the other two are genuinely down).
The /proc/stat diffing still works at the slower cadence — busy units read
94% / 91% GPU, idle ones read 0.

Tests

npx tsc --noEmit clean; npm test 192 passed / 0 failed (190 before, plus
two new cases in server/collectors/__tests__/ssh.test.js covering the shared
master and the multiplex: false opt-out).

Risk

A wedged master would stall that host's collectors instead of just one poll.
The existing per-command timeouts (execFile 10s, 8s for the NV_ERR scan) are
the backstop, and ControlPersist=300 keeps a socket from outliving a rebooted
Spark for long. MaxSessions (default 10) bounds concurrent channels; the
collector loop's in-flight guards keep it to a handful. SSH_MULTIPLEX=0
restores the previous behaviour without a rebuild.

[MikeGibbsOnyx]

Superseded: content landed on main via the 2026-09-08 backlog integration (main tip 950ac31, full suite 312/312 green). Credit preserved in the merge history.