Skip to content

fix(systemd): stop service restart from killing the tmux server - #212

Open
0x79H wants to merge 1 commit into
gbasin:masterfrom
0x79H:fix/systemd-killmode
Open

0x79H wants to merge 1 commit into
gbasin:masterfrom
0x79H:fix/systemd-killmode

Conversation

@0x79H

@0x79H 0x79H commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What

When agentboard starts and no tmux server is running, tmux new-session forks a new server inside the service's cgroup. On systemctl restart, the default KillMode=control-group sends SIGKILL to every process in the cgroup — including the tmux server — destroying all user sessions.

Adding KillMode=process limits the kill to the main bun process, leaving the tmux server (and user sessions) intact across restarts.

Evidence

Journal logs from a production deployment show repeated agentboard.service: Killing process (tmux: server) with signal SIGKILL entries on every service restart, each time destroying all active tmux sessions.

Verification

Single-line change to the generated service template in systemd/install.sh. No code or test changes.

- add KillMode=process so systemctl restart only terminates the main
  bun process, not the tmux server that may have been forked into the
  service cgroup; without this, every restart destroys all user sessions

@gbasin gbasin left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reported restart bug is real, but KillMode=process preserves every Agentboard child rather than isolating the tmux server.

On a graceful SIGTERM, the server attempts to dispose its terminal proxies, so the normal restart path may look correct. On an abnormal exit, forced termination, or failed cleanup, systemd will leave other descendants in the unit cgroup alive. Those descendants can include PTY-backed tmux clients and persistent SSH terminal processes. With Restart=on-failure, a replacement server can then start while those orphaned processes remain. Systemd explicitly discourages KillMode=process for this lifecycle escape behavior.

Please isolate the persistent tmux daemon from the Agentboard service instead. Suitable designs include a separate agentboard-tmux.service or a transient user unit/scope created with systemd-run. Keep Agentboard under the default KillMode=control-group, so its PTY and SSH children are still cleaned reliably. It would also be clearer to run the server directly as ExecStart=$BUN_PATH src/server/index.ts, ensuring the process tracked by systemd is the process with Agentboard's SIGTERM handler rather than relying on bun run exec behavior.

Please verify the final design on Linux with a real systemd user manager and cgroups. At minimum, test:

  1. Start Agentboard with no pre-existing tmux server and create a session.
  2. Restart Agentboard and confirm the tmux server PID and session survive.
  3. Stop Agentboard and document whether tmux is intentionally expected to survive.
  4. Kill Agentboard abnormally, allow Restart=on-failure, and confirm no old tmux-client or SSH child processes remain.
  5. Confirm the replacement server binds successfully and reconnects to the surviving tmux server.
  6. Test logout/login behavior both with and without user lingering, and document the requirement.

Static generation checks can run in CI, but the cgroup and lifecycle assertions need a Linux systemd environment; they cannot be validated meaningfully on macOS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants