Taskmaster

A UNIX job control daemon: a client/server process supervisor in the shape of supervisord.

Running it

cargo build --release

# Daemon. Stays in the foreground and opens its own control shell.
./target/release/taskmasterd -c taskmaster.example.conf.toml

taskmasterd provides the control shell itself, as the mandatory part requires. When its stdin is a terminal you get a prompt directly:

taskmasterd control shell. Type 'help' for commands, 'exit' to shut down.
taskmaster> status

No shell is opened when stdin is not a terminal (a pipe, nohup, a test harness), or when the daemon is not in the terminal's foreground process group (./taskmasterd &) — reading there would raise SIGTTIN and suspend it. In those cases the daemon simply supervises.

The log is mirrored to stderr only when stderr is a terminal. Piping it (taskmasterd 2>&1 | slow) would otherwise fill the 64 KB pipe buffer and block the supervisor thread outright, so the mirror is dropped when it is unsafe. The logfile always has the full record — tail -f it alongside the shell.

Separately — and as a bonus — taskmasterctl is a second program that talks to the daemon over the Unix socket:

# Interactive
./target/release/taskmasterctl -c taskmaster.example.conf.toml

# One-shot: runs the command and exits 0 on success, non-zero on failure
./target/release/taskmasterctl -c taskmaster.example.conf.toml stop web

Both binaries accept -c/--configuration <PATH>, -h/--help and -v/--version, and both offer the same commands.

Shell commands

status [<target>...]     start <target>...     stop <target>...
restart <target>...      reload                shutdown
pid                      version               ping
help [<command>]         exit | quit

A <target> is a program name (all of its instances), a process name such as web_01, or the keyword all. Line editing, history and tab completion are provided by rustyline; completion offers command verbs in the first position and live program/process names fetched from the daemon thereafter.

Signals

Signal	Behaviour
SIGHUP	Reload the config (identical to the reload command)
SIGTERM / SIGINT / SIGQUIT	Graceful shutdown: stop children, unlink socket, remove pidfile, exit 0. A second one escalates straight to SIGKILL rather than hanging
SIGCHLD	Reap (also reaped on every 100 ms tick — see Architecture.md)
SIGUSR2	Reopen the log file, so external logrotate works

Testing

cargo test        # unit tests for the pure functions
./tester/run.sh   # black-box suite driving the real binaries

See Architecture.md for the concurrency model and the reasoning behind it.

CLI Commands

Commands are sent over Unix Domain Sockets (UDS, .sock files) in the following format:

We use NDJSON (Newline Delimited JSON) to communicate over the socket. Sockets are streams not messages, hence if the server receives two messages to quickly, he could merge them instead of splitting them correctly.

Thus, relying on the \n after the message payload makes the communication more reliable.

Both binaries import the same Request / Response types from src/communication/protocol.rs, and the same NDJSON helpers from src/communication/framing.rs, so the two halves cannot drift apart.

Request

• command: one of status, start, stop, restart, reload, shutdown, ping, pid, version
• target: present for status / start / stop / restart. It is a tagged object:
  • {"kind": "all"} — every instance of every program
  • {"kind": "program", "name": "web"} — a program, meaning all its instances
  • {"kind": "process", "name": "web_01"} — one specific instance

A bare name typed at the prompt travels as program; the daemon looks for a group of that name first, then falls back to an instance name. An unknown name is answered with a 404 that names it, never a silent no-op.

Stop every instance of nginx:

{"command": "stop", "target": {"kind": "program", "name": "nginx"}}

Stop everything:

{"command": "stop", "target": {"kind": "all"}}

Responses

Exactly one response per request, always — including for malformed input.

• result: "ok" or "err"
• status: HTTP-style code (200, 400 malformed, 404 unknown target, 409 reload already running, 503 shutting down, 504 daemon did not answer)
• content (on ok): a tagged payload — message, statuses or actions
• error (on err): a human-readable string

{"result": "ok", "status": 200,
 "content": {"type": "message", "data": "pong"}}

{"result": "ok", "status": 200,
 "content": {"type": "statuses", "data": [
   {"name": "web_00", "program": "web", "state": "RUNNING", "pid": 4821,
    "uptime_secs": 134, "exit_status": null,
    "description": "pid 4821, uptime 0:02:14"}]}}

{"result": "err", "status": 404,
 "error": "no such program or process: nope"}

A group operation returns one actions entry per instance, so a partial failure is reported rather than collapsed into a single ambiguous status.

Config file

Treat your config file as an env file. A runnable example is at taskmaster.example.conf.toml.

Unknown keys are a hard parse error, not a silently ignored setting — a typo'd option would otherwise look like it took effect.

The default places where the config file will be sought after, in order:

• $CWD/taskmaster.conf.toml
• ../taskmaster.conf.toml
• $CWD/taskmaster.example.conf.toml

Override with -c/--configuration <PATH> on either binary.

[unix_http_server] Section Values

• file
  • Required: No
  • Default: None
  • A path to a UNIX domain socket on which taskmaster will listen for HTTP requests. taskmasterctl uses HTTP to communicate with taskmasterd over this port. If relative path, it will start where the config file is located.
• chmod
  • Required: No
  • Default: 0o700
  • Change the UNIX permission mode bits of the UNIX domain socket to this value at startup.
• chown
  • Required: No
  • Default: Use the username and group of the user who strats supervisord
  • Change the user and group of the socket file to this value. May be a UNIX username (e.g. chrism) or a UNIX username and group separated by a colon (e.g. chrism:wheel). You can only have one group at a time.

[unix_http_server]
file = "/tmp/taskmaster.sock"
chmod = 0o700
chown = "tomgernez:everyone"

[taskmasterd] Section Values

• logfile
  • Required: No
  • Default: $CWD/taskmaster.log
  • Path to the activity log of the taskmasterd process. Relative path assume start at at where the taskmaster.conf.toml was found.
• pidfile
  • Required: No
  • Default: $CWD/taskmasterd.pid
  • Path to the pid file of the taskmasterd process. Relative path assume start at where the taskmaster.conf.toml was found.
• umask
  • Required: No
  • Default: 0o22
  • Applies to log files created by the server and used as a default for files creates by the programs
• loglevel
  • Required: No
  • Default: info
  • One of debug, info, warn, error. Lines below this level are dropped.
• logfile_maxbytes
  • Required: No
  • Default: 0 (rotation disabled)
  • Rotate the log once it exceeds this size.
• logfile_backups
  • Required: No
  • Default: 0
  • How many rotated files to keep (taskmasterd.log.1, .2, …).
• nodaemon
  • Required: No
  • Default: true
  • true keeps taskmasterd in the foreground, which is what the subject requires and the mode everything is tested in. false daemonizes (double fork, setsid, chdir, std fds redirected to the logfile).
• minfds
  • Required: No
  • Default: 1024
  • Minimum number of available file descriptors that must be available before taskmasterd starts successfully.
• minprocs:
  • Required: No
  • Default: 200
  • Minimum number of available process descriptors that must be available before taskmasterd starts sucessfully.
• user
  • Required: No
  • Default: do not switch user
  • Requires taskmasterd to be launched as root user. If taskmasterd can't switch to the specific user, the program will not be started. The user will be changed using setuid only. This does not start a login shell and does not change env variables like USER or HOME.
• directory
  • Required: No
  • Default: do not cd
  • Only available if daemonized. taskmasterd cd's in that directory before starting.
• environment: Written in [taskmasterd.env], as KEY=VALUE. See below:

[taskmasterd]
logfile = "/tmp/taskmaster.log"
pidfile = "/tmp/taskmaster.pid"
umask = 0o22
nodaemon = true
minfds = 512
minprocs = 100
user = "user"
directory = "/tmp"

[taskmasterctl] Section Values

• serverurl
  • Required: No
  • Default: unix:///tmp/taskmaster.sock
  • For UNIX domain sockets: unix:///absolute/path/to/file.sock
• prompt
  • Required: No
  • Default: taskmaster
• history
  • Required: No
  • Default: /tmp/taskmaster_history.txt
  • Path to the history file

[program:x] Section Values

• command
  • Required: Yes
  • Default: No default
  • Path to the command to execute. If relative, $PATH is searched.
  • Arguments may be written inline (command = "/usr/sbin/nginx -c /etc/nginx.conf"), in which case the string is tokenized with shell-like 'single' and "double" quoting and backslash escapes. There is deliberately no variable expansion, globbing or piping — ask for a shell explicitly with sh -c if you want one.
• args:
  • Required: No
  • Default: None
  • Array of strings. Example: ["-la", "."]. If present it wins, and command is used verbatim as argv[0] with no tokenizing.
• numprocs
  • Required: No
  • Default: 1
  • Taskmaster will start as many instances of this program as named ny numprocs.
• numprocs_start:
  • Required: No
  • Default: 0
  • An integer offset that is used to compute the number at which process_num starts.
• autostart:
  • Required: No
  • Default: true
  • If true, this program will start automatically
• startsecs
  • Required: No
  • Default: 1
  • The total number of seconds which the program needs to stay running after a startup to consider the start successful (moving the process from the STARTING state to the RUNNING state). Set to 0 to indicate that the program needn't stay running for any particular amount of time. Even if a process exits with an "expected" exit code (see exitcodes), the strat will still be considered a failure if the process exits quicker than startsecs
• startretries
  • Required: No
  • Default: 3
  • Number of tries before the program goes into FATAL state. After each failed start, process will be put in BACKOFF. Each retry will take increasingly more time.
• autorestart
  • Required: No
  • Default: unexpected
  • Can be false, true, unexpected. Unexpected makes it restart if the exitcode of the program isn't in exitcodes. It is a STRING, not a boolean.
• exitcodes
  • Required: No
  • Default: 0
  • An array of "expected" exit codes for this program used with autorestart
• stopsignal
  • Required: No
  • Default: TERM
  • Signal used to kill the program when a stop is requested. Can be specified with the signal's name, shortname or number. Ex: "SIGINT", "INT", "2" (2 as a STRING)
• user
  • Required: No
  • Default: do not switch user
  • Requires taskmasterd to be launched as root user. If taskmasterd can't switch to the specific user, the program will not be started. The user will be changed using setuid only. This does not start a login shell and does not change env variables like USER or HOME.
• stdout
  • Required: No
  • Default: /tmp/<program>.stdout
  • Path to the file the child's stdout is appended to. The literal string "discard" (or an empty value) sends it to /dev/null.
  • Children write straight to the file, never through a pipe the daemon would have to drain — so a program producing 50 MB of output cannot stall supervision. The file is opened before the fork, so a bad path fails with a clear error instead of dying inside the child.
• stderr
  • Required: No
  • Default: /tmp/<program>.stderr
  • Same handling as stdout.
• directory:
  • Required: No
  • Default: No chdir (inherit taskmaster's)
  • A file path representing a directory to which taskmasterd should temporarily chdir before starting the child
• umask
  • Required: No
  • Default: No special mask (inherit taskmaster's)
  • An octal number representing the umask of the process.
• serverurl:
  • Required: No
  • Default: AUTO
  • URL passed in the env of the subprocess process as SUPERVISOR_SERVER_URL to allow the subprocess to easily communicate with the internal HTTP server. If provided, it should have the same syntax and structure as the [taskmaskterctl] section option of the same name. If this is set to AUTO, or is unset, taskmaster will automatically construct a server URL.
• environment: Written in [programs.program.env], as KEY=VALUE. See below.

[programs.ls]
command = "/bin/ls"
args = ["-ls", "."]
numprocs = 1
numprocs_start = 1000
autostart = true
startsecs = 5
startretries = 1
autorestart = "unexpected"
exitcodes = [0, 2]
stopsignal = "TERM"
user = "user"
stdout = "/tmp/ls.stdout"
stderr = "/tmp/ls.stderr"
umask = 0o022
serverurl = "unix:///tmp/taskmaster-example-ls.sock"

[programs.ls.env]
TEST = true
ANSWER = 42

Subprocesses

Subprocesses will inherit the env of the shell used to start the supervisord program. Several env vars will be set by taskmasterd itself in the child's env also, including TASKMASTER_ENABLED (a flag indicating the process is under taskmaster control), TASKMASTER_PROCESS_NAME (the config-file-specified process name for the process). These vars may be overriden in the [taskmasterd] section of the config file, or per program [programs.x.env].

See supervisord subprocess-environment for reference.

Process States

See supervisord process-states for reference.

Logs

Format: YYYY-MM-DD HH:MM:SS,mmm TYPE MESSAGE

Types: DEBG, INFO, WARN, ERRO

2026-08-05 14:03:11,204 INFO spawned: 'web_00' with pid 4821
2026-08-05 14:03:12,209 INFO success: 'web_00' entered RUNNING state
2026-08-05 14:05:01,880 WARN exited: 'web_00' (terminated by SIGKILL; not expected)

Timestamps are UTC. Local time would mean reading the tz database, which is a great deal of machinery for a log file; chrono/time would also be a fourth external dependency, and the subject permits three. The civil-date conversion is done by hand in src/logging/timestamp.rs.

The log file is opened with O_APPEND and never truncated: a restarted daemon must not erase its own audit history. If it cannot be opened at startup, the daemon refuses to start rather than running blind.

Lib Choices

Parsing with serde, toml

serde and toml is the best combinaison to parse TOML in Rust. Using the Deserialize macro (also configure it in your Cargo.toml), serde is able to automatically turn a TOML string into a Rust struct.

Jobs

• Job number n may be reffered to as %n.
• %%, %+, % (alone) refer to the shell's notion of the current job. It's either:
  • The last job that ended in the foreground
  • The last job that started in the background
• %- refers to the previous job (see jobs command, it uses + and -)
• A job can be reffered to using a prefix of the name used to start it, or using a substring that appears in its command line. For example, %ce refers to a stopped ce job. %?ce refers to any job containing the string ce in its command line. If it matches more than 1 job, Bash reports an error
• Simply naming a job bring it to the foreground: %1 is synonym for fg %1, and %1 & is synonym for bg %1
• The shell learns immediately whenever a job changes state. Normally, Bash waits until it is aout to print a prompt before reporting changes in a job's status, so as to not interrupt any other output. If -b is enabled, Bash reports immediately. Any trap on SIGCHLD is exectued for each child process that exits.
• Jobs are grouped by ID. All jobs started from a given shell will have the same group ID, the one of the shell, unless put in the background. In that case, they will have their own.

Job Control Builtins

• jobs: list all active jobs. With the -x option, allows to interfere with the Job group ID.
• kill: send signal to jobs or processes. Use kill -l to list all available signals
• disown: remove jobs from the actiev jobs table
• suspend: suspend the execution of this shell until it receives a SIGCONT signal.

Job Control Variables

• auto_resume: controls how the shell interacts with the user and job control (it is not a Bash native feature. zsh, csh, tcsh feature it however). It is used to resume a suspended (CTRL + Z, not background) job using its name, like fg that job.

Architecture

Supervisor: A Process Control System

A client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. Similar programs: launchd, daemontools, runit.

• PID are unreliable, file polling are unreliable, whereas the main process + subprocesses structured allows for great supervision of the processes
• Allows for normal users to control jobs that would require root privileges
• Subprocesses don't daemonize

Supervisor Components

See supervisor components. Below is a short description, but the documentation give more details.

supervisord

Server piece of supervisor. Interacts with the config file. Treat the config file as a .env file: it must be secured as it can contain some secret data. On launch, it daemonizes itself.

See supervisord command-line options for all options.

supervisord may be sent signals, which causes it to perform certain actions. See signals for the list of signals.

supervisorctl

Provides a shell-like interface to the features provided by supervisord. CLI client talks to the server across a UNIX domain socket or an internet (TCP) socket. The server may ask the client for auth credentials before allowing him to perform commands. The client typically uses the same config file as the server, but any configuration file with a [supervisorctl] section in it will work.

• If invoked with arguments, it usually doesn't start the CLI but executes as a command. Example: supervisorctl stop all, and returns the exit code (0 for success, non-zero for an error).
• If invoked without arguments supervisorctl , it starts the CLI with the given parameters. See supervisorctl command-line options for all CLI options.

In CLI mode, supervisorctl can perform multiple actions, see supervisorctl actions

Web Server

If supervisord is started against an internet socket, provide a supervisorctl-like interface but on a web interface. Settings are configured under the [inet_http_server] section of the configuration file.

Supervisor Config File

See file format

• Contains default search paths
• Allow environment variables expansion

[unix_http_server] Section Settings

Configuration parameters for an HTTP server that listens on a UNIX domain socket. If the config file has no [unix_http_server] section, no UNIX domain socket HTTP server will be started.

Configurable values include:

• file: path to a UNIX domain socket, on which supervisor will listen for HTTP/XML-RPC requests. %(here)s can be used, which expands in the directory containing the supervisord config file.
• Other values: chmod, chown, username, password

See values unix_http_server section settings

Example:

[unix_http_server]
file = /tmp/supervisor.sock
chmod = 0777
chown= nobody:nogroup
username = user
password = 123

[inet_http_server] Section Settings

Configuration parameters for an HTTP server that listens on a TCP socket. If the config gile has no [inet_http_server] section, no inet HTTP server will be strated.

Configurable values include:

• port: a TCP host:port value on which supervisor will listen for HTTP/XML-RPC requests.
• Other values: username, password

See values inet_http_server section settings

Example:

[inet_http_server]
port = 127.0.0.1:9001
username = user
password = 123

[supervisord] Section Settings

Global setttings related to the supervisord.

Configurable values include:

• logfile, logfile_maxbytes, logfile_backups, loglevel, pidfile, umask, nodaemon, silent, minfds, minprocs, nocleanup, childlogdir, user, directory, strip_ansi, environment, identifier

See values supervisord section settings

Example:

[supervisord]
logfile = /tmp/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = /tmp/supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200
umask = 022
user = chrism
identifier = supervisor
directory = /tmp
nocleanup = true
childlogdir = /tmp
strip_ansi = false
environment = KEY1="value1",KEY2="value2"

See values supervisord section settings

[supervisorctl] Section Settings

Configuration for the supervisorctl interactive shell program.

• serverurl: URL that should be used to access the supervisord server (e.g. http://localhost:9001, or UNIX domain sockets: unix:///absolute/path/to/file.sock)
• username, password, prompt, history_file see supervisorctl section settings

Example:

[supervisorctl]
serverurl = unix:///tmp/supervisor.sock
username = chris
password = 123
prompt = mysupervisor

[program:x] Section Settings

Settings for the programmes started and controlled by supervisord. The header is a composite, where program tells supervisord that it's a program, and :x tells what's the program to start (here, it's x). It's an error if not present.

See the list of configurable variables program section settings.

• command: path (absolute or relative, if relative then search $PATH). Programs can accept arguments: program foo bar.
• process_name: used to compose the supervisor process name.
• numprocs: the number of instances from that program to start.
• numprocs_start: integer offset used to compute the number at which process_num starts.
• priority: relative priority of the program in the start and shutdown ordering. Defaults to 999. The higher the prio, the higher it strats first and ends last.
• autostart: if true, this program autostart when supervisord is started. Default: true
• startsecs: Total number of seconds which the program needs to stay running after a startup to consider the start successful (mosing the process from the STARTING state to the RUNNING state). 0 for instant.
• startretries: number of serial failure attempts that supervisord will allow when attempting to start the program before giving up the process ito a FATAL state. After each retry, the process will be in BACKOFF state. Each retry attempt takes increasingly more time.
• autorestart: supervisord should restart the process if it exits when in the RUNNING state. Can be:
  • false: never restarted
  • unexpected: restarted if the exit code isn't in exitcodes
  • true: always restarted
• exitcodes: list of "expected" exit codes. Default: 0
• stopsignal: Signal used to kill the program when a stop is requested. Can be the signal's name or number: TERM, HUP, INT, QUIT, KILL, USR1 or USR2. Default: TERM
• stopwaitsecs: number of seconds to wait for the OS to return a SIGCHLD to supervisord after the program has been sent a stopsignal. If this number of seconds elapses before supervisord receives a SIGCHLD from the process, supervisord will attempt to kill it with a final SIGKILL. Deault: 10
• stopasgroup: If true, flag causes supervisor to send the stop signal to the whole process group and implies killasgroup is true. This is useful for programs, such Flask in debug mode, that do not propagate stop signals to their children, leaving them orphaned. Default: false
• killasgroup: If true, when resorting to send SIGKILL to the program to terminate it send it to its whole process group instead, taking care of its children as well, useful e.g with Python programs using multiprocessing. Default: false
• user: Instruct supervisord to use this UNIX user account as the account which runs the program. The user can only be switched if supervisord is run as the root user. If supervisord can't switch to the specified user, the program will NOT be started (user is changed using setuid, so env vars like USER and HOME stay the same).
• redirect_stderr: if true, causes the process' stderr output to be sent back to the supervisord on its stdout file descriptor (in UNIX shell terms, this is the equivalent of executing /the/program 2>&1). Default: false
• stdout_logfile, stdout_logfile_maxbytes, stdout_logfile_backups, stdout_capture_maxbytes, stdout_events_enabled, stdout_syslog, stderr_logfile, stderr_logfile_maxbytes, stderr_logfile_backups, stderr_capture_maxbytes, stderr_events_enabled, stderr_syslog
• environment: list of key / value pairs in the form KEY="val",KEY2="val2" that will be placed in te child process' env. Besides exceptions, the subprocess will inherit the env vars of the shell used to start supervisord.
• directory: a file path representing a directory to which supervisord should temporarily chdir before exec'ing the child.
• umask: Octal number, representing the umask fo the process. Default: inherit supervisor's
• serverurl: the URL passed in the env to the subprocess process as SUPERVISOR_SERVER_URL, to allow the subprocesss to easily communicate with the internal HTTP server. If provide, it should have the same syntax and structure as the [supervisorctl] section option of the same name. If AUTO (or unset), supervisor will automatically construct a serverl URL, giving prefernce to a server that listens on UNIX domain sockets over one that listens on an internet socket.

Example:

[program:cat]
command=/bin/cat
process_name=%(program_name)s
numprocs=1
directory=/tmp
umask=022
priority=999
autostart=true
autorestart=unexpected
startsecs=10
startretries=3
exitcodes=0
stopsignal=TERM
stopwaitsecs=10
stopasgroup=false
killasgroup=false
user=chrism
redirect_stderr=false
stdout_logfile=/a/path
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
stderr_logfile=/a/path
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false
environment=A="1",B="2"
serverurl=AUTO

[include] Section Settings

If the config file contains an [include] section, it must contain a single key named files. The values in this key specify other configuration files to be included within the configuration.

This section is only processed only by supervisord, ignored by supervisorctl.

See details on values include section settings.

• files: A space-separated sequence of file globs. Each file glob may be absolute or relative. If relative, it's relative to the location of the config file which includes it. If [include] section is used, it is required.

Example:

[include]
files = /an/absolute/filename.conf /an/absolute/*.conf foo.conf config??.conf

[group:x] Section Settings

See group section settings

[fcgi-program:x] Section Settings

See fcgi-program section settings

[eventlistener:x] Section Settings

See eventlistener section settings

[rpcinterface:x] Section Settings

See rpcinterface section settings

Resources

• Bash Documentation (see Chap 7 for Job Control)
• SUpervisor: A Process Control System
• daemontools
• launchd
• runit