supervisord - Online in the Cloud

PROGRAM:

NAME

supervisord - supervisord Documentation

Supervisor is a client/server system that allows its users to monitor and control a number
of processes on UNIX-like operating systems.

It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike
some of these programs, it is not meant to be run as a substitute for init as "process id
1". Instead it is meant to be used to control processes related to a project or a
customer, and is meant to start like any other program at boot time.

DOCUMENTATION

supervisord
The server piece of supervisor is named supervisord. It is responsible for starting child
programs at its own invocation, responding to commands from clients, restarting crashed or
exited subprocesseses, logging its subprocess stdout and stderr output, and generating and
handling "events" corresponding to points in subprocess lifetimes.

The server process uses a configuration file. This is typically located in
/etc/supervisord.conf. This configuration file is a "Windows-INI" style config file. It
is important to keep this file secure via proper filesystem permissions because it may
contain unencrypted usernames and passwords.

Running Supervisor
This section makes reference to a BINDIR when explaining how to run the supervisord and
supervisorctl commands. This is the "bindir" directory that your Python installation has
been configured with. For example, for an installation of Python installed via
./configure --prefix=/usr/local/py; make; make install, BINDIR would be /usr/local/py/bin.
Python interpreters on different platforms use a different BINDIR. Look at the output of
setup.py install if you can't figure out where yours is.

Adding a Program
Before supervisord will do anything useful for you, you'll need to add at least one
program section to its configuration. The program section will define a program that is
run and managed when you invoke the supervisord command. To add a program, you'll need to
edit the supervisord.conf file.

One of the simplest possible programs to run is the UNIX cat program. A program section
that will run cat when the supervisord process starts up is shown below.

[program:foo]
command=/bin/cat

This stanza may be cut and pasted into the supervisord.conf file. This is the simplest
possible program configuration, because it only names a command. Program configuration
sections have many other configuration options which aren't shown here. See
programx_section for more information.

Running supervisord
To start supervisord, run $BINDIR/supervisord. The resulting process will daemonize
itself and detach from the terminal. It keeps an operations log at $CWD/supervisor.log by
default.

You may start the supervisord executable in the foreground by passing the -n flag on its
command line. This is useful to debug startup problems.

WARNING:
When supervisord starts up, it will search for its configuration file in default
locations including the current working directory. If you are security-conscious you
will probably want to specify a "-c" argument after the supervisord command specifying
an absolute path to a configuration file to ensure that someone doesn't trick you into
running supervisor from within a directory that contains a rogue supervisord.conf file.
A warning is emitted when supervisor is started as root without this -c argument.

To change the set of programs controlled by supervisord, edit the supervisord.conf file
and kill -HUP or otherwise restart the supervisord process. This file has several example
program definitions.

The supervisord command accepts a number of command-line options. Each of these command
line options overrides any equivalent value in the configuration file.

supervisord Command-Line Options
-c FILE, --configuration=FILE
The path to a supervisord configuration file.

-n, --nodaemon
Run supervisord in the foreground.

-h, --help
Show supervisord command help.

-u USER, --user=USER
UNIX username or numeric user id. If supervisord is started as the root user,
setuid to this user as soon as possible during startup.

-m OCTAL, --umask=OCTAL
Octal number (e.g. 022) representing the umask that should be used by supervisord
after it starts.

-d PATH, --directory=PATH
When supervisord is run as a daemon, cd to this directory before daemonizing.

-l FILE, --logfile=FILE
Filename path to use as the supervisord activity log.

-y BYTES, --logfile_maxbytes=BYTES
Max size of the supervisord activity log file before a rotation occurs. The value
is suffix-multiplied, e.g "1" is one byte, "1MB" is 1 megabyte, "1GB" is 1
gigabyte.

-y NUM, --logfile_backups=NUM
Number of backup copies of the supervisord activity log to keep around. Each
logfile will be of size logfile_maxbytes.

-e LEVEL, --loglevel=LEVEL
The logging level at which supervisor should write to the activity log. Valid
levels are trace, debug, info, warn, error, and critical.

-j FILE, --pidfile=FILE
The filename to which supervisord should write its pid file.

-i STRING, --identifier=STRING
Arbitrary string identifier exposed by various client UIs for this instance of
supervisor.

-q PATH, --childlogdir=PATH
A path to a directory (it must already exist) where supervisor will write its AUTO
-mode child process logs.

-k, --nocleanup
Prevent supervisord from performing cleanup (removal of old AUTO process log files)
at startup.

-a NUM, --minfds=NUM
The minimum number of file descriptors that must be available to the supervisord
process before it will start successfully.

-t, --strip_ansi
Strip ANSI escape sequences from all child log process.

-v, --version
Print the supervisord version number out to stdout and exit.

--profile_options=LIST
Comma-separated options list for profiling. Causes supervisord to run under a
profiler, and output results based on the options, which is a comma-separated list
of the following: cumulative, calls, callers. E.g. cumulative,callers.

--minprocs=NUM
The minimum number of OS process slots that must be available to the supervisord
process before it will start successfully.

Signals
The supervisord program may be sent signals which cause it to perform certain actions
while it's running.

You can send any of these signals to the single supervisord process id. This process id
can be found in the file represented by the pidfile parameter in the [supervisord] section
of the configuration file (by default it's $CWD/supervisord.pid).

Signal Handlers
SIGTERM
supervisord and all its subprocesses will shut down. This may take several seconds.

SIGINT
supervisord and all its subprocesses will shut down. This may take several seconds.

SIGQUIT
supervisord and all its subprocesses will shut down. This may take several seconds.

SIGHUP
supervisord will stop all processes, reload the configuration from the first config
file it finds, and restart all processes.

SIGUSR2
supervisord will close and reopen the main activity log and all child log files.

Runtime Security
The developers have done their best to assure that use of a supervisord process running as
root cannot lead to unintended privilege escalation. But caveat emptor. Supervisor is
not as paranoid as something like DJ Bernstein's daemontools, inasmuch as supervisord
allows for arbitrary path specifications in its configuration file to which data may be
written. Allowing arbitrary path selections can create vulnerabilities from symlink
attacks. Be careful when specifying paths in your configuration. Ensure that the
supervisord configuration file cannot be read from or written to by unprivileged users and
that all files installed by the supervisor package have "sane" file permission protection
settings. Additionally, ensure that your PYTHONPATH is sane and that all Python standard
library files have adequate file permission protections.

Running supervisord automatically on startup
If you are using a distribution-packaged version of Supervisor, it should already be
integrated into the service management infrastructure of your distribution.

There are user-contributed scripts for various operating systems at:
https://github.com/Supervisor/initscripts

There are some answers at Serverfault in case you get stuck: How to automatically start
supervisord on Linux (Ubuntu)

Subprocesses

supervisord's primary purpose is to create and manage processes based on data in its
configuration file. It does this by creating subprocesses. Each subprocess spawned by
supervisor is managed for the entirety of its lifetime by supervisord (supervisord is the
parent process of each process it creates). When a child dies, supervisor is notified of
its death via the SIGCHLD signal, and it performs the appropriate operation.

Nondaemonizing of Subprocesses
Programs meant to be run under supervisor should not daemonize themselves. Instead, they
should run in the foreground. They should not detach from the terminal from which they
are started.

The easiest way to tell if a program will run in the foreground is to run the command that
invokes the program from a shell prompt. If it gives you control of the terminal back,
but continues running, it's daemonizing itself and that will almost certainly be the wrong
way to run it under supervisor. You want to run a command that essentially requires you
to press Ctrl-C to get control of the terminal back. If it gives you a shell prompt back
after running it without needing to press Ctrl-C, it's not useful under supervisor. All
programs have options to be run in the foreground but there's no "standard way" to do it;
you'll need to read the documentation for each program.

Below are configuration file examples that are known to start common programs in
"foreground" mode under Supervisor.

Examples of Program Configurations
Here are some "real world" program configuration examples:

Apache 2.2.6
[program:apache2]
command=/path/to/httpd -c "ErrorLog /dev/stdout" -DFOREGROUND
redirect_stderr=true

Two Zope 2.X instances and one ZEO server
[program:zeo]
command=/path/to/runzeo
priority=1

[program:zope1]
command=/path/to/instance/home/bin/runzope
priority=2
redirect_stderr=true

[program:zope2]
command=/path/to/another/instance/home/bin/runzope
priority=2
redirect_stderr=true

Postgres 8.X
[program:postgres]
command=/path/to/postmaster
; we use the "fast" shutdown signal SIGINT
stopsignal=INT
redirect_stderr=true

OpenLDAP slapd
[program:slapd]
command=/path/to/slapd -f /path/to/slapd.conf -h ldap://0.0.0.0:8888
redirect_stderr=true

Other Examples
Other examples of shell scripts that could be used to start services under supervisord can
be found at http://thedjbway.b0llix.net/services.html. These examples are actually for
daemontools but the premise is the same for supervisor.

Another collection of recipes for starting various programs in the foreground is available
from http://smarden.org/runit/runscripts.html.

pidproxy Program
Some processes (like mysqld) ignore signals sent to the actual process which is spawned by
supervisord. Instead, a "special" thread/process is created by these kinds of programs
which is responsible for handling signals. This is problematic because supervisord can
only kill a process which it creates itself. If a process created by supervisord creates
its own child processes, supervisord cannot kill them.

Fortunately, these types of programs typically write a "pidfile" which contains the
"special" process' PID, and is meant to be read and used in order to kill the process. As
a workaround for this case, a special pidproxy program can handle startup of these kinds
of processes. The pidproxy program is a small shim that starts a process, and upon the
receipt of a signal, sends the signal to the pid provided in a pidfile. A sample
configuration program entry for a pidproxy-enabled program is provided below.

[program:mysql]
command=/path/to/pidproxy /path/to/pidfile /path/to/mysqld_safe

The pidproxy program is put into your configuration's $BINDIR when supervisor is installed
(it is a "console script").

Subprocess Environment
Subprocesses will inherit the environment of the shell used to start the supervisord
program. Several environment variables will be set by supervisord itself in the child's
environment also, including SUPERVISOR_ENABLED (a flag indicating the process is under
supervisor control), SUPERVISOR_PROCESS_NAME (the config-file-specified process name for
this process) and SUPERVISOR_GROUP_NAME (the config-file-specified process group name for
the child process).

These environment variables may be overridden within the [supervisord] section config
option named environment (applies to all subprocesses) or within the per- [program:x]
section environment config option (applies only to the subprocess specified within the
[program:x] section). These "environment" settings are additive. In other words, each
subprocess' environment will consist of:
The environment variables set within the shell used to start supervisord...

... added-to/overridden-by ...

... the environment variables set within the environment global
config option ...

... added-to/overridden-by ...

... supervisor-specific environment variables
(SUPERVISOR_ENABLED, SUPERVISOR_PROCESS_NAME, SUPERVISOR_GROUP_NAME) ..

... added-to/overridden-by ...

... the environment variables set within the per-process
"environment" config option.

No shell is executed by supervisord when it runs a subprocess, so environment variables
such as USER, PATH, HOME, SHELL, LOGNAME, etc. are not changed from their defaults or
otherwise reassigned. This is particularly important to note when you are running a
program from a supervisord run as root with a user= stanza in the configuration. Unlike
cron, supervisord does not attempt to divine and override "fundamental" environment
variables like USER, PATH, HOME, and LOGNAME when it performs a setuid to the user defined
within the user= program config option. If you need to set environment variables for a
particular program that might otherwise be set by a shell invocation for a particular
user, you must do it explicitly within the environment= program config option. An example
of setting these environment variables is as below.

[program:apache2]
command=/home/chrism/bin/httpd -c "ErrorLog /dev/stdout" -DFOREGROUND
user=chrism
environment=HOME="/home/chrism",USER="chrism"

Process States
A process controlled by supervisord will be in one of the below states at any given time.
You may see these state names in various user interface elements in clients.

STOPPED (0)
The process has been stopped due to a stop request or has never been started.

STARTING (10)
The process is starting due to a start request.

RUNNING (20)
The process is running.

BACKOFF (30)
The process entered the STARTING state but subsequently exited too quickly to move to
the RUNNING state.

STOPPING (40)
The process is stopping due to a stop request.

EXITED (100)
The process exited from the RUNNING state (expectedly or unexpectedly).

FATAL (200)
The process could not be started successfully.

UNKNOWN (1000)
The process is in an unknown state (supervisord programming error).

Each process run under supervisor progresses through these states as per the following
directed graph.
[image: Subprocess State Transition Graph] [image] Subprocess State Transition
Graph.UNINDENT

A process is in the STOPPED state if it has been stopped adminstratively or if it has
never been started.

When an autorestarting process is in the BACKOFF state, it will be automatically
restarted by supervisord. It will switch between STARTING and BACKOFF states until it
becomes evident that it cannot be started because the number of startretries has
exceeded the maximum, at which point it will transition to the FATAL state. Each start
retry will take progressively more time.

When a process is in the EXITED state, it will automatically restart:

· never if its autorestart parameter is set to false.

· unconditionally if its autorestart parameter is set to true.

· conditionally if its autorestart parameter is set to unexpected. If it exited with an
exit code that doesn't match one of the exit codes defined in the exitcodes
configuration parameter for the process, it will be restarted.

A process automatically transitions from EXITED to RUNNING as a result of being configured
to autorestart conditionally or unconditionally. The number of transitions between
RUNNING and EXITED is not limited in any way: it is possible to create a configuration
that endlessly restarts an exited process. This is a feature, not a bug.

An autorestarted process will never be automatically restarted if it ends up in the FATAL
state (it must be manually restarted from this state).

A process transitions into the STOPPING state via an administrative stop request, and will
then end up in the STOPPED state.

A process that cannot be stopped successfully will stay in the STOPPING state forever.
This situation should never be reached during normal operations as it implies that the
process did not respond to a final SIGKILL signal sent to it by supervisor, which is
"impossible" under UNIX.

State transitions which always require user action to invoke are these:

FATAL -> STARTING

RUNNING -> STOPPING

State transitions which typically, but not always, require user action to invoke are
these, with exceptions noted:

STOPPED -> STARTING (except at supervisord startup if process is configured to autostart)

EXITED -> STARTING (except if process is configured to autorestart)

All other state transitions are managed by supervisord automatically.

Glossary
daemontools
A process control system by D.J. Bernstein.

launchd
A process control system used by Apple as process 1 under Mac OS X.

runit A process control system.

Superlance
A package which provides various event listener implementations that plug into
Supervisor which can help monitor process memory usage and crash status:
http://pypi.python.org/pypi/superlance.

umask Abbreviation of user mask: sets the file mode creation mask of the current process.
See http://en.wikipedia.org/wiki/Umask.

Use supervisord online using onworks.net services