踩坑: supervisor中的minfds及minprocs参数用途

使用supervisor遇到的一个坑，为此还撕逼了一下午，先填了再说

先来看看minfds及minprocs这两个参数在supervisor官方文档中的说明（官方文档地址http://www.supervisord.org/configuration.html#supervisord-section-values）：

minfds
The minimum number of file descriptors that must be available before supervisord will start successfully. A call to setrlimit will be made to attempt to raise the soft and hard limits of the supervisord process to satisfy minfds. The hard limit may only be raised if supervisord is run as root. supervisord uses file descriptors liberally, and will enter a failure mode when one cannot be obtained from the OS, so it’s useful to be able to specify a minimum value to ensure it doesn’t run out of them during execution. This option is particularly useful on Solaris, which has a low per-process fd limit by default.

Default: 1024

Required: No.

Introduced: 3.0

大致意思：在supervisord成功启动前，最小的文件描述符数字必须是可以使用的。setrlimit参数会试图提高软硬限制来满足supervisord进程的minfds参数值。如果supervisord的运行用户为root，硬限制会被提高。supervisord可以自由地使用文件描述符，当无法从系统获取时将进入错误模式，所以它能够指定一个最小值，用于确保它不会在执行过程中用光文件描述符。这个参数在Solaris中特别有用，Solaris默认情况下具有较低的进程文件描述符限制。（默认值1024，参数为数值，在3.0版本加入）

minprocs
The minimum number of process descriptors that must be available before supervisord will start successfully. A call to setrlimit will be made to attempt to raise the soft and hard limits of the supervisord process to satisfy minprocs. The hard limit may only be raised if supervisord is run as root. supervisord will enter a failure mode when the OS runs out of process descriptors, so it’s useful to ensure that enough process descriptors are available upon supervisord startup.

Default: 200

Required: No.

Introduced: 3.0

大致意思：在supervisord成功启动前，最小的进程描述符数字必须是可以使用的。setrlimit参数会试图提高软硬限制来满足supervisord进程的minprocs参数值。如果supervisord的运行用户为root，硬限制会被提高。当进程描述符超出系统运行的进程描述符时，supervisord会进入错误模式，因此，在启动supervisord时，需要确保有足够的进程描述符可用。（默认值200，参数为数值，在3.0版本加入）

（突然觉得我的英语水平提高了好多，这是错觉么- -//）

个人猜想：minfds配置和Max open files有关，minprocs配置和max user processes有关

接下来用一个例子对这两个参数做一次深入解析：

环境说明：

supervisor版本：3.1.3（epel源安装）

操作系统版本：CentOS 7.3.1611

系统ulimit配置（红色部分为调整配置）：

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15060
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

supervisor配置参数：

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)

为了更好的演示，使用supervisor来做nginx的进程守护，nginx配置（截取部分）：

daemon off;
user nobody nobody;
worker_processes 4;
worker_cpu_affinity 0001 0010 0100 1000;
worker_rlimit_nofile 65535;

events {
use epoll;
worker_connections 65535;
}

supervisor的nginx守护配置：

[program:nginx]
command=/usr/local/nginx/sbin/nginx

接下来启动supervisord

$ systemctl start supervisord

获取supervisord及nginx的pid

$ ps aux |grep -E “supervisord|nginx: master” |grep -v grep
root 2118 0.0 0.3 220852 11968 ? Ss 02:09 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
root 2119 0.1 0.2 54932 10608 ? S 02:09 0:00 nginx: master process /usr/local/nginx/sbin/nginx

用pstree来看下更加直观的进程，可以看到nginx是supervisord的子进程

现在来看下supervisord及nginx进程的limit限制情况

可以看到nginx进程的limit和其父进程supervisord是一致的（Max Processes及Max open files），但与系统的ulimit限制不一致，说明其不受系统ulimit限制所影响。

为了验证我的猜想，我将supervisord中的minfds和minprocs参数值都改为50000

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=50000 ; (min. avail startup file descriptors;default 1024)
minprocs=50000 ; (min. avail process descriptors;default 200)

修改完成后重启supervisord

$ systemctl restart supervisord

重新获取supervisord及nginx的pid

$ ps aux |grep -E “supervisord|nginx: master” |grep -v grep
root 2175 0.0 0.3 220856 11972 ? Ss 02:31 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
root 2176 6.4 0.2 54932 10608 ? S 02:31 0:00 nginx: master process /usr/local/nginx/sbin/nginx

再来看下supervisord及nginx进程的limit限制情况

此时nginx进程的limit和其父进程supervisord仍然是一致的（Max Processes及Max open files），但是其数值全变为50000。

结论：

这个例子很好得说明了我的猜想是正确的，supervisord中参数minfds和minprocs决定了supervisord进程及其守护的子进程的Max Processes及Max open files，并且这个limit限制不受系统ulimit所影响。

经过测试验证，supervisord守护的子进程无法在supervisord配置文件中单独修改minfds和minprocs这两个参数。下面红色部分的配置是无效的

[program:nginx]
command=/usr/local/nginx/sbin/nginx
minfds=65535
minprocs=65535

需要在[supervisord]后面加上配置参数

[supervisord]
minfds=81920
minprocs=81920