在平时运维会遇到类似"Too many open files"(打开太多文件)的提示，是可以通过设置参数来解决资源限制。
在CentOS7上/etc/security/limits.conf配置文件中会有以下说明：#This file sets the resource limits for the users logged in via PAM.#It does not affect resource limits of the system services.
设置方式
内核参数(仅供参考)：file-max(仅供参考，300G内存服务器系统计算的出数字)# echo 39095546 > /proc/sys/fs/file-maxnr_open(仅供参考，默认的基础上乘以10)# echo 10485760 > /proc/sys/fs/nr_open# vim /etc/sysctl.d/my-set.conf ##如果让该配置生效可以sysctl -p /etc/sysctl.d/my-set.conf fs.file-max = 39095546fs.nr_open = 10485760ulimit设置(仅供参考)：# vim /etc/systemd/system.conf ##在配置文件最后添加DefaultLimitNOFILE=65535# systemctl daemon-reexec ##重新加载systemd配置# vim /etc/security/limits.d/my-set.conf ##新建的自定义配置* soft nofile 65535* hard nofile 65535# ulimit -n 65535
生效问题
##当前终端临时生效，可以who am i查看当前所在的终端，重启服务器也失效# ulimit -n 65535 ##当前终端不生效，对新开终端生效，重启服务器也生效# vim /etc/security/limits.d/my-set.conf * soft nofile 65535* hard nofile 65535## 所有终端都生效，应用服务需要重启后才生效，重启服务器也生效# vim /etc/systemd/system.conf ##在配置文件最后添加DefaultLimitNOFILE=65535# systemctl daemon-reexec ## file-max和nr_open内核参数设置类似，所有终端服务都生效，但是重启服务器失效# echo 39095546 > /proc/sys/fs/file-max## 所有终端服务都生效，重启服务器也生效# vim /etc/sysctl.d/my-set.conf fs.file-max = 39095546fs.nr_open = 10485760# sysctl -p /etc/sysctl.d/my-set.conf=============================================
目录
0.环境信息
1. 修改ulimit的nofile值，确定是否对nginx文件句柄的限制
1.1 不修改服务端ulimit值，ab压测
1.2 修改服务端ulimit值，ab压测
2. 修改systemd的LimitNOFILE值，确定是否对nginx文件句柄的限制
2.1 不修改服务端LimitNOFILE值，ab压测
2.2 修改服务端LimitNOFILE值，ab压测
3. file-max值，确定是否对nginx文件句柄的限制
4. nr_open值，确定是否对nginx文件句柄的限制
============================================
0.环境信息
----------------------------------------------
操作系统和应用工具版本
# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) # nginx -Vnginx version: nginx/1.16.1# ab -VThis is ApacheBench, Version 2.3 1430300 $>----------------------------------------------
参数默认值
----------------------------------------------
| 参数 | 描述 | 默认值 |
| ulimit -Sn | 可以打开最大文件描述符的数量，软限制 | 1024 |
| ulimit -Hn | 可以打开最大文件描述符的数量，硬限制 | 4096 |
| fs.nr_open | 单个进程可以分配最大的文件句柄 | 1024*1024=1048576 |
| fs.file-max | 允许整个系统打开的文件句柄的总数 | 基于系统动态计算(通常相当大) |
操作系统内参看
# cat /proc/sys/fs/file-max95548 注：file-max 非固定值，根据系统的资源动态计算，内核官网未找到相关公式或者算法，大概值为grep MemTotal /proc/meminfo | awk '{printf("%d",$2/10)}'，但是值和实际系统计算得出，测试机是1G内存# cat /proc/sys/fs/nr_open ##1024*1024=10485761048576# systemctl show |grep -i nofileDefaultLimitNOFILE=4096# ulimit -n1024# ulimit -Hn4096# prlimit --pid `cat /var/run/nginx.pid` -nRESOURCE DESCRIPTION SOFT HARD UNITSNOFILE max number of open files 1024 4096参数说明
file-max是内核级别强制执行的最大文件描述符(FD)，所有进程都不能超过它；
nr_open和ulimit是在进程级别强制执行的，可以小于file-max；
file-max的数值也可以小于nr_open，但是这样设置没有意义；
但是nr_open>=nofile(ulimit)，必须大于等于。
nofile(ulimit)设置值大于nr_open值的时候会报错，而LimitNOFILE设置值大于nr_open值不会报错，但是重启服务会报错或者将服务ulimit设置为65536；nofile(ulimit)和LimitNOFILE都必须小于等于nr_open值
# cat /proc/sys/fs/nr_open 1048576# ulimit -n 1048577-bash: ulimit: open files: cannot modify limit: Operation not permitted # vim /etc/systemd/system.confDefaultLimitNOFILE=1048577# systemctl daemon-reexec # systemctl show |grep -i nofileDefaultLimitNOFILE=1048577# systemctl restart nginx# prlimit --pid `cat /var/run/nginx.pid` -nRESOURCE DESCRIPTION SOFT HARD UNITSNOFILE max number of open files 65536 65536file-max & file-nr------------------The value in file-max denotes the maximum number of file-handles that the Linux kernel will allocate. When you get lotsof error messages about running out of file handles, you mightwant to increase this limit.Historically,the kernel was able to allocate file handlesdynamically, but not to free them again. The three values infile-nr denote the number of allocated file handles, the numberof allocated but unused file handles, and the maximum number offile handles. Linux 2.6 always reports 0 as the number of freefile handles -- this is not an error, it just means that thenumber of allocated file handles exactly matches the number ofused file handles.Attempts to allocate more file descriptors than file-max arereported with printk, look for "VFS: file-max limit reached".nr_open-------This denotes the maximum number of file-handles a process canallocate. Default value is 1024*1024 (1048576) which should beenough for most machines. Actual limit depends on RLIMIT_NOFILEresource limit.# man ulimitulimit [-HSTabcdefilmnpqrstuvx [limit]]Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.The -H and -S options specify that the hard or soft limit is set for the given resource. A hard limit cannot be increased by a non-root user once it is set; a soft limit may be increased up to the value of the hard limit. If neither -H nor -S is specified, both the soft and hard limits are set. The value of limit can be a number in the unit specified for the resource or one of the special values hard, soft, or unlimited, which stand for the current hard limit, the current soft limit, and no limit, respectively. If limit is omitted, the current value of the soft limit of the resource is printed, unless the -H option is given. When more than one resource is specified, the limit name and unit are printed before the value.内核参数详解：https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/Documentation/admin-guide/sysctl/fs.rst?h=v5.6.13
=============================================
测试过程说明
以nginx服务为例测试，服务端为:192.168.88.111，客户端为：192.168.88.107
服务端为:192.168.88.111
# yum install nginx -y# vim /etc/nginx/nginx.conf ##只修改此处，其他配置不需要修改events {# worker_connections 1024; worker_connections 65535;}# systemctl start nginx# systemctl enable nginx客户端为：192.168.88.107
# yum install httpd-tools -y# ab -n 2000 -c 2000 http://192.168.88.111/index.html客户端也是需要修改ulimit值，如果不修改socket: Too many open files (24)# ulimit -n 8192 ##在客户端操作----------------------------------------------
1. 修改ulimit的nofile值，确定是否对nginx文件句柄的限制
----------------------------------------------
服务端192.168.88.111需要做几个操作
# ulimit -n 1024# ulimit -Hn4096# tailf /var/log/nginx/error.log# prlimit --pid `cat /var/run/nginx.pid` -nRESOURCE DESCRIPTION SOFT HARD UNITSNOFILE max number of open files 1024 4096-------------------
1.1 不修改服务端ulimit值，ab压测
客户端192.168.88.107# ab -n 1000 -c 1000 http://192.168.88.111/index.html服务端192.168.88.111# tailf /var/log/nginx/error.log没有日志输出
客户端192.168.88.107# ab -n 1500 -c 1500 http://192.168.88.111/index.html服务端192.168.88.111# tailf /var/log/nginx/error.log2020/07/25 20:10:21 [crit] 874#0: accept4() failed (24: Too many open files)2020/07/25 20:10:21 [crit] 874#0: *6725 open() "/usr/share/nginx/html/index.html" failed (24: Too many open files), client: 192.168.88.107, server: _, request: "GET /index.html HTTP/1.0", host: "192.168.88.111"2020/07/25 20:10:21 [crit] 874#0: *6725 open() "/usr/share/nginx/html/50x.html" failed (24: Too many open files), client: 192.168.88.107, server: _, request: "GET /index.html HTTP/1.0", host: "192.168.88.111"-------------------
1.2 修改服务端ulimit值，ab压测
# ulimit -n 2048 ##只对当前终端生效# cat /etc/security/limits.d/nginx.conf ##对当前终端不生效，对新终端生效* soft nofile 2048* hard nofile 4096# reboot ## 这里重启，为了让配置文件ulimit生效，来判断ulimit是否对nginx的文件句柄有限制# ulimit -n2048# prlimit --pid `cat /var/run/nginx.pid` -nRESOURCE DESCRIPTION SOFT HARD UNITSNOFILE max number of open files 1024 4096 从prlimit输出结果来看，ulimit的改变，nginx的文件句柄值未改变。做ab压测
客户端192.168.88.107# ab -n 1500 -c 1500 http://192.168.88.111/index.html服务端192.168.88.111# tailf /var/log/nginx/error.log2020/07/25 20:10:21 [crit] 874#0: accept4() failed (24: Too many open files)总结：从prlimit输出结果来看，ulimit的改变，nginx的文件句柄值未改变；以及ab压测，1500压测值没有超过ulimit的2048设置，从服务端nginx的error.log来看，会报错Too many open files，所以ulimit值没有影响nginx服务的文件句柄数。
----------------------------------------------
2. 修改systemd的LimitNOFILE值，确定是否对nginx文件句柄的限制
----------------------------------------------
-------------------
2.1 不修改服务端LimitNOFILE值，ab压测
测试结果和1.1类似
-------------------
2.2 修改服务端LimitNOFILE值，ab压测
修改LimitNOFILE
# ulimit -n1024# ulimit -Hn4096# prlimit --pid `cat /var/run/nginx.pid` -nRESOURCE DESCRIPTION SOFT HARD UNITSNOFILE max number of open files 1024 4096 # systemctl show |grep -i nofileDefaultLimitNOFILE=4096# vim /etc/systemd/system.conf ##文件最后添加设置DefaultLimitNOFILE=8192DefaultLimitNOFILE=8192刷新systemd管理配置，并查看文件句柄相关限制
# systemctl daemon-reexec # systemctl show |grep -i nofileDefaultLimitNOFILE=8192# ulimit -n1024# ulimit -Hn4096# systemctl restart nginx# prlimit --pid `cat /var/run/nginx.pid` -nRESOURCE DESCRIPTION SOFT HARD UNITSNOFILE max number of open files 8192 8192客户端192.168.88.107# ab -n 1500 -c 1500 http://192.168.88.111/index.html服务端192.168.88.111# tailf /var/log/nginx/error.log没有日志输出----------------------------------------------
3. file-max值，确定是否对nginx文件句柄的限制
----------------------------------------------
服务端192.168.88.111
前提条件是修改LimitNOFILE值为8192
# vim /etc/systemd/system.conf ##文件最后添加设置DefaultLimitNOFILE=8192DefaultLimitNOFILE=8192# systemctl daemon-reexec # systemctl show |grep -i nofileDefaultLimitNOFILE=8192在修改file-max
# cat /proc/sys/fs/file-nr 1088 0 95548其中第一个数表示当前系统已分配使用的打开文件描述符数，第二个数为分配后已释放的(目前已不再使用)，第三个数等于file-max。# echo 2500 > /proc/sys/fs/file-max# cat /proc/sys/fs/file-nr1120 0 2500客户端192.168.88.107# ab -n 1500 -c 1500 http://192.168.88.111/index.html服务端192.168.88.111# tailf /var/log/nginx/error.log2020/07/25 22:36:27 [crit] 2327#0: accept4() failed (23: Too many open files in system)2020/07/25 22:36:27 [crit] 2327#0: *26592 open() "/usr/share/nginx/html/index.html" failed (23: Too many open files in system), client: 192.168.88.107, server: _, request: "GET /index.html HTTP/1.0", host: "192.168.88.111"报错信息为Too many open files in system，系统打开太多文件，从/proc/sys/fs/file-nr可知，1120(系统其它进程占用文件句柄数)+1500(ab压测值)>2500(系统内核最大文件句柄限制)，所以会报错。
----------------------------------------------
4. nr_open值，确定是否对nginx文件句柄的限制
----------------------------------------------
服务端192.168.88.111
前提条件是修改LimitNOFILE值为8192和file-max为默认值95548(测试机内存为1G)
# systemctl show |grep -i nofileDefaultLimitNOFILE=8192# cat /proc/sys/fs/file-max 95548# cat /proc/sys/fs/nr_open 1048576修改nr_open
# echo 8000 > /proc/sys/fs/nr_open# systemctl restart nginxJob for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.# echo 8192 > /proc/sys/fs/nr_open# systemctl restart nginx注：nr_open必须要大于等于DefaultLimitNOFILE，否则nginx服务不能启动
# tailf /var/log/messagesJul 26 12:35:04 centos-test01 systemd: Failed at step LIMITS spawning /usr/bin/rm: Operation not permittedJul 26 12:35:04 centos-test01 systemd: nginx.service: control process exited, code=exited status=205Jul 26 12:35:04 centos-test01 systemd: Failed to start The nginx HTTP and reverse proxy server.Jul 26 12:35:04 centos-test01 systemd: Unit nginx.service entered failed state.客户端192.168.88.107# ulimit -n 10000# ab -n 1500 -c 1500 http://192.168.88.111/index.html服务端192.168.88.111# tailf /var/log/nginx/error.log没有日志输出客户端192.168.88.107# ulimit -n 10000# ab -n 8192 -c 8190 http://192.168.88.111/index.html服务端192.168.88.111# tailf /var/log/nginx/error.log020/07/26 12:56:46 [crit] 3438#0: accept4() failed (24: Too many open files)2020/07/26 12:56:46 [crit] 3438#0: *8144 open() "/usr/share/nginx/html/index.html" failed (24: Too many open files), client: 192.168.88.107, server: _, request: "GET /index.html HTTP/1.0", host: "192.168.88.111"2020/07/26 12:56:46 [crit] 3438#0: *8144 open() "/usr/share/nginx/html/50x.html" failed (24: Too many open files), client: 192.168.88.107, server: _, request: "GET /index.html HTTP/1.0", host: "192.168.88.111"只有解除系统对服务限制，才能让服务更好的运行，展示它应有的价值。
1845
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
