linux-Too many open files排查及修复

综述

在Linux下有时会遇到socket/file: can’t open so many files的问题。其实Linux是有文件句柄限制的，而且Linux默认一般都是1024。在生产环境中很容易达到这个值，因此这里就会成为系统的瓶颈。

There are two types of ulimit settings:
The hard limit is the maximum value that is allowed for the soft limit. Any changes to the hard limit require root access.
The soft limit is the value that Linux uses to limit the system resources for running processes. The soft limit cannot be greater than the hard limit.

这里有两种配置：
硬限制（hard limit）是软限制（soft limit）的上限，硬限制只能由root修改。
软限制是linux真正用于限制系统资源使用的值，软限制不能超过硬限制的值

排查手段

已打开文件数查询

1. 确认进程已打开的文件数
   lsof -p $pid | wc -l
2. 系统级已打开文件数

$ cat  /proc/sys/fs/file-nr
7936    0       1608965
3个数字分别代表：
系统已分配文件描述符数量
当前未使用文件描述符数量
系统级文件描述符限制

关于3个数字的官方说明

This (read-only) file contains three numbers: the number of allocated file handles (i.e., the number of files presently opened); the number of free file handles; and the maximum number of file handles (i.e., the same value as /proc/sys/fs/file-max). If the number of allocated file handles is close to the maximum, you should consider increasing the maximum. Before Linux 2.6, the kernel allocated file handles dynamically, but it didn’t free them again. Instead the free file handles were kept in a list for reallocation; the “free file handles” value indicates the size of that list. A large number of free file handles indicates that there was a past peak in the usage of open file handles. Since Linux 2.6, the kernel does deallocate freed file handles, and the “free file handles” value is always zero.

打开文件数限制

1. 进程级文件数限制
   grep “Max open files” /proc/$pid/limits

$ grep "Max open files" /proc/$pid/limits
Max open files            1024                 4096                 files
1024 软限制
4096 硬限制

2. session级的限制
   ulimit -n
3. 用户级限制
   ulimit -u
4. 系统级限制
   cat /proc/sys/fs/file-max
   或者使用命令

$ sysctl fs.file-max
fs.file-max = 1608965

提升打开文件数限制

1. 临时提升session级限制
   ulimit -n 4096
2. session级和用户级-修改/etc/security/limits.conf，然后重新登录，使配置生效

*         hard    nofile      500000       --session级硬限制
*         soft    nofile      500000       --session级软限制
root      hard    nofile      500000       --用户级硬限制
root      soft    nofile      500000       --用户级软限制

3. service级别-修改systemd的脚本参数

[Service]
LimitNOFILE=500000

systemctl daemon-reload
systemctl restart xx 使配置生效
4. 系统级-修改/etc/sysctl.conf

fs.file-max = 500000

sysctl -p 重新加载/etc/sysctl.conf使新配置生效