背景
博主这几天发现服务器负载蹭蹭上涨,特此记录一篇:
借鉴这篇文章:主要原因
阿里云服务器配置:2H4G centos7
docker基础配置包含:taos_img、mysql、nginx、nacos、sentinel、tinyid、jrebel-ls、showdoc、nexus3、jenkins服务
问题点(已解决):
- 服务器异常卡顿
- 命令严重延迟
- TCP连接丢失严重
- 凉凉头开始大
查阅资料后发现以下问题:
命令:netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
[root@iZuf6d9y2x6c8r92unymr1Z bak]# netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
SYN_RECV 4
CLOSE_WAIT 1
ESTABLISHED 373
FIN_WAIT1 1
FIN_WAIT2 426
TIME_WAIT 1013
各个数据及其含义如下:
CLOSED:无连接是活动的或正在进行
LISTEN:服务器在等待进入呼叫
SYN_RECV:一个连接请求已经到达,等待确认
SYN_SENT:应用已经开始,打开一个连接
ESTABLISHED:正常数据传输状态
FIN_WAIT1:应用说它已经完成
FIN_WAIT2:另一边已同意释放
ITMED_WAIT:等待所有分组死掉
CLOSING:两边同时尝试关闭
TIME_WAIT:另一边已初始化一个释放
LAST_ACK:等待所有分组死掉
意思就是我的服务器有消除未被及时释放的TIME_WAIT状态的TCP连接
在执行
[root@iZuf6d9y2x6c8r92unymr1Z bak]# netstat -pnt | grep :9001 | wc -l
981
解决方案:
- 如发现系统存在大量TIME_WAIT状态的连接,通过调整内核参数解决
[root@iZuf6d9y2x6c8r92unymr1Z bak]# vim /etc/sysctl.conf
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
# see details in https://help.aliyun.com/knowledge_detail/39428.html
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
# see details in https://help.aliyun.com/knowledge_detail/41334.html
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1
- 在文件末尾追加了配置:
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
然后执行 /sbin/sysctl -p 让参数生效
[root@iZuf6d9y2x6c8r92unymr1Z bak]# /sbin/sysctl -p
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
过一段时间在查看: