nginx+tomcat结构下，TCP连接会有大量TIME_WAIT

1.查看当前系统中所有TCP的实时运行状态：

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

LAST_ACK 23
SYN_RECV 49
CLOSE_WAIT 6
ESTABLISHED 103195
FIN_WAIT1 69
FIN_WAIT2 928
TIME_WAIT 20045

 

2.TIME_WAIT 数很明显不正常，经从网上查询资料，找到解决方案，需要修改nginx配置文件，修要修改两点：

①upstream中要加入keepalive，表示保活的TCP连接数，如下所示

	upstream backend {
        keepalive 500;
            
	    #A服务器
        server 192.168.1.2:18234 weight=3; 
	    #B服务器
        server 192.168.1.3:18234 weight=3;
	    check interval=3000 rise=2 fall=5 timeout=2000 type=http;
     }

 

②在每一个location下加入proxy_http_version 1.1; 和 proxy_set_header Connection ""; 如下图所示：

		location /testService/testManage/ {
			proxy_http_version 1.1;
			proxy_set_header Connection "";
            proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-Scheme $scheme;   #向下传递https
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_pass http://backend/testService/testManage/;   	
		}

3.经过实际测试发现，在proxy_pass中使用upstream的，TIME_WAIT数量确实减少了，但是不使用upstream依然不会减少，所以在用nginx做代理时，一定要用upstream，即使只转向一个tomcat，也要这么做。

4.使用netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'，如下图所示，依然有三千多的TIME_WAIT，经分析发现是nginx做静态代理的请求，即使加入proxy_http_version 1.1; 和 proxy_set_header Connection ""; 依然没有效果，暂时没有找到好的解决方法

LAST_ACK 30
SYN_RECV 52
CLOSE_WAIT 11
ESTABLISHED 101595
FIN_WAIT1 109
FIN_WAIT2 674
TIME_WAIT 3489

 

5.修改Linux的系统变量，但是感觉用处不大，输入sysctl -p，可以查看当前系统参数

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 100000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.ip_local_port_range = 25000    61000
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 2048
vm.overcommit_memory = 1
net.nf_conntrack_max = 400000
net.netfilter.nf_conntrack_tcp_timeout_established = 10800
kernel.sysrq = 1
net.ipv4.tcp_mem = 1894275  1010280  3030840
net.ipv4.tcp_wmem = 4096 4096 6291456
net.ipv4.tcp_rmem = 4096 4096 6291456
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_keepalive_probes = 5
net.core.netdev_max_backlog = 3000