Time wait bucket overflow

1. Review current connections

• Check which port has the most of TW connections
• Dump all time-wait connections to a file

# ss -antp state time-wait > /tmp/tw-connections-`date +%Y%m%d` 

• Get local TW for the local ports

# cat /tmp/tw-connections-`date +%Y%m%d` | awk '{print $3}' | sort | uniq -c | sort -k 1 -n -r | head

• • Example output

   2619 10.0.20.15:80
    711 127.0.0.1:28080
    496 127.0.0.1:8080
    438 127.0.0.1:18080
     12 10.0.20.15:10050

• • Local Port 80 has 2619 connections on Time-Wait
• Get remote TW for the local ports

# cat /tmp/tw-connections-`date +%Y%m%d` | awk '{print $4}' | sort | uniq -c | sort -k 1 -n -r | head

• Check number of connections that are in TIME_WAIT state.

# wc -l /tmp/tw-connections-`date +%Y%m%d` 

• Add to the escalation request

• Get a summary of current connections

# ss -s 

2. Verify kernel setting for tcp_max_tw_buckets.

# cat /proc/sys/net/ipv4/tcp_max_tw_buckets 

• The correct value is 180000

 180000 

3. Verify kernel setting for tcp_fin_timeout.

# cat /proc/sys/net/ipv4/tcp_fin_timeout

• By our standards it should be 30

 30 

4. If TCP related kernel setting are not properly set, we need to modify those setting manually.

• Check if appropriate kernel setting exist in sysctf.conf file

# cat /etc/sysctl.conf | grep tcp_max_tw_buckets
net.ipv4.tcp_max_tw_buckets = 5000 
# cat /etc/sysctl.conf | grep net.ipv4.tcp_fin_timeout
net.ipv4.tcp_fin_timeout = 30 

• Modify values in /etc/sysctl.conf file

# vi /etc/sysctl.conf 
net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_fin_timeout = 30 

• Apply changes and check current TW max bucket value

# sysctl -p 
# cat /proc/sys/net/ipv4/tcp_max_tw_buckets 
# cat /proc/sys/net/ipv4/tcp_fin_timeout

• Monitor logs and number of Time Wait connections for some time.

转载于:https://my.oschina.net/am313/blog/681017