一场Socket四次握手引发的血案
前奏
Hello Everybody,原谅我是一个标题党。事情是这样的,周末,同事在微信上抛来一个问题,原文如下:
服务端某个服务timewait过多,网上说端口会耗尽,我怎么感觉是fd会耗尽呢,因为一个服务通常只使用一个端口
在记忆的某个角落里,翻出了四次握手的状态图,虽然模模糊糊,但是还是有点印象。
首先,明确一点,主动关闭socket的一端进入timewait阶段,一个socket连接由五元组(目的IP,目的Port,源IP,源Port,协议)唯一标识。对于采用TCP协议的服务端,目的IP、目的Port、协议这三个元素固定不变,对于同一个客户端来说,其源IP固定不变,也就是说唯一能变的就是源Port。而源Port最大为65535,因此有耗尽的可能,第一次回答如下:
五元组(目的IP,目的Port,源IP,源Port,协议),同一个客户端,目的IP、目的Port、源IP、协议不变,端口有数量限制,因此应该是客户端端口会耗尽。
同事紧接着发难:
先只考虑服务端,timewait过多会导致啥情况?
回忆了一下timewait的作用,隐隐约约记得,防止客户端的FIN包积压在链路中,导致socket错乱。即客户端关闭连接后,重用本次五元组;迟来的FIN包影响新连接。使用timewait,锁定五元组(客户端不能重用本次五元组),使得本次会话的所有数据包在链路中消亡。因而,猜想timewait要维护连接状态,fd不会被释放。第二次回答:
timewait 时候fd被霸占,量大后,fd肯定会被耗尽。
作答后,久久不能平静,因为后半截回答主要靠猜想,没有实际论证,总感觉没有把握,于是开始了一系列的论证过程。
Socket 代码
这里仅列出主要代码逻辑,完整代码参加GitHub。
<code class="language-cpp hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)"><span class="hljs-comment" style="color:#75715e;outline:none!important; line-height:1.6;">/* server code */</span><br style="outline:none!important" /><span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">while</span> (<span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">1</span>) {<br style="outline:none!important" /> <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">int</span> len = <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">sizeof</span>(<span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">struct</span> sockaddr);<br style="outline:none!important" /> fd = accept(sfd, &remote, &len);<br style="outline:none!important" /><br style="outline:none!important" /> res = read(fd, buf, <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">sizeof</span>(buf));<br style="outline:none!important" /> <span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">printf</span>(<span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"user data : %s\n"</span>, buf);<br style="outline:none!important" /><br style="outline:none!important" /> <span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">strcpy</span>(buf, <span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"Hello Client"</span>);<br style="outline:none!important" /> res = write(fd, buf, <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">sizeof</span>(buf));<br style="outline:none!important" /> <span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">printf</span>(<span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"send data : %s\n"</span>, buf);<br style="outline:none!important" /><br style="outline:none!important" /> <span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">printf</span>(<span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"client fd: %d\n"</span>, fd);<br style="outline:none!important" /> <span class="hljs-comment" style="color:#75715e;outline:none!important; line-height:1.6;">/* 留出时间间隔给netstat,lsof查看状态 */</span><br style="outline:none!important" /> sleep(<span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">10</span>);<br style="outline:none!important" /><br style="outline:none!important" /> <span class="hljs-comment" style="color:#75715e;outline:none!important; line-height:1.6;">/* 服务端主动close,迫使服务端进入timewait */</span><br style="outline:none!important" /> close(fd);<br style="outline:none!important" /> <span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">printf</span>(<span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"socket closed\n"</span>);<br style="outline:none!important" />}<br style="outline:none!important" /></code>
<code class="language-cpp hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)"><span class="hljs-comment" style="color:#75715e;outline:none!important; line-height:1.6;">/* client code */</span><br style="outline:none!important" /><span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">strcpy</span>(buf, <span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"Hello Server!"</span>);<br style="outline:none!important" />write(sfd, buf, <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">sizeof</span>(buf));<br style="outline:none!important" /><span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">printf</span>(<span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"write < %s > to server\n"</span>, buf);<br style="outline:none!important" /><br style="outline:none!important" />read(sfd, buf, <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">sizeof</span>(buf));<br style="outline:none!important" /><span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">printf</span>(<span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"read < %s > from server\n"</span>, buf);<br style="outline:none!important" /><br style="outline:none!important" /><span class="hljs-comment" style="color:#75715e;outline:none!important; line-height:1.6;">/* server 为10秒,client为20秒,间隔10秒时间差,<br style="outline:none!important" /> 可以查看server进入FIN_WAIT2阶段 */</span><br style="outline:none!important" />sleep(<span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">20</span>);<br style="outline:none!important" /><br style="outline:none!important" /><span class="hljs-comment" style="color:#75715e;outline:none!important; line-height:1.6;">/* shutdown 向server发送FIN, 使server进入timewait阶段 */</span><br style="outline:none!important" />shutdown(sfd, SHUT_RDWR);<br style="outline:none!important" /></code>
NOTE:
client 采用了shutdown关闭连接而不是close,此处有一个坑,见后文。
<code class="language-markdown hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">spch2008@ubuntu:~$ netstat -anp | grep 2016<br style="outline:none!important" />tcp 127.0.0.1:2016 0.0.0.0:* LISTEN 2344/server <br style="outline:none!important" />tcp 127.0.0.1:2016 127.0.0.1:51142 ESTABLISHED 2344/server<br style="outline:none!important" /></code>
<code class="language-markdown hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">spch2008@ubuntu:~$ lsof | grep server<br style="outline:none!important" />server 3u TCP localhost:2016 (LISTEN)<br style="outline:none!important" />server 4u TCP localhost:2016->localhost:51142 (ESTABLISHED)<br style="outline:none!important" /></code>
当Client与Server建立连接后,通过lsof查看,连接套接字为4。10秒钟后,server调用close发送FIN,在client没有返回ACK时,server处于FIN-WAIT1,只可惜,client与server都在本机,几乎没有延时,因此看不到FIN-WAIT1状态。当client 发送的ACK返回时,server 状态转移到 FIN-WAIT2。
<code class="language-markdown hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">spch2008@ubuntu:~$ netstat -anp | grep 2016<br style="outline:none!important" />tcp 127.0.0.1:2016 0.0.0.0:* LISTEN 2344/server <br style="outline:none!important" />tcp 127.0.0.1:2016 127.0.0.1:51141 FIN_WAIT2 - <br style="outline:none!important" /></code>
<code class="language-markdown hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">spch2008@ubuntu:~$ lsof | grep server<br style="outline:none!important" />server 3u TCP localhost:2016 (LISTEN)<br style="outline:none!important" /></code>
FIN-WAIT2状态,通过lsof查看,发现套接字已经被关闭了。也就是说,在进程一端,已经没有办法操作该链接了。因此,也就不存在由于timewait将fd耗尽的可能。但是,在timewait这个阶段,操作系统依然需要维护五元组等相关信息,因此大量timewait链接,会消耗系统资源,内核空间会吃紧。
再经历10秒,client通过shutdown向server发送FIN,server进入timewait阶段。
<code class="language-markdown hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">spch2008@ubuntu:~$ netstat -anp | grep 2016<br style="outline:none!important" />tcp 127.0.0.1:2016 0.0.0.0:* LISTEN 2344/server <br style="outline:none!important" />tcp 127.0.0.1:2016 127.0.0.1:51141 TIME_WAIT - <br style="outline:none!important" /></code>
推理
实际上大量timewait会导致客户端端口耗尽。这里的耗尽指的是客户端A相对于服务端B,端口耗尽。怎么理解呢?
客户端A向服务器B发出大量链接,服务器B就有了大量的TIME_WAIT
。每个TIME_WAIT
都会保存客户端A的源端口。假定客户端A的可用端口为100-109,那么瞬间发出10个请求后,端口(100-109)进入服务器B TIME_WAIT状态锁定。紧接着客户端A创建第11个请求,它只能从(100-109)这10个端口中挑选一个,假如选中端口100,向服务器B发出请求;服务器B判断客户端A的端口100处于TIME_WAIT状态,则拒绝建立连接,连接建立失败。
此时客户端A如果向服务器C发起连接建立请求,那么是可以成功建立连接的。因而说,是客户端A相对于服务器B端口耗尽。
另外,不要将客户端理解为狭义的终端设备,中转系统中的服务器也是客户端。
<code class="language-markdown hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">spch2008@ubuntu:~$ sudo tcpdump -n -i lo port 2016<br style="outline:none!important" />127.0.0.1.2016 > 127.0.0.1.51146: Flags [F.], seq 101, ack 21<br style="outline:none!important" />127.0.0.1.51146 > 127.0.0.1.2016: Flags [.], ack 102, <br style="outline:none!important" /></code>
首先server调用close
函数想client发送FIN,client收到FIN后,回复ACK包,由此,server进行FIN-WAIT2阶段。紧接着,client调用close
函数,但却没有发送FIN包,而是发送了RST
包,抓包如下:
<code class="language-markdown hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">127.0.0.1.51146 > 127.0.0.1.2016: Flags [R.], seq 21, ack 102, <br style="outline:none!important" />127.0.0.1.2016 > 127.0.0.1.51146: Flags [.], ack 21, <br style="outline:none!important" />127.0.0.1.51146 > 127.0.0.1.2016: Flags [R], seq 1003701706, <br style="outline:none!important" /></code>
求助Google
,得到答案,如果缓冲区有未读取的数据,调用close
直接发送RST。
想想也对,TCP是一个全双工协议,我缓冲区里的数据已经不需要了,没有必要发送FIN来告诉你,我要关闭,你赶紧把未发送的数据发送过来。相当于close
主动将连接破坏了,连接破坏,无需四次握手优雅关闭,直接强断。
但是,发送RST就不需要TIME_WAIT
吗? 貌似进入了死循环,求解答。
另外,为什么会有数据没有读取完,代码里server与client都只是读取少量数据,一次read足以啊。仔细看了下,复用多年前的代码,在发送的时候,用的是sizeof
求长度。server的buf为100个字节,client为10个字节,肯定是读不完的啊。被自己坑了一次!
timewait端口复用
说一说我想验证服务端记录端口导致客户端无法建立连接的思路:client向server发请求,直至由于timewait过多不能建立请求;client向server发请求,server 直接rst,没有timeout,这样,如果没有出现不能建立连接的情况,那么就可以肯定,是由于server的timewait保留的端口信息导致客户端无法新建连接。
为了便于实验,我将client的可用端口调小, 只允许有两个可用端口。
<code class="hljs nimrod" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">echo <span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">40000</span> <span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">40001</span> > /<span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">proc</span>/sys/net/ipv4/ip_local_port_range<br style="outline:none!important" /></code>
不允许server复用timewait的端口,一些参数如下。
<code class="language-markdown hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">[root@livecd ~]# cat /proc/sys/net/ipv4/tcp<span class="hljs-emphasis" style="color:#a8a8a2;outline:none!important; line-height:1.6; font-style:italic">_fin_</span>timeout <br style="outline:none!important" />60<br style="outline:none!important" />[root@livecd ~]# cat /proc/sys/net/ipv4/tcp<span class="hljs-emphasis" style="color:#a8a8a2;outline:none!important; line-height:1.6; font-style:italic">_tw_</span>reuse <br style="outline:none!important" />0<br style="outline:none!important" />[root@livecd ~]# cat /proc/sys/net/ipv4/tcp<span class="hljs-emphasis" style="color:#a8a8a2;outline:none!important; line-height:1.6; font-style:italic">_tw_</span>recycle <br style="outline:none!important" />0<br style="outline:none!important" /></code>
client code
<code class="language-cpp hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)"><span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">for</span> (i = <span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">0</span>; i < <span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">3</span>; i++)<br style="outline:none!important" />{<br style="outline:none!important" /> sleep(<span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">1</span>); <span class="hljs-comment" style="color:#75715e;outline:none!important; line-height:1.6;">/* 留出时间,server进入TIME_WAIT */</span><br style="outline:none!important" /> <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">pid_t</span> pid = fork();<br style="outline:none!important" /> <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">if</span> (pid == <span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">0</span>)<br style="outline:none!important" /> {<br style="outline:none!important" /> handler();<br style="outline:none!important" /> <span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">exit</span>(<span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">0</span>);<br style="outline:none!important" /> }<br style="outline:none!important" />}<br style="outline:none!important" /><br style="outline:none!important" /><span class="hljs-function" style="color:#f92672;outline:none!important; line-height:1.6;"><span class="hljs-keyword" style="color:#66d9ef;outline:none!important; line-height:1.6;">void</span> <span class="hljs-title" style="color:#a6e22e;outline:none!important; line-height:1.6;">handler</span><span class="hljs-params" style="color:#f8f8f2;outline:none!important; line-height:1.6;">()</span><br style="outline:none!important" /></span>{<br style="outline:none!important" /> <span class="hljs-comment" style="color:#75715e;outline:none!important; line-height:1.6;">/* 省略非关键代码 */</span><br style="outline:none!important" /> res = connect(sfd, result->ai_addr, result->ai_addrlen);<br style="outline:none!important" /> <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">if</span> (res == -<span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">1</span>) <br style="outline:none!important" /> {<br style="outline:none!important" /> perror(<span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"error"</span>);<br style="outline:none!important" /> <span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">exit</span>(<span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">1</span>);<br style="outline:none!important" /> }<br style="outline:none!important" /><br style="outline:none!important" /> <span class="hljs-built_in" style="color:#e6db74;outline:none!important; line-height:1.6;">printf</span>(<span class="hljs-string" style="color:#e6db74;outline:none!important; line-height:1.6;">"connect\n"</span>);<br style="outline:none!important" />}<br style="outline:none!important" /></code>
server code
<code class="language-cpp hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)"><span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">while</span> (<span class="hljs-number" style="color:#ae81ff;outline:none!important; line-height:1.6;">1</span>) <br style="outline:none!important" />{<br style="outline:none!important" /> <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">int</span> len = <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">sizeof</span>(<span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">struct</span> sockaddr);<br style="outline:none!important" /> fd = accept(sfd, &remote, &len);<br style="outline:none!important" /><br style="outline:none!important" /> read(fd, buf, <span class="hljs-keyword" style="color:#f92672;outline:none!important; line-height:1.6;">sizeof</span>(buf));<br style="outline:none!important" /> close(fd);<br style="outline:none!important" />}<br style="outline:none!important" /></code>
期待显示结果
客户端建立3个连接,由于自身只有两个可用端口,因此,前两个可用创建成功,第三个连接创建失败,connect
函数返回错误。但,完全与期望的相反,3个连接都创建成功。看了一下server,确实是有两个TIME_WAIT
的连接。为什么服务器重用了端口??
<code class="language-markdown hljs" style="font-family:"Source Code Pro",monospace;font-size:undefined; padding:0.5em; color:rgb(248,248,242); display:block; outline:none!important; background:rgb(35,36,31)">[spch2008@livecd ~]$ netstat -an | grep TIME<br style="outline:none!important" />tcp 192.168.88.131:2016 192.168.88.132:40001 TIME_WAIT <br style="outline:none!important" />tcp 192.168.88.131:2016 192.168.88.132:40000 TIME_WAIT<br style="outline:none!important" /></code>
求助stackoverflow,得到大神的回复。
The Time Wait state is used prevent old packets from a previous connection from being accepted into a new connection. It effectively allows enough time for old packets to “die” in the network.
However, a socket in Timeout state can accept a new connection as long as the Initial Sequence Number on the SYN is higher than the last sequence number seen on the socket.
就是说,新建立连接SYN的序号大于TimeWait连接的最后一个序列号,即可用重用。但,不进入TIME_WAIT
状态,怎么让滞留在网络中的FIN Packet
消亡呢? 个人感觉这个地方是有风险的!!!