在 Server 端的底层代码中,下面的函数可以用来在一个连接对象上等待接收指定字节数的数据或者直至连接超时:
01: LRESULT ConnRecv(
02: PCONNECTION pConn, // Connection object
03: LPVOID pBuffer, // Data buffer pointer
04: DWORD dwBufferSize // Buffer size
05: )
06: {
07: DWORD dwReceived = 0;
08: DWORD dwFlags = 0;
09: WSABUF wsaBuffer = { dwBufferSize, (LPSTR)pBuffer };
10: WSAOVERLAPPED wsaOverlapped = { 0, 0, 0, 0, pConn->hIOEvent };
11: DWORD dwWaitResult = 0;
12: WSAEVENT arrEvents[] = { pConn->hExitEvent, pConn->hIOEvent };
13:
14: do
15: {
16: // Start an overlapped I/O
17: int nResult = WSARecv(
18: pConn->skt, // Socket
19: &wsaBuffer, // Winsock buffers
20: 1, // Buffer count
21: &dwReceived, // Number of bytes received
22: &dwFlags, // Flags
23: &wsaOverlapped, // Overlapped structure
24: NULL // Completion routine
25: );
26:
27: // Decide what should we do next according to the return value
28: if (nResult)
29: {
30: // See what happened...
31: if (WSAGetLastError() != WSA_IO_PENDING)
32: // Something wrong, return to caller
33: return SOCKET_ERROR;
34:
35: // Wait for the pending I/O completion or exit signal
36: DWORD dwResult = WSAWaitForMultipleEvents(
37: sizeof(arrEvents) / sizeof(WSAEVENT), // # of events
38: arrEvents, // Event handles
39: FALSE, // Wait all flag
40: pConn->dwIdleTimeout, // Timeout interval
41: FALSE // Alertable flag
42: );
43:
44: // Decide what should we do next according to the return value
45: switch(dwResult)
46: {
47: case WSA_WAIT_EVENT_0: // Exit signal
48: ExitThread(0);
49: case WSA_WAIT_EVENT_0 + 1: // I/O completion
50: {
51: // Get overlapped I/O result
52: if (!WSAGetOverlappedResult(
53: pConn->skt, // Socket
54: &wsaOverlapped, // Overlapped structure
55: &dwReceived, // Bytes received
56: FALSE, // Don't wait I/O completion
57: &dwFlags // Flags
58: ))
59: // Failed to retrieve overlapped result, return
60: return SOCKET_ERROR;
61:
62: // I/O completed successfully, fall through to advance the
63: // buffer pointer
64: break;
65: }
66: case WSA_WAIT_TIMEOUT: // Connection timeout
67: ConnClose(pConn, WSAETIMEDOUT, TRUE);
68: default: // Wait failed, return
69: return SOCKET_ERROR;
70: }
71: }
72:
76: // Advance the buffer pointer
77: wsaBuffer.buf = &wsaBuffer.buf[dwReceived];
78: wsaBuffer.len -= dwReceived;
83:
84: } while(wsaBuffer.len); // Do loop until received desired bytes of data
85: return 0;
86: }
可以看到,函数中并没有包含额外的检测 socket 关闭的代码。这是因为当初设计时曾经认为一旦 socket 被单方面关闭,另一方就可以检测到 WSAECONNRESET 错误,亦即在程序第 17 行的 Winsock 调用将失败,并在第 31 行检测到非 WSA_IO_PENDING 错误,而后来却发现有些时候(事实上是大多数时候)并不是这样的。当我使用自己编制的测试 Client 代码(一个 Console 小程序)对 Server 进行测试的时候,确实会发生 WSAECONNRESET 错误,一切正如当初预想的那样。但是如果我使用 Windows 自带的 Telnet 客户端测试 Server 连通性的时候却发现:当 Telnet 程序关闭后,Server 端的连接线程并没有出错退出,且 Server 程序的 CPU 占用率立即升至 100%。
进入 DEBUG 状态下跟踪,发现当 Telnet 程序关闭以后,第 17 行 WSARecv() 并没有返回任何错误,而是立即成功,并且 dwReceived 值为 0,从而导致连接线程进入了一个死循环。但是自己编写的 Console 测试 Client 端程序却没有这个问题,在关闭程序后 WSARecv() 立即返回错误。这个奇怪的现象一开始令我感到很迷惑,于是想到使用 netstat 命令查看一下连接状态:
Proto | Local Address | Foreign Address | State |
---|
TCP | 127.0.0.1:3716 | 127.0.0.1:27015 | FIN_WAIT_2 |
TCP | 127.0.0.1:27015 | 127.0.0.1:3716 | CLOSE_WAIT |
服务程序在 27015 端口监听(还好,似乎没有 RFC 文档规定这个端口属于 Counter-Strike 专用……),Telnet 客户端程序自动选择了 3716 端口。只不过这两个连接状态似乎不太寻常,于是上网查找。在 Apache 文档中找到了关于 FIN_WAIT_2 的一些解释(文章链接见后“相关文章”部分):
Starting with the Apache 1.2 betas, people are reporting many more connections in the FIN_WAIT_2 state (as reported by netstat) than they saw using older versions. When the server closes a TCP connection, it sends a packet with the FIN bit sent to the client, which then responds with a packet with the ACK bit set. The client then sends a packet with the FIN bit set to the server, which responds with an ACK and the connection is closed. The state that the connection is in during the period between when the server gets the ACK from the client and the server gets the FIN from the client is known as FIN_WAIT_2. See the TCP RFC for the technical details of the state transitions.
看来 TCP 连接关闭的过程并不是我想象中那么简单,需要双方都确认才算是一次“正常的”关闭动作。通过查看 MSDN 中 Graceful Shutdown, Linger Options, and Socket Closure 一章也明确了这一点。这也就不难解释我遇到的现象了:Telnet 程序退出时发出了断开连接的请求,即一个设置了 FIN bit 的包,Server 程序收到这个信号以后在 TCP 底层发送了一个 ACK 包。正如 Apache 文档中所说的那样,FIN_WAIT_2 并不与进程绑定,所以 Telnet 程序发出 FIN 包后立即退出了,其余的过程交由操作系统处理。另外,MSDN 中关于 WSARecv() 函数有下面的解释:
For connection-oriented sockets, WSARecv can indicate the graceful termination of the virtual circuit in one of two ways that depend on whether the socket is byte stream or message oriented. For byte streams, zero bytes having been read (as indicated by a zero return value to indicate success, and lpNumberOfBytesRecvd value of zero) indicates graceful closure and that no more bytes will ever be read. For message-oriented sockets, where a zero byte message is often allowable, a failure with an error code of WSAEDISCON is used to indicate graceful closure. In any case a return error code of WSAECONNRESET indicates an abortive close has occurred.
也就是说,流式套接字在正常关闭的时候,被动关闭一方的 WSARecv() 调用将会立即成功返回,并且指示接收到 0 个字节的数据。这也正是我遇到的情况,而我的问题在于:Server 在接收到 0 个字节的时候并没有检查 socket 是否已经开始关闭了(即 FD_CLOSE 事件已经发生),而是在持续调用 WSARecv() 希望接收到指定字节数的数据后再退出,从而导致进入了死循环。那么,为什么自己编写的客户端测试程序退出的时候不会产生这个问题?知道了套接字关闭的一些细节后,就不难解释这一点了:Windows 自带的 Telnet 程序是一个标准的 Windows 程序,它很可能是可以接收消息的,也就是说它可以对 WM_QUIT 消息作出响应,使得当单击窗口关闭按钮的时候有机会调用 closesocket() 或者 shutdown() 进行一个 Graceful Shutdown;而我写的 Client 端测试程序是一个 Console 程序,是没有消息队列的,不接收任何消息,除非显式调用 closesocket() 或者 shutdown()(而我为了编程简单,恰恰没有这么做),否则进程结束的时候任何资源都将被操作系统回收,包括套接字描述符。而这个过程显然不是那么“优美”的,也就是说在这种情况下进行了一个 Abortive Shutdown,这被认为是一个错误,此时 WSARecv() 才会报告失败(WSAECONNRESET)。
在弄清楚了问题的原因后,就不难提出解决方案了:只要在接收到 0 字节时检查 socket 关闭事件 FD_CLOSE 即可正确处理 Graceful Shutdown 的情况了。修改上述代码 72 行和 83 行之间的部分:
73: // Check transfer result
74: if (dwReceived)
75: {
76: // Advance the buffer pointer
77: wsaBuffer.buf = &wsaBuffer.buf[dwReceived];
78: wsaBuffer.len -= dwReceived;
79: }
80: else
81: // 0 bytes received? We should check for socket closure
82: ConnCheckForClosure(pConn);