Socket编程模型之重叠IO（Overlapped I/O）模型

转载请注明来源：http://blog.csdn.net/caoshiying/article/details/52876090

一、原理

Winsock2的发布使得Socket I/O有了和文件I/O统一的接口。我们可以通过使用Win32文件操纵函数ReadFile和WriteFile来进行Socket I/O。伴随而来的，用于普通文件I/O的重叠I/O模型和完成端口模型对Socket I/O也适用了。这些模型的优点是可以达到更佳的系统性能，但是实现较为复杂，里面涉及较多的C语言技巧。例如我们在完成端口模型中会经常用到所谓的“尾随数据”。

这个模型与上述其他模型不同的是它使用Winsock2提供的异步I/O函数WSARecv。在调用WSARecv时，指定一个WSAOVERLAPPED结构，这个调用不是阻塞的，也就是说，它会立刻返回。一旦有数据到达的时候，被指定的WSAOVERLAPPED结构中的hEvent被Signaled。下面分别讲解关键API函数。

二、HeapAlloc

HeapAlloc是一个Windows API函数。它用来在指定的堆上分配内存，并且分配后的内存不可移动。它分配的内存不能超过4MB。函数原型是：

LPVOID HeapAlloc(HANDLE hHeap, DWORD dwFlags, SIZE_T dwBytes);

hHeap：要分配堆的句柄，可以通过HeapCreate()函数或GetProcessHeap()函数获得。
dwFlags：堆分配时的可选参数，其值可以为以下的一种或多种：

• HEAP_GENERATE_EXCEPTIONS：如果分配错误将会抛出异常，而不是返回NULL。异常值可能是STATUS_NO_MEMORY,表示获得的内存容量不足，或是STATUS_ACCESS_VIOLATION,表示存取不合法。
• HEAP_NO_SERIALIZE：不使用连续存取。
• HEAP_ZERO_MEMORY：将分配的内存全部清零。

dwBytes：要分配堆的字节数。
返回值：如果成功分配内存，返回值为一个指向所分配内存块的首地址的（void*）指针。如果分配内存失败，并且没有指定HEAP_GENERATE_EXCEPTIONS，则返回NULL。如果指定了HEAP_GENERATE_EXCEPTIONS，则抛出异常，而不返回NULL，异常代码有：

• STATUS_NO_MEMORY：由于缺少可用内存或者是堆损坏导致分配失败。
• STATUS_ACCESS_VIOLATION：由于堆损坏或者是不正确的函数参数导致分配失败。

三、GetProcessHeap

GetProcessHeap是一个Windows API函数。它返回调用进程的默认堆句柄。函数原型是：

HANDLE GetProcessHeap(void);

函数没有参数。如果函数成功，返回调用进程的默认内存堆句柄。这个函数允许你从线程的堆动态分配内存，而不必使用HeapCreare函数建立一个堆。如果函数失败，返回 Null。若想，可以调用GetLastError获得更多错误信息。

四、HeapFree

HeapFree是一个Windows API函数。它用来释放堆内存。函数原型是：

BOOL HeapFree(HANDLE hHeap, DWORD dwFlags, LPVOID lpMem);

hHeap：堆内存块释放。这个参数是HeapCreate或GetProcessHeap函数返回的句柄。
dwFlags：指定几个可控释放的内存块。指定以下值将覆盖flOptions中指定参数对应的值,当堆是由使用HeapCreate函数。它的值可以是HEAP_NO_SERIALIZE，表示串行存取将不会被使用。为了确保序列化访问，所有调用这个函数将被禁用，在调用HeapCreate指定HEAP_NO_SERIALIZE。在这种情况下，没有必要在此函数调用另外指定HEAP_NO_SERIALIZE。访问进程堆时没有指定此值。该系统可以创建应用程序的过程中额外的线程，如CTRL + C处理程序，同时访问进程堆。
lpMem：被释放的内存块的指针。这HeapAlloc或HeapReAlloc函数返回的指针。如果这个指针为NULL,则为空。
如果函数成功,返回值是非零。否则为零。应用程序可以调用GetLastError扩展的错误信息。

五、WSARecv

在重叠模型中，接收数据就要靠它了，它的参数也比recv要多，因为要用到重叠结构。函数原型是：

int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine);

s：连接到此服务器的socket对象。
lpBuffers：接收到的数据保存在这里。
dwBufferCount：有几个缓冲区。
// 这里需要一个由WSABUF结构构成的数组
lpNumberOfBytesRecvd：实际收到了多少个字节的数据。如果lpOverlapped参数不为空，那个这个参数可以为空。据MSDN表述，这样做有助于避免潜在的错误。
lpFlags：这个参数很复杂。实在是不好翻译，我贴出MSDN的原文吧。它的值可以是：

• MSG_PEEK：Peeks at the incoming data. The data is copied into the buffer, but is not removed from the input queue. This flag is valid only for nonoverlapped sockets.
• MSG_OOB：Processes OOB data.
• MSG_PARTIAL：This flag is for message-oriented sockets only. On output, this flag indicates that the data specified is a portion of the message transmitted by the sender. Remaining portions of the message will be specified in subsequent receive operations. A subsequent receive operation with the MSG_PARTIAL flag cleared indicates end of sender's message. As an input parameter, this flag indicates that the receive operation should complete even if only part of a message has been received by the transport provider.
• MSG_PUSH_IMMEDIATE：This flag is for stream-oriented sockets only. This flag allows an application that uses stream sockets to tell the transport provider not to delay completion of partially filled pending receive requests. This is a hint to the transport provider that the application is willing to receive any incoming data as soon as possible without necessarily waiting for the remainder of the data that might still be in transit. What constitutes a partially filled pending receive request is a transport-specific matter. In the case of TCP, this refers to the case of incoming TCP segments being placed into the receive request data buffer where none of the TCP segments indicated a PUSH bit value of 1. In this case, TCP may hold the partially filled receive request a little longer to allow the remainder of the data to arrive with a TCP segment that has the PUSH bit set to 1. This flag tells TCP not to hold the receive request but to complete it immediately. Using this flag for large block transfers is not recommended since processing partial blocks is often not optimal. This flag is useful only for cases where receiving and processing the partial data immediately helps decrease processing latency. This flag is a hint rather than an actual guarantee. This flag is supported on Windows 8.1, Windows Server 2012 R2, and later.
• MSG_WAITALL：The receive request will complete only when one of the following events occurs: The buffer supplied by the caller is completely full. The connection has been closed.  The request has been canceled or an error occurred. Be aware that if the underlying transport provider does not support MSG_WAITALL, or if the socket is in a non-blocking mode, then this call will fail with WSAEOPNOTSUPP. Also, if MSG_WAITALL is specified along with MSG_OOB, MSG_PEEK, or MSG_PARTIAL, then this call will fail with WSAEOPNOTSUPP. This flag is not supported on datagram sockets or message-oriented sockets.

lpCompletionRoutine：回调函数，事件通知入口。详情请看下节。

WSA_IO_PENDING是最常见的返回值，这是说明我们的WSARecv操作成功了，但是I/O操作还没有完成，所以我们就需要绑定一个事件来通知我们操作何时完成。如果函数调用发生错误，接收操作立即返回0。这种情况下，这个完成例程将会被列入即将调用的时间表当线程变为警告状态, 否则，函数返回SOCKET_ERROR，我们可以通过WSAGetLastError获取错误代码. WSA_IO_PENDING表示这个重叠操作成功初始化，稍后便会显示。

六、CompletionROUTINE

这个函数是WSARecv函数的参数中回函数的参数格式。它本身是不存在的，用户代码应该按照以下范式声明：

void CALLBACK CompletionROUTINE(DWORD dwError, DWORD cbTransferred, LPWSAOVERLAPPED lpOverlapped, DWORD dwFlags);

CompletionRoutine is a placeholder for an application-defined or library-defined function name. The dwError specifies the completion status for the overlapped operation as indicated by lpOverlapped. The cbTransferred parameter specifies the number of bytes received. The dwFlags parameter contains information that would have appeared in lpFlags if the receive operation had completed immediately. This function does not return a value.

Returning from this function allows invocation of another pending completion routine for this socket. When using WSAWaitForMultipleEvents, all waiting completion routines are called before the alertable thread's wait is satisfied with a return code of WSA_IO_COMPLETION. The completion routines can be called in any order, not necessarily in the same order the overlapped operations are completed. However, the posted buffers are guaranteed to be filled in the same order in which they are specified.

If you are using I/O completion ports, be aware that the order of calls made to WSARecv is also the order in which the buffers are populated. WSARecv should not be called on the same socket simultaneously from different threads, because it can result in an unpredictable buffer order.

七、PEERIO_OPERATION_DATA结构

Windows API定义的LWSAOVERLAPPED结构远远不能满足从主线程到回调线程传递参数的需要，我们需要定义一个自己的数据结构，通过HeapAlloc分配内存，把指针传递到回调函数，结构如下：

typedef struct
{
	WSAOVERLAPPED overlap;
	WSABUF buf;
	char message[SOCKET_MESSAGE_SIZE];
	DWORD received_byte_count;
	DWORD flags;
	SOCKET client;
	iserver_manager *manager;
}PEERIO_OPERATION_DATA, *LPPEERIO_OPERATION_DATA;

overlap：记录原始的值。
buf：缓冲区指针。

message：缓冲区。

received_byte_count：收到的字节数。

flags：旗标。

client：这批缓冲区是哪个客户端发送的。

manager：回调结构。

八、示例代码

接着上面几篇Socket文章写，关于公共代码与反射式客户端请参见： Socket编程模型之简单选择模型。下面是新建的overlapped_server工程，新建了一个overlapped_server_manager类型，继承自iserver_manager接口，头文件完整代码如下：

#pragma once

#include <WinSock2.h>
#include <common_callback.h>
//#include <afxsock.h>

#define SOCKET_MESSAGE_SIZE 1024

typedef struct
{
	WSAOVERLAPPED overlap;
	WSABUF buf;
	char message[SOCKET_MESSAGE_SIZE];
	DWORD received_byte_count;
	DWORD flags;
	SOCKET client;
	iserver_manager *manager;
}PEERIO_OPERATION_DATA, *LPPEERIO_OPERATION_DATA;

class overlapped_server_manager:
	protected iserver_manager
{
private:
	int iport;
	int iclient_count;
	int iaddr_size;
	SOCKET server;
	SOCKET snew_client;
	WSADATA data;
	LPPEERIO_OPERATION_DATA peers[WSA_MAXIMUM_WAIT_EVENTS];
	common_callback callback;
	BOOL brunning;
	BOOL bnew_client;

private:
	void cleanup();
	int find_peer(LPPEERIO_OPERATION_DATA peer);

protected:
	bool accept_by_crt();
	bool accept_by_winapi();
	void receive();

public:
	void shutdown();
	void start_accept();
	void start_receive();
	void completion_routine(DWORD error, DWORD transfered, LPPEERIO_OPERATION_DATA peer, DWORD flags);
	
public:
	overlapped_server_manager();
	virtual ~overlapped_server_manager();
};

void CALLBACK do_completion_routine(DWORD error, DWORD transfered, LPWSAOVERLAPPED overlapped, DWORD flags);

实现文件完整代码如下：

#include "overlapped_server_manager.h"
#include <stdio.h>
#include <tchar.h>
#include <cassert>


overlapped_server_manager::overlapped_server_manager()
{
	iport = 5150;
	iclient_count = 0;
	iaddr_size = sizeof(SOCKADDR_IN);
	ZeroMemory(peers, sizeof(LPPEERIO_OPERATION_DATA));
	WSAStartup(MAKEWORD(2, 2), &data);
	brunning = FALSE;
	bnew_client = FALSE;
	callback.set_manager(this);
}


overlapped_server_manager::~overlapped_server_manager()
{
	cleanup();
}

bool overlapped_server_manager::accept_by_crt()
{
	SOCKET server;
	SOCKADDR_IN server_addr;
	SOCKADDR_IN client_addr;
	int iret = 0;

	server = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
	server_addr.sin_addr.S_un.S_addr = htonl(INADDR_ANY);
	server_addr.sin_family = AF_INET;
	server_addr.sin_port = htons(iport);
	do
	{
		iret = bind(server, (struct sockaddr*)&server_addr, iaddr_size);
		if (iret == SOCKET_ERROR)
		{
			iport++;
			server_addr.sin_port = htons(iport);
		}
	} while (iret == -1);
	listen(server, 3);
	printf("服务启动成功，端口是：%d\n", iport);
	while (brunning)
	{
		snew_client = accept(server, (struct sockaddr*)&client_addr, &iaddr_size);
		if (snew_client == INVALID_SOCKET)
			continue;
		printf("新客户端连接：%s:%d\n", inet_ntoa(client_addr.sin_addr), ntohs(client_addr.sin_port));
		bnew_client = TRUE;
	}
	return true;
}

bool overlapped_server_manager::accept_by_winapi()
{
	
	return true;
}

void overlapped_server_manager::receive()
{
	LPPEERIO_OPERATION_DATA peer = NULL;

	while (brunning)
	{
		if (bnew_client)
		{
			peer = (LPPEERIO_OPERATION_DATA)HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, sizeof(PEERIO_OPERATION_DATA));
			peer->buf.len = SOCKET_MESSAGE_SIZE;
			peer->buf.buf = peer->message;
			peer->client = snew_client;
			peer->manager = this;
			peers[iclient_count] = peer;
			WSARecv(peer->client, &peer->buf, 1, &peer->received_byte_count, &peer->flags, &peer->overlap, &do_completion_routine);
			bnew_client = FALSE;
			iclient_count++;
		}
		SleepEx(1000, TRUE);
	}
}

void overlapped_server_manager::shutdown()
{
	callback.shutdown();
	cleanup();
}

void overlapped_server_manager::cleanup()
{
	int i = 0;

	brunning = FALSE;
	for (i = 0; i < iclient_count; i++)
		HeapFree(GetProcessHeap(), 0, peers[i]);
	iclient_count = 0;
}

void overlapped_server_manager::completion_routine(DWORD error, DWORD transfered, LPPEERIO_OPERATION_DATA peer, DWORD flags)
{
	int index = -1;

	index = find_peer(peer);
	assert(index != -1);
	if (error != 0 || transfered == 0)
	{
		closesocket(peer->client);
		HeapFree(GetProcessHeap(), 0, peer);
		if (index < iclient_count - 1)
			peers[index] = peers[iclient_count - 1];
		peers[iclient_count - 1] = nullptr;
		iclient_count--;
	}
	else
	{
		peer->message[transfered] = 0;
		send(peer->client, peer->message, transfered, 0);
		ZeroMemory(&peer->overlap, sizeof(WSAOVERLAPPED));
		peer->buf.len = SOCKET_MESSAGE_SIZE;
		peer->buf.buf = peer->message;
		WSARecv(peer->client, &peer->buf, 1, &peer->received_byte_count, &peer->flags, &peer->overlap, do_completion_routine);
	}
}

int overlapped_server_manager::find_peer(LPPEERIO_OPERATION_DATA peer)
{
	int index = -1;
	int i = 0;

	for (i = 0; i < iclient_count; i++)
	{
		if (peers[i] != peer)
			continue;
		index = i;
		break;
	}
	return index;
}

void overlapped_server_manager::start_accept()
{
	brunning = TRUE;
	callback.start_accept_by_crt();
}

void overlapped_server_manager::start_receive()
{
	brunning = TRUE;
	callback.start_receive();
}

void CALLBACK do_completion_routine(DWORD error, DWORD transfered, LPWSAOVERLAPPED overlapped, DWORD flags)
{
	auto peer = (LPPEERIO_OPERATION_DATA)overlapped;
	auto pmanager = reinterpret_cast<overlapped_server_manager*>(peer->manager);
	pmanager->completion_routine(error, transfered, peer, flags);
}

int main()
{
	overlapped_server_manager osm;
	osm.start_accept();
	osm.start_receive();
	printf("重叠（Overlapped）I/0模型服务端启动成功。按任意键关闭服务器。\n");
	system("pause");
	osm.shutdown();
	return 0;
}

九、运行效果

十、感想

WSARecv提供recv几个额外的重要功能：可以进行套接字重叠进行重叠recv操作；允许多个接收缓冲区，使其适应于聚集/分散I/O类型；WSARecv函数通过参数 s 指定连接的SOCKETS 或者绑定未连接的SOCKETS 来读取未接收的数据。SOCKETS的本地地址是必须知道的。在服务器应用程序中，SOCKET通常显式通过bind绑定或者通过隐式的通过accept,WSAAccept绑定；在客户点应用程序中，SOCKET通过connect,WSAConnect,Sento,WSASendTo或者WSAJoinLeaf隐式的绑定本地地址。
对于连接和非连接的SOCKET，这个函数通过accept接收的消息限制地址,它仅仅对指定远程地址返回消息，忽略其他地址发送的消息。
对于重叠的SOCKETS，WSARecv通常投递一个或多个可以填充数据的缓冲区，申请完成后等待指定的完成指示（调用的完成例程或事件对象设置）发生，如果这个操作不能马上完成，最终的完成状态可以通过完成例程或WSAGetOverlappedResult获取.
所有的I/O在线程创建时初始化，线程退出时取消，对于重叠的SOCKETS，如果线程关闭，未决的异步操作可能会在完成之前失败。
如果 lpOverlapped ，lpCompletionRoutine 都NULL，SOCKET 在这个函数中被认为是非重叠的SOCKET。对于非重叠的SOCKETS，阻塞的语意与recv相同，且lpOverlapped ，lpCompletionRoutine都会被忽略，任何已经收到和传输缓冲的数据将被复制到指定的用户缓冲区，如果当前没有收到数据或没有任何数据传输，SOCKET将被阻塞，直到接收到数据。Window Socket 2没有为这个函数定义任何的标准阻塞超时机制。
基于字节流的协议栈试图尽可能返回多的可用的缓冲区空间和可接收的数据。然而，单个字节足够疏通这次调用，而且也没有任何可以保证返回多余一个字节。而基于消息的协议，则发送个完整的数据包。（字节流的协议和基于消息的协议的主要区别）。

转载于:https://my.oschina.net/zhtqs/blog/1509794