操作概观
1. 产生一个 I/O completion port。
2. 让它和一个文件handle 产生关联。
3. 产生一堆线程。
4. 让每一个线程都在com pletion port 上等待。
5. 开始对着那个文件handle 发出一些overlapped I/O 请求。
产生一个 I/O Completion Port
HANDLE CreateIoCompletionPort(
HANDLE FileHandle,
HANDLE ExistingCompletionPort,
DWORD CompletionKey,
DWORD NumberOfConcurrentThreads
);
参数
FileHandle 文件或设备(device)的handle。在Window s NT 3.51 之后,此栏位可设定为INVALID_HANDLE_VALUE,于是产生一个没有和任何文件handle 有关系的port。
ExistingCompletionPort 如果此栏位被指定,那么上一栏位FileHandle 就会被加到此port 之上,而不会产生新的port。指定NULL 可以产生一个新的port。
CompletionKey 用户自定义的一个数值,将被交给提供服务的线程。此值和FileHandle 有关联。
NumberOfConcurrentThreads 与此I/O com pletion port 有关联的线程个数。
返回值
如果函数成功,则传回一个I/O com pletion port 的handle。如果函数失败,则传回FALSE。GetLastError() 可以获得更详细的失败原因。
CreateIoCompletionPort() 通常被调用两次。第一次先指定FileHandle 为INVALID_HANDLE_VALUE,并设定ExistingCom pletionPort 为NULL,用以产生一个port。然后再为每一个欲附着上去的文件handle 调用一次CreateIoCompletionPort()。后续的这些调用应该将ExistingCom pletionPort 设定为第一次调用所传回的handle。例如:
HANDLE hPort;
HANDLE hFiles[MAX_FILES];
int index;
//Create the completion port
hPort = CreateIoCompletionPort(
INVALID_HANDLE_VALUE,
NULL,
0, // key
0 // default # of threads
);
// Now associate each file handle
for (index = 0; index < OPEN_FILES; index++)
{
CreateIoCompletionPort(
hFiles[index],
hPort,
0, // key
0 // default # of threads
);
}
产生一堆线程
一旦com pletion port 产生出来,你就可以设立在该port 上等待的那些线程了。I/O completion port 并不自己产生那些线程,它只是使用由你产生的线程。因此,你必须自己以CreateThread() 或_beginthreadex()或AfxBeginThread()产生出线程。
当你一产生这些线程时,它们都应该在com pletion port 上等待。当线程开始为各个“请求”服务时,池子里的线程的组织如下:
目前正在执行的线程
+ 被阻塞的线程
+ 在com pletion port上等待的线程
-----------------------------------------------------------
= 池子里所有线程的个数
因为如此,所以你应该产生比CPU 个数还多的线程。如果你只有一个CPU,而你也只产生了一个线程,那么当该线程阻塞(blocking)时,你的CPU 也变成闲置(idle)的了。由于池子里没有其他线程,completion port 也就没有办法为任何数据包(packets)服务——甚至即使CPU 的能量游刃有余。合理的线程个数应该是CPU 个数的两倍再加2。
在一个 I/O Completion Port 上等待
Worker线程初始化自己之后,它应该调用GetQueuedCompletionStatus()。这个操作像是WaitForSingleO bject() 和
GetOverlappedResult() 的组合。函数规格如下:
BOOL GetQueuedCompletionStatus(
HANDLE CompletionPort,
LPDWORD lpNumberOfBytesTransferred,
LPDWORD lpCompletionKey,
LPOVERLAPPED *lpOverlapped,
DWORD dwMilliseconds
);
参数CompletionPort 将在其上等待的com pletion port。
lpNumberOfBytesTransferred 一个指针,指向DWORD。该DWORD 将收到“被传输的数据字节数”。
lpCompletionKey 一个指针,指向DWORD。该DWORD 将收到由CreateIoCom pletionPort() 所定义的key。
lpOverlapped 这个栏位的名称是个错误。它其实应该命名为lplpOverlapped ,你应该把一个指针的地址放在上面。
overlapped 结构的指针。该结构用以初始化I/O 操作。
dwMilliseconds 等待的最长时间(毫秒)。如果时间终了,
lpOverlapped 将被设为NULL,而函数传回FALSE。
返回值
如果函数成功地将一个com pletion packet 从队列中取出,并完成一个成功的操作,函数将传回TRUE,并填写由lpNum berOfBytesTransferred、lpCompletionKey、lpOverlapped 所指向的变量内容。
发出“Overlapped I/O 请求”
下面这些调用可以启动“能够被一个I/O com pletion port 掌握”的I/O 操作:
ConnectNamePipe()
DeviceIoControl()
LockFileEx()
ReadFile()
TransactNamePipe()
WaitCommEvent()
WriteFile()
为了使用com pletion port,主线程(或任何其他线程)可以对着一个与此completion port 有关联的文件,进行读、写、或其他任何操作。该线程不需要调用WaitForMultipleObjects(),因为池子里的各个线程都曾经调用过GetQueuedCompletionStatus()。一旦I/O 操作完成,一个等待中的线程将会自动被释放,以服务该操作。
#define WIN32_LEAN_AND_MEAN
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
#include <tchar.h>
#include <string.h>
#include <winsock.h>
#include <io.h>
#include "MtVerify.h"
// Pick a port number that seems to be away from all others
#define SERV_TCP_PORT 5554
#define MAXLINE 512
//
// Structure definition
//
// The context key keeps track of how the I/O
// is progressing for each individual file handle.
struct ContextKey
{
SOCKET sock;
// Input
char InBuffer[4];
OVERLAPPED ovIn;
// Output
int nOutBufIndex;
char OutBuffer[MAXLINE];
OVERLAPPED ovOut;
DWORD dwWritten;
};
//
// Global variables
//
HANDLE ghCompletionPort;
//
// Function prototypes
//
void CreateWorkerThreads();
DWORD WINAPI ThreadFunc(LPVOID pvoid);
void IssueRead(struct ContextKey *pCntx);
void CheckOsVersion();
void FatalError(char *s);
///
int main(int argc, char *argv[])
{
SOCKET listener;
SOCKET newsocket;
WSADATA WsaData;
struct sockaddr_in serverAddress;
struct sockaddr_in clientAddress;
int clientAddressLength;
int err;
CheckOsVersion();
err = WSAStartup (0x0101, &WsaData);
if (err == SOCKET_ERROR)
{
FatalError("WSAStartup Failed");
return EXIT_FAILURE;
}
/*
* Open a TCP socket connection to the server
* By default, a socket is always opened
* for overlapped I/O. Do NOT attach this
* socket (listener) to the I/O completion
* port!
*/
listener = socket(AF_INET, SOCK_STREAM, 0);
if (listener < 0)
{
FatalError("socket() failed");
return EXIT_FAILURE;
}
/*
* Bind our local address
*/
memset(&serverAddress, 0, sizeof(serverAddress));
serverAddress.sin_family = AF_INET;
serverAddress.sin_addr.s_addr = htonl(INADDR_ANY);
serverAddress.sin_port = htons(SERV_TCP_PORT);
err = bind(listener,
(struct sockaddr *)&serverAddress,
sizeof(serverAddress)
);
if (err < 0)
FatalError("bind() failed");
ghCompletionPort = CreateIoCompletionPort(
INVALID_HANDLE_VALUE,
NULL, // No prior port
0, // No key
0 // Use default # of threads
);
if (ghCompletionPort == NULL)
FatalError("CreateIoCompletionPort() failed");
CreateWorkerThreads(ghCompletionPort);
listen(listener, 5);
fprintf(stderr, "Echo Server with I/O Completion Ports\n");
fprintf(stderr, "Running on TCP port %d\n", SERV_TCP_PORT);
fprintf(stderr, "\nPress Ctrl+C to stop the server\n");
//
// Loop forever accepting requests new connections
// and starting reading from them.
//
for (;;)
{
struct ContextKey *pKey;
clientAddressLength = sizeof(clientAddress);
newsocket = accept(listener,
(struct sockaddr *)&clientAddress,
&clientAddressLength);
if (newsocket < 0)
{
FatalError("accept() Failed");
return EXIT_FAILURE;
}
// Create a context key and initialize it.
// calloc will zero the buffer
pKey = calloc(1, sizeof(struct ContextKey));
pKey->sock = newsocket;
pKey->ovOut.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
// Set the event for writing so that packets
// will not be sent to the completion port when
// a write finishes.
pKey->ovOut.hEvent = (HANDLE)((DWORD)pKey->ovOut.hEvent | 0x1);
// Associate the socket with the completion port
CreateIoCompletionPort(
(HANDLE)newsocket,
ghCompletionPort,
(DWORD)pKey, // No key
0 // Use default # of threads
);
// Kick off the first read
IssueRead(pKey);
}
return 0;
}
void CreateWorkerThreads()
{
SYSTEM_INFO sysinfo;
DWORD dwThreadId;
DWORD dwThreads;
DWORD i;
GetSystemInfo(&sysinfo);
dwThreads = sysinfo.dwNumberOfProcessors * 2 + 2;
for (i=0; i<dwThreads; i++)
{
HANDLE hThread;
hThread = CreateThread(
NULL, 0, ThreadFunc, NULL, 0, &dwThreadId
);
CloseHandle(hThread);
}
}
//
// Each worker thread starts here.
DWORD WINAPI ThreadFunc(LPVOID pVoid)
{
BOOL bResult;
DWORD dwNumRead;
struct ContextKey *pCntx;
LPOVERLAPPED lpOverlapped;
UNREFERENCED_PARAMETER(pVoid);
// Loop forever on getting packets from
// the I/O completion port.
for (;;)
{
bResult = GetQueuedCompletionStatus(
ghCompletionPort,
&dwNumRead,
&(DWORD)pCntx,
&lpOverlapped,
INFINITE
);
if (bResult == FALSE
&& lpOverlapped == NULL)
{
FatalError(
"ThreadFunc - Illegal call to GetQueuedCompletionStatus");
}
else if (bResult == FALSE
&& lpOverlapped != NULL)
{
// This happens occasionally instead of
// end-of-file. Not sure why.
closesocket(pCntx->sock);
free(pCntx);
fprintf(stderr,
"ThreadFunc - I/O operation failed\n");
}
else if (dwNumRead == 0)
{
closesocket(pCntx->sock);
free(pCntx);
fprintf(stderr, "ThreadFunc - End of file.\n");
}
// Got a valid data block!
// Save the data to our buffer and write it
// all back out (echo it) if we have see a \n
else
{
// Figure out where in the buffer to save the character
char *pch = &pCntx->OutBuffer[pCntx->nOutBufIndex++];
*pch++ = pCntx->InBuffer[0];
*pch = '\0'; // For debugging, WriteFile doesn't care
if (pCntx->InBuffer[0] == '\n')
{
WriteFile(
(HANDLE)(pCntx->sock),
pCntx->OutBuffer,
pCntx->nOutBufIndex,
&pCntx->dwWritten,
&pCntx->ovOut
);
pCntx->nOutBufIndex = 0;
fprintf(stderr, "Echo on socket %x.\n", pCntx->sock);
}
// Start a new read
IssueRead(pCntx);
}
}
return 0;
}
/*
* Call ReadFile to start an overlapped request
* on a socket. Make sure we handle errors
* that are recoverable.
*/
void IssueRead(struct ContextKey *pCntx)
{
int i = 0;
BOOL bResult;
int err;
int numRead;
while (++i)
{
// Request a single character
bResult = ReadFile(
(HANDLE)pCntx->sock,
pCntx->InBuffer,
1,
&numRead,
&pCntx->ovIn
);
// It succeeded immediately, but do not process it
// here, wait for the completion packet.
if (bResult)
return;
err = GetLastError();
// This is what we want to happen, it's not an error
if (err == ERROR_IO_PENDING)
return;
// Handle recoverable error
if ( err == ERROR_INVALID_USER_BUFFER ||
err == ERROR_NOT_ENOUGH_QUOTA ||
err == ERROR_NOT_ENOUGH_MEMORY )
{
if (i == 5) // I just picked a number
{
Sleep(50); // Wait around and try later
continue;
}
FatalError("IssueRead - System ran out of non-paged space");
}
break;
}
fprintf(stderr, "IssueRead - ReadFile failed.\n");
}
//
// Make sure we are running under the right versions
// of Windows NT (3.51, 4.0, or later)
//
void CheckOsVersion()
{
OSVERSIONINFO ver;
BOOL bResult;
ver.dwOSVersionInfoSize = sizeof(OSVERSIONINFO);
bResult = GetVersionEx((LPOSVERSIONINFO) &ver);
if ( (!bResult) ||
(ver.dwPlatformId != VER_PLATFORM_WIN32_NT) )
{
FatalError("ECHOSRV requires Windows NT 3.51 or later.");
}
}
//
// Error handler
//
void FatalError(char *s)
{
fprintf(stdout, "%s\n", s);
exit(EXIT_FAILURE);
}