《Win32多线程程序设计》（14）---操作I/O Completion Port

操作概观

1. 产生一个 I/O completion port。
2. 让它和一个文件handle 产生关联。
3. 产生一堆线程。
4. 让每一个线程都在com pletion port 上等待。
5. 开始对着那个文件handle 发出一些overlapped  I/O 请求。

产生一个 I/O Completion Port

HANDLE CreateIoCompletionPort( 
HANDLE FileHandle,
HANDLE ExistingCompletionPort,
DWORD CompletionKey,
DWORD NumberOfConcurrentThreads
);

参数

FileHandle  文件或设备（device）的handle。在Window s NT 3.51  之后，此栏位可设定为INVALID_HANDLE_VALUE，于是产生一个没有和任何文件handle 有关系的port。

ExistingCompletionPort  如果此栏位被指定，那么上一栏位FileHandle 就会被加到此port  之上，而不会产生新的port。指定NULL 可以产生一个新的port。

CompletionKey  用户自定义的一个数值，将被交给提供服务的线程。此值和FileHandle 有关联。

NumberOfConcurrentThreads  与此I/O com pletion port  有关联的线程个数。

返回值

如果函数成功，则传回一个I/O com pletion port  的handle。如果函数失败，则传回FALSE。GetLastError() 可以获得更详细的失败原因。

CreateIoCompletionPort() 通常被调用两次。第一次先指定FileHandle 为INVALID_HANDLE_VALUE，并设定ExistingCom pletionPort 为NULL，用以产生一个port。然后再为每一个欲附着上去的文件handle  调用一次CreateIoCompletionPort()。后续的这些调用应该将ExistingCom pletionPort 设定为第一次调用所传回的handle。例如：

HANDLE hPort; 
HANDLE hFiles[MAX_FILES]; 
int index; 
//Create the completion port 
hPort = CreateIoCompletionPort( 
INVALID_HANDLE_VALUE, 
NULL, 
0, // key 
0 // default # of threads 
); 
// Now associate each file handle 
for (index = 0; index < OPEN_FILES; index++) 
{ 
CreateIoCompletionPort( 
hFiles[index], 
hPort, 
0, // key 
0 // default # of threads 
); 
} 

产生一堆线程

一旦com pletion port 产生出来，你就可以设立在该port 上等待的那些线程了。I/O completion port  并不自己产生那些线程，它只是使用由你产生的线程。因此，你必须自己以CreateThread() 或_beginthreadex()或AfxBeginThread()产生出线程。

当你一产生这些线程时，它们都应该在com pletion  port 上等待。当线程开始为各个“请求”服务时，池子里的线程的组织如下：

                  目前正在执行的线程

+                被阻塞的线程

+                在com pletion port上等待的线程

-----------------------------------------------------------

=                池子里所有线程的个数

因为如此，所以你应该产生比CPU 个数还多的线程。如果你只有一个CPU，而你也只产生了一个线程，那么当该线程阻塞（blocking）时，你的CPU 也变成闲置（idle）的了。由于池子里没有其他线程，completion port  也就没有办法为任何数据包（packets）服务——甚至即使CPU 的能量游刃有余。合理的线程个数应该是CPU 个数的两倍再加２。

在一个 I/O Completion Port 上等待

Worker线程初始化自己之后，它应该调用GetQueuedCompletionStatus()。这个操作像是WaitForSingleO bject() 和

GetOverlappedResult() 的组合。函数规格如下：

BOOL GetQueuedCompletionStatus( 
HANDLE CompletionPort,
LPDWORD lpNumberOfBytesTransferred,
LPDWORD lpCompletionKey,
LPOVERLAPPED *lpOverlapped,
DWORD dwMilliseconds
);

参数
CompletionPort  将在其上等待的com pletion port。
lpNumberOfBytesTransferred  一个指针，指向DWORD。该DWORD 将收到“被传输的数据字节数”。
lpCompletionKey  一个指针，指向DWORD。该DWORD 将收到由CreateIoCom pletionPort() 所定义的key。
lpOverlapped  这个栏位的名称是个错误。它其实应该命名为lplpOverlapped ，你应该把一个指针的地址放在上面。
overlapped 结构的指针。该结构用以初始化I/O 操作。
dwMilliseconds  等待的最长时间（毫秒）。如果时间终了，
lpOverlapped 将被设为NULL，而函数传回FALSE。
返回值
如果函数成功地将一个com pletion packet  从队列中取出，并完成一个成功的操作，函数将传回TRUE，并填写由lpNum berOfBytesTransferred、lpCompletionKey、lpOverlapped 所指向的变量内容。

发出“Overlapped I/O 请求”

下面这些调用可以启动“能够被一个I/O com pletion port  掌握”的I/O  操作： 

ConnectNamePipe() 
DeviceIoControl() 
LockFileEx() 
ReadFile() 
TransactNamePipe() 
WaitCommEvent() 
WriteFile()

为了使用com pletion port，主线程（或任何其他线程）可以对着一个与此completion port 有关联的文件，进行读、写、或其他任何操作。该线程不需要调用WaitForMultipleObjects()，因为池子里的各个线程都曾经调用过GetQueuedCompletionStatus()。一旦I/O  操作完成，一个等待中的线程将会自动被释放，以服务该操作。

#define WIN32_LEAN_AND_MEAN
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
#include <tchar.h>
#include <string.h>
#include <winsock.h>
#include <io.h>
#include "MtVerify.h"

// Pick a port number that seems to be away from all others
#define SERV_TCP_PORT 5554
#define MAXLINE 512

//
// Structure definition
//

// The context key keeps track of how the I/O
// is progressing for each individual file handle.
struct ContextKey
{
    SOCKET  sock;
    // Input
    char        InBuffer[4];
    OVERLAPPED  ovIn;
    // Output
    int         nOutBufIndex;
    char        OutBuffer[MAXLINE];
    OVERLAPPED  ovOut;
    DWORD       dwWritten;
};

//
// Global variables
//

HANDLE ghCompletionPort;

//
// Function prototypes
//

void CreateWorkerThreads();
DWORD WINAPI ThreadFunc(LPVOID pvoid);
void IssueRead(struct ContextKey *pCntx);
void CheckOsVersion();
void FatalError(char *s);

///

int main(int argc, char *argv[])
{
    SOCKET  listener;
    SOCKET  newsocket;
    WSADATA WsaData;
    struct sockaddr_in serverAddress;
    struct sockaddr_in clientAddress;
    int     clientAddressLength;
    int     err;

    CheckOsVersion();

    err = WSAStartup (0x0101, &WsaData);
    if (err == SOCKET_ERROR)
    {
        FatalError("WSAStartup Failed");
        return EXIT_FAILURE;
    }

    /*
     * Open a TCP socket connection to the server
     * By default, a socket is always opened
     * for overlapped I/O.  Do NOT attach this
     * socket (listener) to the I/O completion
     * port!
     */
    listener = socket(AF_INET, SOCK_STREAM, 0);
    if (listener < 0)
    {
        FatalError("socket() failed");
        return EXIT_FAILURE;
    }

    /*
     * Bind our local address
     */
    memset(&serverAddress, 0, sizeof(serverAddress));
    serverAddress.sin_family      = AF_INET;
    serverAddress.sin_addr.s_addr = htonl(INADDR_ANY);
    serverAddress.sin_port        = htons(SERV_TCP_PORT);

    err = bind(listener,
            (struct sockaddr *)&serverAddress,
            sizeof(serverAddress)
          );
    if (err < 0)
        FatalError("bind() failed");

    ghCompletionPort = CreateIoCompletionPort(
            INVALID_HANDLE_VALUE,
            NULL,   // No prior port
            0,      // No key
            0       // Use default  # of threads
            );
    if (ghCompletionPort == NULL)
        FatalError("CreateIoCompletionPort() failed");

    CreateWorkerThreads(ghCompletionPort);

    listen(listener, 5);

    fprintf(stderr, "Echo Server with I/O Completion Ports\n");
    fprintf(stderr, "Running on TCP port %d\n", SERV_TCP_PORT);
    fprintf(stderr, "\nPress Ctrl+C to stop the server\n");

    //
    // Loop forever accepting requests new connections
    // and starting reading from them.
    //
    for (;;)
    {
        struct ContextKey *pKey;

        clientAddressLength = sizeof(clientAddress);
        newsocket = accept(listener,
                            (struct sockaddr *)&clientAddress,
                            &clientAddressLength);
        if (newsocket < 0)
        {
            FatalError("accept() Failed");
            return EXIT_FAILURE;
        }

        // Create a context key and initialize it.
        // calloc will zero the buffer
        pKey = calloc(1, sizeof(struct ContextKey));
        pKey->sock = newsocket;
        pKey->ovOut.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
        // Set the event for writing so that packets
        // will not be sent to the completion port when
        // a write finishes.
        pKey->ovOut.hEvent = (HANDLE)((DWORD)pKey->ovOut.hEvent | 0x1);

        // Associate the socket with the completion port
        CreateIoCompletionPort(
                (HANDLE)newsocket,
                ghCompletionPort,
                (DWORD)pKey,   // No key
                0              // Use default # of threads
            );

        // Kick off the first read
        IssueRead(pKey);
    }
    return 0;
}

void CreateWorkerThreads()
{
    SYSTEM_INFO  sysinfo;
    DWORD        dwThreadId;
    DWORD        dwThreads;
    DWORD        i;

    GetSystemInfo(&sysinfo);
    dwThreads = sysinfo.dwNumberOfProcessors * 2 + 2;
    for (i=0; i<dwThreads; i++)
    {
        HANDLE hThread;
        hThread = CreateThread(
            NULL, 0, ThreadFunc, NULL, 0, &dwThreadId
            );
        CloseHandle(hThread);
    }
}

//
// Each worker thread starts here.
DWORD WINAPI ThreadFunc(LPVOID pVoid)
{
    BOOL    bResult;
    DWORD   dwNumRead;
    struct ContextKey *pCntx;
    LPOVERLAPPED lpOverlapped;

    UNREFERENCED_PARAMETER(pVoid);

    // Loop forever on getting packets from
    // the I/O completion port.
    for (;;)
    {
        bResult = GetQueuedCompletionStatus(
               ghCompletionPort,
               &dwNumRead,
               &(DWORD)pCntx,
               &lpOverlapped,
               INFINITE
            );

        if (bResult == FALSE
            && lpOverlapped == NULL)
        {
            FatalError(
                "ThreadFunc - Illegal call to GetQueuedCompletionStatus");
        }

        else if (bResult == FALSE
            && lpOverlapped != NULL)
        {
            // This happens occasionally instead of
            // end-of-file. Not sure why.
            closesocket(pCntx->sock);
            free(pCntx);
            fprintf(stderr,
                "ThreadFunc - I/O operation failed\n");
        }

        else if (dwNumRead == 0)
        {
            closesocket(pCntx->sock);
            free(pCntx);
            fprintf(stderr, "ThreadFunc - End of file.\n");
        }

        // Got a valid data block!
        // Save the data to our buffer and write it
        // all back out (echo it) if we have see a \n
        else
        {
            // Figure out where in the buffer to save the character
            char *pch = &pCntx->OutBuffer[pCntx->nOutBufIndex++];
            *pch++ = pCntx->InBuffer[0];
            *pch = '\0';    // For debugging, WriteFile doesn't care
            if (pCntx->InBuffer[0] == '\n')
            {
                WriteFile(
                        (HANDLE)(pCntx->sock),
                        pCntx->OutBuffer,
                        pCntx->nOutBufIndex,
                        &pCntx->dwWritten,
                        &pCntx->ovOut
                    );
                pCntx->nOutBufIndex = 0;
                fprintf(stderr, "Echo on socket %x.\n", pCntx->sock);
            }

            // Start a new read
            IssueRead(pCntx);
        }
    }

    return 0;
}


/*
 * Call ReadFile to start an overlapped request
 * on a socket.  Make sure we handle errors
 * that are recoverable.
 */
void IssueRead(struct ContextKey *pCntx)
{
    int     i = 0;
    BOOL    bResult;
    int     err;
    int     numRead;

    while (++i)
    {
        // Request a single character
        bResult = ReadFile(
                    (HANDLE)pCntx->sock,
                    pCntx->InBuffer,
                    1,
                    &numRead,
                    &pCntx->ovIn
                );

        // It succeeded immediately, but do not process it
        // here, wait for the completion packet.
        if (bResult)
            return;

        err = GetLastError();

        // This is what we want to happen, it's not an error
        if (err == ERROR_IO_PENDING)
            return;

        // Handle recoverable error
        if ( err == ERROR_INVALID_USER_BUFFER ||
             err == ERROR_NOT_ENOUGH_QUOTA ||
             err == ERROR_NOT_ENOUGH_MEMORY )
        {
            if (i == 5) // I just picked a number
            {
                Sleep(50);  // Wait around and try later
                continue;
            }

            FatalError("IssueRead - System ran out of non-paged space");
        }

        break;
    }

    fprintf(stderr, "IssueRead - ReadFile failed.\n");
}

//
// Make sure we are running under the right versions
// of Windows NT (3.51, 4.0, or later)
//
void CheckOsVersion()
{
    OSVERSIONINFO   ver;
    BOOL            bResult;

    ver.dwOSVersionInfoSize = sizeof(OSVERSIONINFO);

    bResult = GetVersionEx((LPOSVERSIONINFO) &ver);

    if ( (!bResult) ||
         (ver.dwPlatformId != VER_PLATFORM_WIN32_NT) )
    {
        FatalError("ECHOSRV requires Windows NT 3.51 or later.");
    }

}

//
// Error handler
//
void FatalError(char *s)
{
    fprintf(stdout, "%s\n", s);
    exit(EXIT_FAILURE);
}