《A Simple IOCP Server/Client Class》整改

《A Simple IOCP Server/Client Class》

 

一．项目概述

1.pre：

预编译器文件stdafx。

 

2.app：MFC主线程。

    BOOL CIOCPApp::InitInstance()

    {

        // ……

        CIOCPDlg dlg; // main dialog

        // ……

    }

 

3.rc：资源文件。

 

4.gui：主对话框以及一些自绘制控件。

    （1）class CHistoryEdit : public CEdit

    （2）class MyListCtrl : public CListCtrl

    （3）class IOCPSettingsDlg : public CDialog

    IOCPSettingsDlg使用UpdateData()<->DDX()实现对IOCP的配置。

    IOCPSettingsDlg包含一个Save按钮（IDOK），但并没有覆写OnOK()响应。

    单击OK按钮将调用基类CDialog的OnOk()函数，在OnOK()中调用了UpdateData(TRUE)，将数据从控件传递到关联的成员变量。

    （4）

    class CIOCPDlg : public CDialog

    {

        CHistoryEdit     m_CtrlLogg;

        MyListCtrl       m_CtrlClientList;

        IOCPSettingsDlg m_ConfigDlg;

        

        MyIOCP           m_iocp; // network i/o handler

    }

 

5.i/o数据结构：

    基于完成端口的重叠I/O网络通信业务逻辑主要涉及“单I/O数据”和“单句柄数据”。

    <1>单I/O数据（per i/o data）

    CIOCPBuffer类包含一个重叠I/O所需要的重叠结构(OVERLAPPED)和I/O缓冲区(WSABUF)，它代表一次重叠I/O操作。

    CIOCPBuffer类还提供了缓冲区管理(m_nUsed)、打包(CreatePackage)、解包(GetPackageInfo)和I/O序列(m_iSequenceNumber)机制。

    class CIOCPBuffer

    {

    public:

        // 用于WSASend/WSARecv的OVERLAPPED结构

        OVERLAPPED m_ol;

 

    private:

        // 用于WSASend/WSARecv的WSABUF结构

        WSABUF m_wsabuf;

 

        // 实际缓冲区

        BYTE m_Buffer[MAXIMUMPACKAGESIZE];

        // 已占用字节数

        UINT m_nUsed;

 

        // I/O类型，取值enum IOType

        int m_Operation;

        // I/O序列号

        int m_iSequenceNumber;

        // I/O在外部列表(IOCPS::m_BufferList)中的位置(指针)

        POSITION m_pPos;

    }

    <2>单句柄数据（per handle data）

    ClientContext结构代表一个TCP连接的客户端（endpoint），它提供了对客户通信的I/O有序化管理。

    struct ClientContext

    {

        // 客户套接字(代表一个TCP连接)

        SOCKET        m_Socket;

 

        // 在该连接上挂起的I/O数，释放该结构前检查以避免Access Violation

        int           m_nNumberOfPendlingIO;

 

        // 发送序列机制

        unsigned int     m_SendSequenceNumber;        // apply for WSASend post:         IOCPS::ASend→IOCPS::SetSendSequenceNumber→CIOCPBuffer::SetSequenceNumber

        unsigned int     m_CurrentSendSequenceNumber; // expect for sequential WSASend operation: IOCPS::OnWrite→IOCPS::GetNextSendBuffer/IOCPS::IncreaseSendSequenceNumber

        BufferMap        m_SendBufferMap;             // out of order writing

 

        // 接收序列机制

        unsigned int  m_ReadSequenceNumber;        // apply for WSARecv post: IOCPS::MakeOrderdRead→CIOCPBuffer::SetSequenceNumber

        unsigned int  m_CurrentReadSequenceNumber; // expect from WSARecv completion: →IOCPS::OnReadCompleted→IOCPS::IncreaseReadSequenceNumber/IOCPS::GetNextReadBuffer

        BufferMap     m_ReadBufferMap;             // out of order reading

 

        // 文件传输模式支持

    #ifdef TRANSFERFILEFUNCTIONALITY

        CFile m_File;                 // file to send/receive

        unsigned int m_iMaxFileBytes; // file size

        unsigned int m_iFileBytes;    // sent/received completed

        BOOL m_bFileSendMode;         // file sender

        BOOL m_bFileReceivedMode;     // file receiver

    #endif

 

        // 组包单元(for assembling)

        CIOCPBuffer* m_pBuffOverlappedPackage;

 

        // 额外数据

        CString m_sReceived;

        int m_iNumberOfReceivedMsg;

        BOOL m_bUpdate;

    };

 

6.序列号机制

每个i/o buffer都有序列号，每个ClientContext都有自己的read/send序列号。

序列号从0~2^31-1回环。

 

二．系统概述

<IOCPS.h><IOCPS.cpp>

|--<MyIOCP.h><MyIOCP.cpp>

 

class IOCPS

{

    // management for client contexts

    ContextMap    m_ContextMap;

    CPtrList      m_FreeContextList;         // pool

    int           m_iMaxNumberOfFreeContext; // pool size

   

    // management for overlapped i/o buffers

    CPtrList      m_BufferList;

    CPtrList      m_FreeBufferList;         // pool

    int           m_iMaxNumberOfFreeBuffer; // pool size

 

    // listener servo

    static UINT ListenerThreadProc(LPVOID pParam);

    HANDLE m_hCompletionPort;

                   

    // i/o dispatcher

    static UINT   IOWorkerThreadProc(LPVOID pParam);

    CPtrList      m_IOWorkerList;

    int           m_nIOWorkers;

    void          ProcessIOMessage(CIOCPBuffer* pOverlapBuff, ClientContext* pContext, DWORD dwSize);

   

    // i/o handler(ProcessJob)

    static UINT   WorkerThreadProc(LPVOID pParam); 

    CMapWordToPtr m_WorkerThreadMap;

    int           m_nOfWorkers;

   

    // job is designed for outgoing writing such as file sending procedure

    CPtrList      m_JobQueueList;                            // job queue

    virtual void  ProcessJob(JobItem* pJob, IOCPS* pServer); // deal with jobs

   

    // assemble(AddAndFlush()) and notify to make custom protocol analyzation(NotifyReceivedPackage())

        // invoked by OnReadCompleted()

    void ProcessPackage(ClientContext* pContext, DWORD dwIoSize, CIOCPBuffer* pOverlapBuff);

};

 

1.system outline

 

    完成端口是一个基于状态机的I/O调度器(iocp dispatcher)，I/O操作(ASend/ARead)中PostQueuedCompletionStatus()。

    I/O调度线程(IOWorkerThreadProc)中，根据状态码(IOType)来协调I/O过程。

 

2.IOCPS::ListenerThreadProc()

    listener servo: wait for "shutdown" event and "accept" event.

 

    监听套接字采用基于事件通知(WSAEventSelect）的重叠I/O模型(WSAAccept）。

    接入的客户套接字采用基于完成端口通知的重叠I/O模型(WSARecv/WSASend）。

    SOCKET clientSocket = WSAAccept(); // 接入连接

    IOCPS::AssociateIncomingClientWithContext(SOCKET clientSocket);

    ↓

    CIOCPBuffer* pOverlapBuff = AllocateBuffer(IOInitialize);

    ::PostQueuedCompletionStatus(); // fire up the iocp dispatcher

   

3.IOCPS::IOWorkerThreadProc()

    i/o dispatcher: watch for the completion port and dispatch i/o completion notification.

   

    (1)ASend()/AZeroByteRead()/ARead()->PostQueuedCompletionStatus()

    (2)PostPackage()->PostQueuedCompletionStatus()

 

4.IOCPS::WorkerThreadProc()

    i/o handler: deal with heavy task.

   

    // job items list

    CPtrList m_JobQueueList;

   

    // Adds a job to the queue.

    BOOL AddJob(JobItem* pJob);

    // Get a Job.

    JobItem* GetJob();

    // Called to do some work.

    virtual inline void ProcessJob(JobItem* pJob, IOCPS* pServer);

    // Clear the Job from the heap.

    inline void FreeJob(JobItem* pJob);

   

CString JobItem::m_Data可扩展为缓冲区char[]，以执行具体任务。

 

三．系统流程

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                               initialization                                |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

IOCPS::AssociateIncomingClientWithContext(SOCKET) // attach tcp socket to ClientContext structure

    EnterIOLoop(ClientContext);

    CIOCPBuffer* pOverlapBuff = AllocateBuffer(IOInitialize);

    ::PostQueuedCompletionStatus();

-------------------------------------------------------------------------------

The initialization for the incoming client will fire up the iocp dispatcher

to take over the charge of its i/o procedure.

-------------------------------------------------------------------------------

 

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                               zero read loop                                |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

IOInitialize->OnInitialize

    AZeroByteRead(); // SetOperation(IOZeroByteRead)

    for (int i=0; i<m_iNumberOfPendlingReads; i++)

    {

        EnterIOLoop(pContext);

        ARead(); // AllocateBuffer(IORead)

    }

 

IOZeroByteRead->OnZeroByteRead

    WSARecv(0); // SetOperation(IOZeroReadCompleted)

 

IOZeroReadCompleted->OnZeroByteReadCompleted

    AZeroByteRead() // post another zero read operation to make a loop

-------------------------------------------------------------------------------

The zero read loop act as a probe to notify application when incoming data pending.

-------------------------------------------------------------------------------

 

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                                  read loop                                  |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

IORead->OnRead

    MakeOrderdRead()

        WSARecv(MAXIMUMPACKAGESIZE) // SetOperation(IOReadCompleted)

 

IOReadCompleted->OnReadCompleted->NotifyReadCompleted

    ProcessPackage()/AddToFile() // custom protocol analyzation for incoming data

    ARead() // post another read operation to make a loop

-------------------------------------------------------------------------------

For each client,we initiate a number of read oepration waiting for incoming data.

The read operation post is automatically made,so it's always ready to receive incoming data.

-------------------------------------------------------------------------------

 

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                               manual write                                  |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

MyIOCP::BuildPackageAndSend

    AllocateBuffer(IOWrite)

    ASend()

        EnterIOLoop()

        PostQueuedCompletionStatus()

 

IOWrite->OnWrite

    WSASend() // SetOperation(IOWriteCompleted)

 

IOWriteCompleted->OnWriteCompleted->NotifyWriteCompleted

-------------------------------------------------------------------------------

The read opration loop is automate machine, but we should invoke ASend() to start up

data sending procedure.

-------------------------------------------------------------------------------

 

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                                iocp packet                                  |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

PostPackage()

    EnterIOLoop()

    SetOperation(IOPostedPackage)

    PostQueuedCompletionStatus()

 

IOPostedPackage->OnPostedPackage->NotifyReceivedPackage

-------------------------------------------------------------------------------

PostPackage() is used to post request into IOCP (simulate received packages).

This function is necessary to split heavy computation operation into several

 parts. (automate machine)

This functions can be used instead of the  function addJob(..).

-------------------------------------------------------------------------------

 

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                            file transmission                                |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

(1)file sender:

MyIOCP::BuildFilePackageAndSend()

    PrepareSendFile() // open a file(m_bFileSendMode = TRUE)

        CIOCPBuffer.CreatePackage(Job_SendFileInfo)

            AllocateBuffer(IOWrite);

            EnterIOLoop();

            PostQueuedCompletionStatus(); // send file information

 

(2)file receiver:

IOCPS::OnReadCompleted()

    ProcessPackage()

        NotifyReceivedPackage()

            PackageFileTransfer()

                PrepareReceiveFile() // create a file(m_bFileReceivedMode = TRUE)

                    AllocateBuffer(IOWrite)

                    CreatePackage(Job_StartFileTransfer)

                    ASend()

   

(3)file sender:

IOCPS::OnReadCompleted()

    ProcessPackage()

        NotifyReceivedPackage()

            PackageStartFileTransfer()

                StartSendFile() // m_bFileSendMode = TRUE

                    AllocateBuffer(IOWrite)

                    EnterIOLoop()

                    SetOperation(IOTransmitFileCompleted)

                    TransmitFile()

 

IOTransmitFileCompleted->OnTransmitFileCompleted->NotifyFileCompleted // file send completed

 

(4)file receiver:

IOCPS::OnReadCompleted()

    AddToFile()

        if (pContext->m_iFileBytes == pContext->m_iMaxFileBytes)

        {

            NotifyFileCompleted(pContext); // file receive completed

        }

 

四．修改完善

参考《NKittyServer》

 

1.去MFC化

(1)CFile              -> FILE

(2)CString            -> std::string

(3)CCriticalSection   -> Lock

(4)CWinThread

    <1>CWinThread*    -> HANDLE

    <2>AfxBeginThread -> CreateThread

    <3>CPtrList       m_IOWorkerList;

    <4>CMapWordToPtr  m_WorkerThreadMap;

    <5>CWinThread*    GetWorker(WORD WorkerID);

(5)CPtrList/CMap -> std adapter

    m_OneIPPerConnectionList                 : CPtrList      -> std::list

    m_BanIPList                              : CPtrList      -> std::list

    m_BufferList                             : CPtrList      -> std::list

    ContextMap                               : CMap          -> std::map

    BufferMap         -> ReorderQueue        : CMap          -> std::priority_queue

    m_FreeContextList -> m_FreeContextVector : CPtrList      -> std::vector

    m_FreeBufferList  -> m_FreeBufferVector  : CPtrList      -> std::vector

    m_IOWorkerList    -> m_IOWorkerVector    : CPtrList      -> std::vector

    m_WorkerThreadMap -> m_WorkerVector      : CMapWordToPtr -> std::vector

    m_JobQueueList    -> m_JobQueue          : CPtrList      -> std::queue

 

2.客户端拔线的情况

TCP一方A进程崩溃或手动终止进程，网络仍处于连接状态，此时A应用层面的进程管理机制将释放连接资源，并向对方B发送RST报文重置连接，对方B收到RST报文，向应用层返回WSAECONNRESET错误。

TCP一方A禁用网络或拔掉网线导致网络中断，A本地状态机能够检测到异常，但是无法向对方B发送RST报文。此时，这条连接就成了“死连接”，直到对方B发送数据才能探测到异常，返回WSAEHOSTUNREACH或WSAECONNABORTED（WSAETIMEDOUT）；如果B也不发数据，那么只有依靠B的TCP心跳（Keep Alive Probe）机制来检测状态，返回WSAENETRESET。

如果2小时内套接口上任一方向都没有数据交换，即无I/O，则B方的TCP自动给对方A发一个心跳包[TCP Keep-Alive]，如果超时无响应，则B认为对方A已放弃该连接，重置（reset）连接回收资源。

对于客户端拔线的“死连接”情况，采用了RegisterWaitForSingleObject的TimeOutCallback函数进行处理。如果有I/O完成，说明连接处于活动状态，置信hAlive事件。如果超过一定时间（如3分钟）无数据进出，则超时(TimerOrWaitFired==TRUE)，在TimeOutCallback中断开该连接。

 

IOCPS::AssociateIncomingClientWithContext

    pContext->hAlive = CreateEvent(NULL,FALSE,FALSE,NULL);

    RegisterWaitForSingleObject(&pContext->hTimeOut, pContext->hAlive, …);

 

IOCPS::OnWriteCompleted

    SetEvent(pContext->hAlive);

 

IOCPS::OnReadCompleted

    SetEvent(pContext->hAlive);

 

IOCPS::AbortiveClose/IOCPS::ReleaseClientContext

    CloseHandle(pContext->hAlive);

 

IOCPS::AbortiveClose/IOCPS::ReleaseClientContext

    UnregisterWaitEx(mp->hTimeOut,NULL);

 

3.WSASend实际完成返回的大小小于请求发送的大小。

IOCPS::OnWriteCompleted()

{

    if (pContext)

    {

        if (pOverlapBuff)

        {

           //发送小于需要发送的大小

            if (pOverlapBuff->GetUsed()!=dwIoSize)

            {

                if (dwIoSize < pOverlapBuff->GetUsed() && dwIoSize > 0)

                {

                    // 这应该是正常的

                    // ReleaseBuffer(pOverlapBuff);

                    if (pOverlapBuff->Flush(dwIoSize) == TRUE)

                    {

                        pOverlapBuff->SetOperation(IOWrite);

                        ASend(pContext,pOverlapBuff);

                    }

                }

            }

            else

            {

            }

        }

    }

}

 

4.WSARecv的自动化处理机制

应该先投递WSARecv(0)，如果返回，表示有数据进入（incoming），投递真正的WSARecv进行读取。这样才发挥了零字节读取的通知作用，同时又不浪费分配CIOCPBuffer的缓冲区(m_Buffer[MAXIMUMPACKAGESIZE])

void IOCPS::OnInitialize(ClientContext* pContext, DWORD dwIoSize, CIOCPBuffer* pOverlapBuff)

{

    // Do some init here..

    // Notify new connection.

    pContext->m_ContextLock.On();

    NotifyNewConnection(pContext);

    pContext->m_ContextLock.Off();

   

    // A ZeroByteLoop. EnterIOLoop is not needed here. Already done in previous call.

    // AZeroByteRead(pContext, pOverlapBuff);

 

    // m_iNumberOfPendlingReads=1 by default.

    for (int i=0; i<m_iNumberOfPendlingReads; i++)

    {

        // EnterIOLoop(pContext); // One for each Read Loop

        // ARead(pContext);

        AZeroByteRead(pContext, pOverlapBuff);

    }

}

 

当ZeroByteReadCompleted完成的时候，投递真正的ARead。

void IOCPS::OnZeroByteReadCompleted(ClientContext* pContext, DWORD dwIoSize, CIOCPBuffer* pOverlapBuff)

{

    if (pContext)

    {

        // Make a Loop.

        // AZeroByteRead(pContext, pOverlapBuff);

        ARead(pContext, pOverlapBuff);

    }

}

 

当ReadCompleted完成的时候，再投递AZeroByteRead(pContext, pOverlapBuff);

void IOCPS::OnReadCompleted(ClientContext* pContext, DWORD dwIoSize, CIOCPBuffer* pOverlapBuff)

{

    // ……

    AZeroByteRead(pContext, pOverlapBuff); // ARead(pContext);

}

 

5.EnterIOLoop()/ExitIOLoop()的对称性计数问题

原来在ReleaseClientContext才调用ExitIOLoop()，应该将I/O挂起计数独立出来，只与I/O过程相关。

修改为投递I/O时EnterIOLoop()，I/O完成时ExitIOLoop()。

(1)需要修改ReleaseContext逻辑。

(2)每一步投入(Post)时EnterIOLoop()，完成时ExitIOLoop()，这样跟踪整个流程状态，而不仅仅是WSASend/WSARecv。

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                                    read                                     |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

<1>IOInitialize

AssociateIncomingClientWithContext()

    EnterIOLoop(pContext, IOInitialize);

ProcessIOMessage()

    ExitIOLoop(pContext, IOInitialize);

    OnInitialize();

 

<2>IOZeroByteRead

OnInitialize()->AZeroByteRead()

    EnterIOLoop(pContext, IOZeroByteRead);

ProcessIOMessage()

    ExitIOLoop(pContext, IOZeroByteRead);

    OnZeroByteRead();

 

<3>IOZeroReadCompleted

OnZeroByteRead()

    EnterIOLoop(pContext, IOZeroReadCompleted);

    WSARecv(0);

ProcessIOMessage()

    ExitIOLoop(pContext, IOZeroReadCompleted);

    OnZeroByteReadCompleted();

 

<4>IORead

OnZeroByteReadCompleted()->ARead() // AssociateIncomingClientWithContext()中分配的缓冲区沿用至今

    EnterIOLoop(pContext, IORead);

ProcessIOMessage()

    ExitIOLoop(pContext, IORead);

    OnRead();

 

<5>IOReadCompleted

OnRead()

    EnterIOLoop(pContext, IOReadCompleted);

    WSARecv(); // MakeOrderRead();

ProcessIOMessage()

    ExitIOLoop(pContext,IOReadCompleted);

    OnReadCompleted();   // ProcessPackage()/AddToFile()->NotifyReceivedMessage()

        AZeroByteRead(); // automate machine

 

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                                    write                                    |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

<1>IOWrite

ASend()

    EnterIOLoop(pContext, IOWrite);

ProcessIOMessage()

    ExitIOLoop(pContext, IOWrite);

    OnWrite();

 

<2>IOWriteCompleted

OnWrite();

    EnterIOLoop(pContext, IOWriteCompleted);

    WSASend();

ProcessIOMessage()

    ExitIOLoop(pContext, IOWriteCompleted);

    OnWriteCompleted();

 

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                                iocp packet                                  |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++   

<1>IOPostedPackage

PostPackage()

    EnterIOLoop(pContext, IOPostedPackage);

ProcessIOMessage()

    ExitIOLoop(pContext, IOPostedPackage);

    OnPostedPackage(); // NotifyReceivedPackage()

 

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

|                            file transmission                                |

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

<1>IOWrite & Job_SendFileInfo

PrepareSendFile()

    pOverlapBuff->CreatePackage(Job_SendFileInfo,iFileSize,sFileName)

    EnterIOLoop(pContext, IOWrite); // 后续同上

 

<2>IOWrite & IOTransmitFileCompleted

StartSendFile()

    EnterIOLoop(pContext, IOTransmitFileCompleted);

    TransmitFile();

ProcessIOMessage()

    ExitIOLoop(pContext, IOTransmitFileCompleted);

    OnTransmitFileCompleted(pContext,pOverlapBuff);

 

6.加解锁条件判断截断问题

在Lock.On()和Lock.Off()过程中遇条件截断提前return，而没有解锁。

BOOL MyIOCP::BuildFilePackageAndSend(int ClientID, CString sFile)

{

    BOOL bRet = FALSE;

    m_ContextMapLock.On();

    ClientContext* pContext = FindClient(ClientID);

 

    if (!pContext)

    {

        m_ContextMapLock.Off(); // 退出解锁

        return FALSE;

    }

 

    bRet = BuildFilePackageAndSend(pContext, sFile);

    m_ContextMapLock.Off();

 

    return bRet;

}

 

其他的地方：

SetWorkers()

    m_WorkerThreadMapLock.On();

    return FALSE; // 在return之前应该解锁

m_WorkerThreadMapLock.Off();

 

五．代码下载

《A Simple IOCP Server/Client Class》