Full Multi-thread Client/Server Socket Class with ThreadPool

Full Multi-thread Client/Server Socket Class with ThreadPool

Ernest Laurentin, 27 Sep 2009 

Complete Client/Server Socket Communication class with threadpool implementation. Easy to use and integrate into C++ application. Linux/UNIX port available.

To run the application as client, type SocketServer.exe /client from the command prompt.

Introduction

Recently, I updated one of my first articles here at The Code Project, the ServerSocket. While the base class (CSocketHandle) is quite stable and easy to use, one has to admit that some of the initial design decisions to keep the communication interface intact are starting to be an issue for newer development.

This is the goal of this article, I present the new and improved version of the communication class and show how you can take advantage of thread pooling to increase performance for your network solutions.

Description

First, I assume that you are already familiar with socket programming and have several years of experience under your belt. If that is not the case, I highly recommend some links that you will find in the reference section that may guide you along the way. For those who are "all fired up and ready to go", please read on. I will try to shed some light on how you can use the new classes to enhance the performance of your system.

Synchronous Sockets

By default, sockets operate in blocking mode, that means you will need a dedicated thread to read/wait for data while another one will write/send data on the other side. This is now made easier for you by using the new template class.

Typically, a client needs only one thread, so there is no problem there but if you are developing server components and need reliable communication or point-to-point link with your clients, sooner or later, you will find that you need multiple threads to handle your requests.

SocketClientImpl

The first template SocketClientImpl encapsulates socket communication from a client perspective. It can be used to communicate using TCP (SOCK_STREAM) or UDP (SOCK_DGRAM). The good news here is that it handles the communication loop and will report data and several important events in an efficient manner. All this to make the task really straightforward for you.

template <typename T, size_t tBufferSize = 2048>
class SocketClientImpl
{
    typedef SocketClientImpl<T, tBufferSize> thisClass;
public:
    SocketClientImpl()
    : _pInterface(0)
    , _thread(0)
    {
    }

    void SetInterface(T* pInterface)
    {
        ::InterlockedExchangePointer(reinterpret_cast<void**>(&_pInterface), pInterface);
    }

    bool IsOpen() const;
    bool CreateSocket(LPCTSTR pszHost, LPCTSTR pszServiceName,
			int nFamily, int nType, UINT uOptions = 0);
    bool ConnectTo(LPCTSTR pszHostName, LPCTSTR pszRemote,
			LPCTSTR pszServiceName, int nFamily, int nType);
    void Close();
    DWORD Read(LPBYTE lpBuffer, DWORD dwSize,
		LPSOCKADDR lpAddrIn = NULL, DWORD dwTimeout = INFINITE);
    DWORD Write(const LPBYTE lpBuffer, DWORD dwCount,
		const LPSOCKADDR lpAddrIn = NULL, DWORD dwTimeout = INFINITE);
    bool StartClient(LPCTSTR pszHost, LPCTSTR pszRemote,
		LPCTSTR pszServiceName, int nFamily, int nType);
    void Run();
    void Terminate(DWORD dwTimeout = 5000L);

    static bool IsConnectionDropped(DWORD dwError);

protected:
    static DWORD WINAPI SocketClientProc(thisClass* _this);
    T*              _pInterface;
    HANDLE          _thread;
    CSocketHandle   _socket;
};

The client interface reports the following events:

class ISocketClientHandler
{
public:
    virtual void OnThreadBegin(CSocketHandle* ) {}
    virtual void OnThreadExit(CSocketHandle* ) {}
    virtual void OnDataReceived(CSocketHandle* , const BYTE* ,
				DWORD , const SockAddrIn& ) {}
    virtual void OnConnectionDropped(CSocketHandle* ) {}
    virtual void OnConnectionError(CSocketHandle* , DWORD ) {}
};

Function	Description
OnThreadBegin	Called when thread starts
OnThreadExit	Called when thread is about to exit
OnDataReceived	Called when new data arrives
OnConnectionDropped	Called when an error is detected. The error is caused by loss of connection or socket being closed.
OnConnectionError	Called when an error is detected.

This interface is in fact quite optional, your program can be implemented as this:

class CMyDialog : public CDialog
{
    typedef SocketClientImpl<CMyDialog> CSocketClient; // CMyDialog handles events!
public:
    CMyDialog(CWnd* pParent = NULL);   // standard constructor
    virtual CMyDialog ();

    // ...
    void OnThreadBegin(CSocketHandle* ) {}
    void OnThreadExit(CSocketHandle* ) {}
    void OnDataReceived(CSocketHandle* , const BYTE* , DWORD , const SockAddrIn& ) {}
    void OnConnectionDropped(CSocketHandle* ) {}
    void OnConnectionError(CSocketHandle* , DWORD ) {}

protected:
    CSocketClient m_SocketClient;
};

SocketServerImpl

The second template SocketServerImpl handles all the communication tasks from a server perspective. In UDP mode, it behaves pretty much the same way as for the client. In TCP, it delegates the management for each connection in a separate pooling thread. The pooling thread template is a modified version that was published under MSDN by Kenny Kerr (^). You should be able to reuse it in your project without any issue. The good thing is it can be used to call a class member from a thread pool. Callbacks can have the following signature:

void ThreadFunc();
void ThreadFunc(ULONG_PTR);

Remember, you need Windows 2000 or higher to use QueueUserWorkItem. That should not be a problem unless you are targeting Windows CE. I was told no one uses Windows 95/98 anymore! 

class ThreadPool
{
    static const int MAX_THREADS = 50;
    template <typename T>
    struct ThreadParam
    {
        void (T::* _function)(); T* _pobject;
        ThreadParam(void (T::* function)(), T * pobject)
        : _function(function), _pobject(pobject) { }
    };
public:
    template <typename T>
    static bool QueueWorkItem(void (T::*function)(),
                                  T * pobject, ULONG nFlags = WT_EXECUTEDEFAULT)
    {
        std::auto_ptr< ThreadParam<T> > p(new ThreadParam<T>(function, pobject) );
        WT_SET_MAX_THREADPOOL_THREADS(nFlags, MAX_THREADS);
        bool result = false;
        if (::QueueUserWorkItem(WorkerThreadProc<T>,
                                p.get(),
                                nFlags))
        {
            p.release();
            result = true;
        }
        return result;
    }

private:
    template <typename T>
    static DWORD WINAPI WorkerThreadProc(LPVOID pvParam)
    {
        std::auto_ptr< ThreadParam<T> > p(static_cast< ThreadParam<T>* >(pvParam));
        try {
            (p->_pobject->*p->_function)();
        }
        catch(...) {}
        return 0;
    }

    ThreadPool();
};

template <typename T, size_t tBufferSize = 2048>
class SocketServerImpl
{
    typedef SocketServerImpl<T, tBufferSize> thisClass;
public:
    SocketServerImpl()
    : _pInterface(0)
    , _thread(0)
    {
    }

    void SetInterface(T* pInterface)
    {
        ::InterlockedExchangePointer(reinterpret_cast<void**>
					(&_pInterface), pInterface);
    }

    bool IsOpen() const
    bool CreateSocket(LPCTSTR pszHost,
		LPCTSTR pszServiceName, int nFamily, int nType, UINT uOptions);
    void Close();
    DWORD Read(LPBYTE lpBuffer, DWORD dwSize,
		LPSOCKADDR lpAddrIn, DWORD dwTimeout);
    DWORD Write(const LPBYTE lpBuffer, DWORD dwCount,
		const LPSOCKADDR lpAddrIn, DWORD dwTimeout);
    bool Lock()
    {
        return _critSection.Lock();
    }

    bool Unlock()
    {
        return _critSection.Unlock();
    }

    bool CloseConnection(SOCKET sock);
    void CloseAllConnections();
    bool StartServer(LPCTSTR pszHost,
		LPCTSTR pszServiceName, int nFamily, int nType, UINT uOptions);
    void Run();
    void Terminate(DWORD dwTimeout);
    void OnConnection(ULONG_PTR s);

    static bool IsConnectionDropped(DWORD dwError);

protected:
    static DWORD WINAPI SocketServerProc(thisClass* _this);
    T*              _pInterface;
    HANDLE          _thread;
    ThreadSection   _critSection;
    CSocketHandle   _socket;
    SocketList      _sockets;
};

The server interface reports the following events:

class ISocketServerHandler
{
public:
    virtual void OnThreadBegin(CSocketHandle* ) {}
    virtual void OnThreadExit(CSocketHandle* )  {}
    virtual void OnThreadLoopEnter(CSocketHandle* ) {}
    virtual void OnThreadLoopLeave(CSocketHandle* ) {}
    virtual void OnAddConnection(CSocketHandle* , SOCKET ) {}
    virtual void OnRemoveConnection(CSocketHandle* , SOCKET ) {}
    virtual void OnDataReceived
	(CSocketHandle* , const BYTE* , DWORD , const SockAddrIn& ) {}
    virtual void OnConnectionFailure(CSocketHandle*, SOCKET) {}
    virtual void OnConnectionDropped(CSocketHandle* ) {}
    virtual void OnConnectionError(CSocketHandle* , DWORD ) {}
};

This interface is also optional, but I hope you will decide to use it as it makes the design cleaner.

Asynchronous Sockets

Windows supports asynchronous sockets. The communication class CSocketHandle makes it accessible to you as well. You will need to define SOCKHANDLE_USE_OVERLAPPED for your project. Asynchronous communication is a non-blocking mode, thus, allows you to handle multiple requests in a single thread. You may also provide multiple read/write buffers to queue your I/O. Asynchronous sockets is a big subject, probably deserves an article by itself but I hope you will consider the design that is currently supported. The functionsCSocketHandle::ReadEx and CSocketHandle::WriteEx give you access to this mode. The latest templateASocketServerImpl shows how to use SocketHandle class in Asynchronous read mode. The main advantage is that in TCP mode, one thread is used to handle all your connections.

Conclusion

In this article, I present the new improvements for the CSocketHandle class. I hope the new interface will make it easier for you. Of course, I'm always open to suggestions, feel free to post your questions and suggestions to the discussion thread.
Enjoy!

Reference

• Windows Sockets
• Berkeley Sockets (Wikipedia)

History

• 02/12/2009 - Initial release (published as separate article)
• 02/17/2009 - Updated Threadpool startup flag for Windows XP
• 03/14/2009 - Fixed hang issue (99% CPU) in templates
• 03/29/2009 - Asynchronous mode Server template (+ Support: WindowsCE, UNIX/Linux)
• 04/05/2009 - Fixed resource leak in server templates
• 08/07/2009 - Fixed Asynchronous mode build (use SOCKHANDLE_USE_OVERLAPPED)
• 09/26/2009 - IPv6 Support (Windows and Linux)

License

This article, along with any associated source code and files, is licensed under The Apache License, Version 2.0