epoll 入门例子 tcp server/client

原文： https://rebootcat.com/2020/09/26/epoll_examples/

复习一下

上一篇博文 epoll原理深入分析 详细分析了 epoll 底层的实现原理，如果对 epoll 原理有模糊的建议先看一下这篇文章。那么本文就开始用 epoll 实现一个简单的 tcp server/client。

本文基于我的 github: https://github.com/smaugx/epoll_examples。

epoll 实现范式

# create listen socket
int listenfd = ::socket();

# bind to local port and ip
int r = ::bind();


# create epoll instance and get an epoll-fd
int epollfd = epoll_create(1);
 
# add listenfd to epoll instance
int r = epoll_ctl(..., listenfd, ...);


# begin epoll_wait, wait for ready socket

struct epoll_event* alive_events =  static_cast<epoll_event*>(calloc(kMaxEvents, sizeof(epoll_event)));

while (true) {
            int num = epoll_wait(epollfd, alive_events, kMaxEvents, kEpollWaitTime);
    
            for (int i = 0; i < num; ++i) {
            int fd = alive_events[i].data.fd;
            int events = alive_events[i].events;

            if ( (events & EPOLLERR) || (events & EPOLLHUP) ) {
                std::cout << "epoll_wait error!" << std::endl;
                // An error has occured on this fd, or the socket is not ready for reading (why were we notified then?).
                ::close(fd);
            } else  if (events & EPOLLRDHUP) {
                // Stream socket peer closed connection, or shut down writing half of connection.
                // more inportant, We still to handle disconnection when read()/recv() return 0 or -1 just to be sure.
                std::cout << "fd:" << fd << " closed EPOLLRDHUP!" << std::endl;
                // close fd and epoll will remove it
                ::close(fd);
            } else if ( events & EPOLLIN ) {
                std::cout << "epollin" << std::endl;
                if (fd == handle_) {
                    // listen fd coming connections
                    OnSocketAccept();
                } else {
                    // other fd read event coming, meaning data coming
                    OnSocketRead(fd);
                }
            } else if ( events & EPOLLOUT ) {
                std::cout << "epollout" << std::endl;
                // write event for fd (not including listen-fd), meaning send buffer is available for big files
                OnSocketWrite(fd);
            } else {
                std::cout << "unknow epoll event!" << std::endl;
            }
        } // end for (int i = 0; ...
    
}

epoll 编程基本是按照上面的范式进行的，这里要注意的是上面的反应的只是单进程或者单线程的情况。

如果涉及到多线程或者多进程，那么通常来说会在 listen() 创建完成之后，创建多线程或者多进程，然后再操作 epoll.

int listenfd = ::socket();


...

int p = fork() # 多进程 或者多线程创建

int r = epoll_ctl(..., listenfd, ...);

...

while(true) {
int num = epoll_wait(epollfd, alive_events, kMaxEvents, kEpollWaitTime);
...
}

同理，多线程版本也是一样，把上面的 fork() 替换成 thread 创建即可。

也就是 listenfd 被添加到了多个进程或者多个线程中，提高吞吐量。这就是基本的 epoll 多进程或者多线程编程范式。

但本文就先讨论单进程（单线程）版本的 epoll 实现。

epoll tcp server

先上代码：

#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <stdio.h>
#include <cstring>
#include <stdlib.h>
#include <sys/epoll.h>
#include <sys/socket.h>
#include <unistd.h>
#include <cassert>

#include <iostream>
#include <string>
#include <thread>
#include <memory>
#include <functional>


namespace mux {

namespace transport {

static const uint32_t kEpollWaitTime = 10; // 10 ms
static const uint32_t kMaxEvents = 100;


typedef struct Packet {
public:
    Packet()
        : msg { "" } {}
    Packet(const std::string& msg)
        : msg { msg } {}
    Packet(int fd, const std::string& msg)
        : fd(fd),
          msg(msg) {}

    int fd { -1 };
    std::string msg;
} Packet;

typedef std::shared_ptr<Packet> PacketPtr;

using callback_recv_t = std::function<void(const PacketPtr& data)>;

class EpollTcpBase {
public:
    EpollTcpBase()                                     = default;
    EpollTcpBase(const EpollTcpBase& other)            = delete;
    EpollTcpBase& operator=(const EpollTcpBase& other) = delete;
    EpollTcpBase(EpollTcpBase&& other)                 = delete;
    EpollTcpBase& operator=(EpollTcpBase&& other)      = delete;
    virtual ~EpollTcpBase()                            = default; 

public:
    virtual bool Start() = 0;
    virtual bool Stop()  = 0;
    virtual int32_t SendData(const PacketPtr& data) = 0;
    virtual void RegisterOnRecvCallback(callback_recv_t callback) = 0;
    virtual void UnRegisterOnRecvCallback() = 0;
};

using ETBase = EpollTcpBase;

typedef std::shared_ptr<ETBase> ETBasePtr;

class EpollTcpServer : public ETBase {
public:
    EpollTcpServer()                                       = default;
    EpollTcpServer(const EpollTcpServer& other)            = delete;
    EpollTcpServer& operator=(const EpollTcpServer& other) = delete;
    EpollTcpServer(EpollTcpServer&& other)                 = delete;
    EpollTcpServer& operator=(EpollTcpServer&& other)      = delete;
    ~EpollTcpServer() override;

    EpollTcpServer(const std::string& local_ip, uint16_t local_port);

public:
    bool Start() override;
    bool Stop() override;
    int32_t SendData(const PacketPtr& data) override;
    void RegisterOnRecvCallback(callback_recv_t callbac