muduo网络库设计与实现(三)

muduo网络库设计与实现(三)

多线程网络库

其实要在前文构建的单线程网络库扩展到多线程是非常简单的。我们回顾一下muduo的多线程框架，有一个main Reactor负责accept连接，然后把连接挂在某个sub Reactor中，这里的连接具体是什么，是TcpConnection对象，所以我们需要做如下修改：

1. 定义一个IO线程池的类EventLoopThreadPool负责管理 sub Reactor。
2. 在TcpServer类中添加一个EventLoopThreadPool对象。
3. 修改TcpServer::newConnection()，将新连接放到 sub Reactor 中处理。

EventLoopThreadPool

class EventLoopThreadPool : noncopyable{
public:
    EventLoopThreadPool(EventLoop* baseLoop);
    ~EventLoopThreadPool();
    void setThreadNum(int numThreads) { numThreads_ = numThreads; }
    void start();
    EventLoop* getNextLoop();

private:
    EventLoop* baseLoop_; // main Reactor的loop
    bool started_;
    int numThreads_; // 多少个sub Reactor，0代表main Reactor也负责处理TcpConnection
    int next_;
    std::vector<EventLoopThread*> threads_;
    std::vector<EventLoop*> loops_;
};

这个类中有两个关键函数，start()和getNextLoop()。会在start()中构建子线程。而getNextLoop()的作用是返回下一个用于处理新连接的EventLoop对象的指针，这里使用简单的round-robin。

void EventLoopThreadPool::start(){
    assert(!started_);
    baseLoop_->assertInLoopThread();
    started_ = true;
    for (int i = 0; i < numThreads_; ++i){
        EventLoopThread* t = new EventLoopThread;
        threads_.push_back(t);
        loops_.push_back(t->startLoop());
    }
}

EventLoop* EventLoopThreadPool::getNextLoop(){
    baseLoop_->assertInLoopThread();
    EventLoop* loop = baseLoop_;
    if(!loops_.empty()){
        // round-robin
        loop = loops_[next_];
        next_ = (next_ + 1) % numThreads_;
    }
    return loop;
}

TcpServer的修改

在单线程中，将TcpServer自用的loop_传给TcpConnection，现在每次从EventLoopThreadPool取出ioLoop来处理新连接，相应的连接的销毁也要作出修改。

void TcpServer::newConnection(int sockfd, const InetAddress& peerAddr){
    loop_->assertInLoopThread();
    char buf[32];
    snprintf(buf, sizeof(buf), "#%d", nextConnId_);
    ++nextConnId_;
    std::string connName = name_ + buf;

    InetAddress localAddr(getLocalAddr(sockfd));

+   EventLoop* ioLoop = threadPool_->getNextLoop();

!   TcpConnectionPtr conn(new TcpConnection(ioLoop, connName, sockfd, localAddr, peerAddr));
    connections_[connName] = conn;
    conn->setConnectionCallback(connectionCallback_);
    conn->setMessageCallback(messageCallback_);
    conn->setCloseCallback(std::bind(&TcpServer::removeConnection, this, std::placeholders::_1));
!   ioLoop->runInLoop(std::bind(&TcpConnection::connectEstablished, conn));
}

TcpClient & Connector

这部分是提供给客户端的编写的简单框架

Connector
Connector负责主动发起连接，不负责创建socket，只负责连接的建立，外部调用Connector::start()就可以发起连接，Connector具有重连的功能和停止连接的功能，连接成功建立后返回到TcpClient。Connector
外部调用Connector::start()后在会调用Connector::connect()，伪代码如下。当需要重试时，会注册定时器，隔一段时间后重新尝试连接。当成功连接后，会使用连接得到的sockfd构建一个Channel对象，注册回调函数后enableWriting()。

void Connector::connect(){
    int sockfd = createNonblocking();
    sockaddr_in addr = serverAddr_.getSockAddrInet();
    int ret = ::connect(sockfd, reinterpret_cast<struct sockaddr*>(&addr), sizeof addr);
    int savedErrno = (ret == 0) ? 0 : errno;
    switch (savedErrno)
    {
        case 连接成功:
            connecting(sockfd);
            break;
        case 需要重试:
            retry(sockfd);
            break;
        case 发生错误:
            close(sockfd);
            abort();
            break;
        default:
            close(sockfd);
            abort();
            break;
  }
}

如果没有出错，sockfd就处于可写状态(内核缓冲区不为满)，而poller_关注了可写事件，所以会调用Connector::handleWrite()，注意这里会调用removeAndResetChannel()，因为现在已经完成连接，TcpClient中会新建TcpConnection，所以原来建立的Channel对象已经不需要使用，需要被析构。如果一切正常，则会调用newConnectionCallback_，这是由TcpClient注册的回调函数，接受的参数是完成连接的sockfd。

void Connector::handleWrite(){
    if (state_ == kConnecting){
        int sockfd = removeAndResetChannel();
        int err = getSocketError(sockfd);
        if(err){
            retry(sockfd);
        }
        else if(isSelfConnect(sockfd)){
            retry(sockfd);
        }
        else{
            setState(kConnected);
            if(connect_){
                newConnectionCallback_(sockfd);
            }
            else{
                close(sockfd);
            }
        }
    }
    else{
        assert(state_ == kDisconnected);
    }
}

TcpClient
一个TcpClient对应一个TcpConnection和一个Connector。主要需要关注其注册到Connector中的回调函数TcpClient::newConnection，其他部分比较简单。这里的处理和在TcpServer中比较类似，使用完成的连接的sockfd建立TcpConnection对象。

void TcpClient::newConnection(int sockfd){
    loop_->assertInLoopThread();
    InetAddress peerAddr(getPeerAddr(sockfd));
    char buf[32];
    snprintf(buf, sizeof buf, ":%s#%d", peerAddr.toHostPort().c_str(), nextConnId_);
    ++nextConnId_;
    std::string connName = buf;
    InetAddress localAddr(getLocalAddr(sockfd));

    TcpConnectionPtr conn(new TcpConnection(loop_, connName, sockfd, localAddr, peerAddr));

    conn->setConnectionCallback(connectionCallback_);
    conn->setMessageCallback(messageCallback_);
    conn->setWriteCompleteCallback(writeCompleteCallback_);
    conn->setCloseCallback(std::bind(&TcpClient::removeConnection, this, std::placeholders::_1));

    {
        MutexLockGuard lock(mutex_);
        connection_ = conn;
    }
    conn->connectEstablished();
}

测试

test1：待线程池的 echo server

#include "TcpServer.h"
#include "EventLoop.h"
#include "InetAddress.h"
#include "utils.h"
#include "base/CurrentThread.h"
#include <stdio.h>

void onConnection(const TcpConnectionPtr& conn)
{
  if (conn->connected())
  {
    printf("onConnection(): tid=%d new connection [%s] from %s\n",
           CurrentThread::tid(),
           conn->name().c_str(),
           conn->peerAddress().toHostPort().c_str());
  }
  else
  {
    printf("onConnection(): tid=%d connection [%s] is down\n",
           CurrentThread::tid(),
           conn->name().c_str());
  }
}

void onMessage(const TcpConnectionPtr& conn,
               Buffer* buf,
               int64_t receiveTime)
{
  printf("onMessage(): tid=%d received %zd bytes from connection [%s] at %s\n",
         CurrentThread::tid(),
         buf->readableBytes(),
         conn->name().c_str(),
         TimeToString(receiveTime).c_str());

  conn->send(buf->retrieveAsString());
}

int main(int argc, char* argv[])
{
  printf("main(): pid = %d\n", getpid());

  InetAddress listenAddr(9981);
  EventLoop loop;

  TcpServer server(&loop, listenAddr);
  server.setConnectionCallback(onConnection);
  server.setMessageCallback(onMessage);
  if (argc > 1) {
    server.setThreadNum(atoi(argv[1]));
  }
  server.start();

  loop.loop();
}

test2：简单的客户端代码，会不断尝试连接上面的 echo server

#include "EventLoop.h"
#include "InetAddress.h"
#include "TcpClient.h"
#include "utils.h"

#include <utility>
#include <functional>
#include <stdio.h>
#include <unistd.h>

std::string message = "Hello\n";

void onConnection(const TcpConnectionPtr& conn)
{
  if (conn->connected())
  {
    printf("onConnection(): new connection [%s] from %s\n",
           conn->name().c_str(),
           conn->peerAddress().toHostPort().c_str());
    conn->send(message);
  }
  else
  {
    printf("onConnection(): connection [%s] is down\n",
           conn->name().c_str());
  }
}

void onMessage(const TcpConnectionPtr& conn, Buffer* buf, int64_t receiveTime)
{
  printf("onMessage(): received %zd bytes from connection [%s] at %s\n",
         buf->readableBytes(),
         conn->name().c_str(),
         TimeToString(receiveTime).c_str());

  printf("onMessage(): %s", buf->retrieveAsString().c_str());
}

int main()
{
  EventLoop loop;
  InetAddress serverAddr("127.0.0.1", 9981);
  TcpClient client(&loop, serverAddr);

  client.setConnectionCallback(onConnection);
  client.setMessageCallback(onMessage);
  client.enableRetry();
  client.connect();
  loop.loop();
}

代码地址

https://github.com/ZhaoxuWang/simple-muduo/tree/main/stage3