http://blog.csdn.net/liuxuejiang158blog/article/details/17056537#comments
注:muduo用C++实现蛮有意思的,其大量使用boost的shared_ptr,bind,function实现对象生命期控制、事件回调机制,且基于对象编程而非面向对象编程。在此记点笔记吧,以备后查。
文字部分:
1 Reactor模式的实现:关键是三个类:Channel,Poller,EventLoop。
class Channel:事件分发器,其记录了描述符fd的注册事件和就绪事件,及就绪事件回调比如可读回调readCallback。其和文件描述符fd是一一对应的关系,但其不拥有fd。当一个fd想要注册事件并在事件就绪时执行相应的就绪事件回调时,首先通过Channel::update(this)->EventLoop::updateChannel(Channel*)->Poller::updateChannel(Channel*)调用链向poll系统调用的侦听事件表注册或者修改注册事件。Channel作为是事件分发器其核心结构是Channel::handleEvent()该函数执行fd上就绪事件相应的事件回调,比如fd可读事件执行readCallback()。Channel应该还具有一个功能是:Channel::~Channel()->EventLoop::removeChannel(Channel*)->Poller::removeChannel(Channel*)将Poller中的Channel*移除防止空悬指针。这是因为Channel的生命周期和Poller/EventLoop不一样长。其关键数据成员:int fd_文件描述符,int events_文件描述符的注册事件,int revents_文件描述符的就绪事件,及事件回调readCallback_,writeCallback...
class Poller
:实现IO multiplexing,其功能仅仅是poll系统调用的简单封装,其生命周期和EventLoop一样长,Poller不拥有Channel支持有Channel*指针(因此必要时候需要Channel自己解注册以防空悬指针)。Poller的关键数据成员是:vector<struct pollfd> pollfds_事件结构体数组用于poll的第一个参数;map<int,Channel*> channels_用于文件描述符fd到Channel的映射便于快速查找到相应的Channel,如poll返回后遍历pollfds_找到就绪事件的fd再通过channels_找到fd的channel然后就可以通过Channel::handleEvent()执行就绪事件回调,值得注意的是Channel中已经记录了fd所以fd和Channel完成了双射。关键的成员函数:Poller::poll(int timeoutMs,vector<Channel*> activeChannels)其调用poll侦听事件集合,并在timoutMs时间内就绪的事件集合通过activeChannels返回。这里Poller::poll()在poll返回后本可以执行Channel::handleEvent()将就绪事件的回调执行了,但是并没有这样做的原因是,Channel::handleEvent()可能修改Poller的两个容器,即添加或删除Channel*,在遍历容器的时候修改容器是非常危险的,同时为了简化Poller,Poller的职责仅仅是IO复用,至于事件分发还是交给Channel自己完成。
class EventLoop
: 事件循环。先看一个调用链:EventLoop::loop()->Poller::poll()通过此调用链获得一个vector<Channel*> activeChannels_的就绪事件集合,再遍历该容器,执行每个Channel的Channel::handleEvent()完成相应就绪事件回调。至此一个完整的Reactor模式即完成。注意这里的Reactor遵循one loop per thread即所在一个线程中完成,故并没有涉及到线程同步机制,以后可能会有其它线程调用这三个类,则通过线程转移函数将一些操作转移到一个线程中完成。(若一个函数既可能加锁情况下使用有可能在未加锁情况下使用,那么就拆成两个函数,需要加锁的函数去调用不需要加锁的函数。线程转移实现就可以通过两个函数实现,如:假设类one隶属于线程B,线程A调用one的方法fun,fun向one注册一个回调,从而将具体操作转移到one的所属线程B中去执行。)
2 定时器: 主要有几个类,Timer定时器包含超时回调,TimerId定时器加上一个唯一的ID,Timestamp时间戳,TimerQueue管理所有的定时器。传统的定时通过select/poll实现,现在通过timerfd实现定时,采用文件描述符实现定时将有利于统一事件源。这些将为EventLoop实现定时功能。
Timer
:定时器,具有一个超时时间和超时回调。超时时间由当前时间戳加上一个超时时间生成一个绝对时间。定时器回调函数timerCallback。
TimerQueue
: 定时器队列,用于管理所有的定时器,当定时器超时后执行相应的Timer::run()定时器回调。采用set<pair<TimeStamp,Timer*> >存储所有未超时的定时器,这里采用pair<TimeStamp,Timer*>的原因是在一个时间点可能有多个时间戳TimeStamp超时,而查找只返回一个。通过给timerfd一个超时时间实现超时计时,通过Channel管理timerfd,然后向EventLoop和Poller注册timerfd的可读事件,当timerfd的可读事件就绪时表明一个超时时间点到了,TimerQueue::handleRead()遍历set容器找出那些超时的定时器并执行Timer::run()实现超时回调。timerfd怎么实现多个定时器超时计时的呢?每次向set插入一个定时器Timer的时候就比较set的头元素的超时时间,若新插入的超时时间小,则更新timerfd的时间,从而保证timerfd始终是set中最近的一个超时时间。当timerfd可读时,需要遍历容器set,因为可能此时有多个Timer超时了(尽管tiemrfd是当前最小的定时时间)。为了复用定时器,每次执行完定时器回调后都要检查定时器是否需要再次定时。这里的关键是采用timerfd实现统一事件源。
3
class EventLoop的改动,实现用户定时回调
:当有了定时器TimerQueue后,EventLoop就可以实现几个定时器接口:EventLoop::runAt(TimeStamp,TimerCallback)在一个绝对时间执行一个回调TimerCallback;EventLoop::runAfter(double delay,TimerCallback)实现一个相对时间回调,其内部实现是当前时间戳TimeStamp加上delay后形成一个绝对时间戳然后调用EventLoop::runAt(); EventLoop::runEvery(double interval,TimerCallback)实现周期回调,这里将使用到TimerQueue执行完超时回调后会检查定时器是否需要再次定时的功能。
4
class EventLoop的改动,实现用户指定任务回调
:EventLoop::runInLoop(boost::function<void()>),若是EventLoop隶属的线程调用EventLoop::runInLoop()则EventLoop马上执行;若是其它线程调用则执行EventLoop::queueInLoop(boost::function<void()>将任务添加到队列中(这里就是前面说的线程转移)。EventLoop如何获得有任务这一事实呢?通过eventfd可以实现线程间通信,具体做法是:其它线程向EventLoop::vector<boost::function<void()> >添加任务T,然后通过EventLoop::wakeup()向eventfd写一个int,eventfd的回调函数EventLoop::handleRead()读取这个int,从而相当于EventLoop被唤醒,此时loop中遍历队列执行堆积的任务。这里采用Channel管理eventfd,Poller侦听eventfd体现了eventfd可以统一事件源的优势。
5
实现线程安全的class TimerQueue
:原来添加定时器的调用链:TimerQueue::addTimer()->TimerQueue::insert()实现添加的,但是这只能实现TimerQueue所属的线程执行,若其它线程想向此IO线程添加一个定时任务是不安全的。
为了实现添加定时器Timer到set的线程安全性,将定时器添加函数TimerQueue::addTimer()分拆为两部分:TimerQueue::addTimer()只负责转发,addTimerInLoop()实现具体的定时器添加。具体的调用链为:TimerQueue::addTimer()->EventLoop::runInLoop(TimerQueue::runInLoop)->TimerQueue::runInLoop()->TimerQueue::insert(),可以看出通过把TimerQueue::runInLoop()这个回调任务添加EventLoop::runInLoop()中从而实现将添加定时器这一操作转移到IO线程来做。TimerQueue::insert()插入一个新的定时器后将检查当前最近的超时时间,若最近的超时时间变了则重置timerfd的计时。
6
class EventLoopThread
: 启动一个线程执行一个EventLoop,其语义和"one loop per thread“相吻合。注意这里用到了互斥量和条件变量,这是因为线程A创建一个EventLoopThread对象后一个运行EventLoop的线程已经开始创建了,可以通过EventLoopThread::startLoop()获取这个EventLoop对象,但是若EventLoop线程还没有创建好,则会出错。所以在创建EventLoop完成后会执行condititon.notify()通知线程A,线程A调用EventLoopThread::startLoop()时调用condition.wai()等待,从而保证获取一个创建完成的EventLoop.毕竟线程A创建的EventLoop线程,A可能还会调用EventLoop执行一些任务回调呢。
7
class Acceptor
: 用于accept一个TCP连接,accept接受成功后通知TCP连接的使用者。Acceptor主要是供TcpServer使用的,其生命期由后者控制。一个Acceptor相当于持有服务端的一个socket描述符,该socket可以accept多个TCP客户连接,这个accept操作就是Acceptor实现的。
这里封装了一些常用的网络相关的数据结构和操作,如class InetAddress表示sockaddr_in的封装,如可以通过ip地址和port端口生成一个sockaddr_in; class Socket封装了部分关于socket套接字的操作,如Socket::bindAddress(InetAddress&)将socket和一个sockaddr_in地址绑定,Socket::accept(InetAddress& peerAddr)将一个socket允许连接一个客户端地址peerAddr,Socket::listen()监听socket,Socket::shutdownWrite()实现关闭socket的写。
Acceptor在构造的时候会创建一个socket描述符acceptSocket_(这是一个Socket类型即socket的RAII封装),并通过一个Channel(注册事件及回调函数)管理acceptSocket_::fd成员(即socket描述符),一旦该socket可读即有TCP客户连接请求,则Channel::handleEvent()将会调用Acceptor::hanleRead()执行accept一个TCP客户连接。Acceptor::handleRead()还会将新的TCP客户连接和客户端地址通过回调函数newConnectionCallback(connfd,peerAddr)传给该TCP客户连接的使用者,通常是TcpServer类,这里的回调函数newConnectionCallback是在Acceptor::setNewConnectionCallback(newConnectionCallback)指定的。值得注意的是这里又是统一事件源的思想,即通过Channel和Poller管理事件。Acceptor::listen()的工作是:启动acceptSocket_::listen()监听socket描述符,并通过Channel::enableReading()将socket的可读事件注册到Poller的事件集合中。
8
class TcpServer:
管理所有的TCP客户连接,TcpServer供用户直接使用,生命期由用户直接控制。用户只需设置好相应的回调函数(如消息处理messageCallback)然后TcpServer::start()即可。
这里先假设每个TCP客户连接由一个类TcpConenction管理(具体执行消息的接收发送之类的),而TcpServer的工作就是管理这些TcpConenction,TcpConnection将在后面给出。假设TcpServer持有boost::scoped_ptr<TcpConnection>的指针TcpConnectionPtr。
TcpServer在构造时接收一个由IP地址和port构成的InetAddress参数,并将此地址传给Acceptor用于接收该地址的TCP连接请求。
TcpServer持有scoped_ptr<Acceptor> acceptor_用于接收TcpServer监听端口上的TCP连接请求,注意Accpetor每次accept连接后都要将新连接的描述符connfd和地址peerAddr返回给使用者,这里TcpServer在构造时通过accptor_->setNewConnectionCallback(bind(&TcpServer::newConnection,this,_1,_2))将TcpServer::newConnection传给Acceptor,acceptor_在接受TCP客户连接后将调用TcpServer::newConnection(connfd,peerAddr),而TcpSrever::newConnection()的主要功能就是为<connfd,peerAddr>创建一个TcpConnection管理该TCP客户连接,并向TcpConnection注册一些回调函数,比如:connectionCallback主要是在TcpServer中由用户指定的一些连接处理函数最后一路经由TcpSrever传到TcpConnection中才被调用,此外还有用户指定的消息处理回调等都是经由TcpServer传给TcpConnection中去具体执行。此外TcpServer::newConnection()中还会执行TcpConnection::connectEstablished()该函数将会使这个具体的TcpConnection连接对应的描述符connfd加入poll的事件表,即也是通过一个Channel管理一个具体的TCP客户连接。用户向TcpServer注册连接回调函数的调用链:用户在创建TcpServer后TcpServer::setConnectionCallback()接收用户注册的连接回调函数;同时在TcpServer创建时会向Acceptor注册回调:TcpServer::TcpServer()->Acceptor::setNewConnecioncallback()后有新TCP连接Acceptor接受连接,并执行回调给连接使用者:Acceptor::handelRead()->newConnection()/Tcpserver::newConnection()->TcpConnection::connectEstablished()/并向TcpConnection注册用户注册的Callback函数。
TcpServer采用map<string,TcpConnectionPtr>管理所有的TCP客户连接,其中string是由TcpServer的服务端地址加上一个int构成表示TcpConnectionPtr的名字。
TcpServer中由用户指定的回调有:connectionCallback当TcpConenction建立时调用(由TcpConnection::connectEstablished()调用connectionCallback())用于执行用户指定的连接回调。messageCallback当TcpConenction有网络消息的时候执行该函数由Channel::handleEvent()->TcpConnection::handleRead()->messageCallback()。writeCompleteCallback由用户指定的当TCP连接上的消息发送完毕时执行的回调。这些函数都是用户在TcpServer创建后通过TcpServer::set*Callback系列函数注册的。当Acceptor接受一个新的TCP连接时执行Acceptor::handleRead()->TcpServer::newConnection()->TcpConnection::set*Callback()这样完成用于指定函数的传递。那么执行呢?这个要在TcpConenction对应的socket事件就绪时可读/可写时由Channel::handEvent()执行这些用户指定的回调。
TcpServer::removeConnection()主要功能从TcpServer中移除一个TcpConnection,但是不能直接移除,而要通过线程转移函数完成。TcpServer::removeConenction()将执行EventLoop::runInLoop(bind(&TcpServer::removeConnectionInLoop)->EventLoop::runInLoop()->TcpServer::removeConnectionInLoop()
TcpServer::removeConenctionInLoop()将一个TcpConnection从TcpServer中移除,并向EventLoop注册回调EventLoop::runInLoop(bind(&TcpConenction::connectDestroyed)),然后执行TcpConnection::connectDestroyed()。
9
class TcpConnection
: 用于管理一个具体的TCP客户连接,比如消息的接收与发送,完成用户指定的连接回调connectionCallback。这里采用shared_ptr管理TcpConnection,因此其public继承boost::enable_shared_from_this<TcpConnection>。
TcpConnection构造时接收参数有TCP连接的描述符sockfd,服务端地址localAddr,客户端地址peerAddr,并通过Socket封装sockfd。且采用Channel管理该sockfd,向Channel注册TcpConection的可读/可写/关闭/出错系列回调函数,用于Poller返回就绪事件后Channel::handleEvent()执行相应事件的回调。
TcpConnection有四个状态:kConnecting正在连接,kConnected已连接,kDisconnecting正在断开,kDisconnected已断开。
TcpConnection:有一些列函数用于TcpServer为连接指定事件回调函数,如TcpConnection::setConnectionCallback/setCloseback等是在TcpServer::newConnection()中注册的回调函数,并且当Acceptor接受一个新TCP连接后执行回调TcpServer::newConnection(),该回调创建一个TcpConenction对象并将用户指定的回调函数通过TcpConnection::set*Callback函数传给TcpConneciton。
TcpConnection有一些列函数用户处理sockfd上的事件回调函数,如TcpConnection::handleRead()是在Poller返回sockfd可读事件时由Channel::handleEvent()调用的。类似还有TcpConnection::handleWrite()等。
TcpConnection::send(string& message)->EventLoop::runInLoop(bind(&TcpConnection::sendInLoop(string& message))->EventLoop::doPendingFunctors()->TcpConnection::sendInLoop(string& message)保证消息发送的线程安全,后者通过write系统调用发送消息。
TcpConnection::shutdown()->EventLoop::runInLoop(bind(&TcpConnection::shutdownInLoop())->EventLoop::doPendingFunctors()->TcpConnection::shutdownInLoop()类似上面,通过线程转移操作实现安全关闭TCP连接。
TcpConnection中增加了用户指定系列回调函数conenctionCallback.messageCallback,writeCompleteCallback这些都是用户通过TcpServer传给TcpConnection,在TcpServer中已经描述过了。当Poller返回TcpConenction对应的Socket就绪事件时Channel::handleEvent()->TcpConnection::handle些列函数->执行这一些列回调。
TcpConnection::closeCallback()不是给用户使用的,而是通知TcpServer或TcpClient移除它们容器中的TcpConnectionPtr。该函数如何设定的呢?当Acceptor接受一个TCP连接时:Channel::handelEvent()->Acceptor::handleRead()->TcpServer::newConenction()中新建一个TcpConnection并通过TcpConnection::setCloseCallback(bind(&TcpSerer::,removeConenction,this,_1))这样才完成将TcpServer::removeChannel()函数传递给TcpConnection::closeCallback()。closeCallback()何时执行呢?当由Channel管理的TcpConnection对应的Socket发生POLLHUP事件(该事件由Poller返回)就绪时,Channel::handleEvent()->TcpConnection::handleClose()->TcpConnection::closeCallback()->TcpServer::removeConnection()。
TcpConnection::connectEstablished()连接建立回调函数(不是用户通过TcpServer指定的connectionCallback),该函数主要的功能:调用Channel::enableReading()将TcpConnection对应的Socket注册到Poller的事件表,执行用户指定的connectionCallback,并将TcpConnection状态置为kConnected。该函数何时被执行呢?回忆前面的Acceptor持有Tcpserver对应的服务端侦听描述符listenfd(由Channel管理),当listenfd可读表明有TCP连接请求,此时Channel::handleEvent()->Acceptor::handleRead()->TcpServer::newConnection()->EventLoop::runInLoop(bind(&TcpConnection::connectEstablished))->EventLoop::queueInLoop()->EventLoop::loop()->EventLoop::doPendingFunctors()->TcpConnection::connectEstablished()。可见TcpServer也是通过向EventLoop::runInLoop添加Tcpconnection::conectEsatablished回调,表明TcpServer可TcpConencion可能不再同一个线程,需要通过线程转移来实现调用。
TcpConenction::connectDestroyed()是TcpConenction析构前调用的最后一个函数,用于通知用户连接已断开。其只要功能是:将TcpConenction状态设为kDisconnected;Channel::disableAll()解除TcpConnection的事件注册,EventLoop::removeChannel()移除该管理TcpConnection的Channel;执行用于指定的回调conenctionCallback。该函数如何调用的呢?这要追溯到TcpServer::newConnection()将TcpServer::removeConenction注册到TcpConnection::closeCallback中,当TcpConnection对应的Socket的POLLHUP事件触发时执行TcpConenction::handleClose()->closeCallback()/TcpServer::removeConenction()->EvetnLoop::runInLoop()->TcpServer::removeInLoop()->EventLoop::runInLoop(bind(&TcpConnection::connectDestroyed))->TcpConnection::connectDestroyed()。
TcpConnection::shutdown()用使用者执行关闭TcpConenction,TcpConnection::shutdown()->EventLoop::runInLoop()->TcpConnection::shutdownInLoop()->socket::shutdown(sockfd,SHUT_WR)//SHUT_WR表示以后的发送将被丢弃。
TcpConnection::handleRead()被Channel::handleEvent()可读事件调用,它的主要工作是通过readv()将socket上的数据读取到Buffer中,并执行用于指定的消息回调messageCallback()。
TcpConnection::handleWrite():被Channel::handleEvent()的可写事件调用,通过write()将Buffer的数据发送出去,若Buffer的数据一次性发送完毕,则执行用户指定的回调writeCompleteCallback(),若一次没有发送完毕,则poll和epoll的LT模式会反复触发可写事件的,所以下次还有机会发送剩余数据。
TcpConnection::handleClose()主要执行Channel::disableAll()和closeCallback()。
TcpConnection有连个Buffer(后面会提到),inputBuffer_,outputBuffer_管理TcpConenction上的数据接收与发送。inputBuffer由TcpConnection::handleRead()调用(Buffer通过readv集中从fd集中读到内存中)。outputBuffer由TcpConnection::handleWrite()通过write()发送到fd上。
TcpConnection::send()->EventLoop::runInLoop()->TcpConenction::runInLoop(),send()可以用户或者其它线程调用,用于发送消息message。这个函数需要先执行线程安全转移到TcpConenction所在的IO线程执行runInLoop()。runInLoop()函数的功能:首先检查TcpConneciton对应的Socket是否注册了可写事件,若注册了可写事件表明outputBuffer_中已经有数据等待发送,为了保证顺序这次的数据只好outputBuffer_.appen()到Buffer中通过Poller返回POLLOUT事件时Channel::handleEvent()->TcpConenction::handleWrite()来发送outputBuffer_的堆积数据。如果Channel::isWriting()返回false则表明此Socket没有向Poller注册POLLOUT事件也就此前没有数据堆积在outputBuffer_中,此次的消息message可以直接通过write发送,但是如果write没有一次性发送完毕,那么message剩余的数据仍要outputBuffer_::append()到Buffer中,并向Poller注册此Socket的POLLOUT事件,以通过TcpConnection::handleWrite()来发送outputBuffer_的堆积数据。无论是sendInLoop()->write还是Channel::handleEvent()->handleWrite(),只要确定发送完message或者outputBuffer_中的数据,那么都要调用用户指定的回调writeCompleteCallback()。
此外TcpConenction还需要忽略SIGPIPE信号,做法是在TcpConenction.cc中定义一个IngoreSigPipe类构造时signal(SIGPIPE,SIG_IGN)。
TcpConnection::setTcpNoDelay()->socketopt(..,TCP_NODELAY..)来关闭Nagle算法。
10 到了这步,
EventLoop和Poller都应该具备成员函数removeC
hannel()用于移除那些管理TcpConenction等的Channel
。这里Poller有个成员vector<struct pollfd> pollfds_当移除Channel时有个trick值得注意:先将要移除的Channel::fd()的pollfd和pollfds_的最后一个元素交换swap()然后再调用pollfds_.pop_back可以避免vector删除时元素的移动。这样删除操作的复杂度为O(1)。
11
class Buffer应用层缓冲区
:在non-blocking+IO multiplexing中应用层缓冲区是必须的。例如:TcpConection向发送100Kb数据,但是write只发送80Kb,剩下20Kb肯定要阻塞线程等待发送了,这和non-blocking矛盾,因此需要设计Buffer充当中间件置于应用层和底层write/read间从而实现non-blocking。这样,应用层只管生成或者读取数据,至于怎样存取数据和数据量则由Buffer完成。Buffer底层是vector<char>,有一个readerIndex和writerIndex分别表示可读位置和可写位置,这两个位置内的区间表示Buffer已有的数据。值得注意的是Buffer的两个trick:每次读取fd上的数据时通过readv一部分读取到Buffer中,一部分读取到找空间char extrabuf[65535]中,若Buffer装满了即extrabuf中有数据,则需要extrabuf中的数据append到Buffer中,这样做可以在初始时每个连接的Buffer的避免过大造成内存浪费,也避免反复调用read的系统开销,每次Buffer不够时通过extrabuf再append到Buffer中使Buffer慢慢变大;还有一个就是Buffer提供了一个前向空间,在消息序列化完毕时可以通过prepend()将消息的大小添加的Buffer的头部。另外Buffer的readerIndex和writerIndex都是移动的,只要在Buffer的空闲空间不够时才加大Buffer的vector,否则可以通过内部腾挪方式即让readerIndex和writerIndex向前移动(数据也跟着移动)这样Buffer就不增大vector.size().
12
class TcpServe的改进
: TcpServer有自己的一个EventLoop用来接收新的TCP客户连接,然后从event loop pool中选一个loop给TCP客户连接(即TcpConnection)。这就需要使用class EventLoopThreadPool来创建多个线程每个线程一个EventLoop(one loop per thread)。
TcpServer::newConnection()在创建一个TcpConnection后从EventLoopTreadPool中选一个EventLoop给这个TcpConnection。
TcpServer::removeConnection()需要拆分成两个函数,因为现在TcpServer和TcpConenction可能不再同一个线程里了,需要通过线程转移函数将移除操作转移到TcpConneciton的IO线程中去。TcpServer::removeConnection()->EventLoop::runInLoop()->TcpServer::removeConnectionInLoop()->EventLoop::runInLoop()->TcpConnection::connectDestroyed()。Tcpserver中erase掉这个TcpConnectionPtr。
13 class Connector:用于发起连接,当socket变得可写时表示连接建立完毕,其间需要处理各种类型的错误。connect返回EAGAIN是真的错误表示暂时没有端口可用,要关闭socket稍后再试;EINPROGRESS是“正在连接”,即使socket可写,也需要用getsockopt(sokfd,SOL_SOCKET,SO_ERROR...)再次确认。超时重连的时间应逐渐延长,需要处理自连接的情况。Connector只负责建立连接,不负责建立TcpConnection,它有一个建立连接回调函数由用户指定(Connector基本是TCP客户端使用,且一个客户端一个Conenctor)。
Connector有三个状态:kDisconnected未连接,kConnecting正在连接,kConnected已连接。
Connector构造时指定一个服务端地址InetAddress和一个事件循环EventLoop,状态设为kDisConnected。指定一个最大重试连接时间。
Connector::start()可以由其它线程调用,该函数内部执行EventLoop::runInLoop(bind(&Connector::startInLoop,this))将通过EventLoop将操作转移到Connector的线程中去。
Connector::startInLoop()判断此Connector还没有连接,则调用Connector::connect()
Connector::connect()创建一个sockfd,并调用connect(sockfd,&serverAddress,sizeof(serverAddress)),然后返回一个errno,根据errno的值进行进一步选择是关闭连接,重师连接还是调用已连接函数。
连接返回errno为EAGAIN则调用Connector::retry()该函数调用EventLoop::runAfter()在一段时间间隔后重试连接。
errno为EINPROGRESS表示正在连接,则调用Connector::connecting()该函数将该为该连接设置一个Channel来管理连接,并向Channel注册Connector::handleWrite,Conenctor::handleError的回调函数。其中Connector::handleWrite()当正在连接kConnecting则用需要测试该连接(已经可写了还kConencting)可能需要进一步重新连接即调用Connector::retry()注意这时候socketfd需要重新分配了,而Conenctor是可以重复使用的。Cionnector::handleError()执行Connector::removeAndResetChannel()和Connector::retry()重试连接。
当用户或其它线程调用Connector::start()->EventLoop::runInLoop()->Connector::startInLoop()->Connector::connect()根据errnor执行下面的情形:
EAGAIN:Connector::retry()->EventLoop::runAfter()延迟重连->Conncetor::startInLoop()
EACCESS/EPERM/EBADF: close(sockfd)
EINPROGRESS:Connector::connecting()该函数向Channel注册回调函数并Channel::enableWriting()关注这个正在连接的socketfd是否可写。
此后当处于“正在连接kConnecting”的sockfd事件就绪时:Channel::handelEvent()->Connector::handleWrite()/Connector::handleError()
Connector::handleWrite()若检测Connector状态仍在kConnecting时需要调用getsocketopt(sockfd,SOL_SOCKET,SO_ERROR...)检测,若返回0则执行用户指定的连接回调函数Connector::newConnectionCallback()。
Connector::handleError()->Conenctor::retry()
14 class TcpClient:每个TcpClinet只管理一个Connector。其内容和TcpServer差不多,TcpClient具备TcpConnection断开连接后重新连接的功能。
TcpClient构造时new一个Connector,并指定一个EventLoop。用户要指定的connectionCallback,messageCallback回调函数。并设置Connector::setNewConnection(bind(&TcpClient::newConnection,this,_1))将连接建立回调赋给Connector::newConnectionCallback。
用户调用TcpClient::connect()发起连接->Conenctor::start()->TcpClient::newConnection().
TcpClinet::newConnection()将new一个TcpConneciton对象conn并设置TcpConnection::setConnecitonCallback/setMessageCallback/setWriteCompleteCallback/setCloseCallback等回调函数。其中setConnectionCallback将用户指定的ConnectionCallback传给TcpConenciton,setMessageCallback将TcpClient中用户指定的messageCallback传给TcpConneciton。TcpConnection::closeCallback是TcpClient::removeConenction。最后TcpClient::newConneciton()将会调用TcpConnection::connectEstablished()。
TcpClient::removeConnection(),由于TcpClient只管理一个Connector也就是一个TcpConenction它们都在一个线程中,所以不涉及操作线程转移。TcpClient::removeConenction()->EventLoop::queueInLoop()->TcpConnection::connectDestroyed().若有重连的必要将执行Connector::restart()。
14 class Epoller和Poller差不多。
代码部分:
注意这里的代码没有muduo网络库全,挂在这里只是做了些注释,算是留个尸体吧。如有必要请参看muduo。
- #include<iostream>
- #include<map>
- #include<string>
- #include<vector>
- #include<utility>
- #include<set>
- #include<deque>
- #include<algorithm>
- #include<boost/any.hpp>
- #include<boost/enable_shared_from_this.hpp>
- #include<boost/noncopyable.hpp>
- #include<boost/scoped_ptr.hpp>
- #include<boost/shared_ptr.hpp>
- #include<boost/weak_ptr.hpp>
- #include<boost/function.hpp>
- #include<boost/static_assert.hpp>
- #include<boost/bind.hpp>
- #include<boost/foreach.hpp>
- #include<boost/ptr_container/ptr_vector.hpp>
- #include<errno.h>
- #include<fcntl.h>
- #include<stdio.h>
- #include<strings.h>
- #include<unistd.h>
- #include<endian.h>
- #include<assert.h>
- #include<stdio.h>
- #include<stdlib.h>
- #include<string.h>
- #include<pthread.h>
- #include<unistd.h>
- #include<poll.h>
- #include<errno.h>
- #include<signal.h>
- #include<stdint.h>
- #include<arpa/inet.h>
- #include<netinet/tcp.h>
- #include<netinet/in.h>
- #include<sys/timerfd.h>
- #include<sys/syscall.h>
- #include<sys/time.h>
- #include<sys/eventfd.h>
- #include<sys/types.h>
- #include<sys/socket.h>
- #include<sys/epoll.h>
- using namespace std;
- using namespace boost;
- # define UINTPTR_MAX (4294967295U)//一个无符号大数
-
-
-
- class Mutex:noncopyable{
- public:
- Mutex(){
- pthread_mutex_init(&mutex,NULL);
- }
- void lock(){
- pthread_mutex_lock(&mutex);
- }
- void unlock(){
- pthread_mutex_unlock(&mutex);
- }
- pthread_mutex_t& get(){
- return mutex;
- }
- private:
- pthread_mutex_t mutex;
- };
-
-
-
- class MutexLockGuard:noncopyable{
- public:
- explicit MutexLockGuard(Mutex& mutex):mutex_(mutex){
- mutex_.lock();
- }
- ~MutexLockGuard(){
- mutex_.unlock();
- }
- private:
- Mutex& mutex_;
- };
-
-
-
- class Condition:noncopyable{
- public:
- explicit Condition(Mutex& mutex):mutex_(mutex){
- pthread_cond_init(&pcond_,NULL);
- }
- ~Condition(){
- pthread_cond_destroy(&pcond_);
- }
- void wait(){
- pthread_cond_wait(&pcond_,&mutex_.get());
- }
- void notify(){
- pthread_cond_signal(&pcond_);
- }
- void notifyALL(){
- pthread_cond_broadcast(&pcond_);
- }
- private:
- Mutex& mutex_;
- pthread_cond_t pcond_;
- };
-
-
-
- class CountDownLatch{
- public:
- CountDownLatch(int count):mutex_(),condition_(mutex_),count_(count){}
- void wait(){
- MutexLockGuard lock(mutex_);
- while(count_>0)
- condition_.wait();
- }
- void countDown(){
- MutexLockGuard lock(mutex_);
- --count_;
- if(count_==0)
- condition_.notifyALL();
- }
- private:
- mutable Mutex mutex_;
- Condition condition_;
- int count_;
- };
-
-
-
- __thread pid_t t_cacheTid=0;
- class Thread:noncopyable{
- public:
- typedef function<void()> ThreadFunc;
- explicit Thread(const ThreadFunc& a,const string& name=string()):started_(false),
- joinded_(false),pthreadID_(0),tid_(new pid_t(0)),func_(a),name_(name){
- }
- ~Thread(){
- if(started_&&!joinded_){
- pthread_detach(pthreadID_);
- }
- }
- void start();
-
-
-
-
-
-
-
-
-
- //###1###使用此处会出错详见http://cboard.cprogramming.com/cplusplus-programming/113981-passing-class-member-function-pthread_create.html
- void join(){
- assert(started_);
- assert(!joinded_);
- joinded_=true;
- pthread_join(pthreadID_,NULL);
- }
- pid_t tid() const{
- if(t_cacheTid==0){
- t_cacheTid=syscall(SYS_gettid);
- }
- return t_cacheTid;
- }
- const string& name() const{
- return name_;
- }
-
- void startThread(){
- func_();
- }
- private:
- bool started_;
- bool joinded_;
- pthread_t pthreadID_;
- shared_ptr<pid_t> tid_;
- ThreadFunc func_;
- string name_;
- };
- void* threadFun(void* arg){
- Thread* thread=static_cast<Thread*>(arg);
- thread->startThread();
- return NULL;
- }
- void Thread::start(){
- assert(!started_);
- started_=true;
- if(pthread_create(&pthreadID_,NULL,threadFun,this)){
- started_=false;
- abort();
- }
- }
-
-
-
-
- template<typename T>
- class ThreadLocal:noncopyable{
- public:
- ThreadLocal(){
- pthread_key_create(&pkey_,&destructor);
- }
- ~ThreadLocal(){
- pthread_key_delete(pkey_);
- }
- T& value(){
- T* perThreadValue=static_cast<T*>(pthread_getspecific(pkey_));
- if(!perThreadValue){
- T* newObj=new T();
- pthread_setspecific(pkey_,newObj);
- perThreadValue=newObj;
- }
- return *perThreadValue;
- }
- private:
- static void destructor(void* x){
- T* obj=static_cast<T*>(x);
- delete obj;
- }
- private:
- pthread_key_t pkey_;
- };
-
-
-
- class ThreadPool:noncopyable{
- public:
- typedef function<void()> Task;
- explicit ThreadPool(const string& name=string()):mutex_(),cond_(mutex_),name_(name),running_(false){
- }
- ~ThreadPool(){
- if(running_){
- stop();
- }
- }
- void start(int numThreads){
- assert(threads_.empty());
- running_=true;
- threads_.reserve(numThreads);
- for(int i=0;i<numThreads;i++){
- threads_.push_back(new Thread(bind(&ThreadPool::runInThread,this)));
- threads_[i].start();
- }
- }
- void stop(){
- running_=false;
- cond_.notifyALL();
- for_each(threads_.begin(),threads_.end(),bind(&Thread::join,_1));
- }
- void run(const Task& task){
- if(running_){
- if(threads_.empty()){
- task();
- }
- else{
- MutexLockGuard guard(mutex_);
- queue_.push_back(task);
- cond_.notify();
- }
- }
- else{
- printf("线程池已停止运行\n");
- }
- }
- bool running(){
- return running_;
- }
- private:
- void runInThread(){
- while(running_){
- Task task(take());
- if(task){
- task();
- }
- }
- }
- Task take(){
- MutexLockGuard guard(mutex_);
- while(queue_.empty()&&running_){
- cond_.wait();
- }
- Task task;
- if(!queue_.empty()){
- task=queue_.front();
- queue_.pop_front();
- }
- return task;
- }
- Mutex mutex_;
- Condition cond_;
- string name_;
- ptr_vector<Thread> threads_;
- deque<Task> queue_;
- bool running_;
- };
-
-
-
- template<typename T>
- class AtomicIntegerT : boost::noncopyable
- {
- public:
- AtomicIntegerT()
- : value_(0){}
- T get() const
- {
- return __sync_val_compare_and_swap(const_cast<volatile T*>(&value_), 0, 0);
- }
- T getAndAdd(T x)
- {
- return __sync_fetch_and_add(&value_, x);
- }
- T addAndGet(T x)
- {
- return getAndAdd(x) + x;
- }
- T incrementAndGet()
- {
- return addAndGet(1);
- }
- void add(T x)
- {
- getAndAdd(x);
- }
- void increment()
- {
- incrementAndGet();
- }
- void decrement()
- {
- getAndAdd(-1);
- }
- T getAndSet(T newValue)
- {
- return __sync_lock_test_and_set(&value_, newValue);
- }
- private:
- volatile T value_;
- };
- typedef AtomicIntegerT<int32_t> AtomicInt32;
- typedef AtomicIntegerT<int64_t> AtomicInt64;
-
- class Channel;
- class Poller;
- class Timer;
- class TimerId;
- class Timestamp;
- class TimerQueue;
- class TcpConnection;
- class Buffer;
-
- typedef shared_ptr<TcpConnection> TcpConnectionPtr;
- typedef function<void()> TimerCallback;
- typedef function<void (const TcpConnectionPtr&)> ConnectionCallback;
- typedef function<void (const TcpConnectionPtr&,Buffer* buf)> MessageCallback;
- typedef function<void (const TcpConnectionPtr&)> WriteCompleteCallback;
- typedef function<void (const TcpConnectionPtr&)> CloseCallback;
-
-
-
- class EventLoop:noncopyable{
- public:
-
-
-
- typedef function<void()> Functor;
- EventLoop();
- ~EventLoop();
- void loop();
- void quit();
-
- void assertInLoopThread(){
- if(!isInLoopThread()){
- abortNotInLoopThread();
- }
- }
- bool isInLoopThread() const{return threadID_==syscall(SYS_gettid);}
- TimerId runAt(const Timestamp& time,const TimerCallback& cb);
- TimerId runAfter(double delay,const TimerCallback& cb);
- TimerId runEvery(double interval,const TimerCallback& cb);
- void runInLoop(const Functor& cb);
- void queueInLoop(const Functor& cb);
- void cancel(TimerId tiemrId);
- void wakeup();
- void updateChannel(Channel* channel);
- void removeChannel(Channel* channel);
- private:
- void abortNotInLoopThread();
- void handleRead();
- void doPendingFunctors();
- typedef vector<Channel*> ChannelList;
- bool looping_;
- bool quit_;
- const pid_t threadID_;
- scoped_ptr<Poller> poller_;
-
- ChannelList activeChannels_;
-
- int wakeupFd_;
- scoped_ptr<Channel> wakeupChannel_;
- Mutex mutex_;
- vector<Functor> pendingFunctors_;
- scoped_ptr<TimerQueue> timerQueue_;
- bool callingPendingFunctors_;
- };
-
-
-
-
- class Poller:noncopyable{
- public:
- typedef vector<Channel*> ChannelList;
- Poller(EventLoop* loop);
- ~Poller();
- Timestamp Poll(int timeoutMs,ChannelList* activeChannels);
- void updateChannel(Channel* channel);
- void assertInLoopThread(){
- ownerLoop_->assertInLoopThread();
- }
- void removeChannel(Channel* channel);
- private:
- void fillActiveChannels(int numEvents,ChannelList* activeChannels) const;
- typedef vector<struct pollfd> PollFdList;
- typedef map<int,Channel*> ChannelMap;
-
- EventLoop* ownerLoop_;
- PollFdList pollfds_;
- ChannelMap channels_;
- };
-
-
-
-
- class Channel:noncopyable{
- public:
- typedef function<void()> EventCallback;
- typedef function<void()> ReadEventCallback;
- Channel(EventLoop* loop,int fd);
- ~Channel();
- void handleEvent();
- void setReadCallback(const ReadEventCallback& cb){
- readCallback_=cb;
- }
- void setWriteCallback(const EventCallback& cb){
- writeCallback_=cb;
- }
- void setErrorCallback(const EventCallback& cb){
- errorCallback_=cb;
- }
- void setCloseCallback(const EventCallback& cb){
- closeCallback_=cb;
- }
- int fd() const{return fd_;}
- int events() const{return events_;}
- void set_revents(int revt){
- revents_=revt;
- }
- bool isNoneEvent() const{
- return events_==kNoneEvent;
- }
- void enableReading(){
- events_|=kReadEvent;
- update();
- }
- void enableWriting(){
- events_|=kWriteEvent;
- update();
- }
- void disableWriting(){
- events_&=~kWriteEvent;
- update();
- }
- void disableAll(){events_=kReadEvent;update();}
- bool isWriting() const{
- return events_&kWriteEvent;
- }
- int index(){return index_;}
- void set_index(int idx){index_=idx;}
- EventLoop* ownerLoop(){return loop_;}
- private:
- void update();
- static const int kNoneEvent;
- static const int kReadEvent;
- static const int kWriteEvent;
- bool eventHandling_;
- EventLoop* loop_;
- const int fd_;
- int events_;
- int revents_;
- int index_;
- ReadEventCallback readCallback_;
- EventCallback writeCallback_;
- EventCallback errorCallback_;
- EventCallback closeCallback_;
- };
-
-
-
-
- class Timestamp{
- public:
- Timestamp():microSecondsSinceEpoch_(0){}
- explicit Timestamp(int64_t microseconds):microSecondsSinceEpoch_(microseconds){}
- void swap(Timestamp& that){
- std::swap(microSecondsSinceEpoch_,that.microSecondsSinceEpoch_);
- }
- bool valid() const{return microSecondsSinceEpoch_>0;}
- int64_t microSecondsSinceEpoch() const {return microSecondsSinceEpoch_;}
- static Timestamp now(){
- struct timeval tv;
- gettimeofday(&tv, NULL);
- int64_t seconds = tv.tv_sec;
- return Timestamp(seconds * kMicroSecondsPerSecond + tv.tv_usec);
- }
- static Timestamp invalid(){return Timestamp();}
- static const int kMicroSecondsPerSecond=1000*1000;
- private:
- int64_t microSecondsSinceEpoch_;
- };
-
- inline bool operator<(Timestamp lhs, Timestamp rhs)
- {
- return lhs.microSecondsSinceEpoch() < rhs.microSecondsSinceEpoch();
- }
-
- inline bool operator==(Timestamp lhs, Timestamp rhs)
- {
- return lhs.microSecondsSinceEpoch() == rhs.microSecondsSinceEpoch();
- }
- inline double timeDifference(Timestamp high, Timestamp low)
- {
- int64_t diff = high.microSecondsSinceEpoch() - low.microSecondsSinceEpoch();
- return static_cast<double>(diff) / Timestamp::kMicroSecondsPerSecond;
- }
- inline Timestamp addTime(Timestamp timestamp, double seconds)
- {
- int64_t delta = static_cast<int64_t>(seconds * Timestamp::kMicroSecondsPerSecond);
- return Timestamp(timestamp.microSecondsSinceEpoch() + delta);
- }
-
-
-
- class TimerId{
- public:
- TimerId(Timer* timer=NULL,int64_t seq=0)
- :timer_(timer),sequence_(seq){}
- friend class TimerQueue;
- private:
- Timer* timer_;
- int64_t sequence_;
- };
-
-
-
- class Timer : boost::noncopyable
- {
- public:
- typedef function<void()> TimerCallback;
-
- Timer(const TimerCallback& cb, Timestamp when, double interval)
- :callback_(cb),expiration_(when),
- interval_(interval),repeat_(interval > 0.0),
- sequence_(s_numCreated_.incrementAndGet()){}
- void run() const {
- callback_();
- }
- Timestamp expiration() const { return expiration_; }
- bool repeat() const { return repeat_; }
- int64_t sequence() const{return sequence_;}
- void restart(Timestamp now);
- private:
- const TimerCallback callback_;
- Timestamp expiration_;
- const double interval_;
- const bool repeat_;
- const int64_t sequence_;
- static AtomicInt64 s_numCreated_;
- };
- AtomicInt64 Timer::s_numCreated_;
- void Timer::restart(Timestamp now){
- if (repeat_){
- expiration_ = addTime(now, interval_);
- }
- else{
- expiration_ = Timestamp::invalid();
- }
- }
-
-
-
- int createTimerfd(){
- int timerfd = ::timerfd_create(CLOCK_MONOTONIC,TFD_NONBLOCK | TFD_CLOEXEC);
- if (timerfd < 0){
- printf("Timderfd::create() error\n");
- }
- return timerfd;
- }
- struct timespec howMuchTimeFromNow(Timestamp when){
- int64_t microseconds = when.microSecondsSinceEpoch()- Timestamp::now().microSecondsSinceEpoch();
- if (microseconds < 100){
- microseconds = 100;
- }
- struct timespec ts;
- ts.tv_sec = static_cast<time_t>(microseconds / Timestamp::kMicroSecondsPerSecond);
- ts.tv_nsec = static_cast<long>((microseconds % Timestamp::kMicroSecondsPerSecond) * 1000);
- return ts;
- }
- void readTimerfd(int timerfd, Timestamp now){
- uint64_t howmany;
- ssize_t n = ::read(timerfd, &howmany, sizeof howmany);
- if (n != sizeof howmany){
- printf("readTimerfd error\n");
- }
- }
- void resetTimerfd(int timerfd, Timestamp expiration)
- {
- struct itimerspec newValue;
- struct itimerspec oldValue;
- bzero(&newValue, sizeof newValue);
- bzero(&oldValue, sizeof oldValue);
- newValue.it_value = howMuchTimeFromNow(expiration);
- int ret = ::timerfd_settime(timerfd, 0, &newValue, &oldValue);
- if (ret){
- printf("timerfd_settime erro\n");
- }
- }
-
-
-
- class TimerQueue : boost::noncopyable
- {
-
-
- public:
- typedef function<void()> TimerCallback;
- TimerQueue(EventLoop* loop);
- ~TimerQueue();
- TimerId addTimer(const TimerCallback& cb,Timestamp when,double interval);
- void cancel(TimerId timerId);
- private:
- typedef pair<Timestamp, Timer*> Entry;
- typedef set<Entry> TimerList;
- typedef pair<Timer*,int64_t> ActiveTimer;
- typedef set<ActiveTimer> ActiveTimerSet;
-
- void handleRead();
- void addTimerInLoop(Timer* timer);
- void cancelInLoop(TimerId timerId);
- std::vector<Entry> getExpired(Timestamp now);
- void reset(const std::vector<Entry>& expired, Timestamp now);
- bool insert(Timer* timer);
-
- EventLoop* loop_;
- const int timerfd_;
- Channel timerfdChannel_;
- TimerList timers_;
- bool callingExpiredTimers_;
- ActiveTimerSet activeTimers_;
- ActiveTimerSet cancelingTimers_;
- };
-
-
-
- TimerQueue::TimerQueue(EventLoop* loop)
- :loop_(loop),timerfd_(createTimerfd()),
- timerfdChannel_(loop, timerfd_),timers_(),
- callingExpiredTimers_(false)
- {
- timerfdChannel_.setReadCallback(bind(&TimerQueue::handleRead, this));
- timerfdChannel_.enableReading();
- }
- TimerQueue::~TimerQueue(){
- ::close(timerfd_);
- for (TimerList::iterator it = timers_.begin();it != timers_.end(); ++it)
- {
- delete it->second;
- }
- }
- TimerId TimerQueue::addTimer(const TimerCallback& cb,Timestamp when,double interval)
- {
- Timer* timer = new Timer(cb, when, interval);
- loop_->runInLoop(bind(&TimerQueue::addTimerInLoop,this,timer));
-
- return TimerId(timer,timer->sequence());
- }
- void TimerQueue::addTimerInLoop(Timer* timer){
- loop_->assertInLoopThread();
- bool earliestChanged=insert(timer);
- if(earliestChanged){
- resetTimerfd(timerfd_,timer->expiration());
- }
- }
- void TimerQueue::cancel(TimerId timerId){
- loop_->runInLoop(bind(&TimerQueue::cancelInLoop,this,timerId));
- }
- void TimerQueue::cancelInLoop(TimerId timerId){
- loop_->assertInLoopThread();
- assert(timers_.size()==activeTimers_.size());
- ActiveTimer timer(timerId.timer_,timerId.sequence_);
- ActiveTimerSet::iterator it=activeTimers_.find(timer);
- if(it!=activeTimers_.end()){
- size_t n=timers_.erase(Entry(it->first->expiration(),it->first));
- assert(n==1);
- (void)n;
- delete it->first;
- activeTimers_.erase(it);
- }
- else if(callingExpiredTimers_){
- cancelingTimers_.insert(timer);
- }
- assert(timers_.size()==activeTimers_.size());
- }
- void TimerQueue::handleRead(){
- loop_->assertInLoopThread();
- Timestamp now(Timestamp::now());
- readTimerfd(timerfd_, now);
- std::vector<Entry> expired = getExpired(now);
- callingExpiredTimers_=true;
- cancelingTimers_.clear();
- for (std::vector<Entry>::iterator it = expired.begin();it!= expired.end(); ++it)
- {
- it->second->run();
- }
- callingExpiredTimers_=false;
- reset(expired, now);
- }
- std::vector<TimerQueue::Entry> TimerQueue::getExpired(Timestamp now)
- {
- assert(timers_.size()==activeTimers_.size());
- std::vector<Entry> expired;
- Entry sentry = std::make_pair(now, reinterpret_cast<Timer*>(UINTPTR_MAX));
- TimerList::iterator it = timers_.lower_bound(sentry);
-
- assert(it == timers_.end() || now < it->first);
- std::copy(timers_.begin(), it, back_inserter(expired));
- timers_.erase(timers_.begin(), it);
- BOOST_FOREACH(Entry entry,expired){
- ActiveTimer timer(entry.second,entry.second->sequence());
- size_t n=activeTimers_.erase(timer);
- assert(n==1);
- (void)n;
- }
- return expired;
- }
-
- void TimerQueue::reset(const std::vector<Entry>& expired, Timestamp now)
- {
- Timestamp nextExpire;
- for (std::vector<Entry>::const_iterator it = expired.begin();it != expired.end(); ++it)
- {
- ActiveTimer timer(it->second,it->second->sequence());
- if (it->second->repeat()&&
- cancelingTimers_.find(timer)==cancelingTimers_.end()){
- it->second->restart(now);
- insert(it->second);
- }
- else{
- delete it->second;
- }
- }
- if (!timers_.empty()){
- nextExpire = timers_.begin()->second->expiration();
- }
- if (nextExpire.valid()){
- resetTimerfd(timerfd_, nextExpire);
- }
- }
- bool TimerQueue::insert(Timer* timer)
- {
- loop_->assertInLoopThread();
- assert(timers_.size()==activeTimers_.size());
- bool earliestChanged = false;
- Timestamp when = timer->expiration();
- TimerList::iterator it = timers_.begin();
- if (it == timers_.end() || when < it->first)
- {
- earliestChanged = true;
- }
- {
- pair<TimerList::iterator,bool> result=
- timers_.insert(Entry(when,timer));
- assert(result.second);
- (void)result;
- }
- {
- pair<ActiveTimerSet::iterator,bool> result=
- activeTimers_.insert(ActiveTimer(timer,timer->sequence()));
- assert(result.second);
- (void)result;
- }
- assert(timers_.size()==activeTimers_.size());
- return earliestChanged;
- }
-
-
-
-
- class IngnoreSigPipe{
- public:
- IngnoreSigPipe(){
- ::signal(SIGPIPE,SIG_IGN);
- }
- };
- IngnoreSigPipe initObj;
- __thread EventLoop* t_loopInThisThread=0;
- const int kPollTimeMs=10000;
- static int createEventfd(){
- int evtfd=eventfd(0,EFD_NONBLOCK|EFD_CLOEXEC);
- if(evtfd<0){
- printf("Failed in eventfd\n");
- abort();
- }
- return evtfd;
- }
- EventLoop::EventLoop()
- :looping_(false),
- quit_(false),
- threadID_(syscall(SYS_gettid)),
- poller_(new Poller(this)),
- timerQueue_(new TimerQueue(this)),
- wakeupFd_(createEventfd()),
- wakeupChannel_(new Channel(this,wakeupFd_)),
- callingPendingFunctors_(false)
- {
- if(!t_loopInThisThread){
- t_loopInThisThread=this;
- }
- wakeupChannel_->setReadCallback(bind(&EventLoop::handleRead,this));
- wakeupChannel_->enableReading();
- }
- EventLoop::~EventLoop(){
- assert(!looping_);
- close(wakeupFd_);
- t_loopInThisThread=NULL;
- }
- void EventLoop::loop(){
- assert(!looping_);
- assertInLoopThread();
- looping_=true;
- quit_=false;
- while(!quit_){
- activeChannels_.clear();
- poller_->Poll(kPollTimeMs,&activeChannels_);
- for(ChannelList::iterator it=activeChannels_.begin();it!=activeChannels_.end();it++){
- (*it)->handleEvent();
- }
- doPendingFunctors();
- }
- looping_=false;
- }
- void EventLoop::quit(){
- quit_=true;
- if(!isInLoopThread()){
- wakeup();
- }
- }
- void EventLoop::updateChannel(Channel* channel){
- assert(channel->ownerLoop()==this);
- assertInLoopThread();
- poller_->updateChannel(channel);
- }
- void EventLoop::abortNotInLoopThread(){
- printf("abort not in Loop Thread\n");
- abort();
- }
- TimerId EventLoop::runAt(const Timestamp& time, const TimerCallback& cb)
- {
- return timerQueue_->addTimer(cb, time, 0.0);
- }
- TimerId EventLoop::runAfter(double delay, const TimerCallback& cb)
- {
- Timestamp time(addTime(Timestamp::now(), delay));
- return runAt(time, cb);
- }
- TimerId EventLoop::runEvery(double interval, const TimerCallback& cb)
- {
- Timestamp time(addTime(Timestamp::now(), interval));
- return timerQueue_->addTimer(cb, time, interval);
- }
- void EventLoop::cancel(TimerId timerId){
- return timerQueue_->cancel(timerId);
- }
- void EventLoop::runInLoop(const Functor& cb){
- if(isInLoopThread()){
- cb();
- }
- else{
- queueInLoop(cb);
- }
- }
- void EventLoop::queueInLoop(const Functor& cb){
- {
- MutexLockGuard lock(mutex_);
- pendingFunctors_.push_back(cb);
- }
- if(!isInLoopThread()||callingPendingFunctors_){
- wakeup();
- }
- }
- void EventLoop::handleRead(){
- uint64_t one=1;
- ssize_t n=read(wakeupFd_,&one,sizeof(one));
- if(n!=sizeof(one)){
- printf("EventLoop::handleRead() error\n");
- }
- }
- void EventLoop::doPendingFunctors(){
- vector<Functor> functors;
- callingPendingFunctors_=true;
- {
- MutexLockGuard lock(mutex_);
- functors.swap(pendingFunctors_);
- }
- for(size_t i=0;i<functors.size();i++){
- functors[i]();
- }
- callingPendingFunctors_=false;
- }
- void EventLoop::wakeup(){
- uint64_t one=1;
- ssize_t n=write(wakeupFd_,&one,sizeof(one));
- if(n!=sizeof(one)){
- printf("EventLoop::wakeup() write error\n");
- }
- }
- void EventLoop::removeChannel(Channel* channel){
- assert(channel->ownerLoop()==this);
- assertInLoopThread();
- poller_->removeChannel(channel);
- }
-
-
-
-
- Poller::Poller(EventLoop* loop):ownerLoop_(loop){}
- Poller::~Poller(){}
- Timestamp Poller::Poll(int timeoutMs,ChannelList* activeChannels){
- int numEvents=poll(&*pollfds_.begin(),pollfds_.size(),timeoutMs);
- Timestamp now(Timestamp::now());
- if(numEvents>0){
- fillActiveChannels(numEvents,activeChannels);
- }
- else if(numEvents==0){
- }
- else{
- printf("Poller::Poll error\n");
- }
- return now;
- }
- void Poller::fillActiveChannels(int numEvents,ChannelList* activeChannels) const{
- for(PollFdList::const_iterator pfd=pollfds_.begin();pfd!=pollfds_.end()&&numEvents>0;++pfd){
- if(pfd->revents>0){
- --numEvents;
- ChannelMap::const_iterator ch=channels_.find(pfd->fd);
- assert(ch!=channels_.end());
- Channel* channel=ch->second;
- assert(channel->fd()==pfd->fd);
- channel->set_revents(pfd->revents);
- activeChannels->push_back(channel);
- }
- }
- }
- void Poller::updateChannel(Channel* channel){
- assertInLoopThread();
- if(channel->index()<0){
- assert(channels_.find(channel->fd())==channels_.end());
- struct pollfd pfd;
- pfd.fd=channel->fd();
- pfd.events=static_cast<short>(channel->events());
- pfd.revents=0;
- pollfds_.push_back(pfd);
- int idx=static_cast<int>(pollfds_.size())-1;
- channel->set_index(idx);
- channels_[pfd.fd]=channel;
- }
- else{
- assert(channels_.find(channel->fd())!=channels_.end());
- assert(channels_[channel->fd()]==channel);
- int idx=channel->index();
- assert(0<=idx&&idx<static_cast<int>(pollfds_.size()));
- struct pollfd& pfd=pollfds_[idx];
- assert(pfd.fd==channel->fd()||pfd.fd==-channel->fd()-1);
- pfd.events=static_cast<short>(channel->events());
- pfd.revents=0;
- if(channel->isNoneEvent()){
- pfd.fd=-channel->fd()-1;
- }
- }
- }
- void Poller::removeChannel(Channel* channel)
- {
- assertInLoopThread();
- assert(channels_.find(channel->fd()) != channels_.end());
- assert(channels_[channel->fd()] == channel);
- assert(channel->isNoneEvent());
- int idx = channel->index();
- assert(0 <= idx && idx < static_cast<int>(pollfds_.size()));
- const struct pollfd& pfd = pollfds_[idx]; (void)pfd;
- assert(pfd.fd == -channel->fd()-1 && pfd.events == channel->events());
- size_t n = channels_.erase(channel->fd());
- assert(n == 1); (void)n;
- if (implicit_cast<size_t>(idx) == pollfds_.size()-1) {
- pollfds_.pop_back();
- } else {
- int channelAtEnd = pollfds_.back().fd;
- iter_swap(pollfds_.begin()+idx, pollfds_.end()-1);
- if (channelAtEnd < 0) {
- channelAtEnd = -channelAtEnd-1;
- }
- channels_[channelAtEnd]->set_index(idx);
- pollfds_.pop_back();
- }
- }
-
-
-
- const int Channel::kNoneEvent=0;
- const int Channel::kReadEvent=POLLIN|POLLPRI;
- const int Channel::kWriteEvent=POLLOUT;
- Channel::Channel(EventLoop* loop,int fdArg)
- :loop_(loop),fd_(fdArg),events_(0),revents_(0),
- index_(-1),eventHandling_(false)
- {}
- void Channel::update(){
- loop_->updateChannel(this);
- }
- Channel::~Channel(){
- assert(!eventHandling_);
- }
- void Channel::handleEvent(){
- eventHandling_=true;
- if(revents_&POLLNVAL){
- printf("Channel::handleEvent() POLLNVAL\n");
- }
- if((revents_&POLLHUP)&&!(revents_&POLLIN)){
- printf("Channel::handle_event() POLLUP\n");
- if(closeCallback_)
- closeCallback_();
- }
- if(revents_&(POLLERR|POLLNVAL)){
- if(errorCallback_)
- errorCallback_();
- }
- if(revents_&(POLLIN|POLLPRI|POLLRDHUP)){
- if(readCallback_) readCallback_();
- }
- if(revents_&POLLOUT){
- if(writeCallback_)
- writeCallback_();
- }
- eventHandling_=false;
- }
-
-
-
-
- class EventLoopThread:noncopyable{
- public:
- EventLoopThread()
- :loop_(NULL),exiting_(false),
- thread_(bind(&EventLoopThread::threadFunc,this)),
- mutex_(),cond_(mutex_){}
- ~EventLoopThread(){
- exiting_=true;
- loop_->quit();
- thread_.join();
- }
- EventLoop* startLoop(){
-
- thread_.start();
- {
- MutexLockGuard lock(mutex_);
- while(loop_==NULL){
- cond_.wait();
- }
- }
- return loop_;
- }
- private:
- void threadFunc(){
- EventLoop loop;
- {
- MutexLockGuard lock(mutex_);
- loop_=&loop;
- cond_.notify();
- }
- loop.loop();
- }
- EventLoop* loop_;
- bool exiting_;
- Thread thread_;
- Mutex mutex_;
- Condition cond_;
- };
-
-
-
- class EventLoopThreadPool:noncopyable{
- public:
- EventLoopThreadPool(EventLoop* baseLoop)
- :baseLoop_(baseLoop),
- started_(false),numThreads_(0),next_(0){}
- ~EventLoopThreadPool(){}
- void setThreadNum(int numThreads){numThreads_=numThreads;}
- void start(){
- assert(!started_);
- baseLoop_->assertInLoopThread();
- started_=true;
- for(int i=0;i<numThreads_;i++){
- EventLoopThread* t=new EventLoopThread;
- threads_.push_back(t);
- loops_.push_back(t->startLoop());
- }
- }
- EventLoop* getNextLoop(){
- baseLoop_->assertInLoopThread();
- EventLoop* loop=baseLoop_;
- if(!loops_.empty()){
- loop=loops_[next_];
- ++next_;
- if(static_cast<size_t>(next_)>=loops_.size())
- next_=0;
- }
- return loop;
- }
- private:
- EventLoop* baseLoop_;
- bool started_;
- int numThreads_;
- int next_;
- ptr_vector<EventLoopThread> threads_;
- vector<EventLoop*> loops_;
- };
-
-
-
- namespace sockets{
-
- inline uint64_t hostToNetwork64(uint64_t host64)
- {
- return htobe64(host64);
- }
- inline uint32_t hostToNetwork32(uint32_t host32)
- {
- return htonl(host32);
- }
- inline uint16_t hostToNetwork16(uint16_t host16)
- {
- return htons(host16);
- }
- inline uint64_t networkToHost64(uint64_t net64)
- {
- return be64toh(net64);
- }
-
- inline uint32_t networkToHost32(uint32_t net32)
- {
- return ntohl(net32);
- }
- inline uint16_t networkToHost16(uint16_t net16)
- {
- return ntohs(net16);
- }
-
- typedef struct sockaddr SA;
- const SA* sockaddr_cast(const struct sockaddr_in* addr){
- return static_cast<const SA*>(implicit_cast<const void*>(addr));
- }
- SA* sockaddr_cast(struct sockaddr_in* addr){
- return static_cast<SA*>(implicit_cast<void*>(addr));
- }
- void setNonBlockAndCloseOnExec(int sockfd){
- int flags = ::fcntl(sockfd, F_GETFL, 0);
- flags |= O_NONBLOCK;
- int ret = ::fcntl(sockfd, F_SETFL, flags);
- flags = ::fcntl(sockfd, F_GETFD, 0);
- flags |= FD_CLOEXEC;
- ret = ::fcntl(sockfd, F_SETFD, flags);
- }
- int createNonblockingOrDie()
- {
- #if VALGRIND
- int sockfd = ::socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
- if (sockfd < 0) {
- printf("socket() error\n");
- }
- setNonBlockAndCloseOnExec(sockfd);
- #else
- int sockfd = ::socket(AF_INET,
- SOCK_STREAM | SOCK_NONBLOCK | SOCK_CLOEXEC,
- IPPROTO_TCP);
- if (sockfd < 0){
- printf("socke() error\n");
- }
- #endif
- return sockfd;
- }
- int connect(int sockfd,const struct sockaddr_in& addr){
- return ::connect(sockfd,sockaddr_cast(&addr),sizeof addr);
- }
- void bindOrDie(int sockfd, const struct sockaddr_in& addr)
- {
- int ret = ::bind(sockfd, sockaddr_cast(&addr), sizeof addr);
- if (ret < 0) {
- printf("bind() error\n");
- }
- }
- void listenOrDie(int sockfd){
- int ret = ::listen(sockfd, SOMAXCONN);
- if (ret < 0){
- printf("listen() error\n");
- }
- }
- int accept(int sockfd, struct sockaddr_in* addr)
- {
- socklen_t addrlen = sizeof *addr;
- #if VALGRIND
- int connfd = ::accept(sockfd, sockaddr_cast(addr), &addrlen);
- setNonBlockAndCloseOnExec(connfd);
- #else
- int connfd = ::accept4(sockfd, sockaddr_cast(addr),
- &addrlen, SOCK_NONBLOCK | SOCK_CLOEXEC);
- #endif
- if (connfd < 0){
- int savedErrno = errno;
- printf("accept error\n");
- switch (savedErrno)
- {
- case EAGAIN:
- case ECONNABORTED:
- case EINTR:
- case EPROTO:
- case EPERM:
- case EMFILE:
- errno = savedErrno;
- break;
- case EBADF:
- case EFAULT:
- case EINVAL:
- case ENFILE:
- case ENOBUFS:
- case ENOMEM:
- case ENOTSOCK:
- case EOPNOTSUPP:
- printf("accept() fatal erro\n");
- break;
- default:
- printf("accpet() unknown error\n");
- break;
- }
- }
- return connfd;
- }
- void close(int sockfd){
- if (::close(sockfd) < 0){
- printf("sockets::close\n");
- }
- }
- void shutdownWrite(int sockfd){
- if(::shutdown(sockfd,SHUT_WR)<0)
- printf("sockets::shutdownWrite() error\n");
- }
- void toHostPort(char* buf, size_t size,const struct sockaddr_in& addr)
- {
- char host[INET_ADDRSTRLEN] = "INVALID";
- ::inet_ntop(AF_INET, &addr.sin_addr, host, sizeof host);
- uint16_t port =networkToHost16(addr.sin_port);
- snprintf(buf, size, "%s:%u", host, port);
- }
- void fromHostPort(const char* ip, uint16_t port,struct sockaddr_in* addr)
- {
- addr->sin_family = AF_INET;
- addr->sin_port = hostToNetwork16(port);
- if (::inet_pton(AF_INET, ip, &addr->sin_addr) <= 0)
- {
- printf("sockets::fromHostPort\n");
- }
- }
- sockaddr_in getLocalAddr(int sockfd)
- {
- struct sockaddr_in localaddr;
- bzero(&localaddr, sizeof localaddr);
- socklen_t addrlen = sizeof(localaddr);
- if (::getsockname(sockfd, sockaddr_cast(&localaddr), &addrlen) < 0)
- {
- printf("getsockname() error\n");
- }
- return localaddr;
- }
- struct sockaddr_in getPeerAddr(int sockfd){
- struct sockaddr_in peeraddr;
- bzero(&peeraddr,sizeof peeraddr);
- socklen_t addrlen=sizeof peeraddr;
- if(::getpeername(sockfd,sockaddr_cast(&peeraddr),&addrlen)<0)
- printf("sockets::getPeerAddr() error\n");
- return peeraddr;
- }
- int getSocketError(int sockfd){
- int optval;
- socklen_t optlen=sizeof optval;
- if(getsockopt(sockfd,SOL_SOCKET,SO_ERROR,&optval,&optlen)<0){
- return errno;
- }
- else{
- return optval;
- }
- }
- bool isSelfConnect(int sockfd){
- struct sockaddr_in localaddr=getLocalAddr(sockfd);
- struct sockaddr_in peeraddr=getPeerAddr(sockfd);
- return localaddr.sin_port==peeraddr.sin_port&&
- localaddr.sin_addr.s_addr==peeraddr.sin_addr.s_addr;
- }
- }
-
-
-
- class InetAddress;
- class Socket:noncopyable{
- public:
- explicit Socket(uint16_t sockfd):sockfd_(sockfd){}
- ~Socket();
- int fd() const{return sockfd_;}
- void bindAddress(const InetAddress& addr);
- void listen();
- int accept(InetAddress* peeraddr);
- void setReuseAddr(bool on);
- void shutdownWrite(){
- sockets::shutdownWrite(sockfd_);
- }
- void setTcpNoDelay(bool on){
- int optval=on?1:0;
- ::setsockopt(sockfd_,IPPROTO_TCP,TCP_NODELAY,&optval,sizeof optval);
- }
- private:
- const int sockfd_;
- };
-
-
-
- class InetAddress{
- public:
- explicit InetAddress(uint16_t port);
- InetAddress(const string& ip,uint16_t port);
- InetAddress(const struct sockaddr_in& addr):addr_(addr){}
- string toHostPort() const;
- const struct sockaddr_in& getSockAddrInet() const{return addr_;}
- void setSockAddrInet(const struct sockaddr_in& addr){addr_=addr;}
- private:
- struct sockaddr_in addr_;
- };
- BOOST_STATIC_ASSERT(sizeof(InetAddress)==sizeof(struct sockaddr_in));
- class Acceptor:noncopyable{
- public:
- typedef function<void(int sockfd,const InetAddress&)> NewConnectionCallback;
- Acceptor(EventLoop* loop,const InetAddress& listenAddr);
- void setNewConnectionCallback(const NewConnectionCallback& cb)
- { newConnectionCallback_=cb;}
- bool listening() const{return listening_;}
- void listen();
- private:
- void handleRead();
- EventLoop* loop_;
- Socket acceptSocket_;
- Channel acceptChannel_;
- NewConnectionCallback newConnectionCallback_;
- bool listening_;
-
- };
-
-
-
- Socket::~Socket()
- {
- sockets::close(sockfd_);
- }
- void Socket::bindAddress(const InetAddress& addr)
- {
- sockets::bindOrDie(sockfd_, addr.getSockAddrInet());
- }
- void Socket::listen()
- {
- sockets::listenOrDie(sockfd_);
- }
- int Socket::accept(InetAddress* peeraddr)
- {
- struct sockaddr_in addr;
- bzero(&addr, sizeof addr);
- int connfd = sockets::accept(sockfd_, &addr);
- if (connfd >= 0)
- {
- peeraddr->setSockAddrInet(addr);
- }
- return connfd;
- }
- void Socket::setReuseAddr(bool on)
- {
- int optval = on ? 1 : 0;
- ::setsockopt(sockfd_, SOL_SOCKET, SO_REUSEADDR,
- &optval, sizeof optval);
- }
-
-
-
- static const in_addr_t kInaddrAny=INADDR_ANY;
- InetAddress::InetAddress(uint16_t port)
- {
- bzero(&addr_, sizeof addr_);
- addr_.sin_family = AF_INET;
- addr_.sin_addr.s_addr = sockets::hostToNetwork32(kInaddrAny);
- addr_.sin_port = sockets::hostToNetwork16(port);
- }
- InetAddress::InetAddress(const std::string& ip, uint16_t port)
- {
- bzero(&addr_, sizeof addr_);
- sockets::fromHostPort(ip.c_str(), port, &addr_);
- }
- string InetAddress::toHostPort() const
- {
- char buf[32];
- sockets::toHostPort(buf, sizeof buf, addr_);
- return buf;
- }
-
-
-
- Acceptor::Acceptor(EventLoop* loop, const InetAddress& listenAddr)
- : loop_(loop),
- acceptSocket_(sockets::createNonblockingOrDie()),
- acceptChannel_(loop, acceptSocket_.fd()),
- listening_(false)
- {
- acceptSocket_.setReuseAddr(true);
- acceptSocket_.bindAddress(listenAddr);
- acceptChannel_.setReadCallback(
- boost::bind(&Acceptor::handleRead, this));
- }
- void Acceptor::listen()
- {
- loop_->assertInLoopThread();
- listening_ = true;
- acceptSocket_.listen();
- acceptChannel_.enableReading();
- }
- void Acceptor::handleRead()
- {
- loop_->assertInLoopThread();
- InetAddress peerAddr(0);
- int connfd = acceptSocket_.accept(&peerAddr);
- if (connfd >= 0) {
- if (newConnectionCallback_) {
- newConnectionCallback_(connfd, peerAddr);
- } else {
- sockets::close(connfd);
- }
- }
- }
-
-
-
- class Buffer{
- public:
- static const size_t kCheapPrepend=8;
- static const size_t kInitialSize=1024;
- Buffer():buffer_(kCheapPrepend+kInitialSize),
- readerIndex_(kCheapPrepend),writerInex_(kCheapPrepend)
- {
- assert(readableBytes()==0);
- assert(writeableBytes()==kInitialSize);
- assert(prependableBytes()==kCheapPrepend);
- }
- void swap(Buffer& rhs){
- buffer_.swap(rhs.buffer_);
- std::swap(readerIndex_,rhs.readerIndex_);
- std::swap(writerInex_,rhs.writerInex_);
- }
- size_t readableBytes() const{
- return writerInex_-readerIndex_;
- }
- size_t writeableBytes() const{
- return buffer_.size()-writerInex_;
- }
- size_t prependableBytes() const{
- return readerIndex_;
- }
- const char* peek() const{
- return begin()+readerIndex_;
- }
- void retrieve(size_t len){
- assert(len<=readableBytes());
- readerIndex_+=len;
- }
- void retrieveUntil(const char* end){
- assert(peek()<=end);
- assert(end<=beginWrite());
- retrieve(end-peek());
- }
- void retrieveAll(){
- readerIndex_=kCheapPrepend;
- writerInex_=kCheapPrepend;
- }
- std::string retrieveAsString(){
- string str(peek(),readableBytes());
- retrieveAll();
- return str;
- }
- void append(const string& str){
- append(str.data(),str.length());
- }
- void append(const char* data,size_t len){
- ensureWriteableBytes(len);
- std::copy(data,data+len,beginWrite());
- hasWritten(len);
- }
- void append(const void* data,size_t len){
- append(static_cast<const char*>(data),len);
- }
- void ensureWriteableBytes(size_t len){
- if(writeableBytes()<len){
- makeSpace(len);
- }
- assert(writeableBytes()>=len);
- }
- char* beginWrite(){
- return begin()+writerInex_;
- }
- const char* beginWrite() const{
- return begin()+writerInex_;
- }
- void hasWritten(size_t len){
- writerInex_+=len;
- }
- void prepend(const void* data,size_t len){
- assert(len<=prependableBytes());
- readerIndex_-=len;
- const char* d=static_cast<const char*>(data);
- std::copy(d,d+len,begin()+readerIndex_);
- }
- void shrink(size_t reserve){
- vector<char> buf(kCheapPrepend+readableBytes()+reserve);
- std::copy(peek(),peek()+readableBytes(),buf.begin()+kCheapPrepend);
- buf.swap(buffer_);
- }
- ssize_t readFd(int fd,int* savedErrno){
- char extrabuf[65536];
- struct iovec vec[2];
- const size_t writeable=writeableBytes();
- vec[0].iov_base=begin()+writerInex_;
- vec[0].iov_len=writeable;
- vec[1].iov_base=extrabuf;
- vec[1].iov_len=sizeof extrabuf;
- const ssize_t n=readv(fd,vec,2);
- if(n<0){
- *savedErrno=errno;
- }
- else if(implicit_cast<size_t>(n)<=writeable){
- writerInex_+=n;
- }
- else{
- writerInex_=buffer_.size();
- append(extrabuf,n-writeable);
- }
- return n;
- }
- private:
- char* begin(){
- return &*buffer_.begin();
- }
- const char* begin() const{
- return &*buffer_.begin();
- }
- void makeSpace(size_t len){
-
- if(writeableBytes()+prependableBytes()<len+kCheapPrepend){
- buffer_.resize(writerInex_+len);
- }
-
- else{
- assert(kCheapPrepend<readerIndex_);
- size_t readable=readableBytes();
- std::copy(begin()+readerIndex_,begin()+writerInex_,begin()+kCheapPrepend);
- readerIndex_=kCheapPrepend;
- writerInex_=readerIndex_+readable;
- assert(readable==readableBytes());
- }
- }
- private:
- vector<char> buffer_;
- size_t readerIndex_;
- size_t writerInex_;
- };
-
- class TcpConnection;
- typedef shared_ptr<TcpConnection> TcpConnectionPtr;
-
-
-
- class TcpConnection:noncopyable,public enable_shared_from_this<TcpConnection>{
- public:
- TcpConnection(EventLoop* loop,const string& name,int sockfd,
- const InetAddress& localAddr,const InetAddress& peerAddr);
- ~TcpConnection();
- EventLoop* getLoop() const{return loop_;}
- const string& name() const{return name_;}
- const InetAddress& localAddr(){return localAddr_;}
- const InetAddress& peerAddress(){return peerAddr_;}
- bool connected() const{return state_==kConnected;}
- void send(const string& message);
- void shutdown();
- void setTcpNoDelay(bool on);
- void setConnectionCallback(const ConnectionCallback& cb){
- connectionCallback_=cb;
- }
- void setMessageCallback(const MessageCallback& cb){
- messageCallback_=cb;
- }
- void setWriteCompleteCallback(const WriteCompleteCallback& cb){
- writeCompleteCallback_=cb;
- }
- void setCloseCallback(const CloseCallback& cb){
-
- closeCallback_=cb;
- }
- void connectEstablished();
-
- void connectDestroyed();
-
-
-
- private:
- enum StateE{kConnecting,kConnected,kDisconnecting,kDisconnected,};
-
- void setState(StateE s){state_=s;}
- void handleRead();
-
- void handleWrite();
- void handleClose();
- void handleError();
- void sendInLoop(const string& message);
- void shutdownInLoop();
- EventLoop* loop_;
- string name_;
- StateE state_;
- scoped_ptr<Socket> socket_;
- scoped_ptr<Channel> channel_;
- InetAddress localAddr_;
- InetAddress peerAddr_;
- ConnectionCallback connectionCallback_;
-
-
- MessageCallback messageCallback_;
- WriteCompleteCallback writeCompleteCallback_;
- CloseCallback closeCallback_;
- Buffer inputBuffer_;
- Buffer outputBuffer_;
- };
-
-
-
- class TcpServer:noncopyable{
- public:
- TcpServer(EventLoop* loop,const InetAddress& listenAddr);
- ~TcpServer();
- void setThreadNum(int numThreads);
- void start();
- void setConnectionCallback(const ConnectionCallback& cb){
- connectionCallback_=cb;
- }
- void setMessageCallback(const MessageCallback& cb){
- messageCallback_=cb;
- }
- void setWriteCompleteCallback(const WriteCompleteCallback& cb){
- writeCompleteCallback_=cb;
- }
- private:
- void newConnection(int sockfd,const InetAddress& peerAddr);
- void removeConnection(const TcpConnectionPtr& conn);
- void removeConnectionInLoop(const TcpConnectionPtr& conn);
- typedef map<string,TcpConnectionPtr> ConnectionMap;
- EventLoop* loop_;
- const string name_;
- scoped_ptr<Acceptor> acceptor_;
- scoped_ptr<EventLoopThreadPool> threadPool_;
- ConnectionCallback connectionCallback_;
- MessageCallback messageCallback_;
- WriteCompleteCallback writeCompleteCallback_;
- bool started_;
- int nextConnId_;
- ConnectionMap connections_;
- };
-
-
-
- TcpConnection::TcpConnection(EventLoop* loop,
- const std::string& nameArg,
- int sockfd,
- const InetAddress& localAddr,
- const InetAddress& peerAddr)
- : loop_(loop),
- name_(nameArg),
- state_(kConnecting),
- socket_(new Socket(sockfd)),
- channel_(new Channel(loop, sockfd)),
- localAddr_(localAddr),
- peerAddr_(peerAddr)
- {
- channel_->setReadCallback(bind(&TcpConnection::handleRead, this));
- channel_->setWriteCallback(bind(&TcpConnection::handleWrite,this));
- channel_->setCloseCallback(bind(&TcpConnection::handleClose,this));
- channel_->setErrorCallback(bind(&TcpConnection::handleError,this));
- }
- TcpConnection::~TcpConnection()
- {
- printf("TcpConnection::%s,fd=%d\n",name_.c_str(),channel_->fd());
- }
- void TcpConnection::send(const string& message){
- cout<<"TcpConnection::send() ##"<<message<<endl;
- if(state_==kConnected){
- if(loop_->isInLoopThread()){
- sendInLoop(message);
- }
- else{
- loop_->runInLoop(bind(&TcpConnection::sendInLoop,this,message));
- }
- }
- }
- void TcpConnection::sendInLoop(const string& message){
-
-
-
-
- loop_->assertInLoopThread();
- ssize_t nwrote=0;
- cout<<message<<endl;
- if(!channel_->isWriting()&&outputBuffer_.readableBytes()==0){
- nwrote=write(channel_->fd(),message.data(),message.size());
- if(nwrote>=0){
- if(implicit_cast<size_t>(nwrote)<message.size()){
- printf("I am going to write more data\n");
- }
- else if(writeCompleteCallback_){
- loop_->queueInLoop(bind(writeCompleteCallback_,shared_from_this()));
- }
- }
- else{
- nwrote=0;
- if(errno!=EWOULDBLOCK){
- printf("TcpConnection::sendInLoop() error\n");
- }
- }
- }
- assert(nwrote>=0);
- if(implicit_cast<size_t>(nwrote)<message.size()){
- outputBuffer_.append(message.data()+nwrote,message.size()-nwrote);
- if(!channel_->isWriting()){
- channel_->enableWriting();
- }
- }
- }
- void TcpConnection::shutdown(){
- if(state_==kConnected){
- setState(kDisconnecting);
- loop_->runInLoop(bind(&TcpConnection::shutdownInLoop,this));
- }
- }
- void TcpConnection::shutdownInLoop(){
- loop_->assertInLoopThread();
- if(!channel_->isWriting()){
- socket_->shutdownWrite();
- }
- }
- void TcpConnection::setTcpNoDelay(bool on){
- socket_->setTcpNoDelay(on);
- }
- void TcpConnection::connectEstablished()
- {
- loop_->assertInLoopThread();
- assert(state_ == kConnecting);
- setState(kConnected);
- channel_->enableReading();
- connectionCallback_(shared_from_this());
- }
- void TcpConnection::handleRead()
- {
- int savedErrno=0;
- ssize_t n =inputBuffer_.readFd(channel_->fd(),&savedErrno);
- if(n>0)
- messageCallback_(shared_from_this(),&inputBuffer_);
- else if(n==0)
- handleClose();
- else{
- errno=savedErrno;
- printf("TcpConnection::hanleRead() error\n");
- handleError();
- }
- }
- void TcpConnection::handleWrite(){
- loop_->assertInLoopThread();
- if(channel_->isWriting()){
- ssize_t n=write(channel_->fd(),outputBuffer_.peek(),outputBuffer_.readableBytes());
- if(n>0){
- outputBuffer_.retrieve(n);
- if(outputBuffer_.readableBytes()==0){
- channel_->disableWriting();
- if(writeCompleteCallback_){
- loop_->queueInLoop(bind(writeCompleteCallback_,shared_from_this()));
- }
- if(state_==kDisconnecting)
- shutdownInLoop();
- }
- else
- printf("I am going to write more data\n");
- }
- else
- printf("TcpConnection::handleWrite()\n");
- }
- else
- printf("Connection is down,no more writing\n");
- }
- void TcpConnection::handleClose(){
- loop_->assertInLoopThread();
- assert(state_==kConnected||state_==kDisconnecting);
- channel_->disableAll();
- closeCallback_(shared_from_this());
- }
- void TcpConnection::handleError(){
- int err=sockets::getSocketError(channel_->fd());
- printf("TcpConnection::handleError() %d %s\n",err,strerror(err));
- }
- void TcpConnection::connectDestroyed(){
- loop_->assertInLoopThread();
- printf("TcpConnection::handleClose() state=%s\n",state_);
- assert(state_==kConnected||state_==kDisconnected);
- setState(kDisconnected);
- channel_->disableAll();
- connectionCallback_(shared_from_this());
- loop_->removeChannel(get_pointer(channel_));
- }
-
-
-
- TcpServer::TcpServer(EventLoop* loop, const InetAddressEventLoop::removeChannel()
- name_(listenAddr.toHostPort()),
- acceptor_(new Acceptor(loop, listenAddr)),
- threadPool_(new EventLoopThreadPool(loop)),
- started_(false),
- nextConnId_(1))
- {
- acceptor_->setNewConnectionCallback(bind(&TcpServer::newConnection, this, _1, _2));
- }
- TcpServer::~TcpServer()
- {
- }
- void TcpServer::setThreadNum(int numThreads){
- assert(numThreads>=0);
- threadPool_->setThreadNum(numThreads);
- }
- void TcpServer::start()
- {
- if (!started_)
- {
- started_ = true;
- }
-
- if (!acceptor_->listening())
- {
- loop_->runInLoop(bind(&Acceptor::listen, get_pointer(acceptor_)));
- }
- }
- void TcpServer::newConnection(int sockfd, const InetAddress& peerAddr)
- {
- loop_->assertInLoopThread();
- char buf[32];
- snprintf(buf, sizeof buf, "#%d", nextConnId_);
- ++nextConnId_;
- string connName = name_ + buf;
- InetAddress localAddr(sockets::getLocalAddr(sockfd));
- EventLoop* ioLoop=threadPool_->getNextLoop();
- TcpConnectionPtr conn(
- new TcpConnection(ioLoop, connName, sockfd, localAddr, peerAddr));
- connections_[connName]=conn;
- conn->setConnectionCallback(connectionCallback_);
- conn->setMessageCallback(messageCallback_);
- conn->setWriteCompleteCallback(writeCompleteCallback_);
- conn->setCloseCallback(bind(&TcpServer::removeConnection,this,_1));
- ioLoop->runInLoop(bind(&TcpConnection::connectEstablished,conn));
-
- }
- void TcpServer::removeConnection(const TcpConnectionPtr& conn){
- loop_->runInLoop(bind(&TcpServer::removeConnectionInLoop,this,conn));
-
- }
- void TcpServer::removeConnectionInLoop(const TcpConnectionPtr& conn){
- loop_->assertInLoopThread();
- size_t n=connections_.erase(conn->name());
- assert(n==1);
- (void)n;
- EventLoop* ioLoop=conn->getLoop();
- ioLoop->queueInLoop(bind(&TcpConnection::connectDestroyed,conn));
- }
-
-
-
- class Connector : boost::noncopyable
- {
- public:
- typedef function<void (int sockfd)> NewConnectionCallback;
- Connector(EventLoop* loop, const InetAddress& serverAddr);
- ~Connector();
- void setNewConnectionCallback(const NewConnectionCallback& cb)
- { newConnectionCallback_ = cb; }
- void start();
- void restart();
- void stop();
- const InetAddress& serverAddress() const { return serverAddr_; }
- private:
- enum States { kDisconnected, kConnecting, kConnected };
-
- static const int kMaxRetryDelayMs = 30*1000;
- static const int kInitRetryDelayMs = 500;
- void setState(States s) { state_ = s; }
- void startInLoop();
- void connect();
- void connecting(int sockfd);
- void handleWrite();
- void handleError();
- void retry(int sockfd);
- int removeAndResetChannel();
- void resetChannel();
- EventLoop* loop_;
- InetAddress serverAddr_;
- bool connect_;
- States state_;
- boost::scoped_ptr<Channel> channel_;
- NewConnectionCallback newConnectionCallback_;
- int retryDelayMs_;
- TimerId timerId_;
- };
-
-
-
- typedef boost::shared_ptr<Connector> ConnectorPtr;
- const int Connector::kMaxRetryDelayMs;
- Connector::Connector(EventLoop* loop, const InetAddress& serverAddr)
- :loop_(loop),
- serverAddr_(serverAddr),
- connect_(false),
- state_(kDisconnected),
- retryDelayMs_(kInitRetryDelayMs)
- {
- }
- Connector::~Connector()
- {
- loop_->cancel(timerId_);
- assert(!channel_);
- }
- void Connector::start()
- {
- connect_ = true;
- loop_->runInLoop(boost::bind(&Connector::startInLoop, this));
- }
- void Connector::startInLoop()
- {
- loop_->assertInLoopThread();
- assert(state_ == kDisconnected);
- if (connect_)
- {
- connect();
- }
- else
- {}
- }
- void Connector::connect()
- {
- int sockfd = sockets::createNonblockingOrDie();
- int ret = sockets::connect(sockfd, serverAddr_.getSockAddrInet());
- int savedErrno = (ret == 0) ? 0 : errno;
- switch (savedErrno)
- {
- case 0:
- case EINPROGRESS:
- case EINTR:
- case EISCONN:
- connecting(sockfd);
- break;
-
- case EAGAIN:
- case EADDRINUSE:
- case EADDRNOTAVAIL:
- case ECONNREFUSED:
- case ENETUNREACH:
- retry(sockfd);
- break;
-
- case EACCES:
- case EPERM:
- case EAFNOSUPPORT:
- case EALREADY:
- case EBADF:
- case EFAULT:
- case ENOTSOCK:
- sockets::close(sockfd);
- break;
-
- default:
- sockets::close(sockfd);
-
- break;
- }
- }
- void Connector::restart()
- {
- loop_->assertInLoopThread();
- setState(kDisconnected);
- retryDelayMs_ = kInitRetryDelayMs;
- connect_ = true;
- startInLoop();
- }
- void Connector::stop()
- {
- connect_ = false;
- loop_->cancel(timerId_);
- }
- void Connector::connecting(int sockfd)
- {
- setState(kConnecting);
- assert(!channel_);
- channel_.reset(new Channel(loop_, sockfd));
- channel_->setWriteCallback(bind(&Connector::handleWrite, this));
- channel_->setErrorCallback(bind(&Connector::handleError, this));
- channel_->enableWriting();
- }
- int Connector::removeAndResetChannel()
- {
- channel_->disableAll();
- loop_->removeChannel(get_pointer(channel_));
- int sockfd = channel_->fd();
- loop_->queueInLoop(bind(&Connector::resetChannel, this));
- return sockfd;
- }
- void Connector::resetChannel()
- {
- channel_.reset();
- }
- void Connector::handleWrite()
- {
- if (state_ == kConnecting)
- {
- int sockfd = removeAndResetChannel();
- int err = sockets::getSocketError(sockfd);
- if (err)
- retry(sockfd);
- else if (sockets::isSelfConnect(sockfd))
- retry(sockfd);
- else
- {
- setState(kConnected);
- if (connect_)
- newConnectionCallback_(sockfd);
- else
- sockets::close(sockfd);
- }
- }
- else
- {
- assert(state_ == kDisconnected);
- }
- }
-
- void Connector::handleError()
- {
- assert(state_ == kConnecting);
-
- int sockfd = removeAndResetChannel();
- int err = sockets::getSocketError(sockfd);
- retry(sockfd);
- }
-
- void Connector::retry(int sockfd)
- {
- sockets::close(sockfd);
- setState(kDisconnected);
- if (connect_){
- timerId_ = loop_->runAfter(retryDelayMs_/1000.0,
- boost::bind(&Connector::startInLoop, this));
- retryDelayMs_ = std::min(retryDelayMs_ * 2, kMaxRetryDelayMs);
- }
- else
- {}
- }
-
-
-
- typedef boost::shared_ptr<Connector> ConnectorPtr;
- class TcpClient : boost::noncopyable
- {
- public:
- TcpClient(EventLoop* loop,
- const InetAddress& serverAddr,
- const string& name);
- ~TcpClient();
- void connect();
- void disconnect();
- void stop();
- TcpConnectionPtr connection() const
- {
- MutexLockGuard lock(mutex_);
- return connection_;
- }
-
- EventLoop* getLoop() const { return loop_; }
- bool retry() const;
- void enableRetry() { retry_ = true; }
- void setConnectionCallback(const ConnectionCallback& cb)
- { connectionCallback_ = cb; }
- void setMessageCallback(const MessageCallback& cb)
- { messageCallback_ = cb; }
- void setWriteCompleteCallback(const WriteCompleteCallback& cb)
- { writeCompleteCallback_ = cb; }
- #ifdef __GXX_EXPERIMENTAL_CXX0X__
- void setConnectionCallback(ConnectionCallback&& cb)
- { connectionCallback_ = cb; }
- void setMessageCallback(MessageCallback&& cb)
- { messageCallback_ = cb; }
- void setWriteCompleteCallback(WriteCompleteCallback&& cb)
- { writeCompleteCallback_ = cb; }
- #endif
- private:
- void newConnection(int sockfd);
- void removeConnection(const TcpConnectionPtr& conn);
- EventLoop* loop_;
- ConnectorPtr connector_;
- const string name_;
- ConnectionCallback connectionCallback_;
- MessageCallback messageCallback_;
- WriteCompleteCallback writeCompleteCallback_;
- bool retry_;
- bool connect_;
- int nextConnId_;
- mutable MutexLock mutex_;
- TcpConnectionPtr connection_;
- };
- namespace detail
- {
- void removeConnection(EventLoop* loop, const TcpConnectionPtr& conn)
- {
- loop->queueInLoop(boost::bind(&TcpConnection::connectDestroyed, conn));
- }
- void removeConnector(const ConnectorPtr& connector)
- {
-
- }
- }
- TcpClient::TcpClient(EventLoop* loop,
- const InetAddress& serverAddr,
- const string& name)
- : loop_(CHECK_NOTNULL(loop)),
- connector_(new Connector(loop, serverAddr)),
- name_(name),
- connectionCallback_(defaultConnectionCallback),
- messageCallback_(defaultMessageCallback),
- retry_(false),
- connect_(true),
- nextConnId_(1)
- {
- connector_->setNewConnectionCallback(
- boost::bind(&TcpClient::newConnection, this, _1));
- }
-
- TcpClient::~TcpClient()
- {
- TcpConnectionPtr conn;
- {
- MutexLockGuard lock(mutex_);
- conn = connection_;
- }
- if (conn)
- {
- CloseCallback cb = boost::bind(&detail::removeConnection, loop_, _1);
- loop_->runInLoop(
- boost::bind(&TcpConnection::setCloseCallback, conn, cb));
- }
- else
- {
- connector_->stop();
- loop_->runAfter(1, boost::bind(&detail::removeConnector, connector_));
- }
- }
- void TcpClient::connect()
- {
- connect_ = true;
- connector_->start();
- }
- void TcpClient::disconnect()
- {
- connect_ = false;
- {
- MutexLockGuard lock(mutex_);
- if (connection_)
- {
- connection_->shutdown();
- }
- }
- }
- void TcpClient::stop()
- {
- connect_ = false;
- connector_->stop();
- }
- void TcpClient::newConnection(int sockfd)
- {
- loop_->assertInLoopThread();
- InetAddress peerAddr(sockets::getPeerAddr(sockfd));
- char buf[32];
- snprintf(buf, sizeof buf, ":%s#%d", peerAddr.toIpPort().c_str(), nextConnId_);
- ++nextConnId_;
- string connName = name_ + buf;
- InetAddress localAddr(sockets::getLocalAddr(sockfd));
- TcpConnectionPtr conn(new TcpConnection(loop_,
- connName,
- sockfd,
- localAddr,
- peerAddr));
-
- conn->setConnectionCallback(connectionCallback_);
- conn->setMessageCallback(messageCallback_);
- conn->setWriteCompleteCallback(writeCompleteCallback_);
- conn->setCloseCallback(
- boost::bind(&TcpClient::removeConnection, this, _1));
- {
- MutexLockGuard lock(mutex_);
- connection_ = conn;
- }
- conn->connectEstablished();
- }
- void TcpClient::removeConnection(const TcpConnectionPtr& conn)
- {
- loop_->assertInLoopThread();
- assert(loop_ == conn->getLoop());
- {
- MutexLockGuard lock(mutex_);
- assert(connection_ == conn);
- connection_.reset();
- }
- loop_->queueInLoop(boost::bind(&TcpConnection::connectDestroyed, conn));
- if (retry_ && connect_)
- {
- connector_->restart();
- }
- }
-
-
-
-
- class Epoller:noncopyable{
- public:
- typedef vector<Channel*> ChannelList;
- Epoller(EventLoop* loop)
- :ownerLoop_(loop),
- epollfd_(::epoll_create1(EPOLL_CLOEXEC)),
- events_(kInitEventListSize)
- {
- if(epollfd_<0){
- printf("Epoller::epoll_create1() error\n");
- abort();
- }
- }
- ~Epoller(){
- ::close(epollfd_);
- }
- Timestamp poll(int timeoutMs,ChannelList* activeChannels){
- int numEvents=::epoll_wait(epollfd_,&*events_.begin(),
- static_cast<int>(events_.size()),timeoutMs);
- Timestamp now(Timestamp::now());
- if(numEvents>0){
- fillActiveChannels(numEvents,activeChannels);
- if(implicit_cast<size_t>(numEvents)==events_.size()){
- events_.resize(events_.size()*2);
- }
- else if(numEvents==0){}
- else{
- printf("Epoller::epoll_wait() error\n");
- }
- }
- return now;
- }
- void updateChannel(Channel* channel){
- assertInLoopThread();
- const int index=channel->index();
- if(index==-1||index==2){
- int fd=channel->fd();
- if(index==-1){
- assert(channels_.find(fd)==channels_.end());
- channels_[fd]=channel;
- }
- else{
- assert(channels_.find(fd)!=channels_.end());
- assert(channels_[fd]==channel);
- }
- channel->set_index(1);
- update(EPOLL_CTL_ADD,channel);
- }
- else{
- int fd=channel->fd();
- (void)fd;
- assert(channels_.find(fd)!=channels_.end());
- assert(channels_[fd]==channel);
- assert(index==1);
- if(channel->isNoneEvent()){
- update(EPOLL_CTL_DEL,channel);
- channel->set_index(2);
- }
- else{
- update(EPOLL_CTL_MOD,channel);
- }
- }
- }
- void removeChannel(Channel* channel){
- assertInLoopThread();
- int fd=channel->fd();
- assert(channels_.find(fd)!=channels_.end());
- assert(channels_[fd]==channel);
- assert(channel->isNoneEvent());
- int index=channel->index();
- assert(index==1||index==2);
- size_t n=channels_.erase(fd);
- (void)n;
- assert(n==1);
- if(index==1){
- update(EPOLL_CTL_DEL,channel);
- }
- channel->set_index(-1);
- }
- void assertInLoopThread(){
- ownerLoop_->assertInLoopThread();
- }
- private:
- static const int kInitEventListSize=16;
- void fillActiveChannels(int numEvents,ChannelList* activeChannels) const
- {
- assert(implicit_cast<size_t>(numEvents)<=events_.size());
- for(int i=0;i<numEvents;i++){
- Channel* channel=static_cast<Channel*>(events_[i].data.ptr);
- int fd=channel->fd();
- ChannelMap::const_iterator it=channels_.find(fd);
- assert(it!=channels_.end());
- assert(it->second==channel);
- channel->set_revents(events_[i].events);
- activeChannels->push_back(channel);
- }
- }
- void update(int operation,Channel* channel){
- struct epoll_event event;
- bzero(&event,sizeof event);
- event.events=channel->events();
- event.data.ptr=channel;
- int fd=channel->fd();
- if(::epoll_ctl(epollfd_,operation,fd,&event)<0){
- if(operation==EPOLL_CTL_DEL){
- printf("Epoller::update() EPOLL_CTL_DEL error\n");
- }
- else{
- printf("Epoller::update() EPOLL_CTL_ error\n");
- }
- }
- }
- typedef vector<struct epoll_event> EventList;
- typedef map<int,Channel*> ChannelMap;
-
- EventLoop* ownerLoop_;
- int epollfd_;
- EventList events_;
- ChannelMap channels_;
- };