twisted reactor 实现源码解析

最新推荐文章于 2022-08-07 11:15:42 发布

akuibpt23191

最新推荐文章于 2022-08-07 11:15:42 发布

阅读量279

点赞数

文章标签： python 网络

原文链接：http://www.cnblogs.com/wodeboke-y/p/11247074.html

版权

twisted reactor 实现源码解析

1. reactor源码解析

1.1. 案例分析代码：

from twisted.internet import protocol
from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor

class Chat(LineReceiver):

    def __init__(self, users):
        self.users = users
        self.name = None
        self.state = "GETNAME"

    def connectionMade(self):
        self.sendLine(b'Whats your name')

    def connectionLost(self, reason):
        if self.name in self.users:
            del self.users[self.name]

    def lineReceived(self, line):
        if self.state == "GETNAME":
            self.handle_GETNAME(line)
        else:
            self.handle_CHAT(line)

    def handle_GETNAME(self, name):
        if name in self.users:
            self.sendLine(b"Name taken, please choose another.")
            return
        self.sendLine(b"Welcome, %s!" % (name,))
        self.name = name
        self.users[name] = self
        self.state = "CHAT"

    def handle_CHAT(self, message):
        message = "<%s> %s" % (self.name, message)
        pr_type(message)
        print(self.users)
        for name, protocol in self.users.items():
            if protocol != self:
                protocol.sendLine(message.encode())

class ChatFactory(Factory):

    def __init__(self):
        self.users = {} # maps user names to Chat instances

    def buildProtocol(self, addr):
        return Chat(self.users)

def main():
    # 监听端口
    reactor.listenTCP(8123, ChatFactory())
    reactor.run()

if __name__ == '__main__':
    main()

1.2. reactor代码解析

已有文档追踪到reactor.run()，所以前面部分不在赘述。

reactor已经找到了，下面看一下它在运行后做什么：

res = reactor.run
pr_type(res)

结果：

在github中查找得到：twisted/internet/base.py

class _SignalReactorMixin(object):

def startRunning(self, installSignalHandlers=True):

"""

PosixReactorBase的父类_SignalReactorMixin和ReactorBase都有该函数，但是_SignalReactorMixin在前，安装mro顺序的话，会先调用_SignalReactorMixin中的。

"""

self._installSignalHandlers = installSignalHandlers

ReactorBase.startRunning(self)

def run(self, installSignalHandlers=True):

self.startRunning(installSignalHandlers=installSignalHandlers)

self.mainLoop()

def mainLoop(self):

while self._started:

try:

while self._started:

# Advance simulation time in delayed event

# processors.

self.runUntilCurrent()

t2 = self.timeout()

t = self.running and t2

self.doIteration(t) #实现事务监听循环

except:

log.msg("Unexpected error in main loop.")

log.err()

else:

log.msg('Main loop terminated.')

这里就是主循环了。

主循环做了几件事：

监听：self.doIteration(t)

监听实现：self.doIteration(t)

#twisted/src/twisted/internet/selectreactor.py

@implementer(IReactorFDSet)
class SelectReactor(posixbase.PosixReactorBase, _extraBase):
    """
    A select() based reactor - runs on all POSIX platforms and on Win32.
    @ivar _reads: A set containing L{FileDescriptor} instances which will be
        checked for read events.
    @ivar _writes: A set containing L{FileDescriptor} instances which will be
        checked for writability.
    """

    def __init__(self):
      """
        Initialize file descriptor tracking dictionaries and the base class.
        """
        self._reads = set()
        self._writes = set()
        posixbase.PosixReactorBase.__init__(self)

    def doSelect(self, timeout):
        """
        Run one iteration of the I/O monitor loop.
        This will run all selectables who had input or output readiness
        waiting for them.
        """
        try:
            r, w, ignored = _select(self._reads,
                                    self._writes,
                                    [], timeout)
        except ValueError:
            # Possibly a file descriptor has gone negative?
            self._preenDescriptors()
            return
        except TypeError:
            # Something *totally* invalid (object w/o fileno, non-integral
            # result) was passed
            log.err()
            self._preenDescriptors()
            return
        except (select.error, socket.error, IOError) as se:
            # select(2) encountered an error, perhaps while calling the fileno()
            # method of a socket. (Python 2.6 socket.error is an IOError
            # subclass, but on Python 2.5 and earlier it is not.)
            if se.args[0] in (0, 2):
                # windows does this if it got an empty list
                if (not self._reads) and (not self._writes):
                    return
                else:
                    raise
            elif se.args[0] == EINTR:
                return
            elif se.args[0] == EBADF:
                self._preenDescriptors()
                return
            else:
                # OK, I really don't know what's going on. Blow up.
                raise

        _drdw = self._doReadOrWrite
        _logrun = log.callWithLogger
        for selectables, method, fdset in ((r, "doRead", self._reads),
                                           (w,"doWrite", self._writes)):
            for selectable in selectables:
                # if this was disconnected in another thread, kill it.
                # ^^^^ --- what the !@#*? serious! -exarkun
                if selectable not in fdset:
                    continue
                # This for pausing input when we're not ready for more.
                _logrun(selectable, _drdw, selectable, method)

    doIteration = doSelect

def _doReadOrWrite(self, selectable, method):
    try:
        why = getattr(selectable, method)()
    except:
        why = sys.exc_info()[1]
        log.err()
    if why:
        self._disconnectSelectable(selectable, why, method == "doRead")

说明：

实验平台是win所以这里调用的是selectReactor

这里有一个封装，在win平台下执行的是doIteration = doSelect

在Unix平台下就是doIteration = doPoll

当然也可以手动选择，只要平台支持。

另外_select实质也是一个封装，只要是selector循环，最后底层的都是select.select()，具体可见twisted/internet/selectreactor.py

r, w, e = select.select(r, w, w, timeout)

return r, w + e, []

在这里，实质就是分别对返回的readable,writable队列执行read和write。

那么假如客户端有连接请求了，就会调用的doRead方法；

在这里，selectable实质上是Port类的实例

@implementer(interfaces.IListeningPort)
class Port(base.BasePort, _SocketCloser):
    """
    A TCP server port, listening for connections.
    """

    def doRead(self):
        """
        Called when my socket is ready for reading.
        This accepts a connection and calls self.protocol() to handle the
        wire-level protocol.
        """
        try:
            if platformType == "posix":
                numAccepts = self.numberAccepts
            else:
                # win32 event loop breaks if we do more than one accept()
                # in an iteration of the event loop.
                numAccepts = 1

            with _BuffersLogs(self._logger.namespace,
                              self._logger.observer) as bufferingLogger:
                accepted = 0
                clients = _accept(bufferingLogger,
                                  range(numAccepts),
                                  self.socket,
                                  _reservedFD)

                for accepted, (skt, addr) in enumerate(clients, 1):
                    fdesc._setCloseOnExec(skt.fileno())

                    if len(addr) == 4:
                        # IPv6, make sure we get the scopeID if it
                        # exists
                        host = socket.getnameinfo(
                            addr,
                            socket.NI_NUMERICHOST | socket.NI_NUMERICSERV)
                        addr = tuple([host[0]] + list(addr[1:]))

                    protocol = self.factory.buildProtocol(
                        self._buildAddr(addr))
                    if protocol is None:
                        skt.close()
                        continue
                    s = self.sessionno
                    self.sessionno = s + 1
                    transport = self.transport(
                        skt, protocol, addr, self, s, self.reactor)
                    protocol.makeConnection(transport)

            # Scale our synchronous accept loop according to traffic
            # Reaching our limit on consecutive accept calls indicates
            # there might be still more clients to serve the next time
            # the reactor calls us. Prepare to accept some more.
            if accepted == self.numberAccepts:
                self.numberAccepts += 20
            # Otherwise, don't attempt to accept any more clients than
            # we just accepted or any less than 1.
            else:
                self.numberAccepts = max(1, accepted)
        except BaseException:
            # Note that in TLS mode, this will possibly catch SSL.Errors
            # raised by self.socket.accept()
            #
            # There is no "except SSL.Error:" above because SSL may be
            # None if there is no SSL support. In any case, all the
            # "except SSL.Error:" suite would probably do is log.deferr()
            # and return, so handling it here works just as well.
            log.deferr()

解析：

clients = _accept(bufferingLogger,range(numAccepts),self.socket,_reservedFD)

返回client,addr。

def _accept(logger, accepts, listener, reservedFD):
    """
    Return a generator that yields client sockets from the provided
    listening socket until there are none left or an unrecoverable
    error occurs.
    """

client, address = listener.accept()

yield client, address

然后

transport = self.transport(skt, protocol, addr, self, s, self.reactor)

protocol.makeConnection(transport)

transport实际是调用下述方法

def _makeTransport(self):
    """
    Create a L{Client} bound to this L{Connector}.
    @return: a new L{Client}
    @rtype: L{Client}
    """
    return Client(self.host, self.port, self.bindAddress, self, self.reactor)

总而言之，transport指向CLIENT，它是一个Port实例，包含host,port,bindaddress等套接字信息。

另外，reactor.listenTCP(8123, ChatFactory())

会调用它的def startListening(self):

在其中会调用self.startReading()，将port实例添加到reactor的read序列中，具体见下面的章节

class _BaseTCPClient(object):
    """
    Code shared with other (non-POSIX) reactors for management of outgoing TCP
    connections (both TCPv4 and TCPv6).
    @note: In order to be functional, this class must be mixed into the same
        hierarchy as L{_BaseBaseClient}. It would subclass L{_BaseBaseClient}
        directly, but the class hierarchy here is divided in strange ways out
        of the need to share code along multiple axes; specifically, with the
        IOCP reactor and also with UNIX clients in other reactors.
    @ivar _addressType: The Twisted _IPAddress implementation for this client
    @type _addressType: L{IPv4Address} or L{IPv6Address}
    @ivar connector: The L{Connector} which is driving this L{_BaseTCPClient}'s
        connection attempt.
    @ivar addr: The address that this socket will be connecting to.
    @type addr: If IPv4, a 2-C{tuple} of C{(str host, int port)}. If IPv6, a
        4-C{tuple} of (C{str host, int port, int ignored, int scope}).
    @ivar createInternetSocket: Subclasses must implement this as a method to
        create a python socket object of the appropriate address family and
        socket type.
    @type createInternetSocket: 0-argument callable returning
        C{socket._socketobject}.
    """

    _addressType = address.IPv4Address

    def __init__(self, host, port, bindAddress, connector, reactor=None):
        # BaseClient.__init__ is invoked later
        self.connector = connector
        self.addr = (host, port)

        whenDone = self.resolveAddress
        err = None
        skt = None

        if abstract.isIPAddress(host):
            self._requiresResolution = False
        elif abstract.isIPv6Address(host):
            self._requiresResolution = False
            self.addr = _resolveIPv6(host, port)
            self.addressFamily = socket.AF_INET6
            self._addressType = address.IPv6Address
        else:
            self._requiresResolution = True
        try:
            skt = self.createInternetSocket()
        except socket.error as se:
            err = error.ConnectBindError(se.args[0], se.args[1])
            whenDone = None
        if whenDone and bindAddress is not None:
            try:
                if abstract.isIPv6Address(bindAddress[0]):
                    bindinfo = _resolveIPv6(*bindAddress)
                else:
                    bindinfo = bindAddress
                skt.bind(bindinfo)
            except socket.error as se:
                err = error.ConnectBindError(se.args[0], se.args[1])
                whenDone = None
        self._finishInit(whenDone, skt, err, reactor)

再看一下protocol

protocol.makeConnection(transport)

protocol = self.factory.buildProtocol(self._buildAddr(addr))

这里使用的protocol是LineReceiver，看一下它的makeConnection

# /twisted/internet/protocol.py

@_oldStyle
class BaseProtocol:
    """
    This is the abstract superclass of all protocols.
    Some methods have helpful default implementations here so that they can
    easily be shared, but otherwise the direct subclasses of this class are more
    interesting, L{Protocol} and L{ProcessProtocol}.
    """
    connected = 0
    transport = None

    def makeConnection(self, transport):
        """
        Make a connection to a transport and a server.
        This sets the 'transport' attribute of this Protocol, and calls the
        connectionMade() callback.
        """
        self.connected = 1
        self.transport = transport
        self.connectionMade()

调用self.connectionMade()

这是自己写的代码，发送一条信息。

def connectionMade(self):
self.sendLine(b'Whats your name')

# /twisted/protocols/basic.py

class LineReceiver(protocol.Protocol, _PauseableMixin):
    """
    A protocol that receives lines and/or raw data, depending on mode.
    """
    def sendLine(self, line):
        """
        Sends a line to the other end of the connection.
        @param line: The line to send, not including the delimiter.
        @type line: C{bytes}
        """
        return self.transport.write(line + self.delimiter)

1.3. listenTCP()

回到示例中。

def main():
    # 监听端口
    reactor.listenTCP(8123, ChatFactory())
    reactor.run()

reactor.listenTCP()注册了一个监听事件，它是父类PosixReactorBase中方法。

# twisted/internet/posixbase.py
@implementer(IReactorTCP, IReactorUDP, IReactorMulticast)
class PosixReactorBase(_SignalReactorMixin, _DisconnectSelectableMixin,
                       ReactorBase):
    """
    A basis for reactors that use file descriptors.
    @ivar _childWaker: L{None} or a reference to the L{_SIGCHLDWaker}
        which is used to properly notice child process termination.
    """
    # IReactorTCP

    def listenTCP(self, port, factory, backlog=50, interface=''):
        p = tcp.Port(port, factory, backlog, interface, self)
        p.startListening()
        return p

# twisted/internet/tcp.py
@implementer(interfaces.IListeningPort)
class Port(base.BasePort, _SocketCloser):
    """
    A TCP server port, listening for connections.
    """
    def startListening(self):
        """Create and bind my socket, and begin listening on it.
        创建套接字，开始监听。
        This is called on unserialization, and must be called after creating a
        server to begin listening on the specified port.
        """
        _reservedFD.reserve()
        if self._preexistingSocket is None:
            # Create a new socket and make it listen
            try:
                # 声明一个非阻塞socket
               skt = self.createInternetSocket()
                if self.addressFamily == socket.AF_INET6:
                    addr = _resolveIPv6(self.interface, self.port)
                else:
                    addr = (self.interface, self.port)
                # 绑定
                skt.bind(addr)
            except socket.error as le:
                raise CannotListenError(self.interface, self.port, le)
            # 监听
            skt.listen(self.backlog)
        else:
            # Re-use the externally specified socket
            skt = self._preexistingSocket
            self._preexistingSocket = None
            # Avoid shutting it down at the end.
            self._shouldShutdown = False

        # Make sure that if we listened on port 0, we update that to
        # reflect what the OS actually assigned us.
        self._realPortNumber = skt.getsockname()[1]

        log.msg("%s starting on %s" % (
                self._getLogPrefix(self.factory), self._realPortNumber))

        # The order of the next 5 lines is kind of bizarre. If no one
        # can explain it, perhaps we should re-arrange them.
        self.factory.doStart()
        self.connected = True
        self.socket = skt
        self.fileno = self.socket.fileno
        self.numberAccepts = 100

        self.startReading()

    def createInternetSocket(self):
        """(internal) Create a non-blocking socket using
        self.addressFamily, self.socketType.
        """
        s = socket.socket(self.addressFamily, self.socketType)
        s.setblocking(0)
        fdesc._setCloseOnExec(s.fileno())
        return s

整个逻辑很简单，和正常的server端一样，创建套接字、绑定、监听。不同的是将套接字的描述符添加到了reactor的读集合。那么假如有了client连接过来的话，reactor会监控到，然后触发事件处理程序。

看一下self.startReading()

# /twisted/internet/abstract.py
@implementer(
    interfaces.IPushProducer, interfaces.IReadWriteDescriptor,
    interfaces.IConsumer, interfaces.ITransport,
    interfaces.IHalfCloseableDescriptor)
class FileDescriptor(_ConsumerMixin, _LogOwner):
    """
    An object which can be operated on by select().
    This is an abstract superclass of all objects which may be notified when
    they are readable or writable; e.g. they have a file-descriptor that is
    valid to be passed to select(2).
    """
    def startReading(self):
        """Start waiting for read availability.
        """
        self.reactor.addReader(self)

def addReader(self, reader):
    """
    Add a FileDescriptor for notification of data available to read.
    """
    self._reads.add(reader)

在此将transport加入_reads序列

然后在每次select时会对相应SOCKET及message进行处理，就是调用transport的doRead()方法，该方法在tcp.py/Connection类下。

# twisted/internet/tcp.py
@implementer(interfaces.ITCPTransport, interfaces.ISystemHandle)
class Connection(_TLSConnectionMixin, abstract.FileDescriptor, _SocketCloser,
                 _AbortingMixin):

    def doRead(self):
        try:
            # 接收数据
            data = self.socket.recv(self.bufferSize)
        except socket.error as se:
            if se.args[0] == EWOULDBLOCK:
                return
            else:
                return main.CONNECTION_LOST

        return self._dataReceived(data)

    def _dataReceived(self, data):
        if not data:
            return main.CONNECTION_DONE
        # 调用我们自定义protocol的dataReceived方法处理数据
        rval = self.protocol.dataReceived(data)
        if rval is not None:
            offender = self.protocol.dataReceived
            warningFormat = (
                'Returning a value other than None from %(fqpn)s is '
                'deprecated since %(version)s.')
            warningString = deprecate.getDeprecationWarningString(
                offender, versions.Version('Twisted', 11, 0, 0),
                format=warningFormat)
            deprecate.warnAboutFunction(offender, warningString)
        return rval

在其中调用自定义protocol的

rval = self.protocol.dataReceived(data)

至此，整个循环算是跑起来了。

2. 其它

`2.1.` implementer(IReactorFDSet)

这个装饰器是一个接口实现，具体可见https://www.cnblogs.com/wodeboke-y/p/11216029.html

# twisted/internet/selectreactor.py

@implementer(IReactorFDSet)

class SelectReactor(posixbase.PosixReactorBase, _extraBase)

implementer表示SelectReactor实现了IReactorFDSet接口的方法，这里用到了zope.interface，它是python中的接口实现。

IReactorFDSet接口主要对描述符的获取、添加、删除等操作的方法。这些方法看名字就能知道意思，略过。

# twisted/internet/interfaces.py

class IReactorFDSet(Interface):

    def addReader(reader):

    def addWriter(writer):

    def removeReader(reader):

    def removeWriter(writer):

    def removeAll():

    def getReaders():

    def getWriters():

3. 总结

简单点描述的话，twisted创建两个socket，端口号分别为port,port+1，使用系统的select（或其它IO复用函数）作为轮询主体;

然后，对上层处理机进行封装，形成protocol及factory，reactor负责循环。

PS：阅读没有架构图的代码太坑了。。。。。

转载于:https://www.cnblogs.com/wodeboke-y/p/11247074.html

akuibpt23191

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
twisted reactor 实现源码解析

twisted reactor 实现源码解析1. reactor源码解析1.1. 案例分析代码：from twisted.internet import protocolfrom twisted.internet.protocol import Factoryfrom twisted.protocols.basic import LineR...
复制链接

扫一扫