Python 网络编程攻略 第二章

2.1简介

先来了解几个基本库

SocketServer

先看一下文档：

Python文档

The SocketServer module simplifies the task of writing network servers.

There are four basic server classes: TCPServer uses the Internet TCP protocol, which provides for continuous streams of data between the client and server. UDPServer uses datagrams, which are discrete packets of information that may arrive out of order or be lost while in transit. The more infrequently used UnixStreamServer and UnixDatagramServer classes are similar, but use Unix domain sockets; they’re not available on non-Unix platforms. 

These four classes process requests synchronously; each request must be completed before the next request can be started. This isn’t suitable if each request takes a long time to complete, because it requires a lot of computation, or because it returns a lot of data which the client is slow to process. The solution is to create a separate process or thread to handle each request; theForkingMixIn and ThreadingMixIn mix-in classes can be used to support asynchronous behaviour.

Creating a server requires several steps. First, you must create a request handler class by subclassing the BaseRequestHandler class and overriding its handle() method; this method will process incoming requests. Second, you must instantiate one of the server classes, passing it the server’s address and the request handler class. Finally, call the handle_request() or serve_forever() method of the server object to process one or many requests.

When inheriting from ThreadingMixIn for threaded connection behavior, you should explicitly declare how you want your threads to behave on an abrupt shutdown. The ThreadingMixIn class defines an attribute daemon_threads, which indicates whether or not the server should wait for thread termination. You should set the flag explicitly if you would like threads to behave autonomously; the default is False, meaning that Python will not exit until all threads created by ThreadingMixIn have exited.

就是说SocketServer主要有四个类，TCPServer和UDPServer是主要用的，但是都是只能够同步操作；需要异步操作呢，就是为每一个request建立一个线程去操作，就要用到ForkingMixIn或者ThreadMixIn这两个mix-in类。

+------------+
| BaseServer |
+------------+
      |
      v
+-----------+        +------------------+
| TCPServer |------->| UnixStreamServer |
+-----------+        +------------------+
      |
      v
+-----------+        +--------------------+
| UDPServer |------->| UnixDatagramServer |
+-----------+        +--------------------+

建立一个服务器有这么几步操作：

1、从BaseRequestHandler继承一个类，用来request handle的，重载它的handle()方法，这个方法用来解决服务器需要解决的问题。

2、实例化server类，就是前面提到的那四个server类。根据提示，应该是同时继承了一个异步通信的类/mix-in类，和一个server类，比如这样：

class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass

mix-in类要放在server类的前面，因为前者重载了后者的一些属性。

实例化的时候，给它传递服务器地址和request handler，根据前面一章节的学习，应该都知道服务器地址就是这样子的(ip,port)。

3、最后调用handler_request()或者server_forever()方法

最后还提了一下，如果继承的是ThreadingMixIn类，需要特地指出在突然关闭的情况下，定义一个daemon_threads，来确定服务器是否要等待线程结束。

前面一章基本都只是套接字对应套接字进行的操作，这一章节由于前面引入了服务器，服务器因为继承了mixin类和server类，在初始化当中已经传递了address==》(ip,port)，应该和我们自己写的client类的构造器当中一样，都建立好了套接字，也就不用管了。在另外的handler类当中，self.request应该就是我们在服务器当中需要的socket，大家可以通过下面的代码感受一下。

2.2 在套接字服务器程序中使用ForkingMixIn

<span style="font-size:18px;"><span style="font-size:18px;"># 2_1_forking_mixin_socket.server.py

import os
import socket
import threading
import SocketServer

SERVER_HOST = 'localhost'
SERVER_PORT = 0 # tell the kernel to pick up a port dynamically
BUF_SIZE = 1024
ECHO_MSG = 'Hello echo server!'

class ForkingClient():
    """ A client to test forking server """
    def __init__(self, ip, port):
        # Create a socket
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        # Connect to the server
        self.sock.connect((ip,port))

    def run(self):
        """ Client playing with the server"""
        # Send the data to server
        current_process_id = os.getpid()
        print("PID %s Sending echo message to the server: %s" %(current_process_id, ECHO_MSG))
        sent_data_length = self.sock.send(ECHO_MSG)
        # print("Sent: %d characters, so far ..." %sent_data_length)
        print("Client ready to receive 1")

        # Display server response
        response = self.sock.recv(BUF_SIZE)
        print("Client ready to receive 2")
        print("PID %s received: %s" %(current_process_id, response))

    def shutdown(self):
        """ Cleanup the client socket"""
        self.sock.close()



class ForkingServerRequestHandler(SocketServer.BaseRequestHandler):
    def handle(self):
        # Send the echo back to the client
        data = self.request.recv(BUF_SIZE)
        current_process_id = os.getpid()
        response = '%s : %s' %(current_process_id, data)
        print("OK 1")
        print("Server sending response [current_process_id: data] = [%s]" %response)
        print("OK 2")
        self.request.send("received " + response)
        print("OK 3")


    
class ForkingServer(SocketServer.ForkingMixIn, SocketServer.TCPServer):
    """ Nothing to add here, inherited everything necessary from parents"""
    pass

def main():
    # Launch the server
    server = ForkingServer( (SERVER_HOST, SERVER_PORT), ForkingServerRequestHandler)
    ip, port = server.server_address # Retrieve the port number
    server_thread = threading.Thread(target=server.serve_forever)
    server_thread.setDaemon(True) # Don't hang on exit
    server_thread.start()
    print("Server loop running PID: %s" %os.getpid())

    # Launch the client(s)
    client1 = ForkingClient(ip, port)
    client1.run()

    client2 = ForkingClient(ip, port)
    client2.run()

    # Clean them up
    server.shutdown()
    client1.shutdown()
    client2.shutdown()
    server.socket.close()

if __name__ == '__main__':
    main()</span></span>

python 2_1_forking_mixin_socket_server.py

即可看到结果

这个和2.3一起讲一下，关于异步通信的方式，有几种常用的

ForkingMixIn 分叉方式。对只有一个收到的连接，主进程会生成相应的子进程进行处理，主进程依旧监听，比较耗费资源。

ThreadingMixIn 开线程方式。产生线程，而非进程，共享内存，效率高；但是大量使用线程，会造成线程之间的数据同步，导致服务程序时区效应。

专门的异步IO 在一定的时间短内察看已有的连接，并且进行处理。有asyncore和asynchat模块来实现异步IO处理。这个功能依赖于select()和poll()方法，这两个方法定义在select模块当中。

2.3在套接字服务器中使用ThreadingMixIn

<span style="font-size:18px;"><span style="font-size:18px;"># 2_2_threading_mixin_socket_server.py

import os
import socket
import threading
import SocketServer

SERVER_HOST = 'localhost'
SERVER_PORT = 0
BUF_SIZE = 1024

def client(ip, port, message):
    """ A client to test threading mixin server"""
    # Connect to the server
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM )
    sock.connect((ip, port))
    try:
        sock.sendall(message)
        response = sock.recv(BUF_SIZE)
        print("Client received: %s" %response)
    finally:
        sock.close()

class ThreadTCPRequestHandler(SocketServer.BaseRequestHandler):
    """ An example of threaded TCP request handler"""
    def handle(self):
        data = self.request.recv(1024)
        current_thread = threading.current_thread()
        response = "%s: %s" %(current_thread.name, data)
        self.request.sendall(response)

class ThreadTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
    """ Nothing to add here, inherited everything necessary from parents"""
    pass

if __name__ == '__main__':
    # Run server
    server = ThreadTCPServer((SERVER_HOST, SERVER_PORT), ThreadTCPRequestHandler)
    ip, port = server.server_address

    # Start a thread with the server --- one thread per request
    server_thread = threading.Thread(target=server.serve_forever)
    # Exit the server thread when the main thread exists
    server_thread.daemon = True
    server_thread.start()
    print(ip + "   " + str(port))
    print("Server loop running on thread: %s" %server_thread.name)

    # Run the clients
    client(ip, port, "Hello from Client 1")
    client(ip, port, "Hello from Client 2")
    client(ip, port, "Hello from Client 3")

    # Server cleanup
    server.shutdown()</span></span>

python 2_2_threading_mixin_socket_server.py

即可看到结果

基本思路就是：

1、client短发出message，server端通过handler接收。

2、然后current_thread.name: message，再返回给client端。

3、client接受，然后打印client received: current_thread.name : message

2.4 使用select.select编写一个聊天室服务器

<span style="font-size:18px;"># 2_3_chat_server_with_select

import select
import socket
import sys
import signal
import cPickle
import struct
import argparse

SERVER_HOST = 'localhost'
CHAT_SERVER_NAME = 'server'

# some utilities
def send(channel, *args):
    buffer = cPickle.dumps(args)
    value = socket.htonl(len(buffer))
    size = struct.pack("L",value)
    channel.send(size)
    channel.send(buffer)

def receive(channel):
    size = struct.calcsize("L")
    size = channel.recv(size)
    try:
        size = socket.ntohl(struct.unpack("L",size)[0])
    except struct.error as e:
        return ''
    buf = ""
    while len(buf) < size:
        buf = channel.recv(size- - len(buf) )
    return cPickle.loads(buf)[0]

class ChatServer(object):
    """ An example chat server using select """
    def __init__(self, port, backlog=5):
        self.clients = 0
        self.clientmap = {}
        self.outputs = [] # list output socktets
        self.server = socket.socket(socket.AF_INET , socket.SOCK_STREAM )
        self.server.setsockopt(socket.SOL_SOCKET , socket.SO_REUSEADDR , 1)
        self.server.bind((SERVER_HOST,port))
        print("Server listening to pot: %s ..." %port)
        self.server.listen(backlog)
        # Catch keyboard interrupts
        signal.signal(signal.SIGINT, self.sighandler)

    def sighandler(self, signum, frame):
        """ Clean up client outputs"""
        # Close the server
        print("Shutting down the server")
        # Close existing client sockets
        for output in self.outputs:
            output.close()
        self.server.close()

    def get_client_name(self, client):
        """ Return the name of the client """
        info = self.clientmap[client]
        host, name = info[0][0], info[1]
        return '@'.join((name, host))

    def run(self):
        inputs = [self.server, sys.stdin]
        self.outputs = []
        running = True
        while running:
            try:
                readable, writeable, exceptional = select.select(inputs, self.outputs, [])
            except select.error, e:
                break

            for sock in readable:
                if sock == self.server:
                    # handle the server socket
                    client, address = self.server.accept()
                    print("Chat server: got connection %d from %s"
                            %(client.fileno(),address))
                    # Read the login name
                    cname = receive(client).split('NAME: ')[1]

                    # Compute client name and send back
                    self.clients += 1
                    send(client, 'CLIENT: ' + str(address[0]))
                    inputs.append(client)
                    self.clientmap[client] = (address, cname)
                    # Send joining information to other clients
                    msg = "\n(Connected: New client (%d) from %s)" % (self.clients,
                                                                      self.get_client_name(client))
                    for output in self.outputs:
                        send(output, msg)
                    self.outputs.append(client)
                        
                elif sock == sys.stdin:
                    # handle standard input
                    junk = sys.stdin.readline()
                    running = False
                else:
                    # handle all other sockets
                    try:
                        data = receive(sock)
                        if data:
                            # Send as new client's message ...
                            msg = "\n#[" + self.get_client_name(sock) + ']>>' + data
                            # Send data to all except ourself
                            for output in self.outputs:
                                if output != sock:
                                     send(output, msg)
                        else:
                            print ("Chat server: %d hung up" %sock.fileno())
                            self.clients -= 1
                            sock.close()
                            inputs.remove(sock)
                            self.outputs.remove(sock)

                            # Sending client leaving info to others
                            msg = '\n(Now hung up: Client from %s)' %self.get_client_name(sock)
                            for output in self.outputs:
                                send(output, msg)
                    except socket.error as e:
                        # Remove
                        inputs.remove(sock)
                        self.outputs.remove(sock)
        self.server.close()

class ChatClient(object):
    """ A command line chat client using select """

    def __init__(self, name, port, host=SERVER_HOST):
        self.name = name
        self.connected = False
        self.host = host
        self.port = port
        # Initial prompt
        self.prompt='[' + '@'.join( (name, socket.gethostname().split('.')[0] )) + ']> '
        # Connect to server at port
        try:
            self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            self.sock.connect((host, self.port))
            print("Now connected to cha server@ port %d" %self.port)
            self.connected = True
            # Send my name
            send(self.sock, 'NAME: ' + self.name)
            data =receive(self.sock)
            # Contains client address, set it
            addr = data.split('CLIENT: ')[1]
            self.prompt = '[' + '@'.join((self.name, addr)) + ']>'
        except socket.error, e:
            print("Failed to connect to chat server @ port %d" %self.port)
            print(e)
            sys.exit(1)

    def run(self):
        """ Chat client main loop"""
        while self.connected:
            try:
                sys.stdout.write(self.prompt)
                sys.stdout.flush()
                # Wait for input from stdin and socket
                readable, writeable, exceptional = select.select([0, self.sock], [], [])
                for sock in readable:
                    if sock == 0:
                        data = sys.stdin.readline().strip()
                        if data:
                            send(self.sock, data)
                    elif sock == self.sock:
                        data = receive(self.sock)
                        print(data)
                        if not data:
                            print("Client shutting down.")
                            self.connected = False
                            break
                        else:
                            sys.stdout.write(data + "\n")
                            sys.stdout.flush()
            except KeyboardInterrupt:
                print " Client interrupted. "
                self.sock.close()
                break

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description = 'Socket Server Example with Select')
    parser.add_argument('--name', action="store", dest="name", required=True)
    parser.add_argument('--port', action="store", dest='port', type=int, required=True)
    given_args = parser.parse_args()
    port = given_args.port
    name = given_args.name
    if name == CHAT_SERVER_NAME:
        server = ChatServer(port)
        server.run()
    else:
        client = ChatClient(name=name, port=port)
        client.run()
</span>

运行：

python 2_3_chat_server_with_select.py --name=server --port=8800

python 2_3_chat_server_with_select.py --name=client1 --port=8800

python 2_3_chat_server_with_select.py --name=client2 --port=8800

select用于对指定的文件描述符进行监视，并在文件描述符集改变的适合做出响应。

文件描述符集合，可以接受的文件描述符有：

sys.stdin

open()和popen()方法得到的对象

socket.socket()返回的Socket对象

这三个的共同特征就是可以用fileno()来获取描述符

select.select(rlist, wlist, xlist [, timeout])

若timeout未设定，这个调用会一直阻塞到前三个集合中至少有一个文件描述符准备好为止。

poll()

select采用bitmap索引方式处理文件描述符，而poll()仅仅处理感兴趣的文件描述符

select.poll()返回一个Polling对象，有register(),unregister(),poll()方法。register()和unregister()分别注册和一处一个文件描述符或者fileno()得到的值，注册之后，用poll()获取一个{fileno:event}列表。其中event是表示指定发生的事件，是位掩码，用为操作&与系统模块已经提供的参数，来判断是否执行相对于的操作——event & select.POLLIN。模块参数如下：

select.POLLIN 含有可读的数据

select.POLLPRI 含有紧急可读的数据

select.POLLOUT 含有需要写处的数据

select.POLLERR 发生错误

select.HUP 连续断开

select.VAL 错误的请求

2.5 使用select.epoll夺路服用Web服务器

# 2_4_simple_web_server_with_epoll.py

import socket
import select
import argparse

SERVER_HOST = 'localhost'

EOL1 = b'\n\n'
EOL2 = b'\n\r\n'

SERVER_RESPONSE = b"""HTTP/1.1 200 OK\r\nDate: Min, 1 Apr 2013 01:01:01
GMT\r\nContent-Type: text/plain \r\nCpntent-Length: 25\r\n\r\n
Hello from Epoll Server"""

class EpollServer(object):
    """ A socket server using Epoll"""

    def __init__(self, host=SERVER_HOST, port=0):
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.sock.bind((host, port))
        self.sock.listen(1)
        self.sock.setblocking(0)
        self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
        print("Started Epoll Server")
        self.epoll = select.epoll()
        self.epoll.register(self.sock.fileno(), select.EPOLLIN)

    def run(self):
        """Execute epoll server operation"""
        try:
            connections = {}
            requests = {}
            responses = {}
            while True:
                events = self.epoll.poll(1)
                for fileno,event in events:
                    if fileno == self.sock.fileno():
                        connection, address = self.sock.accept()
                        connection.setblocking(0)
                        self.epoll.register(connection.fileno(), select.EPOLLIN)
                        connections[connection.fileno()] = connection
                        requests[connection.fileno()] = b''
                        responses[connection.fileno()] = SERVER_RESPONSE
                    elif event & select.EPOLLIN:
                        requests[fileno] += connections[fileno].recv(1024)
                        if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
                            self.epoll.modify(fileno, select.EPOLLOUT)
                            print('-'*40 + '\n' + requests[fileno].decode()[:-2])
                    elif event & select.EPOLLOUT:
                        bytewritten = connections[fileno].send(responses[fileno])
                        responses[fileno] = responses[fileno][bytewritten:]
                        if len(responses[fileno]) == 0:
                            self.epoll.modify(fileno, 0)
                            connections[fileno].shutdown(socket.SHUT_RDWR)
                    elif event & select.EPOLLHUP:
                        self.epoll.unregister(fileno)
                        connections[fileno].close()
                        del connections[fileno]
        finally:
            self.epoll.unregister(self.sock.fileno())
            self.epoll.close()
            self.sock.close()

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Socket Server Example with Epoll')
    parser.add_argument('--port', action="store", dest="port", type=int, required=True)
    given_args = parser.parse_args()
    port = given_args.port
    server = EpollServer(host=SERVER_HOST, port=port)
    server.run()

运行：

python 2_4_simple_web_server_with_epoll.py --port=8800

然后随意打开一个浏览器，输入地址： localhost:8800

self.epoll.poll() 返回的是{fileno:event}

connections存储的是{fileno:socket}

还有几个模块也很有用，比如dasyncore和diesel，不过这章就先到这里啦。