python 网络编程笔记

 学习路线：socket-->SocketServer-->Twisted(基于select函数或poll函数)

1.    Socket模块

部分函数：

1.connection, address = socket.accept()

服务器套接字通过socket的accept方法等待客户请求一个连接。

accept方法返回一个含有两个元素的元组(connection,address)。第一个元素connection是新的socket对象，服务器必须通过它与客户通信；第二个元素address是客户的Internet地址。

2.socket.listen(backlog) 设置一个listen***队列的大小***

backlog指定了最多连接数，至少为1，接到连接请求后，这些请求必须排队，如果队列已满，则拒绝请求。

3.socket.connect((host,port) )

使用socket的connect方法连接服务器。

4.socket.setblocking(False) 是否设置为阻塞

5.nthol(x)和ntohs(x)函数要求一个网络字节顺序的数值并把它转换为当前主机字节顺序的相同数值，而htonl(x)和htons(x)则相反：

socket.htons(20000) #转换为一个16位的值

socket.htonl(20000) #转换为一个32位的值

6.setsockopt(level,name,value)和getsockopt(level,name[,buflen])方法来设置和获取套接字的属性。

level参数指定了选项应用于哪一层。level的取值以SOL_开头（SOL_SOCKET,SOL_TCP等等）；name表明你涉及的是哪个选项；

value是该选项传入的数值。

对getsockopt，不指定buflen参数意味你要求一个数字值，并返回这个值。如果你提供了buflen，getsockopt返回代表一个缓存的字符串，它的最大长度是buflen的字节数。下面的例子设置了一个socket的用于发送的缓存尺寸为64KB：

s.setsockopt(SOL_SOCKET,SO_SNDBUF,65535)

又如socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR , 1)实现端口重用

[附：C语言实现端口重用：

setsockopt()

设置调用closesocket()后,仍可继续重用该socket。调用closesocket()一般不会立即关闭socket，而经历TIME_WAIT的过程。

BOOL bReuseaddr = TRUE;

setsockopt( s, SOL_SOCKET, SO_REUSEADDR, (const char* )&bReuseaddr, sizeof( BOOL ) );

如果要已经处于连接状态的soket在调用closesocket()后强制关闭，不经历TIME_WAIT过程：

BOOL bDontLinger= FALSE;

setsockopt( s, SOL_SOCKET, SO_DONTLINGER, (const char* )&bDontLinger, sizeof( BOOL ) ); 

]

 socket server and client例子：

2.    SocketServer模块

SocketServer模块简单化了编写网络服务器的工作。它提供了四个基本的服务类：TCPServer(使用TCP协议)、UDPServer(使用数据报)、UnixStreamServer、UnixDatagramServer。UnixStreamServer和UnixDatagramServer用于类Unix平台。

这四个类处理请求都使用同步的方法，也就是说，在下一个请求处理开始之前当前的请求处理必须已完成。

用SocketServer创建一个服务器需要三步：

1、通过子类化BaseRequestHandler类和覆盖它的handle()方法来创建一个请求处理器类，用于处理进来的请求；

2、实例化服务类如TCPServer，并传递给它参数：服务器地址和请求处理器类；

3、调用服务实例对象的handle_request()或serve_forever()方法去处理请求。

下面使用SocketServer用同步的方法写一个最简单的服务器：

from SocketServer import TCPServer, StreamRequestHandler
#第一步。其中StreamRequestHandler类是BaseRequestHandler类的子类，它为流socket定义了
#rfile和wfile方法
class Handler(StreamRequestHandler):
     def handle(self):#处理到来的请求
         addr = self.request.getpeername()
         print 'Got connection from', addr
         self.wfile.write('Thank you for connecting')#send msg to client

#第二步。其中''代表运行服务器的主机
server = TCPServer(('', 12345), Handler)
#第三步。serve_forever()导致进入循环状态
#[server_forever 只是反复调用handle_request而已。]
server.serve_forever()

3.    非阻塞或异步网络编程

序：

服务器端使用阻塞或同步的方法一次只能连接一个客户端，处理完成后才能连接下一个客户端。

例如：Socket方式下Server端程序的执行过程：

一旦服务端socket调用了listen方法，就进入了临听状态，然后通常使用一个无限的循环：1、开始接受客房端的连接，这通过调用accept方法来实现。调用了这个方法后将处于阻塞状态（等待客户端发起连接）直到一个客户端连接，连接后，accept返回形如(client,address)的一个元组，其中client是一个用于与客户端通信的socket，address是客户端的形如xxx.xxx.xxx.xxx:xxx的地址；2、然后服务端处理客户端的请求；3、处理完成之后又调用1。

实现非阻塞主要有三种方法：forking、threading、异步I/O。

Forking和threading的方法非常简单，通过使用SocketServer服务类的mix-in类就可以实现。forking只适用于类Unix平台；threading需要注意内存共享的问题。

要实现异步I/O，我们可以通过使用框架asyncore/asynchat或Twisted，它们都是基于select函数或poll函数（poll只适于类Unix系统）的。select和poll函数都来自select模块。

Twisted是一个事件驱动型的网络引擎。事件驱动编程是一种编程范式，这里程序的执行流由外部事件来决定。它的特点是包含一个事件循环，当外部事件发生时使用回调机制来触发相应的处理。另外两种常见的编程范式是（单线程）同步以及多线程编程。

 

使用 select ：

在python中，select函数是一个对底层操作系统的直接访问的接口。它用来监控sockets、files和pipes，等待IO完成（Waiting for I/O completion）。当有可读、可写或是异常事件产生时，select可以很容易的监控到。

select.select（rlist, wlist, xlist[, timeout]） 传递三个参数，一个为输入而观察的文件对象列表，一个为输出而观察的文件对象列表和一个观察错误异常的文件列表。第四个是一个可选参数，表示超时秒数(如果超时值没有给出的话，select将处于阻塞状态（也就是等待）直到有文件描述符准备动作。如果超时值给出了，那么select只阻塞给定的时间。如果超时值是0的话，那么将不阻塞。)。其返回3个tuple，每个tuple都是一个准备好的对象列表，它和前边的参数是一样的顺序。

[附：C语言select函数原型

int select(int maxfdp,fd_set*readfds,fd_set *writefds,fd_set *errorfds,struct timeval *timeout);

]

 例子：

1.fork server 略

2.threading server

#threading服务器

from SocketServer import TCPServer, ThreadingMixIn, StreamRequestHandler

class Server(ThreadingMixIn, TCPServer): pass

class Handler(StreamRequestHandler):
    def handle(self):
        addr = self.request.getpeername()
        print 'Got connection from', addr
        self.wfile.write('Thank you for connecting')

server = Server(('', 12345), Handler)
server.serve_forever()

3.select server 

'''
The echo server example from the socket section can be extanded to watche for more than
one connection at a time by using select() .The new version starts out by creating a nonblocking
TCP/IP socket and configuring it to listen on an address
'''
import select
import socket
import Queue
 
#create a socket
server = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.setblocking(False)
#set option reused  #0119 get reused port
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR  , 1)

host=socket.gethostname()
port=10001
server_address= (host,port) #socket
server.bind(server_address)
 
server.listen(10)
 
#sockets from which we except to read
inputs = [server]#0119 server is the listen socket what we created
 
#sockets from which we expect to write
outputs = []
 
#Outgoing message queues (socket:Queue)
message_queues = {}
 
#A optional parameter for select is TIMEOUT
timeout = 20
 
while inputs:
    print "waiting for next event"
    readable , writable , exceptional = select.select(inputs, outputs, inputs, timeout)
 
    # When timeout reached , select return three empty lists
    if not (readable or writable or exceptional) :
        print "Time out ! "
        break;    
    for s in readable :
        if s is server: #0119 s is the ---listenning socket---
            # A "readable" socket is ready to accept a connection
            connection, client_address = s.accept()
            print "    connection from ", client_address
            connection.setblocking(0)
            inputs.append(connection)
            message_queues[connection] = Queue.Queue()
        else:#0119 s is the ---accepting sockets(the element in inputs list except for 'server') 
            data = s.recv(1024)
            if data :
                print " received " , data , "from ",s.getpeername()
                message_queues[s].put(data) #0119  put an item(received data) in the queue(message_queues[s])
                # Add output channel for response    
                if s not in outputs:
                    outputs.append(s)
            else:
                #Interpret empty result as closed connection(***)
                print "  closing", client_address
                if s in outputs :
                    outputs.remove(s)
                inputs.remove(s)  #0119 inputs.remove(s)  001
                s.close()
                #remove message queue 
                del message_queues[s]
    for s in writable:
        try:
            next_msg = message_queues[s].get_nowait()#0119 Queue.get_nowait() is equivalent to Queue.get(False)
        except Queue.Empty:  #(***)
            print " " , s.getpeername() , 'queue empty'
            outputs.remove(s)
        else:
            print " sending " , next_msg , " to ", s.getpeername()
            s.send(next_msg)
     
    for s in exceptional:
        print " exception condition on ", s.getpeername()
        #stop listening for input on the connection
        inputs.remove(s)   #0119 inputs.remove(s)  002
        if s in outputs:
            outputs.remove(s)
        s.close()
        #Remove message queue
        del message_queues[s]

测试的client代码：

'''
The example client program uses some sockets to demonstrate how the server
with select() manages multiple connections at the same time . The client
starts by connecting each TCP/IP socket to the server
'''
 
import socket
 
messages = ["This is the message" ,
            "It will be sent" ,
            "in parts "]
 
print "Connect to the server"

host=socket.gethostname()
port=10001
server_address= (host,port) #socket
 
#Create a TCP/IP sock
 
socks = []
 
for i in range(4):
    socks.append(socket.socket(socket.AF_INET,socket.SOCK_STREAM))
 
for s in socks:
    while 1:#0119 
        err=s.connect(server_address)
        print err
        if err==-1:
            continue
        else:
            break
 
counter = 0
for message in messages :
    #Sending message from different sockets
    for s in socks:
        counter+=1
        print "  %s sending %s" % (s.getpeername(),message+" version "+str(counter))
        s.send(message+" version "+str(counter))
    #Read responses on both sockets
    for s in socks:
        data = s.recv(1024)
        print " %s received %s" % (s.getpeername(),data)
        if not data:
            print "closing socket ",s.getpeername()
            s.close()

测试运行结果：

server端：

>>> ================================ RESTART ================================
>>> 
waiting for next event
    connection from  ('172.22.144.167', 29326)
waiting for next event
    connection from  ('172.22.144.167', 29327)
waiting for next event
    connection from  ('172.22.144.167', 29328)
waiting for next event
    connection from  ('172.22.144.167', 29329)
waiting for next event
 received  This is the message version 1 from  ('172.22.144.167', 29326)
waiting for next event
 received  This is the message version 2 from  ('172.22.144.167', 29327)
 received  This is the message version 3 from  ('172.22.144.167', 29328)
 received  This is the message version 4 from  ('172.22.144.167', 29329)
 sending  This is the message version 1  to  ('172.22.144.167', 29326)
waiting for next event
  ('172.22.144.167', 29326) queue empty
 sending  This is the message version 2  to  ('172.22.144.167', 29327)
 sending  This is the message version 3  to  ('172.22.144.167', 29328)
 sending  This is the message version 4  to  ('172.22.144.167', 29329)
waiting for next event
  ('172.22.144.167', 29327) queue empty
  ('172.22.144.167', 29328) queue empty
  ('172.22.144.167', 29329) queue empty
waiting for next event
 received  It will be sent version 5 from  ('172.22.144.167', 29326)
 received  It will be sent version 6 from  ('172.22.144.167', 29327)
 received  It will be sent version 7 from  ('172.22.144.167', 29328)
 received  It will be sent version 8 from  ('172.22.144.167', 29329)
waiting for next event
 sending  It will be sent version 5  to  ('172.22.144.167', 29326)
 sending  It will be sent version 6  to  ('172.22.144.167', 29327)
 sending  It will be sent version 7  to  ('172.22.144.167', 29328)
 sending  It will be sent version 8  to  ('172.22.144.167', 29329)
waiting for next event
  ('172.22.144.167', 29326) queue empty
  ('172.22.144.167', 29327) queue empty
  ('172.22.144.167', 29328) queue empty
  ('172.22.144.167', 29329) queue empty
waiting for next event
 received  in parts  version 9 from  ('172.22.144.167', 29326)
 received  in parts  version 10 from  ('172.22.144.167', 29327)
 received  in parts  version 11 from  ('172.22.144.167', 29328)
 received  in parts  version 12 from  ('172.22.144.167', 29329)
waiting for next event
 sending  in parts  version 9  to  ('172.22.144.167', 29326)
 sending  in parts  version 10  to  ('172.22.144.167', 29327)
 sending  in parts  version 11  to  ('172.22.144.167', 29328)
 sending  in parts  version 12  to  ('172.22.144.167', 29329)
waiting for next event
  ('172.22.144.167', 29326) queue empty
  ('172.22.144.167', 29327) queue empty
  ('172.22.144.167', 29328) queue empty
  ('172.22.144.167', 29329) queue empty
waiting for next event
Time out ! 
>>> 

client 端：

>>> ================================ RESTART ================================
>>> 
Connect to the server
None
None
None
None
  ('172.22.144.167', 10001) sending This is the message version 1
  ('172.22.144.167', 10001) sending This is the message version 2
  ('172.22.144.167', 10001) sending This is the message version 3
  ('172.22.144.167', 10001) sending This is the message version 4
 ('172.22.144.167', 10001) received This is the message version 1
 ('172.22.144.167', 10001) received This is the message version 2
 ('172.22.144.167', 10001) received This is the message version 3
 ('172.22.144.167', 10001) received This is the message version 4
  ('172.22.144.167', 10001) sending It will be sent version 5
  ('172.22.144.167', 10001) sending It will be sent version 6
  ('172.22.144.167', 10001) sending It will be sent version 7
  ('172.22.144.167', 10001) sending It will be sent version 8
 ('172.22.144.167', 10001) received It will be sent version 5
 ('172.22.144.167', 10001) received It will be sent version 6
 ('172.22.144.167', 10001) received It will be sent version 7
 ('172.22.144.167', 10001) received It will be sent version 8
  ('172.22.144.167', 10001) sending in parts  version 9
  ('172.22.144.167', 10001) sending in parts  version 10
  ('172.22.144.167', 10001) sending in parts  version 11
  ('172.22.144.167', 10001) sending in parts  version 12
 ('172.22.144.167', 10001) received in parts  version 9
 ('172.22.144.167', 10001) received in parts  version 10
 ('172.22.144.167', 10001) received in parts  version 11
 ('172.22.144.167', 10001) received in parts  version 12
>>>