python socket connect 并发_Python套接字压力并发

我需要一个可以处理至少数万个并发套接字连接的

Python TCP服务器.我试图在多处理器和多线程模式下测试Python SocketServer包功能,但两者都远远没有达到预期的性能.

首先,我将描述客户端,因为这两种情况都很常见.

client.py

import socket

import sys

import threading

import time

SOCKET_AMOUNT = 10000

HOST, PORT = "localhost", 9999

data = " ".join(sys.argv[1:])

def client(ip, port, message):

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sock.connect((ip, port))

while 1:

sock.sendall(message)

time.sleep(1)

sock.close()

for i in range(SOCKET_AMOUNT):

msg = "test message"

client_thread = threading.Thread(target=client, args=(HOST, PORT, msg))

client_thread.start()

多处理器服务器：

foked_server.py

import os

import SocketServer

class ForkedTCPRequestHandler(SocketServer.BaseRequestHandler):

def handle(self):

cur_process = os.getpid()

print "launching a new socket handler, pid = {}".format(cur_process)

while 1:

self.request.recv(4096)

class ForkedTCPServer(SocketServer.ForkingMixIn, SocketServer.TCPServer):

pass

if __name__ == "__main__":

HOST, PORT = "localhost", 9999

server = ForkedTCPServer((HOST, PORT), ForkedTCPRequestHandler)

print "Starting Forked Server"

server.serve_forever()

多线程服务器：

threaded_server.py

import threading

import SocketServer

class ThreadedTCPRequestHandler(SocketServer.BaseRequestHandler):

def handle(self):

cur_thread = threading.current_thread()

print "launching a new socket handler, thread = {}".format(cur_thread)

while 1:

self.request.recv(4096)

class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):

pass

if __name__ == "__main__":

HOST, PORT = "localhost", 9999

server = ThreadedTCPServer((HOST, PORT), ForkedTCPRequestHandler)

print "Starting Threaded Server"

server.serve_forever()

在第一种情况下,使用forked_server.py,只创建了40个进程,其中大约20个进程在一段时间内开始出现以下错误：

error: [Errno 104] Connection reset by peer

在客户端.

线程版本更耐用,可容纳超过4000个连接,但最终开始显示

gaierror: [Errno -5] No address associated with hostname

测试是在我的本地机器上进行的,Kubuntu 14.04 x64在内核v3.13.0-32上进行.这些是我为提高系统的一般性能而采取的步骤：

>提高文件句柄的内核限制：sysctl -w fs.file-max = 10000000

>增加连接积压,sysctl -w net.core.netdev_max_backlog = 2500

>提高最大连接数,sysctl -w net.core.somaxconn = 250000

所以,问题是：

>如果测试正确,我可以依靠这些结果吗？我是所有这些网络/套接字的新手,所以请在我的结论中纠正我.

>在重负载系统中,多处理器/多线程方法真的不可行吗？

>如果是,我们还有哪些选择？异步方法？ Tornado / Twisted / Gevent框架？