python进程数上限_在多处理python中限制进程数

限制并发连接数的最简单方法是使用线程池：#!/usr/bin/env python

from itertools import izip, repeat

from multiprocessing.dummy import Pool # use threads for I/O bound tasks

from urllib2 import urlopen

def fetch(url_data):

try:

return url_data[0], urlopen(*url_data).read(), None

except EnvironmentError as e:

return url_data[0], None, str(e)

if __name__=="__main__":

pool = Pool(20) # use 20 concurrent connections

params = izip(urls, repeat(data)) # use the same data for all urls

for url, content, error in pool.imap_unorderred(fetch, params):

if error is None:

print("done: %s: %d" % (url, len(content)))

else:

print("error: %s: %s" % (url, error))

503 Service Unavailable是服务器错误。它可能无法处理负载。在

Name or service not known是一个dns错误。如果需要多次请求，请安装/启用本地缓存dns服务器。在