Python爬虫多线程爬取

仅作参考

仅使用threadpool库：

task_pool = threadpool.ThreadPool(50)#开50个线程
        requests = threadpool.makeRequests(self.threading_thread, self.list_all)
#self.threading_thread为请求代码所在函数，self.list_all所传参数，不支持元组类型
        # print(requests)
        for req in requests:
            task_pool.putRequest(req)
        task_pool.wait()