python 多进程爬虫
top_beyond
自动化工程师正在向爬虫进军
展开
-
一.BeautifulSoup 多进程抓取智联招聘信息,并且存储到mongodb
# coding:utf-8import reimport requestsimport urllib2import datetimefrom bs4 import BeautifulSoupfrom pymongo import MongoClientfrom multiprocessing import Pool#开启数据库cn=MongoClient('localhost原创 2017-06-30 11:54:47 · 966 阅读 · 0 评论 -
一.多进程爬取百思不得姐图片
#coding:utf-8import requestsfrom requests.exceptions import HTTPError, ConnectionErrorimport reimport os,timefrom multiprocessing import Process,JoinableQueueclass get_Url(): def __init__(se原创 2017-06-30 13:35:33 · 914 阅读 · 0 评论 -
二.python 多进程总结
一.多进程 def1.import timeimport multiprocessingdef run(data): print time.ctime(time.time()) print dataif __name__=='__main__': p1=multiprocessing.Process(target=run,args=('11',))原创 2017-07-04 17:28:28 · 703 阅读 · 0 评论