scrapy User-Agent随机更换
首先在settings.py文件末尾添加用户代理池,如:
USER_AGENTS=['Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11','Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0','Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1','Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50','Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50','Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11','Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)']
然后在middlewares.py中将原有的代码注释掉,来自定义随即更换useragent类:
import random
class RandomUserAgentMiddleware(object):
def process_request(self,request,spider):
ua = random.choice(spider.settings.get('USER_AGENTS'))
request.headers['User-Agent']=ua
class Middleware(object):
def process_response(self,request,response,spider):
return response
最后在settings.py中将DOWNLOADER_MIDDLEWARES解除注释,然后换上自定义的middleware
将爬取的数据导入mongodb数据库
在settings.py末尾添加数据库信息,如:
MONGODB_HOST='127.0.0.1' #定义数据库的地址,端口,名称
MONGODB_PORT=27017
MONGODB_NAME= <数据库名>
MONGODB_COLLECTIONS= <集合名>
然后在piplines.py中添加如下代码:
from scrapy.utils.project import get_project_settings #导入该模块,使能使用settings.py中的数据
import pymongo
class Pipeline(object):
def __init__(self):
settings = get_project_settings()
host=settings['MONGODB_HOST']
port=settings['MONGODB_PORT']
dbname=settings['MONGODB_NAME']
client=pymongo.MongoClient(host=host,port=port) #链接到mongodb
tdb=client[dbname] #创建数据库
self.post=tdb[settings['MONGODB_COLLECTIONS']] #指定集合
def process_item(self, item, spider):
info=dict(item)
self.post.insert(info) #插入数据
return item
最后在settings.py中将ITEM_PIPELINES解除注释,换上自定义的piplines