scrapy
用户需要写的只有spider模块和item pipelines模块
downloader模块,scheduler模块,engine模块已有实现
只经过简单的配置就可实现框架功能,但不用于测试,先用request再移植到scrapy中进行爬取
- engine:控制所有模块间的数据流,根据条件触发事件
- downloader:根据请求下载网页
- scheduler:对所有的爬取请求进行调度管理,决定谁先爬谁后爬,爬过的url如何处理,去重,过滤
- downloader middleware:实施engine,scheduler和downloader之间用户可配置的控制,用来修改,丢弃,新增请求或响应,用户可修改配置代码
- spider:解析 downloader返回的响应,产生爬取项(scraped item),产生额外的爬取请求
- item pipelines:以流水线方式处理spider产生的爬取项,可能操作包括清理,检验查重爬取项中的html数据,将数据存储到数据库
- spider middlewaare:对请求和爬取项的再处理,修改,丢弃新增请求或爬取项
requests VS. Scrapy
相同点:1.两者都可以进行页面请求和爬取,python爬虫的两个重要技术路线
2.两者可用性都号,文档丰富,入门简单
3.两者都没有处理js,提交表单,应对验证码等功能(可扩展)
requests | Scrapy |
---|---|
页面级爬虫 | 网站级爬虫 |
功能库 | 框架 |
并发性考虑不足 | 并发性好 |
重点在于页面下载 | 重点在于爬虫结构 |
定制灵活 | 一般定制灵活,深度定制困难 |
request发请求等待响应的时候无法处理其他事
爬取速度快慢要根据具体情况而定
Request和Response类
Request
Request对象表示一个HTTP请求
由Spider生成,由downloader执行
class scrapy.http.Request()
属性或方法 | 说明 |
---|---|
url | Request对应的请求url地址 |
callback | 指定本次请求响应的回调函数 |
method | 对应的请求方法,‘GET’‘POST’ |
proxy | 代理 |
headers | 字典类型的请求头 |
cookies | cookie,添加时需要键值对形式(" “:” ") |
encoding | 编码方式 |
priority | 默认为0 |
dont_filter | 是否过滤请求(已发送),默认True |
errback | 请求失败的回调函数,可被try,except代替 |
body | 请求内容主体,字符串类型 |
meta | 用户添加的扩展信息,字典类型,在Scrapy内部模块间传递信息使用 |
.copy() | 复制该请求 |
.FormRequest() | 发送POST请求 |
Response
Response对象表示一个HTTP响应
由downloader生成,由spider处理
class scrapy.http.Response()
属性或方法 | 说明 |
---|---|
url | Response对应的请求url地址 |
status | HTTP状态码,默认是200 |
text | 文本数据 |
meta | 响应的meta,从中可以拿到响应的cookies |
headers | response对应的头部信息 |
body | response对应的内容信息,二进制流数据,用于存储媒体流文件 |
flags | 一组标记 |
request | 产生Response类型对应的Request对象 |
.copy() | 复制该响应 |
.xpath() | 用来做xpath解析,还需要用.getall()或者.extract()获取内容 |
scrapy源码
spider
ctrl+鼠标单击查看函数信息
class Spider(object_ref):
"""Base class for scrapy spiders. All spiders must inherit from this
class.
"""
#所有spider的基类
name = None #类属性,爬虫名
custom_settings = None
def __init__(self, name=None, **kwargs):
if name is not None:
self.name = name#如果爬虫名非空,按照输入的赋值
elif not getattr(self, 'name', None):#如果没有名字则抛出异常
raise ValueError("%s must have a name" % type(self).__name__)
self.__dict__.update(kwargs)
if not hasattr(self, 'start_urls'):#如果没有start_urls则设置为空列表
self.start_urls = []
@property
#日志记录
def logger(self):
logger = logging.getLogger(self.name)
return logging.LoggerAdapter(logger, {'spider': self})
def log(self, message, level=logging.DEBUG, **kw):
"""Log the given message at the given log level
This helper wraps a log call to the logger within the spider, but you
can use it directly (e.g. Spider.logger.info('msg')) or use any other
Python logger too.
"""
self.logger.log(level, message, **kw)
#创建crawler
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = cls(*args, **kwargs)
spider._set_crawler(crawler)
return spider
#设置crawler,抛出信号
def _set_crawler(self, crawler):
self.crawler = crawler
self.settings = crawler.settings
crawler.signals.connect(self.close, signals.spider_closed)
#起始请求方法
def start_requests(self):
cls = self.__class__
if method_is_overridden(cls, Spider, 'make_requests_from_url'):
warnings.warn(
"Spider.make_requests_from_url method is deprecated; it "
"won't be called in future Scrapy releases. Please "
"override Spider.start_requests method instead (see %s.%s)." % (
cls.__module__, cls.__name__
),
)
for url in self.start_urls:#从start_urls中取出url,start_urls中可写入多个url
yield self.make_requests_from_url(url)
else:
for url in self.start_urls:
yield Request(url, dont_filter=True)
def make_requests_from_url(self, url):#根据url创建request对象
""" This method is deprecated. """
return Request(url, dont_filter=True)#默认发请求不过滤,没有cookie,headers等参数,需要携带时需要重载
def parse(self, response):
raise NotImplementedError('{}.parse callback is not defined'.format(self.__class__.__name__))#必须重写parse方法,否则抛出异常
@classmethod
def update_settings(cls, settings):
settings.setdict(cls.custom_settings or {}, priority='spider')
@classmethod
def handles_request(cls, request):
return url_is_from_spider(request.url, cls)
@staticmethod
def close(spider, reason):
closed = getattr(spider, 'closed', None)
if callable(closed):#如果属性中有close那么就关闭爬虫
return closed(reason)
Item
Item对象表示一个从HTML页面中提取的信息内容
由spider生成,由Item pipeline处理
Item类似字典类型,可以按照字典类型操作
pipelines
方法名不能改,参数不要改,函数从spider中接收item并存储
settings
存放spider的位置
全局的user-agent,除非被headers中的参数覆盖,电脑和浏览器不变全局user-agent也不变,当访问网站次数过多时可能需要更换
启用pipelines,参数为优先级,数字越小优先级越高
是否遵守robots协议
全局请求头
最大并发连接数,linux为1024,windows为512
配置爬虫中间件,键为中间件所在位置,值为优先级
配置下载器中间件,键为中间件所在位置,值为优先级
配置爬取深度,不配置
不启用cookies,不配置
不使用telnet链接即拨号上网,不配置
是否允许拓展,不配置
限速操作,不配置
设置缓存,一般不配置
日志配置
在setting中设置
是否显示日志
日志级别(向下兼容)
1. DEBUG : 程序基本调试信息
2. INFO : 基本提示信息
3. WARNING : 警告
4. ERROR : 一般错误
5. CRITICAL : 严重错误,可能引起程序崩溃
主动抛出日志
断点续传
原理:调度器的内存队列和磁盘队列
内存队列:在内存中保存request对象的容器
特点:快,无法持久化
磁盘队列:在磁盘中保存request对象的容器
特点:慢,持久化
scrapy默认使用内存队列,如果配置jobdir可以把调度器中的数据进行持久化
JOBDIR=" "#设置持久化的目录
request如果边爬取边存数据则也算持久化(使用update)
middleware
# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
from scrapy import signals
class LearnSpiderMiddleware(object):
#spider中间件不是所有方法都需要被定义,如果没有定义则pass
#创建爬虫时调用
@classmethod
def from_crawler(cls, crawler):
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)#信号提示爬虫启动
return s
#当每一个响应穿过爬虫中间件进入spider时被调用
def process_spider_input(self, response, spider):
# 返回none或者抛出异常
return None
#处理完response之后调用
def process_spider_output(self, response, result, spider):
# 必须返回一个可迭代的request对象或者item对象或者字典
for i in result:
yield i
#process_spider_input抛出异常时调用
def process_spider_exception(self, response, exception, spider):
# 返回none或者request或者item或者字典
# 不需要注意,之后为pass
pass
#start_requests被调用时
def process_start_requests(self, start_requests, spider):
#类似于process_spider_output() method,
#只能返回request
for r in start_requests:
yield r
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
#下载器中间件,可以在发送给downloader之前添加数据
class LearnDownloaderMiddleware(object):
#spider中间件不是所有方法都需要被定义,如果没有定义则pass
#创建时调用
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
#当request穿过下载器中间件(未到达)时调用
def process_request(self, request, spider):
# Must either:
# 返回None: 继续
# 返回Response给spider
# 返回Request给调度器
# 抛出异常调用process_exception()
return None
#当Response从下载器返回时调用
def process_response(self, request, response, spider):
# Must either;
# 返回Response给spider
# 返回Request给调度器
# 抛出异常调用process_exception()
return response
#异常处理
def process_exception(self, request, exception, spider):
# 返回None: 跳过处理异常
# 返回Response:停止处理异常的动作链
# 返回Request:停止处理异常的动作链
return response
pass
#提示爬虫启动
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
scheduler
class Scheduler(object):
"""
调度器可让多个requests入队,并且让下一个request去下载,并具有去重机制
优先级和队列不会被调度器执行,需要用户去设置
(defined by :setting:`SCHEDULER_PRIORITY_QUEUE`) uses these priorities
to dequeue requests in a desired order.
调度器使用两个优先级队列实例,默认使用磁盘队列,当磁盘队列不能处理时使用内存队列
:setting:`SCHEDULER_MEMORY_QUEUE` and
:setting:`SCHEDULER_DISK_QUEUE` allow to specify lower-level queue classes
which PriorityQueue instances would be instantiated with, to keep requests
on disk and in memory respectively.
总之,调度器是一个池由优先级队列和应用反馈逻辑的调度器,并且可以控制去重策略
"""
def __init__(self, dupefilter, jobdir=None, dqclass=None, mqclass=None,
logunser=False, stats=None, pqclass=None, crawler=None):
self.df = dupefilter#去重策略
self.dqdir = self._dqdir(jobdir)#设置在磁盘队列中的工作路径
self.pqclass = pqclass
self.dqclass = dqclass
self.mqclass = mqclass
self.logunser = logunser
self.stats = stats
self.crawler = crawler
@classmethod
def from_crawler(cls, crawler):
settings = crawler.settings#导入配置
dupefilter_cls = load_object(settings['DUPEFILTER_CLASS'])#setting中去重策略的类
dupefilter = create_instance(dupefilter_cls, settings, crawler)#创建去重策略
pqclass = load_object(settings['SCHEDULER_PRIORITY_QUEUE'])#加载优先级队列
if pqclass is PriorityQueue:#优先级在优先级队列中抛出警告
warnings.warn("SCHEDULER_PRIORITY_QUEUE='queuelib.PriorityQueue'"
" is no longer supported because of API changes; "
"please use 'scrapy.pqueues.ScrapyPriorityQueue'",
ScrapyDeprecationWarning)
from scrapy.pqueues import ScrapyPriorityQueue
pqclass = ScrapyPriorityQueue
dqclass = load_object(settings['SCHEDULER_DISK_QUEUE'])#加载磁盘队列
mqclass = load_object(settings['SCHEDULER_MEMORY_QUEUE'])#加载内存队列
logunser = settings.getbool('LOG_UNSERIALIZABLE_REQUESTS',
settings.getbool('SCHEDULER_DEBUG'))#记录日志
return cls(dupefilter, jobdir=job_dir(settings), logunser=logunser,
stats=crawler.stats, pqclass=pqclass, dqclass=dqclass,
mqclass=mqclass, crawler=crawler)#生成包装好的对象
def has_pending_requests(self):
return len(self) > 0
def open(self, spider):#打开爬虫
self.spider = spider
self.mqs = self._mq()
self.dqs = self._dq() if self.dqdir else None
return self.df.open()
def close(self, reason):#关闭爬虫
if self.dqs:#如果磁盘队列中没有值则关闭
state = self.dqs.close()
self._write_dqs_state(self.dqdir, state)
return self.df.close(reason)
def enqueue_request(self, request):
if not request.dont_filter and self.df.request_seen(request):#如果requests已重复并在已爬取的队列中,就抛出日志,并return false
self.df.log(request, self.spider)
return False
dqok = self._dqpush(request)#不重复则push进dq
if dqok:#如果磁盘队列非空就应用磁盘队列
self.stats.inc_value('scheduler/enqueued/disk', spider=self.spider)
else:#否则request push进内存队列
self._mqpush(request)
self.stats.inc_value('scheduler/enqueued/memory', spider=self.spider)
self.stats.inc_value('scheduler/enqueued', spider=self.spider)
return True
def next_request(self):
request = self.mqs.pop()#从内存队列中取出request交给spider
if request:#有值
self.stats.inc_value('scheduler/dequeued/memory', spider=self.spider)
else:#无值则从磁盘队列中取
request = self._dqpop()
if request:
self.stats.inc_value('scheduler/dequeued/disk', spider=self.spider)
if request:#不管是dq还是mq拿出了request则加入到已采集的队列中防止爬到重复对象
self.stats.inc_value('scheduler/dequeued', spider=self.spider)
return request
def __len__(self):
return len(self.dqs) + len(self.mqs) if self.dqs else len(self.mqs)
def _dqpush(self, request):
if self.dqs is None:
return
try:
self.dqs.push(request, -request.priority)
except ValueError as e: # non serializable request
if self.logunser:
msg = ("Unable to serialize request: %(request)s - reason:"
" %(reason)s - no more unserializable requests will be"
" logged (stats being collected)")
logger.warning(msg, {'request': request, 'reason': e},
exc_info=True, extra={'spider': self.spider})
self.logunser = False
self.stats.inc_value('scheduler/unserializable',
spider=self.spider)
return
else:
return True
def _mqpush(self, request):
self.mqs.push(request, -request.priority)
def _dqpop(self):
if self.dqs:
return self.dqs.pop()
def _newmq(self, priority):
""" Factory for creating memory queues. """
return self.mqclass()
def _newdq(self, priority):
""" Factory for creating disk queues. """
path = join(self.dqdir, 'p%s' % (priority, ))
return self.dqclass(path)
def _mq(self):
""" Create a new priority queue instance, with in-memory storage """
return create_instance(self.pqclass, None, self.crawler, self._newmq,
serialize=False)
def _dq(self):
""" Create a new priority queue instance, with disk storage """
state = self._read_dqs_state(self.dqdir)
q = create_instance(self.pqclass,
None,
self.crawler,
self._newdq,
state,
serialize=True)
if q:
logger.info("Resuming crawl (%(queuesize)d requests scheduled)",
{'queuesize': len(q)}, extra={'spider': self.spider})
return q
def _dqdir(self, jobdir):
""" Return a folder name to keep disk queue state at """
if jobdir:
dqdir = join(jobdir, 'requests.queue')
if not exists(dqdir):
os.makedirs(dqdir)
return dqdir
def _read_dqs_state(self, dqdir):
path = join(dqdir, 'active.json')
if not exists(path):
return ()
with open(path) as f:
return json.load(f)
def _write_dqs_state(self, dqdir, state):
with open(join(dqdir, 'active.json'), 'w') as f:
json.dump(state, f)
常用命令
scrapy是为持续运行设计的专业爬虫框架,提供操作的scrapy命令行,在scrapy框架下一个工程是一个最大单元相当于一个大的scrapy框架,在scrapy框架中可以有多个爬虫,每一个爬虫相当于一个scrapy模块
>scrapy<command>[options][args]
命令 | 说明 | 格式 |
---|---|---|
startproject | 创建一个新工程 | scrapy startproject <name>[dir] |
genspider | 创建一个爬虫 | scrapy genspider [options]<name><domain> |
settings | 获取爬虫配置信息 | scrapy settings [options] |
crawl | 运行一个爬虫 | scrapy crawl <spider> |
list | 运行工程中所有爬虫 | scrapy list |
shell | 启动url调试命令行 | scrapy shell [url] |
为什么使用命令行
- 命令行更容易自动化,适合脚本控制
- 给程序员使用,功能更重要
生成爬虫
1.建立一个scrapy爬虫工程
2.在工程中产生一个scrapy爬虫
会在工程目录里的spiders中生成新的py文件
demo.py
# -*- coding: utf-8 -*-
import scrapy
class DemoSpider(scrapy.Spider):#必须继承于scrapy.spider
name = 'demo'
#命令中赋值name
allowed_domains = ['python123.io']
#命令中提交的域名,爬虫在爬取网站时只能爬这个网站以下的相关链接
start_urls = ['http://python123.io/']
#爬取的初始页面
def parse(self, response):
#用于处理响应,解析内容形成字典,发现新的url爬取请求
pass
3.配置爬虫
class DemoSpider(scrapy.Spider):
name = 'demo'
start_urls = ['https://python123.io/ws/demo.html']
def parse(self, response):
#response为返回内容对应的对象
fname = response.url.split('/')[-1]
with open(fname,'wb') as f:
f.write(reponse.body)
self.log('Saved file %s.' % name)
pass
4.运行爬虫
yield关键字
主要与循环在一起使用
yield= 生成器
生成器是一个不断产生值的函数
包含yield语句的函数是一个生成器
生成器每次产生一个值,函数被冻结,被唤醒后再产生一个值
下次执行时从被冻结处开始执行
为什么要有生成器
- 节省存储空间
- 响应更迅速
- 使用更灵活
UA,cookies,headers添加
- spider中生成Request对象的时候传入headers
- 在settings中配置全局UA
- 在settings中配置全局headers
中间件添加
股票实例
步骤
1.建立工程和Spider模块
命令行中输入命令:
scrapy startproject Stocks
cd Stocks
scrapy genspider demo quote.cfi.cn/stockList.aspx#http://之后的东西
修改spiders/stocks.py文件
2.编写Spider
- 配置demo.py文件
- 修改对返回页面的处理
- 修改对新增url爬取请求的处理
3.编写pipelines
- 配置pipelines.py文件
- 定义对爬取项(Scraped Item)的处理类
- 配置ITEM_PIPELINES选项settings.py文件中找到参数ITEM_PIPELINES并修改为自己写的
代码
spider
# -*- coding: utf-8 -*-
import scrapy
import re
class DemoSpider(scrapy.Spider):
name = 'demo'
start_urls = ['http://quote.cfi.cn/stockList.aspx/']
def parse(self, response):
for href in response.css('a::attr(href)').extract():
try:
stock = re.findall(r'\d{6}',href)[0]
url = 'http://quote.cfi.cn/' + stock + '.html'
yield scrapy.Request(url, callback = self.parse_stock)
except:
continue
pass
def parse_stock(self, response):
infoDict = {}
stockInfo = response.css('#act_quote')[0]
#id查找使用#
nmae1 = stockInfo.css('.Lfont')[0]
name = nmae1.css('::text').extract()
Item = response.css('#quotetab_stock')[0]
item = Item.css('td')
count = 0
for Info in item:
if(count==10):
break
info ="".join(Info.css('::text').extract())
# print(info)
key = info.split(':')[0]
# print(key)
val = info.split(':')[1]
infoDict[key] = val
count=count+1
infoDict.update({'股票名称':name[0].split()[0]})
yield infoDict
#class查找使用.
pipelines
class StocksPipeline(object):
#对每一个item进行处理的过程
def process_item(self, item, spider):
return item
class StocksInfoPipeline(object):
def open_spider(self,spider):
#爬虫被调用时启动的方法
self.f = open('股票.txt','w')
def close_spider(self,spider):
self.f.close()
def process_item(self, item, spider):
try :
line = str(dict(item))+'\n'
self.f.write(line)
except:
pass
return item
招聘网站实例
spider
# -*- coding: utf-8 -*-
from ..items import LearnItem
import scrapy
class Job51Spider(scrapy.Spider):
name = 'job51' #
# allowed_domains = ['51job.com']
# start_urls = ['https://search.51job.com/list/211000,000000,0000,00,9,99,%25E9%2594%2580%25E5%2594%25AE,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=']
def start_requests(self):#分页,添加cookie,headers等情况不适用start_url,需要重载start_requests
for i in range(50):
url = "http://search.51job.com/list/020000,000000,0000,00,9,99,%2B,2," + str(i) + ".html?lang=c&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare="
yield scrapy.Request(url=url)
#for循环中使用return则循环结束
def parse(self, response):
job_name = response.xpath("//div[@class='el']/p/span/a/@title").getall()
#selector对象getall()方法,返回列表
c_name = response.xpath("//div[@class='el']/span/a/text()").getall()
item = LearnItem()
#实例化对象,此时item为字典
for name in job_name:
index = job_name.index(name)
#获取索引值
new_cname = c_name[index]
#公司名称
item['j_name']=name
#往item对象存工作名
item['c_name']=new_cname
yield item
items
import scrapy
class LearnItem(scrapy.Item):
# define the fields for your item here like:
j_name = scrapy.Field()
#工作名
c_name = scrapy.Field()
#公司名
pass
pipelines
class LearnPipeline(object):
def process_item(self, item, spider):
print(item)
return item
小说实例
scrapy异步框架处理小说内容会乱序
spider
import scrapy
from ..items import BiqugeItem
class BiqugeSpider(scrapy.Spider):
name = "biquge"
def start_requests(self):
url = "http://www.xbiquge.la/xiaoshuodaquan/"
yield scrapy.Request(url,callback=self.parse)
#设置回调函数为自身
def parse(self, response):
book_urls = response.xpath("//div[@class='novellist']//a/@href").getall()
#获取列表页所有小说的url
for book_url in book_urls:
yield scrapy.Request(url=book_url,callback=self.parse1)
#start
def parse1(self, response):
chapter_urls = response.xpath("//div[@id='list']/dl//dd/a/@href").getall()
#获取章节url
new_cha_urls = ["http://www.xbiquge.la"+i for i in chapter_urls]
#拼接具体章节url
for cha_url in new_cha_urls:
yield scrapy.Request(url=cha_url,callback=self.parse2)
def parse2(self, response):
item = BiqugeItem()
cont = response.xpath("//div[@id='content']/text()").getall()
cha_name = response.xpath("//h1/text()").getall()[0]
#获取章节名字
s = ""
#空字符串来存放小说内容
for i in cont:
s += i.strip()
#去空格
item['content']=s
item['cha_name']=cha_name
yield item
items
class BiqugeItem(scrapy.Item):
# define the fields for your item here like:
content = scrapy.Field()
#内容
cha_name = scrapy.Field()
#章节名
pipelines
class BiqugePipeline(object):
def process_item(self, item, spider):
print(item)
return item