文章目录
Scrapy简介
通用爬虫框架流程
Scrapy 框架流程
Scrapy组件
Scrapy主要包括了以下组件:
- 引擎(Scrapy)
用来处理整个系统的数据流处理, 触发事务(框架核心) - 调度器(Scheduler)
用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL(抓取网页的网址或者说是链接)的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址 - 下载器(Downloader)
用于下载网页内容, 并将网页内容返回给蜘蛛(Scrapy下载器是建立在twisted这个高效的异步模型上的) - 爬虫(Spiders)
爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面 - 项目管道(Pipeline)
负责处理爬虫从网页中抽取的实体,主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后,将被发送到项目管道,并经过几个特定的次序处理数据。 - 下载器中间件(Downloader Middlewares)
位于Scrapy引擎和下载器之间的框架,主要是处理Scrapy引擎与下载器之间的请求及响应。 - 爬虫中间件(Spider Middlewares)
介于Scrapy引擎和爬虫之间的框架,主要工作是处理蜘蛛的响应输入和请求输出。 - 调度中间件(Scheduler Middewares)
介于Scrapy引擎和调度之间的中间件,从Scrapy引擎发送到调度的请求和响应。
Scrapy运行流程
- 引擎从调度器中取出一个链接(URL)用于接下来的抓取
- 引擎把URL封装成一个请求(Request)传给下载器
- 下载器把资源下载下来,并封装成应答包(Response)
- 爬虫解析Response
- 解析出实体(Item),则交给实体管道进行进一步的处理
- 解析出的是链接(URL),则把URL交给调度器等待抓取
Scrapy的安装
Linux下的安装(包括mac)
pip install scrapy
Windows下的安装
1. 下载twisted
http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
2. 安装wheel
pip3 install wheel
3. 安装twisted
进入下载目录,执行 pip3 install Twisted‑18.7.0‑cp36‑cp36m‑win_amd64.whl
4. 安装pywin32
pip3 install pywin32
5. 安装scrapy
pip3 install scrapy
基本命令
1. scrapy startproject 项目名称
在当前目录中创建一个项目文件
2. scrapy genspider [-t template] <name> <domain>
创建爬虫应用
如:
scrapy gensipider -t basic oldboy oldboy.com
scrapy gensipider -t xmlfeed autohome autohome.com.cn
或者简单直接:
scrapy gensipider app名 要爬取的域名
PS:
查看所有命令:scrapy gensipider -l
查看模板命令:scrapy gensipider -d 模板名称
3. scrapy list
展示爬虫应用列表
4. scrapy crawl 爬虫应用名称
运行单独爬虫应用
备注:
scrapy crawl 应用名称 表示以日志的形式运行爬虫应用,可以在后面加 --nolog 取消日志
scrapy crawl 名称 --nolog
项目文件说明
- scrapy.cfg 项目的主配置信息。(真正爬虫相关的配置信息在settings.py文件中)
- items.py 设置数据存储模板,用于结构化数据,如:Django的Model
- pipelines 数据处理行为,如:一般结构化的数据持久化
- settings.py 配置文件,如:递归的层数、并发数,延迟下载等
- spiders 爬虫目录,如:创建文件,编写爬虫规则
项目案例
项目介绍
为了充分利用网上大数据资源,让用户能够方便利用影视信息,采用基于 Scrapy 框架的爬虫技术,开发了检索电影信息的搜索引擎。对豆瓣网站的影视信息进行爬取,以方便用户准确获取最新的电影信息。
项目代码
以“豆瓣电影”为爬取目 标,爬取网站中的影视信息。主要包括网站排名 “ Top250 ”和喜剧、动作类电影的电影名称、电影评分、电影导演, 电影上映时间以及电影评语。
创建工程
scrapy startproject DouBan
创建爬虫程序
cd DouBan/
scrapy genspider douban 'douban.com'
自动创建目录及文件
编写爬虫文件(douban.py)
# -*- coding: utf-8 -*-
import scrapy
from scrapy import Request
from DouBan.items import DoubanItem
import copy
class DoubanSpider(scrapy.Spider):
name = 'douban'
allowed_domains = ['douban.com']
# start_urls = ['http://douban.com/']
start_urls = ['https://movie.douban.com/top250']
url = 'https://movie.douban.com/top250'
def parse(self, response):
items = DoubanItem()
# with open('douban.html', 'w') as f:
# f.write(response.text)
movies = response.xpath("//ol[@class='grid_view']/li")
for movie in movies:
title = movie.xpath(".//span[@class='title']/text()").extract()[0]
rating_num = movie.xpath(".//span[@class='rating_num']/text()").extract()[0]
# <span class="inq">希望让人自由。</span>
inq = movie.xpath(".//span[@class='inq']/text()").extract()
if inq:
inq = inq[0]
items['inq'] = inq
items['rating_num'] = rating_num
items['title'] = title
# 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p1454261925.jpg'
items['image_url'] = movie.xpath('.//div[@class="pic"]/a/img/@src').extract()[0]
# print("image url: ", item['image_url'])
items['detail_url'] = movie.xpath('.//div[@class="hd"]//a/@href').extract()[0]
# print("detail url: ", item['detail_url'])
# yield items
yield Request(items['detail_url'], meta={'item': copy.deepcopy(items)}, callback=self.detailParse)
# """
# <span class="next">
# <link rel="next" href="?start=50&filter=">
# <a href="?start=50&filter=">??></a>
# </span>
# """
# nextLink = response.xpath('.//span[@class="next"]/link/@href').extract() # 返回列表
# if nextLink:
# nextLink = nextLink[0]
# print('Next Link: ', nextLink)
# yield Request(self.url + nextLink, callback=self.parse)
def detailParse(self, response):
items = response.meta['item']
# print(items, '111111111111')
items['movieLength'] = response.xpath(".//span[@property='v:runtime']/text()").extract()[0]
print(items, '333333333333333')
yield copy.deepcopy(items)
编辑item文件
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class DoubanItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
inq = scrapy.Field()
rating_num = scrapy.Field()
title = scrapy.Field()
image_url = scrapy.Field()
detail_url = scrapy.Field()
image_path = scrapy.Field()
movieLength = scrapy.Field()
编辑pipelines文件
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
import json
import pymysql
import scrapy
from scrapy.exceptions import DropItem
from scrapy.pipelines.images import ImagesPipeline
class DoubanPipeline(object):
def process_item(self, item, spider):
return item
class AddScoreNum(object):
def process_item(self, item, spider):
if item['rating_num']:
rating_num = float(item['rating_num'])
item['rating_num'] = str(rating_num + 1)
return item
else:
raise Exception('没有爬取到rating_num')
class JsonWriterPipeline(object):
def open_spider(self, spider):
self.file = open('douban.json', 'w')
def process_item(self, item, spider):
line = json.dumps(dict(item), indent=4, ensure_ascii=False)
self.file.write(line)
return item
def close_spider(self, spider):
self.file.close()
class MysqlPipeline(object):
def open_spider(self, spider):
self.connect = pymysql.connect(
host='127.0.0.1',
port=3306,
db='scrapyProject',
user='root',
passwd='westos',
charset='utf8',
use_unicode=True
)
self.cursor = self.connect.cursor()
a = "create table if not exists douBanTop250(title varchar(50) unique,rating_num float,inq varchar(100));"
print(a)
self.cursor.execute(a)
def process_item(self, item, spider):
insert_sqli = "insert into douBanTop250(title, rating_num, inq) " \
"values('%s', '%s', '%s');" \
% (item['title'], item['rating_num'], item['inq'])
print(insert_sqli)
try:
self.cursor.execute(insert_sqli)
print(111)
except Exception as e:
self.connect.rollback()
print(222)
else:
self.connect.commit()
print(333)
return item
def close_spider(self, spider):
self.connect.commit()
self.cursor.close()
self.connect.close()
class MyImagesPipeline(ImagesPipeline):
def get_media_requests(self, item, info):
yield scrapy.Request(item['image_url'])
def item_completed(self, results, item, info):
"""
:param results:
[(True, {'url': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p1454261925.jpg',
'path': 'full/e9cc62a6d6a0165314b832b1f31a74ca2487547a.jpg',
'checksum': '5d77f59d4d634b795780b2138c1bf572'})]
:param item:
:param info:
:return:
"""
# for result in results:
# print("result: ", result)
# isok = True/False
image_paths = [x['path'] for isok, x in results if isok]
# print("image_paths: ", image_paths[0])
if not image_paths:
raise DropItem("Item contains no images")
item['image_path'] = image_paths[0]
return item
编辑settings文件
# -*- coding: utf-8 -*-
# Scrapy settings for DouBan project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'DouBan'
SPIDER_MODULES = ['DouBan.spiders']
NEWSPIDER_MODULE = 'DouBan.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'DouBan (+http://www.yourdomain.com)'
from fake_useragent import UserAgent
ua = UserAgent()
USER_AGENT = ua.random
# Obey robots.txt rules
# ROBOTSTXT_OBEY = True
ROBOTSTXT_OBEY = False
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'DouBan.middlewares.DoubanSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'DouBan.middlewares.DoubanDownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'scrapy.pipelines.images.ImagesPipeline': 1,
'scrapy.pipelines.files.FilesPipeline': 2,
'DouBan.pipelines.MyImagesPipeline': 2,
'DouBan.pipelines.AddScoreNum': 100,
'DouBan.pipelines.JsonWriterPipeline': 200,
'DouBan.pipelines.MysqlPipeline': 200
}
IMAGES_STORE = './images'
# FILES_STORE = './files'
IMAGES_EXPIRES = 30
# FILES_EXPIRES = 90
IMAGES_THUMBS = {
'small': (50, 50),
'big': (270, 270)
}
IMAGES_MIN_HEIGHT = 110
IMAGES_MIN_WIDTH = 110
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
项目效果
- 实现了页面解析爬取
- 爬取电影短评,分数,电影名,电影时长
- 对电影分数实现加一操作
- item数据保存为json文件
- 下载保存图片:电影缩略图,电影大图