浅析Scrapy框架数据持久化

将数据保存到MongoDB,MySQL和图片下载

先来看一下流程图
数据持久化是最后一道步骤,由pipeline完成,我们可以在这里将爬取到的有效的信息保存到数据库,或是将需要的图片下载到本地.温习一下将数据保存到MongoDB,MySQL和图片下载的步骤.

准备工作

• 数据库:

MySQL事先建表,用来保存信息,而MongoDB则无需事先创建集合
在 items.py 中定义字段,例如:

class News163Item(scrapy.Item):
    title = scrapy.Field()
    href = scrapy.Field()
    content = scrapy.Field()
    pic_url = scrapy.Field()

• 在 setting.py 中:

注册管道类,设置权重

ITEM_PIPELINES = {
    'news163.pipelines.ImgPipeLine': 300,
    'news163.pipelines.MongoPipeline': 301,
    'news163.pipelines.MysqlPipeline': 302,
}

提供连接数据库所需的信息

MONGO_URI = 'localhost'   #  MongoDB
MONGO_DB = 'wynews'

MYSQL_HOST = 'localhost'    # MySQL
MYSQL_PORT = 3306
MYSQL_DATABASE = 'news'
MYSQL_USER = 'root'
MYSQL_PWD = ''

图片路径

IMAGES_STORE = './mvs'

OK,准备工作完成先切入正题

书写管道类

写入MongoDB

1. init()方法： 初始化属性host和库名
2. 类方法from_crawler():获取配置内的 MONGO_URI 和 MONGO_DB ， 给属性赋值
3. open_spider():方法:连接数据库
4. process_item():方法:拦截item，插入数据
5. close_spider():关闭连接

例如：

import pymongo
class MongoPipeline(object):
    def __init__(self, mongo_uri, mongo_db):
    #初始化属性
    self.mongo_uri = mongo_uri
    self.mongo_db = mongo_db

	@classmethod
	def from_crawler(cls, crawler):
	#给属性赋值
    	return cls(
        	mongo_uri=crawler.settings.get('MONGO_URI'),
        	mongo_db=crawler.settings.get('MONGO_DB')
    		)

	def open_spider(self, spider):
		#连接数据库
    	self.client = pymongo.MongoClient(self.mongo_uri)
    	self.db = self.client[self.mongo_db]

	def process_item(self, item, spider):
    	self.db['n163'].insert(dict(item))  #方括号内是集合名
    	return item

	def close_spider(self, spider):
    	self.client.close()

写入MySQL

1. init():初始化属性:host,端口,数据库,用户名,密码
2. 类方法from_crawler():给属性赋值
3. open_spider():连接数据库,获取cursor对象
4. process_item():拦截item,写SQL语句并执行插入数据
5. close_spider():关闭连接

例子:

from pymysql import connect
class MysqlPipeline(object):
	def __init__(self, mysql_host, mysql_port, mysql_database, mysql_user, mysql_pwd):
    	self.mysql_host = mysql_host
    	self.mysql_port = mysql_port
    	self.mysql_database = mysql_database
    	self.mysql_user = mysql_user
    	self.mysql_pwd = mysql_pwd
	@classmethod
	def from_crawler(cls, crawler):
    	return cls(
        	mysql_host=crawler.settings.get('MYSQL_HOST'),
        	mysql_port=crawler.settings.get('MYSQL_PORT'),
        	mysql_database=crawler.settings.get('MYSQL_DATABASE'),
        	mysql_user=crawler.settings.get('MYSQL_USER'),
        	mysql_pwd=crawler.settings.get('MYSQL_PWD')
    		)

	def open_spider(self, spider):
    	self.conn = connect(host=self.mysql_host, port=self.mysql_port, database=self.mysql_database,
        	                user=self.mysql_user,
            	            password=self.mysql_pwd, charset='utf8')
    	self.cur = self.conn.cursor()

	def process_item(self, item, spider):
    	data = dict(item)
    	keys = ','.join(data.keys())
    	values = ','.join(['%s'] * len(data))
    	sql = 'insert into %s (%s) values (%s)' % ('news', keys, values)
    	self.cur.execute(sql, tuple(data.values()))
    	self.conn.commit()
    	return item

	def close_spider(self, spider):
    	self.cur.close()
    	self.conn.close()

在process_item()构造的SQL语句不会受具体字段个数的影响

def process_item(self, item, spider):
	data = dict(item)
	keys = ','.join(data.keys())
	values = ','.join(['%s'] * len(data))   # 按键值对个数生成占位符放在'()'中
	sql = 'insert into %s (%s) values (%s)' % ('news', keys, values)    # 将表名,字段名和键值对个数*占位符 替换sql中的占位符
	self.cur.execute(sql, tuple(data.values()))   # 拿出item中的所有值,替换占位符
	self.conn.commit()
	return item

下载图片

继承使用Scrapy中自带的管道类:ImagesPipeline

1. get_media_requests():拦截item,从中拿到与下载图片有关的信息并发出请求
2. file_path():写图片名(在setting.py中以指定下载路径),ps:windows文件名中不能有下列符

“?”、“、”、“╲”、“/”、“*”、““”、“”“、“<”、“>”、“|”

from scrapy.pipelines.images import ImagesPipeline
class ImgPipeLine(ImagesPipeline):
	def get_media_requests(self, item, info):
    	yield scrapy.Request(url=item['pic_url'], meta={'title': item['title']})

	def file_path(self, request, response=None, info=None):
    	file_name = request.meta['title'].replace('?', '') + '.jpg'
    	return file_name

	def item_completed(self, results, item, info):
    	return item