将数据保存到MongoDB,MySQL和图片下载
先来看一下流程图
数据持久化是最后一道步骤,由pipeline完成,我们可以在这里将爬取到的有效的信息保存到数据库,或是将需要的图片下载到本地.温习一下将数据保存到MongoDB,MySQL和图片下载的步骤.
准备工作
- 数据库:
MySQL事先建表,用来保存信息,而MongoDB则无需事先创建集合
在 items.py 中定义字段,例如:
class News163Item(scrapy.Item):
title = scrapy.Field()
href = scrapy.Field()
content = scrapy.Field()
pic_url = scrapy.Field()
- 在 setting.py 中:
注册管道类,设置权重
ITEM_PIPELINES = {
'news163.pipelines.ImgPipeLine': 300,
'news163.pipelines.MongoPipeline': 301,
'news163.pipelines.MysqlPipeline': 302,
}
提供连接数据库所需的信息
MONGO_URI = 'localhost' # MongoDB
MONGO_DB = 'wynews'
MYSQL_HOST = 'localhost' # MySQL
MYSQL_PORT = 3306
MYSQL_DATABASE = 'news'
MYSQL_USER = 'root'
MYSQL_PWD = ''
图片路径
IMAGES_STORE = './mvs'
OK,准备工作完成先切入正题
书写管道类
写入MongoDB
- init()方法: 初始化属性host和库名
- 类方法from_crawler():获取配置内的 MONGO_URI 和 MONGO_DB , 给属性赋值
- open_spider():方法:连接数据库
- process_item():方法:拦截item,插入数据
- close_spider():关闭连接
例如:
import pymongo
class MongoPipeline(object):
def __init__(self, mongo_uri, mongo_db):
#初始化属性
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
@classmethod
def from_crawler(cls, crawler):
#给属性赋值
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DB')
)
def open_spider(self, spider):
#连接数据库
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def process_item(self, item, spider):
self.db['n163'].insert(dict(item)) #方括号内是集合名
return item
def close_spider(self, spider):
self.client.close()
写入MySQL
- init():初始化属性:host,端口,数据库,用户名,密码
- 类方法from_crawler():给属性赋值
- open_spider():连接数据库,获取cursor对象
- process_item():拦截item,写SQL语句并执行插入数据
- close_spider():关闭连接
例子:
from pymysql import connect
class MysqlPipeline(object):
def __init__(self, mysql_host, mysql_port, mysql_database, mysql_user, mysql_pwd):
self.mysql_host = mysql_host
self.mysql_port = mysql_port
self.mysql_database = mysql_database
self.mysql_user = mysql_user
self.mysql_pwd = mysql_pwd
@classmethod
def from_crawler(cls, crawler):
return cls(
mysql_host=crawler.settings.get('MYSQL_HOST'),
mysql_port=crawler.settings.get('MYSQL_PORT'),
mysql_database=crawler.settings.get('MYSQL_DATABASE'),
mysql_user=crawler.settings.get('MYSQL_USER'),
mysql_pwd=crawler.settings.get('MYSQL_PWD')
)
def open_spider(self, spider):
self.conn = connect(host=self.mysql_host, port=self.mysql_port, database=self.mysql_database,
user=self.mysql_user,
password=self.mysql_pwd, charset='utf8')
self.cur = self.conn.cursor()
def process_item(self, item, spider):
data = dict(item)
keys = ','.join(data.keys())
values = ','.join(['%s'] * len(data))
sql = 'insert into %s (%s) values (%s)' % ('news', keys, values)
self.cur.execute(sql, tuple(data.values()))
self.conn.commit()
return item
def close_spider(self, spider):
self.cur.close()
self.conn.close()
在process_item()构造的SQL语句不会受具体字段个数的影响
def process_item(self, item, spider):
data = dict(item)
keys = ','.join(data.keys())
values = ','.join(['%s'] * len(data)) # 按键值对个数生成占位符放在'()'中
sql = 'insert into %s (%s) values (%s)' % ('news', keys, values) # 将表名,字段名和键值对个数*占位符 替换sql中的占位符
self.cur.execute(sql, tuple(data.values())) # 拿出item中的所有值,替换占位符
self.conn.commit()
return item
下载图片
继承使用Scrapy中自带的管道类:ImagesPipeline
- get_media_requests():拦截item,从中拿到与下载图片有关的信息并发出请求
- file_path():写图片名(在setting.py中以指定下载路径),ps:windows文件名中不能有下列符
“?”、“、”、“╲”、“/”、“*”、““”、“”“、“<”、“>”、“|”
from scrapy.pipelines.images import ImagesPipeline
class ImgPipeLine(ImagesPipeline):
def get_media_requests(self, item, info):
yield scrapy.Request(url=item['pic_url'], meta={'title': item['title']})
def file_path(self, request, response=None, info=None):
file_name = request.meta['title'].replace('?', '') + '.jpg'
return file_name
def item_completed(self, results, item, info):
return item