获取到影片信息之后,下一步就是要把获取到的信息进行保存了。网上很多的案例都是保存成json格式,这里我想用Mysql服务器来保存。
1、首先安装好Mysql数据库,建好filminfo表和字段。
2、在items.py文件中添加你需要保存到数据库中的信息,定义相对应的class,生成item类对象。
class DoubanItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
type = scrapy.Field()
id = scrapy.Field()
title = scrapy.Field()
year = scrapy.Field()
directer = scrapy.Field()
scenarist = scrapy.Field()
actor = scrapy.Field()
genre = scrapy.Field()
area = scrapy.Field()
language = scrapy.Field()
ReleaseDate = scrapy.Field()
runtime = scrapy.Field()
anname = scrapy.Field()
imdb = scrapy.Field()
(1)python3.6使用的Mysql数据库模块是pymysql,不支持2.x支持的MySQLdb 模块;
(2)在pymysql.connect里面设置连接服务器的参数;
(3)sql语句执行后,需要commit()执行一下
def __init__(self):
self.conn = pymysql.connect(host='localhost',port=3306,user="root",passwd="******",db = "filmandtv",charset='utf8')
self.cursor = self.conn.cursor()
insert_sql = '''
insert into filminfo(type,filmid,title,year,directer,scenarist,actor,genre,area,language,ReleaseDate,runtime,anname,imdb,imagepath,plotsummary) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
'''
self.cursor.execute(insert_sql,(item["type"],item["id"],item["title"],item["year"],directer,scenarist,actor,genre,area,language,ReleaseDate,runtime,anname,item["imdb"],filmrelative_path,item["plot_summary"]))
self.conn.commit()
4、还有很重要的一步,就是在spider里设置返回items的信息
return DoubanItem(type=type,id=id,title=title,year=year,directer=directer,scenarist=scenarist,actor=actor,genre=genre,area=area,language=language,ReleaseDate=ReleaseDate,runtime=runtime,anname=anname,imdb=imdb,poster_url=poster_url,plot_summary=plot_summary)
5、运行一下,数据就保存下来了!成功!