python 3 scrapy_python3-scrapy学习之保存到本地json

python3-scrapy学习之保存到本地json

龙行    Python    2018-8-10    2176    1评论

多的不说直接上代码，做个记录

//scrapy Pipeline 保存数据到json

from scrapy.exporters import JsonItemExporter

class JsonWithEncodingPipeline(object):

def __init__(self):

self.file = open('article.json', 'wb') # 必须二进制写入

self.exporter = JsonItemExporter(self.file, encoding='utf-8', ensure_ascii=False)

# 开始写入

self.exporter.start_exporting()

def open_spider(self, spider):

print('爬虫开始')

pass

def process_item(self, item, spider):

self.exporter.export_item(item)

return item

def close_spider(self, spider):

# 完成写入

self.exporter.finish_exporting()

self.file.close()

#setting.py 管道开启JsonWithEncodingPipeline

ITEM_PIPELINES = {

'articlepider.pipelines.JsonWithEncodingPipeline': 2,

记得一定要开启管道哦

评论一下

赞助站长

赞助站长X