scrapy 学习 002

爬取数据写入 mongodb

编辑 pipelines

• 编辑 pipelines.py 添加

class BiqugePipeline(object):
    def open_spider(self, spider):
        client = MongoClient()
        self.collection = client["biquge"]["novellist"]
        self.client = client

    def process_item(self, item, spider):
        self.collection.insert(dict(item))
        return item
    
    def close_spider(self, spider):
        self.client.close()

启用 pipelines

• 编辑 settings.py 添加

ITEM_PIPELINES = {
    'tutorial.pipelines.BiqugePipeline': 300,
}

查看 mongodb