scrapy爬数据到mongodb

最新推荐文章于 2023-06-30 13:05:31 发布

AuroraPetard

最新推荐文章于 2023-06-30 13:05:31 发布

阅读量324

点赞数

分类专栏： python

本文链接：https://blog.csdn.net/qq_38250124/article/details/79829681

版权

python 专栏收录该内容

23 篇文章 0 订阅

订阅专栏

首先创建一个数据库

use learn 然后插入点数据，要不然看不到表

db.learn.insert({"test":'1'})

然后创建集合类似于表

db.createCollection(name)

然后在setting中

ITEM_PIPELINES = {
    'quote.pipelines.QuotePipeline': 300,
}

MONGODB_HOST = '127.0.0.1'
MONGODB_PORT = 27017
MONGODB_DBNAME = 'learn'
MONGODB_COLLNAME = 'python'

pipeline中

import pymongo
from scrapy.conf import settings


class QuotePipeline(object):
    def __init__(self):
        host = settings['MONGODB_HOST']
        port = settings['MONGODB_PORT']
        db_name = settings['MONGODB_DBNAME']
        client = pymongo.MongoClient(host=host, port=port)
        db = client[db_name]
        self.post = db[settings['MONGODB_COLLNAME']]

    def process_item(self, item, spider):
        quotess = dict(item)
        self.post.insert(quotess)
        return item