Scrapy 中的 MongoDB 安装、使用、可视化入门

MongoDB 的安装

https://www.mongodb.org/downloads

mkdir data

mongod --dbpath ./data

MongoDB 的可视化插件安装

https://www.mongovue.com/

pymongo 的安装

pip install pymongo

或

easy_install pymongo

在程序中使用的代码骗子

import pymongo

connection = pymongo.MongoClient()

tdb = connection.Tianxiaoxiao #连接的名字叫Tianxiaoxiao 且不能为变量

post_info = tdb.test #test不能为变量，test为table

post_info.insert(xxx)

post_info.remove(xxx)

scrapy中配置MongoDB，我这里之说pipelins，有问题请评论

def __init__(self):

host = settings['MONGODB_HOST']

port = settings['MONGODB_PORT']

dbName = settings['MONGODB_DBNAME']

client = pymongo.MongoClient(host=host, port=port)

tdb = Client[dbName]

self.post = tdb[settings['MONGODB_DOCNAME']]

def process_item(self, item, spider)

bookInfo = dict(item)

self.post.insrt(bookInfo)

return item