scrapy+mongodb实现爬取的小说入库

MongoDB数据库的下载安装

在官网下载https://www.mongodb.com/download-center/community

可视化工具Studio 3T下载https://robomongo.org/
创建数据库

点击New Connection

创建新的表

scrapy+mongodb

1、setting.py

DOWNLOADER_MIDDLEWARES = {
    #'Douluodalu.middlewares.DouluodaluDownloaderMiddleware': 543,
    'Douluodalu.middlewares.MyUserAgentMiddleware':543,
}

2、item.py

import scrapy

class DouluodaluItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title=scrapy.Field() #章节名
    chapter=scrapy.Field() #内容
    pass

3、pipeline.py

import pymongo
  #从settings.py中导入mongodb的配置信息
class DouluodaluPipeline(object):
    def __init__(self):
        #配置mongo
        host='127.0.0.1'#本地地址
        port=27017     #端口
        client=pymongo.MongoClient(host=host,port=port)
        db = client["dldl"]  #数据库名
        self.post = db["douluo1"] #表名
    def process_item(self, item, spider):
        self.post.insert(item)  #插入字典
        return item