scrapy爬虫基本步骤总结

参考资料：https://www.imooc.com/learn/1017
以下内容仅为学习笔记
一、新建项目
1、在cmd中输入：
scrapy startproject 项目名称
即可在当前目录下建立项目
2、用pycharm打开项目，可在settings.py中进行一些设置：
添加USER_AGENT
ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 0.5（随意）
……
3、在cmd中进入项目内的spiders文件夹，输入：
scrapy genspider 爬虫名称 要爬虫的网站的域名
即可得到一个爬虫文件
注意爬虫名和项目名要不一样
4、在项目中新建一个main.py，添加如下代码：
from scrapy import cmdline
cmdline.execute(‘scrapy crawl douban_spider -o test.csv’.split())
之后运行main.py即可将爬取的数据导入test.csv文件中

二、明确目标
在items.py的item类下定义要抓取的数据结构
格式：name = scrapy.Field()

三、spider的编写
1、在爬虫文件内更改入口url
start_urls = [‘入口url’]
入口url必须在允许的域名范围内（allowed_domains）
2、在parse函数下编写规则

    def parse(self, response):
        # xpath最外层和内部的引号必须不一样
        movie_list = response.xpath('//div[@class="article"]//ol[@class="grid_view"]//li')
        for i_item in movie_list:
            douban_item = DoubanItem()
            douban_item['serial_number'] = i_item.xpath('.//div[@class="pic"]//em/text()').extract_first()
            douban_item['movie_name'] = i_item.xpath('.//div[@class="info"]//div[@class="hd"]//a/span[1]/text()').extract_first()
            content = i_item.xpath('.//div[@class="info"]//div[@class="bd"]/p[1]/text()').extract()
            content_s = ""
            for i_content in content:
                content_s += "".join((i_content.split())) + " "
            douban_item['introduction'] = content_s
            douban_item['star'] = i_item.xpath('.//div[@class="info"]//div[@class="bd"]//div[@class="star"]/span[2]/text()').extract_first()
            douban_item['comments_num'] = i_item.xpath('.//div[@class="info"]//div[@class="bd"]//div[@class="star"]/span[4]/text()').extract_first()
            douban_item['description'] = i_item.xpath('.//div[@class="info"]//div[@class="bd"]//p[@class="quote"]/span/text()').extract_first()
            # print(douban_item)
            yield douban_item
        next_link = response.xpath('//div[@class="article"]//span[@class="next"]/link/@href').extract()
        if next_link:
            next_link = next_link[0]
            yield scrapy.Request("https://movie.douban.com/top250" + next_link, callback=self.parse)

四、将数据存入MongoDB
1、在settings.py中做如下定义：
mongo_host = ‘127.0.0.1’
mongo_port = 27017
mongo_db_name = ‘xxx’
mongo_db_collection = ‘xxxxx’
2、取消settings.py中pipelines相关代码的注释
3、在pipelines.py中编写如下代码：

class DoubanPipeline(object):
    def __init__(self):
        host = mongo_host
        port = mongo_port
        dbname = mongo_db_name
        sheetname = mongo_db_collection
        client = pymongo.MongoClient(host=host, port=port)
        mydb = client[dbname]
        self.post = mydb[sheetname]
    def process_item(self, item, spider):
        data = dict(item)
        self.post.insert(data)
        return item

五、自定义下载中间件
1、代理ip（暂略）
2、user_agent

class my_useragent(object):
    def process_request(self, request, spider):
        USER_AGENT_LIST = [
            'MSIE (MSIE 6.0; X11; Linux; i686) Opera 7.23',
            'Opera/9.20 (Macintosh; Intel Mac OS X; U; en)',
            'Opera/9.0 (Macintosh; PPC Mac OS X; U; en)',
            'iTunes/9.0.3 (Macintosh; U; Intel Mac OS X 10_6_2; en-ca)',
            'Mozilla/4.76 [en_jp] (X11; U; SunOS 5.8 sun4u)',
            'iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2)',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20120813 Firefox/16.0',
            'Mozilla/4.77 [en] (X11; I; IRIX;64 6.5 IP30)',
            'Mozilla/4.8 [en] (X11; U; SunOS; 5.7 sun4u)'
        ]
        agent = random.choice(USER_AGENT_LIST)
        request.headers['User_Agent'] = agent

3、取消settings.py中downloader middlewares相关代码的注释，仿照格式添加自定义的中间件：
‘项目名称.middlewares.自定义下载中间件的名称’: 优先级,
优先级的值越小，优先度越高
不同的自定义中间件的优先级不能一样