scrapy实战——利用CrawlSpider爬取腾讯社招全部岗位信息（进行有一定深度的爬取）

经过scrapy的简单学习，我们实现这样一个爬虫：爬取腾讯社招的全部岗位信息，将粗略的大致信息保存在tencent.json文件中，将岗位的进一步具体信息（职责、要求）保存在positiondescribe.json文件中。
即，我们需要两个item进行页面信息的保存，同时要继承CrawlSpider对页面链接进行相应提取。
项目目录如下：(创建名为TencntSpider的项目)

TencentSpider
    │  items.py
    │  middlewares.py
    │  pipelines.py
    │  settings.py
    │  __init__.py
    │
    ├─spiders
    │  │  tencent.py
    │  │  __init__.py
    │  │
    │  └─__pycache__
    │          tencent.cpython-36.pyc
    │          __init__.cpython-36.pyc
    │
    └─__pycache__
            items.cpython-36.pyc
            pipelines.cpython-36.pyc
            settings.cpython-36.pyc
            __init__.cpython-36.pyc

难点主要在：
1. 对多个item的处理：在pipelines文件中对传入的item做判断：利用class.name的方法可以对类名进行判断！
2. 对于spider中rules的书写，要清楚我们需要过滤的链接或页面。
3. 对于爬虫文件中parse方法的书写：要记得一旦继承了CrawlSpider类便不能再重写parse方法，我们要自己编写parse方法，因为我们提取了链接之后，要对链接进行跟进处理，进入详细信息的页面，所以我们要写两个parse方法！
items.py代码如下：

# items.py
# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class TencentspiderItem(scrapy.Item):
    # 职位名
    positionName = scrapy.Field()
    # 详情链接
    positionLink = scrapy.Field()
    # 职位类别
    positionType = scrapy.Field()
    # 招聘人数
    peopleNum = scrapy.Field()
    # 工作地点
    workLocation = scrapy.Field()
    # 发布时间
    publishTime = scrapy.Field()

class PositionDescribe(scrapy.Item):
    # 职位名
    positionName = scrapy.Field()
    # 职位类别
    positionType = scrapy.Field()
    # 招聘人数
    peopleNum = scrapy.Field()
    # 工作地点
    workLocation = scrapy.Field()
    # 职责
    duty = scrapy.Field()
    # 要求
    requirement = scrapy.Field()

tencent.py代码如下：

# tencent.py
# -*- coding: utf-8 -*-
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from TencentSpider.items import TencentspiderItem
from TencentSpider.items import PositionDescribe


class TencentSpider(CrawlSpider):
    name = 'tencent'
    allowed_domains = ['hr.tencent.com']
    start_urls = ['https://hr.tencent.com/position.php?&start=0#a']

    rules = (
        Rule(LinkExtractor(allow=r'&start=\d+'), callback='tencentParse', follow=True),
        Rule(LinkExtractor(allow=r'/position_detail.php?'), callback='positionParse', follow=True)
    )

    def tencentParse(self, response):
        jobs_list = response.xpath('//tr[@class="even"or@class="odd"]')

        for node in jobs_list:
            item = TencentspiderItem()
            name = node.xpath('./td[1]/a/text()').extract()[0]
            link = node.xpath('./td[1]/a/@href').extract()[0]
            type = ''.join(node.xpath('./td[2]/text()').extract())
            num = node.xpath('./td[3]/text()').extract()[0]
            location = node.xpath('./td[4]/text()').extract()[0]
            date = node.xpath('./td[5]/text()').extract()[0]
            item['positionName'] = name
            item['positionLink'] = 'https://hr.tencent.com/' + str(link)
            item['positionType'] = type
            item['peopleNum'] = num
            item['workLocation'] = location
            item['publishTime'] = date

            yield item

    def positionParse(self, response):
        item = PositionDescribe()
        name = response.xpath('//td[@id="sharetitle"]/text()').extract()
        location = response.xpath('//tr[@class="c bottomline"]/td[1]/text()').extract()
        type = response.xpath('//tr[@class="c bottomline"]/td[2]/text()').extract()
        num = response.xpath('//tr[@class="c bottomline"]/td[3]/text()').extract()
        s = ''
        duties = response.xpath('//table//tr[3]//ul/li/text()').extract()
        for duty in duties:
            s += duty
        requirements = response.xpath('//table//tr[4]//ul/li/text()').extract()
        q = ''
        for require in requirements:
            q += require

        # 职位名
        item['positionName'] = name
        # 职位类别
        item['positionType'] = type
        # 招聘人数
        item['peopleNum'] = num
        # 工作地点
        item['workLocation'] = location
        # 职责
        item['duty'] = s
        # 要求
        item['requirement'] = q

        yield item

pipelines.py代码如下：

# pipelines.py
# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
import json

class TencentspiderPipeline(object):

    def __init__(self):
        # isinstance
        self.file = open('tencent.json', 'a', encoding='utf-8')
        self.file2 = open('positiondescribe.json', 'a', encoding='utf-8')

    def process_item(self, item, spider):
        if item.__class__.__name__ == 'TencentspiderItem':
            jsontext = json.dumps(dict(item), ensure_ascii=False) + ',\n'
            self.file.write(jsontext)
        else:
            jsontext = json.dumps(dict(item), ensure_ascii=False) + ',\n'
            self.file2.write(jsontext)

        return item

    def close_spider(self, spider):
        self.file.close()
        self.file2.close()

主要便是这三个文件，同时爬取之后，会生成两个json文件。