pyspider爬虫框架之拉勾网招聘信息爬取

需求

1. 遍历所有职位目录
2. 点击职位分类，进入之后按照地区抓取，职位名称，发布时间，薪酬，工作年限要求，学历要求，招聘公司，所属行业，所处轮次
3. 进入职位详情页，抓取HR聊天意愿（用时），简历处理，活跃时段。

代码

代码有详细的注解，就不一步一步讲解了，再有个问题是，如果没有代理，慎用，IP容易被禁

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Created on 2018-08-17 14:49:15
# Project: lagou

from pyspider.libs.base_handler import *
import re
import datetime
from pymongo import MongoClient

DB_NAME = 'research'
DB_COL = 'lagou_recruit'
db = client[DB_NAME]
col = db[DB_COL]


class Handler(BaseHandler):

    crawl_config = {
        "headers": {
  "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"
                  },
        "proxy": "http://localhost:6666"  ## 搭建的代理服务
    }

    url = 'https://www.lagou.com/'

    def format_date(self, date):
        return datetime.datetime.strptime(date, '%Y-%m-%d')

    def get_today(self):
        return datetime.datetime.strptime(datetime.datetime.now().strftime('%Y-%m-%d'), '%Y-%m-%d')

    @every(minutes=24 * 60)
    def on_start(self):
        self.crawl(self.url, callback=self.index_page)



    @config(age=60)
    def index_page(self, response):
        page = response.etree

        ## 职位分类列表
        cat_list = page.xpath("//div[@class='mainNavs']/div[@class='menu_box']/div[@class='menu_sub dn']/dl")
        ## 遍历每个类别
        for each in cat_list:
            ## 大类
            b_title = each.xpath("./dt/span/text()")[0]
            print('-----------',b_title,'------------')

            ##子类列表
            sub_list = each.xpath("./dd/a")
            for sub in sub_list:
                sub_title = sub.xpath("./text()")[