需求
- 遍历所有职位目录
- 点击职位分类,进入之后按照地区抓取,职位名称,发布时间,薪酬,工作年限要求,学历要求,招聘公司,所属行业,所处轮次
- 进入职位详情页,抓取HR聊天意愿(用时),简历处理,活跃时段。
代码
代码有详细的注解,就不一步一步讲解了,再有个问题是,如果没有代理,慎用,IP容易被禁
#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Created on 2018-08-17 14:49:15
# Project: lagou
from pyspider.libs.base_handler import *
import re
import datetime
from pymongo import MongoClient
DB_NAME = 'research'
DB_COL = 'lagou_recruit'
db = client[DB_NAME]
col = db[DB_COL]
class Handler(BaseHandler):
crawl_config = {
"headers": {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"
},
"proxy": "http://localhost:6666" ## 搭建的代理服务
}
url = 'https://www.lagou.com/'
def format_date(self, date):
return datetime.datetime.strptime(date, '%Y-%m-%d')
def get_today(self):
return datetime.datetime.strptime(datetime.datetime.now().strftime('%Y-%m-%d'), '%Y-%m-%d')
@every(minutes=24 * 60)
def on_start(self):
self.crawl(self.url, callback=self.index_page)
@config(age=60)
def index_page(self, response):
page = response.etree
## 职位分类列表
cat_list = page.xpath("//div[@class='mainNavs']/div[@class='menu_box']/div[@class='menu_sub dn']/dl")
## 遍历每个类别
for each in cat_list:
## 大类
b_title = each.xpath("./dt/span/text()")[0]
print('-----------',b_title,'------------')
##子类列表
sub_list = each.xpath("./dd/a")
for sub in sub_list:
sub_title = sub.xpath("./text()")[