网络爬虫--24.【selenium实战】实现拉勾网爬虫之--分析接口获取数据

一. 思路概述

1.拉勾网采用Ajax技术，加载网页时会向后端发送Ajax异步请求，因此首先找到数据接口；

2.后端会返回json的数据，分析数据，找到单个招聘对应的positionId，拼接出具体的url；

3.解析url，使用xpath获取数据。

二. 分析数据接口

三. 详细代码

#encoding: utf-8

import requests
from lxml import etree
import time
import re

headers = {
   
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36',
    "Referer": 'https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput=',
    "Cookie": 'user_trace_token=20200524155122-083f0df9-307f-44ee-8266-fceb6af1028b; _ga=GA1.2.728245592.1590306686; LGUID=20200524155125-3d15f82a-9111-4024-98da-9c1422b856dd; JSESSIONID=ABAAAECABFAACEA42866F7C54F46B971C0B595641303F73; WEBTJ-ID=20200524155129-17245aa50d74ec-0946ac3961652a-f7d1d38-1327104-17245aa50d980; RECOMMEND_TIP=true; _gid=GA1.2.369120051.1590306690; index_location_city=%E5%8C%97%E4%BA%AC; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%2217245aab6d623b-09a44787242115-f7d1d38-1327104-17245aab6d7237%22%2C%22%24device_id%22%3A%2217245aab6d623b-09a44787242115-f7d1d38-1327104-17245aab6d7237%22%7D; sajssdk_2015_cross_new_user=1; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1590306685,1590324851; LGSID=20200524205410-e3cbacc7-ae0f-4d56-a775-67d45085e31d; PRE_UTM=m_cf_cpt_baidu_pcbt; PRE_HOST=sp0.baidu.com; PRE_SITE=https%3A%2F%2Fsp0.baidu.com%2F9q9JcDHa2gU2pMbgoY3K%2Fadrc.php%3Ft%3D06KL00c00fA1VH%5F0w3dM0FNkUsKEU%5FGX00000rUBx7C00000xs000y.THL0oUhY1x60UWY3rj04n1cdnNtzndqCmyqxTAT0T1dhmymYmhm1P10snH6dmvR10ZRqfbwDfHuDPWnsnRP7f1uDPbfLrDc1rHPDPjfknY7ArjD0mHdL5iuVmv-b5Hn1PWRzP1DYnHThTZFEuA-b5HDv0ARqpZwYTZnlQzqLILT8UA7MULR8mvqVQvk9UhwGUhTVTA7Muiqsmzq1uy7zmv68pZwVUjqdIAdxTvqdThP-5ydxmvuxmLKYgvF9pywdgLKWmMf0mLFW5HR4PWRs%26tpl%3Dtpl%5F11534%5F22672%5F17382%26l%3D1518133990%26attach%3Dlocation%253D%2526linkName%253D%2525E6%2525A0%252587%2525E5%252587%252586%2525E5%2525A4%2525B4%2525E9%252583%2525A8-%2525E6%2525A0%252587%2525E9%2525A2%252598-%2525E4%2525B8%2525BB%2525E6%2525A0%252587%2525E9%2525A