Python爬取拉勾网数据，实现数据保存至文件或数据库

最新推荐文章于 2020-03-17 14:15:19 发布

tiandi345

最新推荐文章于 2020-03-17 14:15:19 发布

阅读量1.4k

点赞数

分类专栏： python 文章标签： lagou

本文链接：https://blog.csdn.net/tiandi345/article/details/85870864

版权

本文展示了如何使用Python爬取拉勾网的职位数据，数据包括公司ID、公司全称、公司简称等信息，并将数据保存到txt文件中。此外，文章还提及了将数据保存至MySQL数据库的相关操作。

摘要由CSDN通过智能技术生成

Python爬取拉钩网数据一一一保存数据至文件

import requests
import time
import json

def get_data(url,page,lang_name):
header = {
'Content-Language': 'zh-CN',
'Content-Type': 'application/json;charset=UTF-8',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36',
'Referer': 'https://www.lagou.com/jobs/list_Hadoop?px=default&city=%E5%85%A8%E5%9B%BD'}
Cookies = {
'Cookie': '_ga=GA1.2. 1864083170.1542538584; user_trace_token=20181118185622-96224099-eb20-11e8-a648-525400f775ce; LGUID=20181118185622-96224871-eb20-11e8-a648-525400f775ce; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221672676ed41274-09ce2d0583d289-3f674604-2073600-1672676ed42739%22%2C%22%24device_id%22%3A%221672676ed41274-09ce2d0583d289-3f674604-2073600-1672676ed42739%22%7D; showExpriedIndex=1; showExpriedCompanyHome=1; showExpriedMyPublish=1; index_location_city=%E6%B7%B1%E5%9C%B3; WEBTJ-ID=20181207230154-1678930781a414-046c47d4f0aadf-3f674604-2073600-1678930781b154; _gid=GA1.2.538058677.1544194914; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1543851889,1543853652,1544080965,1544194915; LGSID=20181207230155-098306d9-fa31-11e8-8ce7-5254005c3644; PRE_UTM=m_cf_cpt_baidu_pc; PRE_HOST=www.baidu.com; PRE_SITE=https%3A%2F%2Fwww.baidu.com%2Fs%3Fwd%3D%25E6%258B%2589%25E9%2592%25A9%25E7%25BD%2591%26rsv_spt%3D1%26rsv_iqid%3D0xa0da035100015b9c%26issp%3D1%26f%3D8%26rsv_bp%3D0%26rsv_idx%3D2%26ie%3Dutf-8%26tn%3Dbaiduhome_pg%26rsv_enter%3D1%26rsv_sug3%3D6%26rsv_sug1%3D5%26rsv_sug7%3D100; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Flp%2Fhtml%2Fcommon.html%3Futm_source%3Dm_cf_cpt_baidu_pc; JSESSIONID=ABAAABAAAGGABCB09F792E3C88B0DF4712407BBD57C7D1A; _putrc=3CC2AD8D77BBA3E1; login=true; unick=%E6%9D%A8%E8%83%9C; hasDeliver=390; gate_login_token=c200afceae9db4a3f7c79f4414fb4daccc005448