使用Spark分析拉勾网招聘信息(二): 获取数据

要获取什么样的数据?

我们要获取的数据,是指那些公开的,可以轻易地获取地数据.如果你有完整的数据集,肯定是极好的,但一般都很难通过还算正当的方式轻易获取.单就本系列文章要研究的实时招聘信息来讲,能获取最近一个月的相关信息,已是足矣.

如何获取数据?

爬虫,也是可以的,作为一个备选方案.但是,我注意到拉勾网本身的数据,是通过ajax请求更新的,所以批量获取变得更加简单.基于ajax请求来获取数据,方式有很多,这里我演示其中的自认为较为简单通用的一种: 使用 curl 模拟 ajax 请求获取数据.

注意,以下的步骤演示全部基于 Mac 版的 * Google Chrome* 浏览器,其他浏览器部分操作可能会有些许差异.最后一步会给出 提取出的通用 curl 脚本,直接其实也是可以的,如果对步骤不是很关心.

1.找到目标城市和目标职位,然后按”最新排序”,参考链接: http://www.lagou.com/jobs/list_iOS?px=new&city=北京#order

2.双指击/右击 页面,弹出快捷菜单,选择”检查”,以进入浏览器调试界面,切换到调试器的 network -> xhr 标签下.

3.cmd + R 刷新页面,此时会捕捉到此页面发出的xhr请求.找到 http://www.lagou.com/jobs/positionAjax.json 开头的请求,并双指击/右击,选择 copy as cUrl.

这个 curl代码非常长,对于本次分析来说,最关键的是 末尾的 pn=1&kd=iOS,分别代表着页面和职位,动态设置,即可获取更多职位的更多数据了,文章的其他篇幅,会单独分析.

curl 'http://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: user_trace_token=20160522122749-8a4d6717-1fd5-11e6-963e-5254005c3644; LGUID=20160522122749-8a4d6cb3-1fd5-11e6-963e-5254005c3644; tencentSig=8513357824; LGMOID=20160818212815-33C56329AA2FB6D809D557FD6CC1DE3C; JSESSIONID=E0C5692414B160F3BEFFCFF8D240B693; _gat=1; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=http%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; ctk=1472587366; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1470032014,1470032429,1471526897,1471752875; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1472587368; LGSID=20160831035434-92fffed3-6eeb-11e6-8c02-5254005c3644; LGRID=20160831040247-b90762b6-6eec-11e6-a745-525400f775ce; _ga=GA1.2.1654849521.1463891270; SEARCH_ID=506c8097626343ab9b5a2c1139807a2c' -H 'Origin: http://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: http://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data 'first=true&pn=1&kd=iOS' --compressed

4.讲上一步中的curl指令复制到终端,桥下回车键,即可看到输出.

{"success":true,"requestId":null,"msg":null,"resubmitToken":null,"content":{"pageNo":1,"pageSize":15,"positionResult":{"totalCount":974,"resultSize":15,"locationInfo":{"city":"北京","district":null,"queryByGisCode":false,"businessZone":null,"locationCode":null},"queryAnalysisInfo":{"positionName":"ios","companyName":null,"usefulCompany":false,"industryName":null},"strategyProperty":{"name":"dm-csearch-newSimScorer","id":1},"result":[{"companyId":129801,"companyShortName":"言之有物科技","createTime":"2016-08-30 19:28:12","positionId":1857486,"positionAdvantage":"一线公司，技术驱动，免费三餐，超期望回报","salary":"25k-50k","score":0,"workYear":"不限","education":"本科","city":"北京","positionName":"iOS高级研发工程师/Lead","companyLogo":"i/image/M00/43/4E/CgqKkVeDGsuAXz0gAAA4XeGAAHQ390.png","financeStage":"成长型(A轮)","industryField":"移动互联网,电子商务","jobNature":"全职","approve":1,"companySize":"15-50人","district":null,"companyLabelList":["股票期权","扁平管理","美女多","领导好"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"19:28发布","gradeDescription":null,"companyFullName":"北京言之有物科技有限公司","businessZones":null,"imState":"today","lastLogin":1472556472000,"publisherId":5092848,"explain":null,"plus":null,"pcShow":0},{"companyId":133,"companyShortName":"猎豹移动","createTime":"2016-08-30 19:09:34","positionId":2151896,"positionAdvantage":"明星产品 超赞年终奖 靠谱领导","salary":"15k-30k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"iOS","companyLogo":"image1/M00/39/70/CgYXBlWo3nqABJTsAADJ3hn5gmE062.jpg","financeStage":"上市公司","industryField":"移动互联网,信息安全","jobNature":"全职","approve":1,"companySize":"500-2000人","district":"朝阳区","companyLabelList":["带薪年假","美女前台","超赞年终奖","一公里工作圈"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"19:09发布","gradeDescription":null,"companyFullName":"北京金山网络科技有限公司","businessZones":["姚家园","十里堡","高碑店"],"imState":"today","lastLogin":1472555392000,"publisherId":129969,"explain":null,"plus":null,"pcShow":0},{"companyId":107608,"companyShortName":"MUM计算机","createTime":"2016-08-30 19:03:24","positionId":1963945,"positionAdvantage":"帮助程序员赴美做IT，享受高薪高品质生活","salary":"10k-20k","score":0,"workYear":"不限","education":"本科","city":"北京","positionName":"IOS程序员赴美项目推广员","companyLogo":"i/image/M00/00/C2/CgqKkVZVHmSAWPtRAASUg0iUVuI932.jpg","financeStage":"初创型(不需要融资)","industryField":"教育","jobNature":"全职","approve":0,"