【Python爬虫历程】简单爬虫实战练习1

今天来爬取拉钩网站来做实战练习

如果按照之前的直接使用的urlopen直接来爬取数据，可能会被网站的反爬虫机制给识别，如下代码

from urllib import request

resp=request.urlopen('https://www.lagou.com/jobs/positionAjax.json?city=%E6%9D%AD%E5%B7%9E&needAddtionalResult=false')
print(resp.read())

结果如下：

 

 这是因为被检测到时爬虫程序，会反馈出的一些的错误信息。

 

这就需要加上请求头以及请求数据等内容进行爬虫

# from urllib import request
#
# resp=request.urlopen('https://www.lagou.com/jobs/positionAjax.json?city=%E6%9D%AD%E5%B7%9E&needAddtionalResult=false')
# print(resp.read())

from urllib import request,parse

url='https://www.lagou.com/jobs/positionAjax.json?city=%E6%9D%AD%E5%B7%9E&needAddtionalResult=false'
headers = {
    'Accept':'application/json, text/javascript, */*; q=0.01',
    'Referer':'https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput=',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.4098.3 Safari/537.36',
    'Cookie':'user_trace_token=20190823091901-fd5287bd-c543-11e9-a504-5254005c3644; LGUID=20190823091901-fd528c54-c543-11e9-a504-5254005c3644; LG_LOGIN_USER_ID=e045585e162dc778c44ed0a28d10681ab56e6e164005627e; LG_HAS_LOGIN=1; index_location_city=%E6%9D%AD%E5%B7%9E; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%2216dbf0f0c381-064661d79b3e71-46531b29-1440000-16dbf0f0c39142%22%2C%22%24device_id%22%3A%2216dbf0f0c381-064661d79b3e71-46531b29-1440000-16dbf0f0c39142%22%7D; PRE_UTM=; PRE_HOST=; PRE_SITE=https%3A%2F%2Fwww.lagou.com%2Fhangzhou; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist%5Fpython%3FlabelWords%3D%26fromSearch%3Dtrue%26suginput%3D; _gid=GA1.2.2135446708.1585361539; _gat=1; _ga=GA1.2.1509779664.1566523142; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1585380500; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1585380500; X_HTTP_TOKEN=3277d2c74afe2e2c70508358516e9efc593003644e; LGSID=20200328152815-0eb6fadf-626e-43c1-8476-5e666830effe; LGRID=20200328152828-fc6dc1b3-201a-40ff-823a-8d235e266219'
}
data={
    'first':"true",
    'pn':1,
    'kd':"python"
}

req = request.Request(url,headers=headers,data=parse.urlencode(data).encode('utf-8'),method='POST')
resp = request.urlopen(req)
print(resp.read().decode('utf-8'))

结果放在json在线解析看一下：

在这里补充一下：如果访问错误之后可以尝试加一下请求头中的数据等，然后进行爬虫，每个网站的反爬虫的机制可能不一样。