python翻页爬取京东商品评价
以小米手机评论为案例
1.找到目标的url
2.检查响应结果
3.解析json数据,发现相应数据无法成功解析
注意:json数据一般都是以大括号或中括号开头或者结尾的。这里我们发现,响应的json数据是有干扰信息的。我们需要将这些干扰信息去除,然后才能进行在线解析。(而我们在程序中可以通过正则实现将干扰信息去除)
解析结果如下:
由上图,我们可以得到目标数据的jsonpath语法
4.最后对比前三页的url,找到翻页的规律:
https://club.jd.com/comment/productPageComments.action?callback=fetchJSON_comment98&productId=10025642911830&score=0&sortType=5&page=0&pageSize=10&isShadowSku=0&fold=1
https://club.jd.com/comment/productPageComments.action?callback=fetchJSON_comment98&productId=10025642911830&score=0&sortType=5&page=1&pageSize=10&isShadowSku=0&rid=0&fold=1
https://club.jd.com/comment/productPageComments.action?callback=fetchJSON_comment98&productId=10025642911830&score=0&sortType=5&page=2&pageSize=10&isShadowSku=0&rid=0&fold=1
由上,找到翻页规律在于page参数
解析完毕,开始上代码:
import requests
import jsonpath
import re
import json
if __name__ == '__main__':
# 输入要爬取评论的页数
pages = int(input('请输入要爬取的页数:'))
# 创建for循环进行翻页
for i in range(pages):
page = i
# 确认目标的url
url = f'https://club.jd.com/comment/productPageComments.action?callback=fetchJSON_comment98&productId=10025642911830&score=0&sortType=5&page={page}&pageSize=10&isShadowSku=0&fold=1'
# 创建请求头参数
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36',
'Referer':'https://item.jd.com/',
'Cookie':'__jdu=738122922; shshshfpa=6d2e8f0f-4894-6aa0-4b01-cff3a242b1f4-1610016278; shshshfpb=zA27bQPLMCh6Pu%20y8W8M9ew%3D%3D; unpl=V2_ZzNtbUUHFBx9AEUAfR5YUGIFFl8RV0BAdVxFVHoeVQVkAhsPclRCFnUUR1RnGlUUZwYZX0FcQB1FCEdkeBBVAWMDE1VGZxBFLV0CFSNGF1wjU00zQwBBQHcJFF0uSgwDYgcaDhFTQEJ2XBVQL0oMDDdRFAhyZ0AVRQhHZH0bVQVkABpeQWdzEkU4dlxyHlUAYzMTbUNnAUEpD05VchlaSGEBG11BVEsWdjhHZHg%3d; __jdv=76161171|baidu-pinzhuan|t_288551095_baidupinzhuan|cpc|0f3d30c8dba7459bb52f2eb5eba8ac7d_0_6af8992e765d4753b12d1e210681208c|1611815549947; areaId=7; ipLoc-djd=7-420-421-0; __jda=122270672.738122922.1610016207.1610097257.1611815550.6; __jdc=122270672; shshshfp=985dab7abfa1bb52f9468aa423f30a12; 3AB9D23F7A4B3C9B=QGPEIVJE44XKXCOI2WZ6PAHNP2MMLBPQLIXQ4EWDPIL3O4SB6PWJXOAGGGTLLRLIZHO2AZRAC4UBTGFYFI76BQ5X4A; jwotest_product=99; JSESSIONID=CE22768EFA71FB9CD36F9B9BBFF2902C.s1; __jdb=122270672.8.738122922|6.1611815550; shshshsID=df7926d2035282c451cca5bca0543fd2_8_1611817303661'
}
# 发送请求,获取响应
response = requests.get(url,headers=headers)
# 通过正则去除json外多余的部分
content = response.text
print(content)
str_data = re.findall(r'fetchJSON_comment98\((.*?)\);',content)[0]
# 将数据转换成python数据
py_data = json.loads(str_data)
# 提取id和评论
id_data = jsonpath.jsonpath(py_data,'$.comments[*].nickname')
comment_data = jsonpath.jsonpath(py_data,'$.comments[*].content')
for i in range(len(id_data)):
dict_ = {}
dict_[id_data[i]] = comment_data[i]
json_data = json.dumps(dict_,ensure_ascii=False)+',\n'
with open(f'翻页爬取京东商品用户名及评论共{pages}页.json','a',encoding='utf-8')as f:
f.write(json_data)