爬取淘宝产品评论及分析
需求
爬取淘宝天猫店,某个单品的评论,根据评论内容生成词云,以及对评论做情感分析
分析
1.登录淘宝。找到某个产品,点击评论。
2.进入评论页面后,按F12调出开发者工具,在Network 找list_det。打开后点击Preview。可以找到评论数据,数据类型是json。一页评论20条(怎么判断是ajax异步加载?个人判断是点击第二页的时,ie中的url没有改变从而推测是ajax异步加载。)
-
构建ajax异步加载分析:
ajax请问url 就是红色框的Request URL ;用的是get 请求。可以用request库中request.get()。
分析url内容:Request URL:
https://rate.tmall.com/list_detail_rate.htm?itemId=576746298719&spuId=1042574054&sellerId=2328901391&order=3¤tPage=1&append=0&content=1&tagId=&posi=&picture=&groupId=&ua=098%23E1hvAQvEvbQvUvCkvvvvvjiPnLSytj1bPFLUsjnEPmPwgjECRsMyQjtWPsMZQjEV3QhvCvvhvvvEvpCWvPiIvvwdD7zwaXTAVA1ldExrVut%2BCNLy0bvqrADn9W2%2BFfmtEpch1RFE%2BFIlBqevrf9gKFnfIWFvgEkXw6cv6R29DCOrvGjxs4hZebyCvm9vvvvvphvvvvvv9X1vpvFjvvmmZhCv2CUvvUEpphvWOpvv9DCvpv13kphvC99vvOCgoTyCvv9vvUvTZhOHSQhCvvOvCvvvphvPvpvhvv2MMsyCvvpvvvvviQhvCvvv9UU%3D&needFold=0&_ksTS=1597997731033_540&callback=jsonp541url构成:
itemId: 576746298719
spuId: 1042574054
sellerId: 2328901391
order: 3
currentPage: 1
append: 0
content: 1
tagId:
posi:
picture:
groupId:
ua: 098#E1hvAQvEvbQvUvCkvvvvvjiPnLSytj1bPFLUsjnEPmPwgjECRsMyQjtWPsMZQjEV3QhvCvvhvvvEvpCWvPiIvvwdD7zwaXTAVA1ldExrVut+CNLy0bvqrADn9W2+FfmtEpch1RFE+FIlBqevrf9gKFnfIWFvgEkXw6cv6R29DCOrvGjxs4hZebyCvm9vvvvvphvvvvvv9X1vpvFjvvmmZhCv2CUvvUEpphvWOpvv9DCvpv13kphvC99vvOCgoTyCvv9vvUvTZhOHSQhCvvOvCvvvphvPvpvhvv2MMsyCvvpvvvvviQhvCvvv9UU=
needFold: 0
_ksTS: 1597997731033_540
callback: jsonp541
解释:itemId:商品ID,spuId:产品id,sellerId:卖家ID,currentPage:评论页数,ua: 用户名(这个是淘宝有它自己的算法,动态的),_ksTS:时间戳,callback:不知道什么鬼。。。
继续分析url构成及请求:
第一页url
Request URL:
https://rate.tmall.com/list_detail_rate.htm?itemId=576746298719&spuId=1042574054&sellerId=2328901391&order=3¤tPage=1&append=0&content=1&tagId=&posi=&picture=&groupId=&ua=098%23E1hvAQvEvbQvUvCkvvvvvjiPnLSytj1bPFLUsjnEPmPwgjECRsMyQjtWPsMZQjEV3QhvCvvhvvvEvpCWvPiIvvwdD7zwaXTAVA1ldExrVut%2BCNLy0bvqrADn9W2%2BFfmtEpch1RFE%2BFIlBqevrf9gKFnfIWFvgEkXw6cv6R29DCOrvGjxs4hZebyCvm9vvvvvphvvvvvv9X1vpvFjvvmmZhCv2CU