python多页爬取获取淘宝商品（莫小仙酸辣粉）的评论

目标：获取莫小仙酸辣粉的2000条评论

第一步：找到评论所存储的位置，如果找不到可以翻页后再查找

第二步：找到其header中的cookie,referer,user-agent参数
另外注意：url中表示页码变更的参数只有currentPage,如粗线所示

https://rate.tmall.com/list_detail_rate.htm?itemId=635947040812&spuId=0&sellerId=3994116855&order=3&currentPage=3&append=0&content=1&tagId=&posi=&picture=0&groupId=&ua=098%23E1hvrvvxvxIvUvCkvvvvvjiWPLSy6jtERFMpgjnEPmPZ1j3bP2zvtjiUn2FhsjYURvhvCvvvvvm%2BvpvEvvFrkRcDBn85dvhvmpvUNQoXeZo0v8QCvvyvm2026rEhByIgvpvhvvvvvUvCvvswPRld%2BYMwznsMEDu%2BvpvEvv12kHBZvhvBRvhvCvvvvvm%2BvpvEvvAYvK5ivC2mi9hvCvvv9UURvpvhvv2MMQvCvvOvCvvvphmgvpvIvvvvWhCvv8vvvUHHphvUl9vvvQCvpvACvvv2vhCv2RvvvUE%2BphvUOT9Cvv9vvUvGYJCg9I9CvvOUvvVva6wUvpCW2mgGvvawljc6D40fjLR1DfeaRFB%2Bm7zOaNoAdcZISWkQD404jomU%2BOkQ%2Bull8PoxfaoKD7UbDO0U%2Bb8rJoL6D7zZd3wspqVD64vBhL9Cvvpvvvvv39hvCvvhvvmvvpvWz2s7cDsNznswNCY4dvhvmpvWRgR7D9pTtgvCvCLwPVXuxrMwzPTLmlSSXY11zUC49p%3D%3D&itemPropertyId=&itemPropertyIndex=&userPropertyId=&userPropertyIndex=&rateQuery=&location=&needFold=0&_ksTS=1617946926249_1361&callback=jsonp1362

第三步：运用正则表达式，注意这里需要尝试将result打印一下，打印后的结果是与网页界面所看到的不同的

re.compile(’“rateContent”:"(.*?)",“fromMall”’)表示从rateContent到fronMall之间的文本

完整代码如下

import requests
import csv
import time
import re

def get_url(url):
    global page
    print(page)
    headers={
        'user-agent': '自己的user-agent',
        'cookie': '自己的cookie',
        'referer':'自己的'
    }

    response=requests.get(url=url,headers=headers,timeout=3)
    result=response.text
    return result

def parse_url():
    global dit1
    htmltext=get_url(url)
    Conment=[]
    comment=re.compile('"rateContent":"(.*?)","fromMall"')
    Comment=re.findall(comment,htmltext)
    for comments in Comment:
        dit1={
                # '日期':Date,
            '评论':comments
        }
        save()

def save():
    csv_writer.writerow(dit1)

def main():
    parse_url()

if __name__=='__main__':
    global page
    page=0
    f=open('D://莫小仙酸辣粉评价.csv','a+',encoding='utf-8',newline='')
    csv_writer = csv.DictWriter(f, fieldnames=['评论'])#'日期','购买商品种类',
    csv_writer.writeheader()
    for i in range(1,100):
        page+=1
        url='https://rate.tmall.com/list_detail_rate.htm?itemId=635947040812&spuId=0&sellerId=3994116855&order=3&currentPage='+str(i)+'&append=0&content=1&tagId=&posi=&picture=0&groupId=&ua=098%23E1hvrvvxvxIvUvCkvvvvvjiWPLSy6jtERFMpgjnEPmPZ1j3bP2zvtjiUn2FhsjYURvhvCvvvvvm%2BvpvEvvFrkRcDBn85dvhvmpvUNQoXeZo0v8QCvvyvm2026rEhByIgvpvhvvvvvUvCvvswPRld%2BYMwznsMEDu%2BvpvEvv12kHBZvhvBRvhvCvvvvvm%2BvpvEvvAYvK5ivC2mi9hvCvvv9UURvpvhvv2MMQvCvvOvCvvvphmgvpvIvvvvWhCvv8vvvUHHphvUl9vvvQCvpvACvvv2vhCv2RvvvUE%2BphvUOT9Cvv9vvUvGYJCg9I9CvvOUvvVva6wUvpCW2mgGvvawljc6D40fjLR1DfeaRFB%2Bm7zOaNoAdcZISWkQD404jomU%2BOkQ%2Bull8PoxfaoKD7UbDO0U%2Bb8rJoL6D7zZd3wspqVD64vBhL9Cvvpvvvvv39hvCvvhvvmvvpvWz2s7cDsNznswNCY4dvhvmpvWRgR7D9pTtgvCvCLwPVXuxrMwzPTLmlSSXY11zUC49p%3D%3D&itemPropertyId=&itemPropertyIndex=&userPropertyId=&userPropertyIndex=&rateQuery=&location=&needFold=0&_ksTS=1617946926249_1361&callback=jsonp1362'
        main()
        time.sleep(1)
    print('下载完成')

实现效果如图：