2021-11-16爬取淘宝商品信息时如何获取cookie

爬取淘宝商品信息时如何获取cookie

一、基本环境

1.win10系统
2.火狐浏览器
3.编程软件anaconda
4.淘宝的robots：https://www.taobao.com/robots.txt

二、参考代码，嵩天python网络爬虫与信息提取

#CrowTaobaoPrice.py
import requests
import re

def getHTMLText(url):
    headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0',

    #获取cookie时不能用搜索商品后的cookie，必须用刚打开淘宝时的cookie
    'Cookie': '#获取cookie时不能用搜索商品后的cookie，我用的是刚打开淘宝时的cookie'
    }

    try:
        r = requests.get(url, timeout=30, headers=headers)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        #print(r.text)
        return r.text
    except:
        return ""
    
def parsePage(ilt, html):
    try:
        plt = re.findall(r'\"view_price\"\:\"[\d\.]*\"',html)
        tlt = re.findall(r'\"raw_title\"\:\".*?\"',html)
        for i in range(len(plt)):
            price = eval(plt[i].split(':')[1])
            title = eval(tlt[i].split(':')[1])
            ilt.append([price , title])
    except:
        print("")

def printGoodsList(ilt):
    tplt = "{:4}\t{:8}\t{:16}"
    print(tplt.format("序号", "价格", "商品名称"))
    count = 0
    for g in ilt:
        count = count + 1
        print(tplt.format(count, g[0], g[1]))
        
def main():
    goods = '书包 '
    depth = 2
    start_url = 'https://s.taobao.com/search?q=' + goods
                #https://s.taobao.com/search?q=
    infoList = []
    for i in range(depth):
        try:
            url = start_url + '&s=' + str(44*i)
            html = getHTMLText(url)
            parsePage(infoList, html)
        except:
            continue
    printGoodsList(infoList)
    
main()

三、如何获取cookie

1.确保当前页面是在淘宝网首页

2.获取cookie与headers，依次点击浏览器右上方三横