Python爬虫-DAY5淘宝页面爬取

最新推荐文章于 2024-10-01 22:51:33 发布

Lucky小小吴

最新推荐文章于 2024-10-01 22:51:33 发布

阅读量937

点赞数

分类专栏： Python

本文链接：https://blog.csdn.net/weixin_47126666/article/details/112979868

版权

python

Python 专栏收录该内容

22 篇文章 2 订阅

订阅专栏

申明：本章主要是个人的学习心得笔记

学习来源：中国MOOC课网

https://www.icourse163.org/learn/BIT-1001870001?tid=1461946455#/learn/content?type=detail&id=1238729654&cid=1259474386

遇到的一些编程问题：

1、只是输出了前面的表头：

可能性：

1、未能获取到文章信息

2、储存到列表中出错

3、输出出错

检查了很久，发现②③都没啥问题，最后去查了查淘宝的Robots

发现了淘宝是不允许爬取的，所以这里可以用一点小技巧——Cookie，对其进行爬虫

①在这里你需要在getHTMLText()添加一个元素——cookie

kv = {''}
r = requests.get(url, headers=kv,timeout = 30)

这样就🆗了

import requests
import re

#获取网页信息
def getHTMLText(url):
    try:
        kv = {'cookie':'miid=54413211854779671; cna=HRLpFzER4yYCARst14JDWCNn; thw=cn; t=db4804aa9a667a1bba5880d737e4cc35; _m_h5_tk=577622958c904e76bb652af2ae19f618_1611290682421; _m_h5_tk_enc=db23b8ad421ba8675eeb84cd43f66b00; xlly_s=1; cookie2=1698842a25b960385350377a6dbd9e49; _tb_token_=3aa9973717753; _samesite_flag_=true; sgcookie=E100EmoEKdhKZM9bnIQmko%2BZ0dYeTDle93doNmHmrY18KOVy3Ff%2FMftfkUeE2XvuEYM%2FFnUESwRjYGHeafJNxsCRrw%3D%3D; uc3=vt3=F8dCuAFVpbogSNAzelk%3D&lg2=UtASsssmOIJ0bQ%3D%3D&nk2=rXvxJKtliBeBGP0OJ0Nen%2FY%3D&id2=UU8A589XV%2FqXuQ%3D%3D; csg=c03d79ac; lgc=%5Cu5434%5Cu57F9%5Cu683918316397383; dnk=%5Cu5434%5Cu57F9%5Cu683918316397383; skt=86f39eecb8b986a3; existShop=MTYxMTI4MzE0OQ%3D%3D; uc4=id4=0%40U22HurUbZE6Ez1z0mWVIpooTlzSe&nk4=0%40r4qgLtVUE2SY5h4X%2FA%2FRt0pNT1NIYGPlpQaDww%3D%3D; tracknick=%5Cu5434%5Cu57F9%5Cu683918316397383; _cc_=VT5L2FSpdA%3D%3D; enc=GxQBlNuymk8JB2g7a9CVh%2FkUKah7wP3TbcbNDelegYedavxdIKuyw1Je%2B8obwhsN%2Fugd39AKOHDUiY9kdQzX0g%3D%3D; hng=CN%7Czh-CN%7CCNY%7C156; mt=ci=203_1; isg=BI6OVWWhYvPqqNbbjOASRnX232RQD1IJ2DD_2LjX-hFMGy51IJ-iGTTZU0d3GEoh; l=eBNCd-wmjj39xxJLBOfanurza77OSIRYYuPzaNbMiOCP92CB5nq5W6Gayu86C3GVh6YDR3Rj0APyBeYBqQAonxv92j-la_kmn; tfstk=cZPfBP4Y-nxj_IBwgr_yg5UvxNlOwpXSytizl--xKs5xoD101BoIzE2MVQHKP','user-agent':'Mozilla/5.0'}
        r = requests.get(url, headers=kv,timeout = 30)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        return r.text
    except:
        return ""

#获取关键商品信息到列表中
def parsePage(ilt, html):
    try:
        plt = re.findall(r'\"view_price\"\:\"[\d\.]*\"',html)
        tlt = re.findall(r'\"raw_title\"\:\".*?\"',html) #获取商品的货名
        for i in range(len(plt)):
            price = eval(plt[i].split(':')[1])
            title = eval(tlt[i].split(':')[1])
            ilt.append([price, title])
    except:
        print("")

#打印
def printGoodsList(ilt):
    tplt = "{:4}\t{:8}\t{:16}"
    print(tplt.format("序号","价格","商品名称"))
    count = 0
    for g in ilt:
        count = count + 1
        print(tplt.format(count,g[0],g[1]))

#主函数
def main():
    goods = '运动鞋'
    depth = 3  #成功后改成5
    start_url = 'https://s.taobao.com/search?q=' + goods
    infoList = []   #建立一个列表，存储信息
    for i in range(depth):
        try:
            url = start_url + '&s=' + str(44*i)
            html = getHTMLText(url)   #获取网页信息
            parsePage(infoList, html)
        except:
            continue
    printGoodsList(infoList)#打印信息

main()