Python汽车用户消费投诉数据爬取

最新推荐文章于 2022-01-13 15:52:14 发布

qq_48566899

最新推荐文章于 2022-01-13 15:52:14 发布

阅读量372

点赞数

分类专栏： python

本文链接：https://blog.csdn.net/qq_48566899/article/details/115562664

版权

python 专栏收录该内容

64 篇文章 0 订阅

订阅专栏

http://tousu.315che.com/tousulist/serial/7805/0/0/0/8.htm

from  selenium import webdriver
import requests
from lxml import etree
import os
if not os.path.exists('./汽车用户消费投诉数据'):
    os.mkdir('./汽车用户消费投诉数据')
#driver=webdriver.Firefox()
#driver.get('http://tousu.315che.com/tousudetail/104189/')       #任务一
wz='http://tousu.315che.com/tousulist/serial/7805/0/0/0/%d.htm'
ua = {'User-Agent':'Mozilla/5.0 (Mozilla/5.0 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36 Edg/88.0.705.53'}
for page in range(1,376):
    wz1=format(wz%page)
    hq = requests.get(url=wz1, headers=ua).text
    jx = etree.HTML(hq)
    lst=jx.xpath('//div[@class="tousu-filter-list"]/ul[@class="clearfix"]/li/a/@href')
    for i in lst:
        wz2=i
        hq2 = requests.get(url=wz2, headers=ua)
        hq2.encoding='utf-8'
        hq2=hq2.text
        jx = etree.HTML(hq2)
        lst2=jx.xpath('//div[@class="article box1"]/div[@class="complaints-appeal-info"]/p/text()')
        for n in lst2:
            dz='./汽车用户消费投诉数据/数据.csv'
            with open(dz,'a',encoding='utf-8')as fp:
                fp.write(n+'\n')
                fp.close()

具体步骤

进入爬取页面

获取详情页面url
即由http://tousu.315che.com/tousulist/serial/7805/0/0/0/8.htm
进入http://tousu.315che.com/tousudetail/117828/


wz='http://tousu.315che.com/tousulist/serial/7805/0/0/0/%d.htm'
ua = {'User-Agent':'Mozilla/5.0 (Mozilla/5.0 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36 Edg/88.0.705.53'}

hq = requests.get(url=wz1, headers=ua).text
 jx = etree.HTML(hq)
lst=jx.xpath('//div[@class="tousu-filter-list"]/ul[@class="clearfix"]/li/a/@href')

在这里插入图片描述

详情页面获取投诉信息

for i in lst:
        wz2=i
        hq2 = requests.get(url=wz2, headers=ua)
        hq2.encoding='utf-8'
        hq2=hq2.text
        jx = etree.HTML(hq2)
        lst2=jx.xpath('//div[@class="article box1"]/div[@class="complaints-appeal-info"]/p/text()')

在这里插入图片描述

翻页循环

for page in range(1,376):
    wz1=format(wz%page)
    hq = requests.get(url=wz1, headers=ua).text

在这里插入图片描述