day19hw

最新推荐文章于 2024-09-30 14:16:32 发布

weixin_54146878

最新推荐文章于 2024-09-30 14:16:32 发布

阅读量134

点赞数

文章标签： python chrome 爬虫

本文链接：https://blog.csdn.net/weixin_54146878/article/details/127431415

版权

这段代码展示了如何使用requests和BeautifulSoup库从京东搜索结果页面抓取电子商品的名称，然后通过Selenium模拟浏览器行为获取商品详情页的用户评价（好评、中评、差评）。数据被写入CSV文件进行记录。

摘要由CSDN通过智能技术生成

import requests
from bs4 import BeautifulSoup
from selenium.webdriver import Chrome
from re import findall
import csv
def ak():
    headers={'user-agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'}
    r = requests.get('https://search.jd.com/Search?keyword=%E7%94%B5%E8%84%91&enc=utf-8&wq=%E7%94%B5%E8%84%91&pvid=1d04bc957fe048bca55a829aceed9df3',headers=headers)
    soup = BeautifulSoup(r.text,'lxml')
    ul1 = soup.select('#J_goodsList>.gl-warp clearfix')
    data1 = []
    for x in ul1:
        ht = x.select('div>div>a').attrs['href']
        r1 = requests.get('ht')
        s1 = BeautifulSoup(r1.text,'lxml')
        names = s1.select_one('.p-parameter>ul>li')
        b=Chrome()
        cli = b.find_elements_by_css_selector('.detail>div>div>ul')[-2]
        cli.click()
        good0 = s1.select_one('.tab-main small>.filter-list')[4]
        good = good0.findall(r'<em>(.+)<em>')
        mid0 = s1.select_one('.tab-main small>.filter-list')[5]
        mid = mid0.findall(r'<em>(.+)<em>')
        bad0 = s1.select_one('.tab-main small>.filter-list')[6]
        bad = bad0.findall(r'<em>(.+)<em>')
        data1.append([names,good,mid,bad])
        writer.writerows(data1)
        print('一页数据获取成功!')


if __name__ == '__main__':
    writer = csv.writer(open('files/computer.csv', 'w', encoding='utf-8', newline=''))
    writer.writerow(['名称', '好评', '中评', '差评'])