使用python selenium实现url不变的情况下，翻页爬取页面数据

网络安全小时光

已于 2023-03-06 15:51:58 修改

阅读量3.1k

点赞数 2

分类专栏： Python 文章标签： python selenium 开发语言

于 2023-03-06 15:35:56 首次发布

本文链接：https://blog.csdn.net/zg_111/article/details/129363802

版权

Python 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

import requests
from bs4 import BeautifulSoup
import openpyxl
from selenium import webdriver
from selenium.webdriver.common.by import By
import time

wt = openpyxl.Workbook()
ws = wt.create_sheet('sheet11')
ws.append(['LIST'])
browser = webdriver.Edge()
url = 'https://www.freebuf.com/articles/security-management'
html = browser.get(url)
# print(browser.page_source)
nextpagebutton = browser.find_element(By.XPATH, "//li[@class = ' ant-pagination-next']/a")
for r in range(1,16):
    soup = BeautifulSoup(browser.page_source, 'html.parser')
    spans = soup.find_all('span', class_ = 'title text-line-1')
    browser.execute_script("arguments[0].click();" ,nextpagebutton)
    time.sleep(1)


    print('--------------------正在打印第' + str(r) + '页数据---------------------------')
    for span in spans:
        LIST = span.text.strip()
        print(LIST)
        ws.append([LIST])

print('success')
browser.close()
wt.save('demo.xlsx')
wt.close()