使用Selenium进行高效网页数据采集

最新推荐文章于 2024-09-06 10:01:05 发布

Peter高效办公有大招

最新推荐文章于 2024-09-06 10:01:05 发布

阅读量686

点赞数 3

文章标签： selenium 测试工具 web

本文链接：https://blog.csdn.net/qq_32516133/article/details/140217144

版权

1. Selenium简介

Selenium是一个强大的Web自动化测试工具,同时也是一个优秀的数据采集工具。它可以模拟真实用户操作浏览器,处理动态加载的内容,填写表单,点击按钮等,这使得它在处理复杂网页时特别有优势。

2. 安装Selenium

首先,我们需要安装Selenium库和WebDriver:

pip install selenium

然后,下载与您的浏览器版本匹配的WebDriver(如ChromeDriver),并将其添加到系统PATH中。

3. 基本使用

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 初始化WebDriver
driver = webdriver.Chrome()

# 打开网页
driver.get("https://www.example.com")

# 查找元素
element = driver.find_element(By.ID, "example-id")

# 点击元素
element.click()

# 输入文本
input_element = driver.find_element(By.NAME, "search")
input_element.send_keys("Selenium")

# 等待元素出现
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "result")))

# 获取文本
text = element.text

# 关闭浏览器
driver.quit()

4. 处理动态内容

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 等待动态内容加载
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.ID, "dynamic-content")))

# 获取动态内容
content = element.text

driver.quit()

5. 处理下拉菜单

from selenium.webdriver.support.ui import Select

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 查找下拉菜单元素
select_element = driver.find_element(By.ID, "dropdown")

# 创建Select对象
select = Select(select_element)

# 选择选项
select.select_by_visible_text("Option 1")
# 或者
select.select_by_value("value1")
# 或者
select.select_by_index(1)

driver.quit()

6. 处理iframe

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 切换到iframe
iframe = driver.find_element(By.TAG_NAME, "iframe")
driver.switch_to.frame(iframe)

# 在iframe中操作
element = driver.find_element(By.ID, "inside-iframe")
element.click()

# 切回主文档
driver.switch_to.default_content()

driver.quit()

7. 处理多个窗口

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 打开新窗口
driver.execute_script("window.open('');")

# 切换到新窗口
driver.switch_to.window(driver.window_handles[-1])

# 在新窗口中操作
driver.get("https://www.anotherdomain.com")

# 切回原窗口
driver.switch_to.window(driver.window_handles[0])

driver.quit()

8. 处理JavaScript

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 执行JavaScript
driver.execute_script("alert('Hello, Selenium!');")

# 获取页面标题
title = driver.execute_script("return document.title;")

# 滚动到页面底部
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

driver.quit()

9. 处理弹出窗口

from selenium.webdriver.common.alert import Alert

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 切换到弹出窗口
alert = Alert(driver)

# 获取弹出窗口文本
alert_text = alert.text

# 接受弹出窗口
alert.accept()

# 或者取消弹出窗口
# alert.dismiss()

driver.quit()

10. 实际应用示例：爬取商品信息

让我们通过一个实际的例子来展示如何使用Selenium爬取电商网站的商品信息:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import csv

def scrape_product_info(url):
    driver = webdriver.Chrome()
    driver.get(url)

    # 等待商品列表加载
    wait = WebDriverWait(driver, 10)
    products = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "product-item")))

    product_data = []

    for product in products:
        try:
            name = product.find_element(By.CLASS_NAME, "product-name").text
            price = product.find_element(By.CLASS_NAME, "product-price").text
            rating = product.find_element(By.CLASS_NAME, "product-rating").text
            
            product_data.append({
                "name": name,
                "price": price,
                "rating": rating
            })
        except:
            continue

    driver.quit()
    return product_data

def save_to_csv(data, filename):
    keys = data[0].keys()
    with open(filename, 'w', newline='', encoding='utf-8') as output_file:
        dict_writer = csv.DictWriter(output_file, keys)
        dict_writer.writeheader()
        dict_writer.writerows(data)

# 使用函数
url = "https://www.example-ecommerce.com/products"
product_info = scrape_product_info(url)
save_to_csv(product_info, "product_data.csv")