Python网页正文提取神器: readability库详解

最新推荐文章于 2025-02-27 22:15:38 发布

engchina

最新推荐文章于 2025-02-27 22:15:38 发布

阅读量2k

点赞数 16

分类专栏： LINUX 文章标签： python 开发语言 readability

本文链接：https://blog.csdn.net/engchina/article/details/140179476

版权

LINUX 专栏收录该内容

851 篇文章

订阅专栏

Python网页正文提取神器: readability库详解

在网络爬虫和数据分析领域,我们经常需要从HTML页面中提取主要内容,去除广告、导航栏等无关元素。readability是一个强大的Python库,专门用于从HTML中提取文章的主要内容和元数据。本文将详细介绍readability库的使用方法和基本概念。

1. readability简介

readability-lxml是Mozilla开发的Readability.js的Python移植版本。它使用启发式算法来识别和提取网页中的主要内容,同时去除广告、侧边栏等干扰元素。

主要特点:

从HTML中提取文章正文
识别文章标题、作者、发布日期等元数据
支持多种语言
可以处理不规范的HTML

2. 安装

使用pip安装readability-lxml:

pip install readability-lxml

3. 基本使用

3.1 提取文章内容

from readability import Document
import requests

# 获取网页内容
url = "https://example.com/article"
response = requests.get(url)
html_content = response.text

# 创建Document对象
doc = Document(html_content)

# 获取文章标题
title = doc.title()

# 获取文章正文
content = doc.summary()

print(f"标题: {title}")
print("正文:")
print(content)

3.2 获取元数据

# 获取文章短标题
short_title = doc.short_title()

# 获取文章描述
description = doc.description()

print(f"短标题: {short_title}")
print(f"描述: {description}")

4. 高级功能

4.1 自定义选项

readability允许你自定义一些参数来调整提取算法:

from readability import Document

options = {
    'min_text_length': 100,  # 最小文本长度
    'retry_length': 250,     # 重试时的最小长度
    'char_threshold': 500,   # 字符阈值
    'keep_classes': [],      # 保留的CSS类名
    'clean_conditionally': True,  # 是否进行条件清理
    'remove_empty_nodes': True,   # 是否移除空节点
}

doc = Document(html_content, **options)
content = doc.summary()

4.2 处理非标准HTML

readability能够处理一些不太规范的HTML:

from readability import Document

# 处理缺少closing tags的HTML
html = "<html><body><p>This is a paragraph<p>This is another paragraph</body></html>"
doc = Document(html)
content = doc.summary()
print(content)

5. 实际应用示例

5.1 批量提取新闻文章

import requests
from readability import Document
import csv

def extract_article(url):
    response = requests.get(url)
    doc = Document(response.text)
    return {
        'title': doc.title(),
        'content': doc.summary(),
        'url': url
    }

urls = [
    "https://example.com/news1",
    "https://example.com/news2",
    "https://example.com/news3"
]

articles = [extract_article(url) for url in urls]

# 将结果保存到CSV文件
with open('articles.csv', 'w', newline='', encoding='utf-8') as file:
    writer = csv.DictWriter(file, fieldnames=['title', 'content', 'url'])
    writer.writeheader()
    writer.writerows(articles)

5.2 创建简单的阅读模式

from flask import Flask, request, render_template_string
from readability import Document
import requests

app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def reader_mode():
    if request.method == 'POST':
        url = request.form['url']
        response = requests.get(url)
        doc = Document(response.text)
        title = doc.title()
        content = doc.summary()
        return render_template_string("""
            <h1>{{ title }}</h1>
            {{ content|safe }}
        """, title=title, content=content)
    return '''
        <form method="post">
            URL: <input type="text" name="url">
            <input type="submit" value="Read">
        </form>
    '''

if __name__ == '__main__':
    app.run(debug=True)

6. 性能优化

对于大量网页的处理,可以考虑以下优化策略:

使用异步请求库如aiohttp来并发获取网页内容
使用多进程处理大量文章
将提取的内容缓存到数据库中,避免重复处理

import asyncio
import aiohttp
from readability import Document
from concurrent.futures import ProcessPoolExecutor

async def fetch(url, session):
    async with session.get(url) as response:
        return await response.text()

def process_html(html):
    doc = Document(html)
    return doc.summary()

async def extract_articles(urls):
    async with aiohttp.ClientSession() as session:
        html_contents = await asyncio.gather(*[fetch(url, session) for url in urls])
    
    with ProcessPoolExecutor() as executor:
        results = list(executor.map(process_html, html_contents))
    
    return results

urls = ["https://example.com/article1", "https://example.com/article2", ...]
results = asyncio.run(extract_articles(urls))