爬取某付费网站文档保存为html文件

最新推荐文章于 2024-05-03 14:25:28 发布

努力学习各种软件

最新推荐文章于 2024-05-03 14:25:28 发布

阅读量667

点赞数 7

分类专栏：爬虫案例文章标签： python 爬虫

本文链接：https://blog.csdn.net/m0_57265868/article/details/135873733

版权

爬虫案例专栏收录该内容

27 篇文章 5 订阅

订阅专栏

import requests
import re
import os
import parsel


html_str = """
<!doctype html>
<html lang="en">
<head>
    <meta charset = "utf-8">
    <title>Document</title>
</head>
<body>
{article}
</body>
</html>
"""
html_filename = 'html\\'
if not os.path.exists(html_filename):
    os.mkdir(html_filename)
url = 'https://www.chinawenwang.com/zlist-66-1.html'  #文章列表页的url地址
headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
        }
response = requests.get(url=url,headers=headers)

href = re.findall('<h2><a href="(.*?)" class="juhe-page-left-div-link">',response.text) # 提取文章的url地址
for link in href:
    response_1 = requests.get(url=link,headers=headers)
    selector = parsel.Selector(response_1.text)
    title = selector.css('.content-page-header-div h1::text').get()
    content = selector.css(('.content-page-main-content-div')).get()
    article = html_str.format(article = content)
    with open(html_filename+title+'.html',mode='w',encoding='utf-8') as f:
        f.write(article)
        print(title)

结果展示：