Python爬取网页文本内容

最新推荐文章于 2024-04-27 16:08:02 发布

湛蓝9979

最新推荐文章于 2024-04-27 16:08:02 发布

阅读量776

点赞数

文章标签： python 爬虫开发语言

本文链接：https://blog.csdn.net/weixin_56452427/article/details/128852671

版权

利用Python爬取当前网页所有文本内容

摘要由CSDN通过智能技术生成

# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup


def second_pro(text):
    last_sen = []
    while 1:
        last_sen.append(text.split(",")[-1])
        other_list = text.split(",")[:-1]
        # 特定位置超长，需要处理
        if len(other_list) == 1:
            last_sen.append(other_list[0].split("，")[-1])
            other_list = other_list[0].split("，")[:-1]
        if len(",".join(other_list)) < 508:
            return ",".join(other_list), last_sen
        text = ",".join(other_list[:-1])


def maching(text):
    hold_l = []
    con_text = []
    for one in text.split(","):
        hold_l.append(one)
        if len(",".join(hold_l)) > 508:
            model_text, last_sen = second_pro(",".join(hold_l))
            hold_l = []
            hold_l.append(",".join(last_sen))
            con_text.append(model_