正则+Beautifulsoup爬取起点

zzx666love

于 2024-04-12 17:43:27 发布

阅读量316

点赞数 5

文章标签： beautifulsoup

本文链接：https://blog.csdn.net/zzx666love/article/details/137689228

版权

看他们说起点反扒机制比较牛逼就去试了试看能不能爬取起点，然后成功了。以下是代码

import requests
from bs4 import  BeautifulSoup
import  re
header={'User-Agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0'
        ,'Cookie':+自己的cookie
}
url='https://www.qidian.com/chapter/1037068783/753437190/'

response=requests.get(url,headers=header).text
# 使用request进行爬取
# contents=re.findall('5f2><p>(.*?)</main>',response)[0].split('</p><p>')
# title=re.findall(' <title>(.*?)小说在线阅读 - 起点中文网</title>',response)
# print(title)
# for content  in contents:
# 	print(content)
# 使用Beautifulsoup进行爬取
soup=BeautifulSoup(response,'html.parser')
# print(soup)
# print("--------soup输出完毕--------")
contents=soup.find_all('p')
title=soup.select_one('title')
print(title.text)
for content in contents:
	print(content.text)