爬虫解析网页

最新推荐文章于 2023-08-29 15:12:06 发布

neu-gwh

最新推荐文章于 2023-08-29 15:12:06 发布

阅读量190

点赞数

分类专栏：爬虫

本文链接：https://blog.csdn.net/sdauguanweihong/article/details/103227216

版权

爬虫专栏收录该内容

7 篇文章 0 订阅

订阅专栏

1、正则表达式获取博客标题

import requests
import re
link="http://www.santostang.com/"
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36',
         'Host':'www.santostang.com'
         }
r=requests.get(link,headers=headers)
html=r.text
title_list=re.findall('<h1 class="post-title"><a href=.*?>(.*?)</a></h1>',html)
print(title_list)

2、beautifulsoup方式

import requests
import re

from bs4 import BeautifulSoup

link="http://www.santostang.com/"
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36',
         'Host':'www.santostang.com'
         }
r=requests.get(link,headers=headers)
html=r.text
soup=BeautifulSoup(r.text,'lxml')
first_title=soup.find("h1",class_="post-title").a.string.strip()
print('第一篇文章标题是',first_title)
title_list=soup.find_all("h1",class_="post-title")
for i in range(len((title_list))):
    title=title_list[i].a.string.strip()
    print("第%s篇文章标题为%s" % (i+1,title))

3。lxml方式

import requests
import re
from lxml import etree
from bs4 import BeautifulSoup

link="http://www.santostang.com/"
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36',
         'Host':'www.santostang.com'
         }
r=requests.get(link,headers=headers)
html=etree.HTML(r.text)
title_list=html.xpath('//h1[@class="post-title"]/a/text()')
print(title_list)