python 自学第二课：使用BeautifulSoup抓取链接正则表达式-CSDN博客

python 自学第二课：使用BeautifulSoup抓取链接正则表达式

具体的查看BeautifulSoup文档（根据自己的安装的版本查看对应文档）
文档链接https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.zh.html#

#!/usr/bin/env python
# -*- coding:utf-8 -*-
import io  
import sys
from urllib import request
from bs4 import BeautifulSoup
import re
sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='utf8') #改变标准输出的默认编码  
resp = request.urlopen("http://news.baidu.com/").read().decode("utf-8")
soup =BeautifulSoup(resp,"html.parser")
listUrls=soup.find_all("a",href=re.compile(".*\/\/news\.baidu.*"))
for url in listUrls:
    print (url["href"])

最后效果：

http://news.baidu.com/view.html
http://news.baidu.com/advanced_news.html
http://news.baidu.com/pianhao.html
http://news.baidu.com/n?bypass=lamp&m=pagesother&v=newsgx
http://news.baidu.com/n?cmd=6&loc=0&name=%B1%B1%BE%A9
http://news.baidu.com/history.html
http://news.baidu.com/newscode.html
http://news.baidu.com/licence.html

转载于:https://www.cnblogs.com/jieshendada/p/7843785.html