Python爬虫（七）学习提取网页中所有链接

最新推荐文章于 2024-08-17 00:42:37 发布

Frank Kong

最新推荐文章于 2024-08-17 00:42:37 发布

阅读量1.6w

点赞数 6

分类专栏： Python 文章标签： python 爬虫

本文链接：https://blog.csdn.net/kh971024/article/details/78637294

版权

Python 专栏收录该内容

16 篇文章 2 订阅

订阅专栏

import re
import urllib.request


def getlink(url):
    headers = ("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36")
    opener = urllib.request.build_opener()
    opener.addheaders = [headers]
    urllib.request.install_opener(opener)
    file = urllib.request.urlopen(url).read()
    file = file.decode('utf-8')
    pattern = '(https?://[^\s)";]+(\.(\w|/)*))'
    link = re.compile(pattern).findall(file)
    #去重
    #link = list(set(link))
    return link


url = "http://blog.csdn.net/"
linklist = getlink(url)
for link in linklist:
    print(link[0])
print(len(linklist))