本文介绍了Python中使用html.parser和urllib库进行网页爬取的基本步骤，包括解析HTML、提取链接以及如何递归爬取多页内容。LinkParser类实现了从HTML标签中获取链接的功能，通过getLinks方法获取页面所有链接，并用spider函数实现爬虫逻辑，寻找特定词汇并跟踪链接。示例代码展示了如何在实际项目中应用这些技术。
标签：html.parser HTMLParser
urllib.request urlopen
urllib parse
LinkParser(HTMLParser):
handle_starttag(, tag, attrs):
tag == :
(key, value) attrs:
key == :
newUrl = parse.urljoin(.baseUrl, value)
.links = .links + [newUrl]
getLinks(, url):
.links = []
.baseUrl = url
response = urlopen(url)
response.getheader()==:
htmlBytes = response.read()
htmlString = htmlBytes.decode()
.feed(htmlString)
htmlString, .links
:
,[]
spider(url, word, maxPages):
pagesToVisit = [url]
numberVisited = foundWord = numberVisited < maxPages pagesToVisit != [] foundWord:
numberVisited = numberVisited +url = pagesToVisit[]
pagesToVisit = pagesToVisit[:]
:
(numberVisited, , url)
parser = LinkParser()
data, links = parser.getLinks(url)
pagesToVisit = pagesToVisit + links
data.find(word)>-:
foundWord = pagesToVisit = pagesToVisit + links
()
:
()
foundWord:
(, word, , url)
:
()
spider(,,)
标签：
原文地址：http://my.oschina.net/u/215677/blog/523343
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
