exceptionHTMLParser.HTMLParseErrorHTMLParser is able to handle broken markup, but in some cases it might raise this exception when it encounters an error while parsing.This
exception provides three attributes: msg is a briefmessage explaining the error, lineno is the number of the line onwhich the broken construct was detected, and
offset is the number ofcharacters into the line at which the construct starts.下面仅就重点进行解释,与html解析的pythonlib除了HTMLParser,还有htmllib,htmllib不同于HTMLParser的一点是htmllib基于SGMLParser#!/usr/bin/python
#a simple htmlparser demo
from HTMLParser import HTMLParser
class SimpleParser(HTMLParser):
def handle_starttag(self,tag,attr): print tag
def handle_endtag(self,tag): print tag
def handle_data(self,data): print data
simpleparser = SimpleParser()
simpleparser.feed("
")这是一个很简单的例子,但是HTMLParser的基本原理也就是这样,当然,为了更好的完成工作我们还要处理一些非标准的html,因为现在浏览器的发展是在不断迎合识别各种非标准的html,所以造成了今天我们对html的解析变得异常复杂,我们会看到
...然后没有终止标记这样的非标准格式,当然,像xhtml这种语言是比较严格的,不会允许这种情况的出现,下面给出一个在《python基础网络编程》的例子,这个例子中除了处理这种不标准的html标记外,还同时用
HTMLParser.handle_entityref(name)This method is called to process a named character reference of the form&name; (e.g.
>), where name is a general entity reference(e.g.
‘gt‘).
处理特殊符号,因为在过滤中会自动把这一类的符号给过滤掉,不能转义也不能作为字面值出现:
下面是代码:
#!/usr/bin/env python
from HTMLParser import HTMLParser
#handle entity like &
from htmlentitydefs import entitydefs
import sys
class TitleParser(HTMLParser):
def __init__(self):
self.taglevels = []
#处理title ul li 只有开始无结束标记的情况
self.handledtags = ['title', 'ul', 'li']
self.processing = None
HTMLParser.__init__(self)
def handle_starttag(self, tag, attrs):
if len(self.taglevels) and self.taglevels[-1] == tag:
self.handle_endtag(tag)
self.taglevels.append(tag)
if tag in self.handledtags:
self.data = ''
self.processing = tag
if tag == 'ul':
#do print
def handle_data(self, data):
if self.processing:
self.title += data
def handle_endtag(self, tag):
if not tag in self.taglevels:
return
while len(self.taglevels):
starttag = self.taglevels.pop()
if starttag in self.handledtags:
self.finishprocessing(starttag)
if starttag == tag:
break
def cleanse(self):
self.data = re.sub('\s+', ' ', self.data)
def finishprocessing(self, tag):
self.cleanse()
if tag == 'title' and tag == self.processing:
print "Dom title", self.data
elif tag == 'ul':
print "List ended"
elif tag == 'li' and tag == self.processing:
print "List item", self.data
self.processing = None
def gettitle(self):
return self.title
#处理特殊值,如果在映射表中有对应的,即采用映射的值,否则为字面值
def handle_entityref(self, name):
if entitydefs.has_key(name):
self.handle_data(entitydefs[name])
else:
self.handle_data('&' + name + ';')
def handle_charref(self, name):
try:
charnum = int(name)
except ValueError:
return
if charnum < 1 or charnum > 255:
return
self.handle_data(chr(charnum))
fd = open(sys.argv[1])
tp = TitleParser()
tp.feed(fd.read())
print tp.gettitle()