python解析html的库_python 解析html基础 HTMLParser库,方法,及代码实例

exceptionHTMLParser.HTMLParseErrorHTMLParser is able to handle broken markup, but in some cases it might raise this exception when it encounters an error while parsing.This

exception provides three attributes: msg is a briefmessage explaining the error, lineno is the number of the line onwhich the broken construct was detected, and

offset is the number ofcharacters into the line at which the construct starts.下面仅就重点进行解释，与html解析的pythonlib除了HTMLParser，还有htmllib，htmllib不同于HTMLParser的一点是htmllib基于SGMLParser#!/usr/bin/python

#a simple htmlparser demo

from HTMLParser import HTMLParser

class SimpleParser(HTMLParser):

def handle_starttag(self,tag,attr): print tag

def handle_endtag(self,tag): print tag

def handle_data(self,data): print data

simpleparser = SimpleParser()

simpleparser.feed("

")

这是一个很简单的例子，但是HTMLParser的基本原理也就是这样，当然，为了更好的完成工作我们还要处理一些非标准的html，因为现在浏览器的发展是在不断迎合识别各种非标准的html，所以造成了今天我们对html的解析变得异常复杂，我们会看到

...然后没有终止标记这样的非标准格式，当然，像xhtml这种语言是比较严格的，不会允许这种情况的出现，下面给出一个在《python基础网络编程》的例子，这个例子中除了处理这种不标准

的html标记外，还同时用

HTMLParser.handle_entityref(name)This method is called to process a named character reference of the form&name; (e.g.

>), where name is a general entity reference(e.g.

‘gt‘).

处理特殊符号，因为在过滤中会自动把这一类的符号给过滤掉，不能转义也不能作为字面值出现:

下面是代码:

#!/usr/bin/env python

from HTMLParser import HTMLParser

#handle entity like &

from htmlentitydefs import entitydefs

import sys

class TitleParser(HTMLParser):

def __init__(self):

self.taglevels = []

#处理title ul li 只有开始无结束标记的情况

self.handledtags = ['title', 'ul', 'li']

self.processing = None

HTMLParser.__init__(self)

def handle_starttag(self, tag, attrs):

if len(self.taglevels) and self.taglevels[-1] == tag:

self.handle_endtag(tag)

self.taglevels.append(tag)

if tag in self.handledtags:

self.data = ''

self.processing = tag

if tag == 'ul':

#do print

def handle_data(self, data):

if self.processing:

self.title += data

def handle_endtag(self, tag):

if not tag in self.taglevels:

return

while len(self.taglevels):

starttag = self.taglevels.pop()

if starttag in self.handledtags:

self.finishprocessing(starttag)

if starttag == tag:

break

def cleanse(self):

self.data = re.sub('\s+', ' ', self.data)

def finishprocessing(self, tag):

self.cleanse()

if tag == 'title' and tag == self.processing:

print "Dom title", self.data

elif tag == 'ul':

print "List ended"

elif tag == 'li' and tag == self.processing:

print "List item", self.data

self.processing = None

def gettitle(self):

return self.title

#处理特殊值，如果在映射表中有对应的，即采用映射的值，否则为字面值

def handle_entityref(self, name):

if entitydefs.has_key(name):

self.handle_data(entitydefs[name])

else:

self.handle_data('&' + name + ';')

def handle_charref(self, name):

try:

charnum = int(name)

except ValueError:

return

if charnum < 1 or charnum > 255:

return

self.handle_data(chr(charnum))

fd = open(sys.argv[1])

tp = TitleParser()

tp.feed(fd.read())

print tp.gettitle()