最近对python的解析HTML的能力叹为观止,毕竟python这样强大的工具根本不缺乏解析html和xml的第三方库,我使用的是lxml当然还有其他比较好使的如:HTMLParser、BeautifulSoup等,鉴于我比较喜欢xpath这样的语法表达所以这里使用的也是lxml
要做的就是对于解析后的DOM树形成一个完整的xpath路径的列表,列表中的每一个路径都是从根节点到叶子节点的一条完整的路径,当然这里的表达形式也不是唯一的,在这里我要得到DOM树的最大深度,思路很简单做法也很简单,就是遍历这样的一个xpath列表得到最大长度的那个数值就行了,这里给出来简单的实现:
#!usr/bin/env python
#encoding:utf-8
'''
__author__:沂水寒城
功能:python处理xpath得到页面的DOM树深度
'''
from get_all_node_xpath import *
def get_tree_max_deepth(all_xpath_list):
'''
得到一个HTML页面形成的xpath列表中最大长度,即DOM树的最大深度
'''
tree_deepth_list=[]
for one_xpath in all_xpath_list:
tree_deepth_list.append(len(one_xpath.split('/')[1:]))
return max(tree_deepth_list)
if __name__ == '__main__':
with open('../baidu.txt') as f:
baidu=f.read()
baidu_tree, baidu_xpath_list=get_clean_allnodes_xpath(baidu)
max_tree_deepth=get_tree_max_deepth(baidu_xpath_list)
for one_xpath in baidu_xpath_list:
print one_xpath
print 'max_tree_deepth is:', max_tree_deepth
结果如下:
/html
/html/head
/html/head/meta[1]
/html/head/meta[2]
/html/head/meta[3]
/html/head/meta[4]
/html/head/title
/html/body
/html/body/p
/html/body/p/comment()[1]
/html/body/p/comment()[2]
/html/body/p/comment()[3]
/html/body/p/meta
/html/body/div[1]
/html/body/div[1]/div[1]
/html/body/div[1]/div[1]/div
/html/body/div[1]/div[1]/div/div[1]
/html/body/div[1]/div[1]/div/div[1]/div
/html/body/div[1]/div[1]/div/div[1]/div/div[1]
/html/body/div[1]/div[1]/div/div[1]/div/a
/html/body/div[1]/div[1]/div/div[1]/div/form
/html/body/div[1]/div[1]/div/div[1]/div/form/span[1]
/html/body/div[1]/div[1]/div/div[1]/div/form/span[2]
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/div
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/div/span
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/ul
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/ul/li[1]
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/ul/li[1]/a
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/ul/li[2]
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/ul/li[2]/a
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/ul/li[3]
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/ul/li[4]
/html/body/div[1]/div[1]/div/div[1]/div/form/span[3]/span/ul/li[4]/a
/html/body/div[1]/div[1]/div/div[1]/div/div[2]
/html/body/div[1]/div[1]/div/div[2]
/html/body/div[1]/div[1]/div/div[2]/a[1]
/html/body/div[1]/div[1]/div/div[2]/a[2]
/html/body/div[1]/div[1]/div/div[2]/a[3]
/html/body/div[1]/div[1]/div/div[3]
/html/body/div[1]/div[1]/div/div[3]/a[1]
/html/body/div[1]/div[1]/div/div[3]/a[2]
/html/body/div[1]/div[1]/div/div[3]/a[3]
/html/body/div[1]/div[1]/div/div[3]/a[4]
/html/body/div[1]/div[1]/div/div[3]/a[5]
/html/body/div[1]/div[1]/div/div[3]/a[6]
/html/body/div[1]/div[1]/div/div[3]/a[7]
/html/body/div[1]/div[1]/div/div[3]/a[8]
/html/body/div[1]/div[1]/div/div[3]/a[9]
/html/body/div[1]/div[2]
/html/body/div[1]/div[2]/a[1]
/html/body/div[1]/div[2]/a[2]
/html/body/div[1]/div[2]/a[3]
/html/body/div[1]/div[2]/a[4]
/html/body/div[1]/div[2]/a[5]
/html/body/div[1]/div[2]/a[6]
/html/body/div[1]/div[2]/a[7]
/html/body/div[1]/div[2]/a[8]
/html/body/div[1]/div[2]/a[9]
/html/body/div[1]/div[3]
/html/body/div[1]/div[3]/div
/html/body/div[1]/div[3]/div/div
/html/body/div[1]/div[3]/div/div/div[1]
/html/body/div[1]/div[3]/div/div/div[2]
/html/body/div[1]/div[3]/div/div/div[2]/p
/html/body/div[1]/div[4]
/html/body/div[1]/div[4]/div
/html/body/div[1]/div[4]/div/div
/html/body/div[1]/div[4]/div/div/p[1]
/html/body/div[1]/div[4]/div/div/p[1]/a[1]
/html/body/div[1]/div[4]/div/div/p[1]/a[2]
/html/body/div[1]/div[4]/div/div/p[1]/a[3]
/html/body/div[1]/div[4]/div/div/p[1]/a[4]
/html/body/div[1]/div[4]/div/div/p[2]
/html/body/div[1]/div[4]/div/div/p[2]/a[1]
/html/body/div[1]/div[4]/div/div/p[2]/a[2]
/html/body/div[1]/div[4]/div/div/p[2]/a[3]
/html/body/div[1]/div[5]
/html/body/div[2]
/html/body/div[3]
/html/body/div[4]
max_tree_deepth is: 13
接下来可以继续计算得到更多的相关数据,如枝叶的均值、方差。