htmlparser对html页面处理的算法

主要是如下几种方式

  采用Visitor方式访问Html

try {

    Parser parser = new Parser();

    parser.setURL(”http://www.google.com”);

    parser.setEncoding(parser.getEncoding());

    NodeVisitor visitor = new NodeVisitor() {

        public void visitTag(Tag tag) {

            System.out.println (”testVisitorAll()  Tag name is :”

                    + tag.getTagName() + ” \n Class is :”

                    + tag.getClass());

        }

    };

    parser.visitAllNodesWith(visitor);

} catch (ParserException e) {

    e.printStackTrace();

}

  采用Filter方式访问html

try {

    NodeFilter filter = new NodeClassFilter(LinkTag.class);

    Parser parser = new Parser();

    parser.setURL(”http://www.google.com”);

    parser.setEncoding(parser.getEncoding());

    NodeList list = parser.extractAllNodesThatMatch(filter);

    for (int i = 0; i < list.size(); i++) {

        LinkTag node = (LinkTag) list.elementAt(i);

        System.out.println(”testLinkTag() Link is :” + node.extractLink());

    }

} catch (Exception e) {

    e.printStackTrace();

}

  采用org.htmlparser.beans方式

另外htmlparser 还在org.htmlparser.beans中对一些常用的方法进行了封装，以简化操作，例如：

Parser parser = new Parser();

LinkBean linkBean = new LinkBean();

linkBean.setURL(”http://www.google.com”);

URL[] urls = linkBean.getLinks();

for (int i = 0; i < urls.length; i++) {

    URL url = urls[i];

    System.out.println (”testLinkBean() -url  is :” + url);

}

 

Htmlparser关键包结构说明

    htmlparser其实核心代码并不多，好好研究一下其代码，弥补文档不足的问题。同时htmlparser的代码注释和单元测试用例还是很齐全的，也有助于了解htmlparser的用法。

 

3.1、org.htmlparser

    定义了htmlparser的一些基础类。其中最为重要的是Parser类。

    Parser是htmlparser的最核心的类，其构造函数提供了如下：Parser.createParser (String html, String charset)、 Parser ()、Parser (Lexer lexer, ParserFeedback fb)、Parser (URLConnection connection, ParserFeedback fb)、Parser (String resource, ParserFeedback feedback)、 Parser (String resource)

  各构造函数的具体用法及含义可以查看其代码，很容易理解。

  Parser常用的几个方法：

•         elements获取元素

    Parser parser = new Parser (”http://www.google.com”);

    for (NodeIterator i = parser.elements (); i.hasMoreElements (); )

      processMyNodes (i.nextNode ());

•       parse (NodeFilter filter)：通过NodeFilter方式获取

•       visitAllNodesWith (NodeVisitor visitor)：通过Nodevisitor方式

•       extractAllNodesThatMatch (NodeFilter filter)：通过NodeFilter方式

3.2、org.htmlparser.beans

    对Visitor和Filter的方法进行了封装，定义了针对一些常用html元素操作的bean，简化对常用元素的提取操作。

    包括：FilterBean、HTMLLinkBean、HTMLTextBean、LinkBean、StringBean、BeanyBaby等。

3.3、org.htmlparser.nodes

    定义了基础的node，包括：AbstractNode、RemarkNode、TagNode、TextNode等。

3.4、org.htmlparser.tags

    定义了htmlparser的各种tag。

3.5、org.htmlparser.filters

    定义了htmlparser所提供的各种filter，主要通过extractAllNodesThatMatch (NodeFilter filter)来对html页面指定类型的元素进行过滤，包括：AndFilter、CssSelectorNodeFilter、HasAttributeFilter、HasChildFilter、HasParentFilter、HasSiblingFilter、IsEqualFilter、LinkRegexFilter、LinkStringFilter、NodeClassFilter、NotFilter、OrFilter、RegexFilter、StringFilter、TagNameFilter、XorFilter

3.6、org.htmlparser.visitors

   定义了htmlparser所提供的各种visitor，主要通过visitAllNodesWith (NodeVisitor visitor)来对html页面元素进行遍历，包括：HtmlPage、LinkFindingVisitor、NodeVisitor、ObjectFindingVisitor、StringFindingVisitor、TagFindingVisitor、TextExtractingVisitor、UrlModifyingVisitor

 

3.7、org.htmlparser.parserapplications

   定义了一些实用的工具，包括LinkExtractor、SiteCapturer、StringExtractor、WikiCapturer，这几个类也可以作为htmlparser使用样例。

3.8、org.htmlparser.tests

   对各种功能的单元测试用例，也可以作为htmlparser使用的样例。

 

 

 

1 . 逻辑关系：与或非

AndFilter()            Creates a new instance of an AndFilter.

AndFilter(NodeFilter[] predicates)            Creates an AndFilter that accepts nodes acceptable to all given filters.

AndFilter(NodeFilter left, NodeFilter right)            Creates an AndFilter that accepts nodes acceptable to both filters.

 

 

OrFilter()            Creates a new instance of an OrFilter.

OrFilter(NodeFilter[] predicates)            Creates an OrFilter that accepts nodes acceptable to any of the given filters.

OrFilter(NodeFilter left, NodeFilter right)            Creates an OrFilter that accepts nodes acceptable to either filter.

 

 

 

OrFilter()            Creates a new instance of an OrFilter.

OrFilter(NodeFilter[] predicates)            Creates an OrFilter that accepts nodes acceptable to any of the given filters.

OrFilter(NodeFilter left, NodeFilter right)            Creates an OrFilter that accepts nodes acceptable to either filter.

 

2. 内容

StringFilter：功能简单有限；复杂功能可使用RegexFilter (正则表达式)

 

StringFilter()            Creates a new instance of StringFilter that accepts all string nodes.

StringFilter(String pattern)            Creates a StringFilter that accepts text nodes containing a string.

StringFilter(String pattern, boolean sensitive)            Creates a StringFilter that accepts text nodes containing a string.

StringFilter(String pattern, boolean sensitive, Locale locale)            Creates a StringFilter that accepts text nodes containing a string.

RegexFilter()            Creates a new instance of RegexFilter that accepts string nodes matching the regular expression ".*" using the FIND strategy.

RegexFilter(String pattern)            Creates a new instance of RegexFilter that accepts string nodes matching a regular expression using the FIND strategy.

RegexFilter(String pattern, int strategy)            Creates a new instance of RegexFilter that accepts string nodes matching a regular expression.

 

3 标签

TagNameFilter()利用标签名过滤 : div ,img , ...

NodeClassFilter()利用标签类别 ：LinkTag.class ...

HasAttributeFilter()利用属性 ：HasAttributeFilter(“class”, “className”)

LinkRegexFilter（）用正则表达式匹配链接

 

TagNameFilter()            Creates a new instance of TagNameFilter.

TagNameFilter(String name)            Creates a TagNameFilter that accepts tags with the given name.

NodeClassFilter()            Creates a NodeClassFilter that accepts Html tags.

NodeClassFilter(Class cls)            Creates a NodeClassFilter that accepts tags of the given class.

HasAttributeFilter()            Creates a new instance of HasAttributeFilter.

HasAttributeFilter(String attribute)            Creates a new instance of HasAttributeFilter that accepts tags with the given attribute.

HasAttributeFilter(String attribute, String value)            Creates a new instance of HasAttributeFilter that accepts tags with the given attribute and value.

LinkRegexFilter(String regexPattern)            Creates a LinkRegexFilter that accepts LinkTag nodes containing a URL that matches the supplied regex pattern.

LinkRegexFilter(String regexPattern, boolean caseSensitive)            Creates a LinkRegexFilter that accepts LinkTag nodes containing a URL that matches the supplied regex pattern.

LinkStringFilter(String pattern)            Creates a LinkStringFilter that accepts LinkTag nodes containing a URL that matches the supplied pattern.

LinkStringFilter(String pattern, boolean caseSensitive)            Creates a LinkStringFilter that accepts LinkTag nodes containing a URL that matches the supplied pattern.

 

4 层次关系

HasParentFilter()            Creates a new instance of HasParentFilter.

HasParentFilter(NodeFilter filter)            Creates a new instance of HasParentFilter that accepts nodes with the direct parent acceptable to the filter.

HasParentFilter(NodeFilter filter, boolean recursive)            Creates a new instance of HasParentFilter that accepts nodes with a parent acceptable to the filter.

HasChildFilter()            Creates a new instance of a HasChildFilter.

HasChildFilter(NodeFilter filter)            Creates a new instance of HasChildFilter that accepts nodes with a direct child acceptable to the filter.

HasChildFilter(NodeFilter filter, boolean recursive)            Creates a new instance of HasChildFilter that accepts nodes with a child acceptable to the filter.