htmlUnit使用

htmlUnit介绍:

htmlunit 是一款开源的java 页面分析工具，读取页面后，可以有效的使用htmlunit分析页面上的内容。项目可以模拟浏览器运行，被誉为java浏览器的开源实现。是一个没有界面的浏览器，运行速度迅速。官方网址:

http://htmlunit.sourceforge.net/

htmlUnit示例:

pom依赖:

 <dependency>
            <groupId>net.sourceforge.htmlunit</groupId>
            <artifactId>htmlunit</artifactId>
            <version>2.26</version>
        </dependency>

测试代码：

import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlInput;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import org.junit.Test;

import java.io.IOException;


public class BaiduDemo {
    @Test
    public void test() throws IOException {
        WebClient webClient = new WebClient();
        webClient.getOptions().setJavaScriptEnabled(false);
        webClient.getOptions().setCssEnabled(false);
        webClient.getOptions().setUseInsecureSSL(false);


        //获取页面
        String url ="https://www.baidu.com";
        HtmlPage page = webClient.getPage(url);

        System.out.println("页面文本:"+page.getTitleText());


        //获取页面元素
        HtmlInput htmlInput = page.getHtmlElementById("kw");
        System.out.println(htmlInput.asText());
        htmlInput.setValueAttribute("test");

        HtmlInput btn = page.getHtmlElementById("su");
        HtmlPage page2 = btn.click();

        System.out.println("页面2:"+page2.getTitleText());
    }
    
}

获取cookie:

BasicCookieStore cookieStore = new BasicCookieStore();
        Set<Cookie> cookies2 =webClient.getCookieManager().getCookies();
        for (Cookie cookie : cookies2) {
            cookieStore.addCookie(cookie.toHttpClient());
        }

通过htmlUnit获取的cookie信息发送https请求:

入参: 

cookieStore //通过htmlUnit获取的登陆信息

 url参数:要访问的页面参数

protected String sendRequest(BasicCookieStore cookieStore,String url) throws Exception {
        if (null == cookieStore) {
            return null;
        }

        //设置cookie
        HttpClientContext context = HttpClientContext.create();
        context.setCookieStore(cookieStore);

        //忽略SSL证书验证

        SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {
            //信任所有
            public boolean isTrusted(X509Certificate[] chain,
                                     String authType) throws CertificateException {
                return true;
            }
        }).build();  
        CloseableHttpClient httpClient = HttpClients.custom().setSSLContext(sslContext).build();
        HttpGet httpGet = new HttpGet(url);

        //发送请求并获取response
        CloseableHttpResponse response = httpClient.execute(httpGet, context);
        HttpEntity entity = response.getEntity();

        return EntityUtils.toString(entity);
    }

htmlUnit应用场景

todo

相关资料:

http://blog.csdn.net/lifj07/article/details/8638098