java爬取http类型的网站比较容易实现,因为不需要建立证书的通道,直接通过httpclient访问链接获取相应源码就可以获取相关数据,现在我们可以通过证书的方式,实现java爬取https网站的相关数据。
1.下载网站的证书
以CSDN博客为例 谷歌浏览器右键网页——点击检查——点击security出现如下界面
点击view certificate
点击证书路径 我们可以发现证书如下:
想要导出证书 需要IE浏览器得支持 我们使用IE浏览器重新访问该网站
点击工具——Internet选项
可以找到相关证书:
接下来点击导出
选择保存的目录以及文件名
导出即可
2.java命令行将证书生成秘钥库
进入JDK 安装bin目录,命令行运行:keytool -import -alias Root -file d:/test.cer -keystore “d:/test.keystore” -storepass 123456。转换证书
3.编写代码
项目目录如下:红色框框圈住的为需要的包,下载地址为:https://download.csdn.net/download/qq_36706878/11855431
代码如下:
public static Document sendHttp(String url) {
String html = "";
CloseableHttpClient httpclient = null;
CloseableHttpResponse response = null;
try {
SSLConnectionSocketFactory sslsf = createSSLConnSocketFactory();
httpclient = HttpClients.custom()
.setSSLSocketFactory(sslsf).build();
HttpGet httpget = new HttpGet(url);
httpget.addHeader(HttpHeaders.USER_AGENT,
"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0");
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(10000).setConnectTimeout(10000).build();// 设置请求和传输超时时间
httpget.setConfig(requestConfig);
System.out.println("Executing request " + httpget.getRequestLine());
response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
int resStatu = response.getStatusLine().getStatusCode();// 返回码
if (resStatu == HttpStatus.SC_OK) {// 200正常 其他就不对
// 获得相应实体
if (entity != null) {
html = EntityUtils.toString(entity, "UTF-8");
html = html.replace(" ", " ");
}
}
EntityUtils.consume(entity);
} catch(Exception e){
e.printStackTrace();
}finally{
if(response!=null){
try {
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(httpclient!=null){
try {
httpclient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Document document = Jsoup.parse(html);
return document;
}
// ssl通道证书的创建
private static SSLConnectionSocketFactory createSSLConnSocketFactory()
throws Exception {
SSLContext sslcontext = SSLContexts
.custom()
.loadTrustMaterial(
new File(
"E:/key.keystore"),
"123456".toCharArray(), new TrustSelfSignedStrategy()) //文件和密码要对应
.build();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
sslcontext, new String[] { "TLSv1" }, null,
SSLConnectionSocketFactory.getDefaultHostnameVerifier());
return sslsf;
}
测试结果: