目前webmagic
最新版是0.7.3版本,在爬取只支持TLS1.2
的https
站点会报错,作者说会在0.7.4版本发布修复错误,但是等了3年了都没等到0.7.4版本的发布.
javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
at sun.security.ssl.Alert.createSSLException(Alert.java:131)
at sun.security.ssl.Alert.createSSLException(Alert.java:117)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:187)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:154)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1198)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1107)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:400)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:372)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:85)
at us.codecraft.webmagic.Spider.processRequest(Spider.java:404)
at us.codecraft.webmagic.Spider.access$000(Spider.java:61)
at us.codecraft.webmagic.Spider$1.run(Spider.java:320)
at us.codecraft.webmagic.thread.CountableThreadPool$1.run(CountableThreadPool.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
目前的解决办法是:临时适配方式,修改HttpClientGenerator中的buildSSLConnectionSocketFactory方法,重写自己实现的HttpClientDownloader,并设置到Spider中。修改内容如下:
return new SSLConnectionSocketFactory(createIgnoreVerifySSL(), new String[]{"SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"},
null,
new DefaultHostnameVerifier())
目前我已经把源码修改好了,并编译为jar包,有被同样问题困扰的朋友可以直接下载使用.
提取码:webm
jar包使用方法:将下载好的 jar包,放在你的maven仓库的us\codecraft\webmagic-core\0.7.3路径下,替换原有jar包即可解决问题.