关于webmagic爬虫报错javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version

最新推荐文章于 2024-07-08 16:16:09 发布

君を見つけて

最新推荐文章于 2024-07-08 16:16:09 发布

阅读量2.4k

点赞数

分类专栏：爬虫

本文链接：https://blog.csdn.net/chinatopno1/article/details/109993015

版权

爬虫专栏收录该内容

6 篇文章 1 订阅

订阅专栏

目前 webmagic最新版是0.7.3版本，在爬取只支持 TLS1.2的 https站点会报错

javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[na:na]
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[na:na]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:311) ~[na:na]
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:291) ~[na:na]
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:184) ~[na:na]
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1460) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1368) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:437) ~[na:na]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:85) ~[webmagic-core-0.7.3.jar:na]
at us.codecraft.webmagic.Spider.processRequest(Spider.java:404) ~[webmagic-core-0.7.3.jar:na]
at us.codecraft.webmagic.Spider.access$000(Spider.java:61) ~[webmagic-core-0.7.3.jar:na]
at us.codecraft.webmagic.Spider$1.run(Spider.java:320) ~[webmagic-core-0.7.3.jar:na]
at us.codecraft.webmagic.thread.CountableThreadPool $1 . r u n (C o u n t a b l e T h r e a d P o o l . j a v a : 74) [w e b m a g i c - c o r e - 0.7.3 . j a r : n a] a t j a v a . b a s e / j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r . r u n W o r k e r (T h r e a d P o o l E x e c u t o r . j a v a : 1128) [n a : n a] a t j a v a . b a s e / j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r$ Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:830) ~[na:na]

目前作者说会在0.7.4版本发布修复错误，目前的解决办法是：
临时适配方式，修改HttpClientGenerator中的buildSSLConnectionSocketFactory方法

return new SSLConnectionSocketFactory(createIgnoreVerifySSL(), new String[]{"SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"}, null, new DefaultHostnameVerifier())