spark中遇到mysql链接超时的问题解决

最新推荐文章于 2023-10-12 20:55:36 发布

涟漪海洋

最新推荐文章于 2023-10-12 20:55:36 发布

阅读量1.5k

点赞数

分类专栏：开发问题技术实战文章标签： scala spark mysql

本文链接：https://blog.csdn.net/hu10131013/article/details/120262123

版权

技术实战同时被 2 个专栏收录

35 篇文章 0 订阅

订阅专栏

开发问题

33 篇文章 0 订阅

订阅专栏

背景

最近使用scala写spark的运行jar，不小心遇到一个比较傻的坑，主要是mysql方面的报错，

报错如下

2021-09-13 10:26:12.426  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   : Exception in thread "main" com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   : The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2498)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2535)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.sql.DriverManager.getConnection(DriverManager.java:664)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.sql.DriverManager.getConnection(DriverManager.java:247)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.lang.reflect.Method.invoke(Method.java:498)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   : Caused by: java.net.ConnectException: 连接超时 (Connection timed out)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.PlainSocketImpl.socketConnect(Native Method)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.Socket.connect(Socket.java:589)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.Socket.connect(Socket.java:538)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.Socket.<init>(Socket.java:434)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.Socket.<init>(Socket.java:244)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    ... 29 more
  2021-09-13 10:26:12.555  INFO 22591 --- [launcher-proc-4] c.a.n.e.s.impl.SparkSubmitServiceImpl    : stateChanged:FAILED

原因分析

根据错误提示，大致应该是使用mysql连接不上，或者是连接超时

编者的真实错误原因

因为做了开发环境和测试环境的配置，且，测试环境和开发环境的网络不通，不小心使用的是测试环境的mysq数据库配置，知道是这个原因，把配置文件改成开发环境的即可

原因总结

1、需要连接的目标mysql服务器是否存在(或者是配置信息是否有误)

2、mysql的链接是否真的是超时的

和链接或者是连接池中的配置做比较

# 查看mysql的链接超时时间(单位是秒)
mysql> show global variables like 'wait_timeout';

可直接使用命令修改

mysql> set global wait_timeout=1814400;

比较搞笑的错误，耽误了这么老久，记录下。。。。。

当然，如果是设置了超时间之后仍然超时，那就可能是代码或者是网络的问题了

涟漪海洋

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
spark中遇到mysql链接超时的问题解决

背景最近使用scala写spark的运行jar，不小心遇到一个比较傻的坑，主要是mysql方面的报错，报错如下2021-09-13 10:26:12.426 INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application : Exception in thread "main" com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications .
复制链接

扫一扫

专栏目录