Hive连接Spark报错java.sql.SQLException: null, message from server: "Host 'datanode03' is blocked becaus

最新推荐文章于 2024-01-15 17:02:31 发布

笨鸟先飞-天道酬勤

最新推荐文章于 2024-01-15 17:02:31 发布

阅读量1.7k

点赞数 2

分类专栏：大数据踩过的坑文章标签： Hive Sql Mysql

本文链接：https://blog.csdn.net/dehu_zhou/article/details/81737947

版权

大数据踩过的坑专栏收录该内容

10 篇文章 1 订阅

订阅专栏

背景：

线上一些任务大部分使用Spark Sql来处理Hive的数据；今天任务由于数据量增大，任务耗时也增加，因此导致多个任务同时运行。但是后来任务迟迟运行不完，因此去查看任务日志；

16-08-2018 10:16:47 CST java.sql.SQLException: null,  message from server: "Host 'datanode03' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'"
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:963)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1042)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2253)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2284)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2083)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:806)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47)
16-08-2018 10:16:47 CST 	at sun.reflect.GeneratedConstructorAccessor54.newInstance(Unknown Source)
16-08-2018 10:16:47 CST 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
16-08-2018 10:16:47 CST 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:410)
16-08-2018 10:16:47 CST 	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:328)
16-08-2018 10:16:47 CST 	at java.sql.DriverManager.getConnection(DriverManager.java:664)
16-08-2018 10:16:47 CST 	at java.sql.DriverManager.getConnection(DriverManager.java:208)
16-08-2018 10:16:47 CST 	at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
16-08-2018 10:16:47 CST 	at com.jolbox.bonecp.BoneCP.obtainInternalConnection(BoneCP.java:269)
16-08-2018 10:16:47 CST 	at com.jolbox.bonecp.ConnectionHandle.(ConnectionHandle.java:242)
16-08-2018 10:16:47 CST 	at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
16-08-2018 10:16:47 CST 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
16-08-2018 10:16:47 CST 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
16-08-2018 10:16:47 CST 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
16-08-2018 10:16:47 CST 	at java.lang.Thread.run(Thread.java:745)
16-08-2018 10:16:48 CST 18/08/16 10:16:48 ERROR BoneCP: Failed to acquire connection to jdbc:mysql://datanode15:3306/hive?characterEncoding=UTF-8. Sleeping for 7000 ms. Attempts left: 0
16-08-2018 10:16:48 CST java.sql.SQLException: null,  message from server: "Host 'fg03' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'"
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:963)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1042)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2253)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2284)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2083)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:806)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47)
16-08-2018 10:16:48 CST 	at sun.reflect.GeneratedConstructorAccessor54.newInstance(Unknown Source)
16-08-2018 10:16:48 CST 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
16-08-2018 10:16:48 CST 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:410)
16-08-2018 10:16:48 CST 	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:328)
16-08-2018 10:16:48 CST 	at java.sql.DriverManager.getConnection(DriverManager.java:664)
16-08-2018 10:16:48 CST 	at java.sql.DriverManager.getConnection(DriverManager.java:208)
16-08-2018 10:16:48 CST 	at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
16-08-2018 10:16:48 CST 	at com.jolbox.bonecp.BoneCP.obtainInternalConnection(BoneCP.java:269)
16-08-2018 10:16:48 CST 	at com.jolbox.bonecp.ConnectionHandle.(ConnectionHandle.java:242)
16-08-2018 10:16:48 CST 	at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
16-08-2018 10:16:48 CST 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
16-08-2018 10:16:48 CST 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
16-08-2018 10:16:48 CST 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
16-08-2018 10:16:48 CST 	at java.lang.Thread.run(Thread.java:745)
16-08-2018 10:16:48 CST 18/08/16 10:16:48 ERROR PoolWatchThread: Error in trying to obtain a connection. Retrying in 7000ms

关键的一段：java.sql.SQLException: null, message from server: "Host 'datanode03' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'"

怀疑是Mysql的问题

百度查询Mysql问题

原因：同一个ip在短时间内产生太多（超过mysql数据库max_connect_errors的最大值）中断的数据库连接而导致的阻塞；

解决方法：
   1、提高允许的max_connection_errors数量：
　　① 进入Mysql数据库查看max_connection_errors： show variables like '%max_connect_errors%';
　 ② 修改max_connection_errors的数量为1000： set global max_connect_errors = 1000;
　　③ 查看是否修改成功：show variables like '%max_connect_errors%';

   2、使用mysqladmin flush-hosts 命令清理一下hosts文件（不知道mysqladmin在哪个目录下可以使用命令查找：whereis mysqladmin）；
　　① 在查找到的目录下使用命令修改：mysqladmin --socket=/tmp/kkimdb.sock --port=3306 -uhyman -p flush-hosts
　　备注：配置有master/slave主从数据库的要把主库和从库都修改一遍的（我就吃了这个亏明明很容易的几条命令结果折腾了大半天）；
   mysql> flush hosts; 也可以
      max_connect_errors是一个MySQL中与安全有关的计数器值，它负责阻止过多尝试失败的客户端以防止暴力破解密码的情况。max_connect_errors的值与性能并无太大关系。
      默认情况下，my.cnf文件中可能没有此行，如果需要设置此数值，手动添加即可。

原文链接：https://blog.csdn.net/fanyun_01/article/details/50922922

功能与作用
一般来说建议数据库服务器不监听来自网络的连接，仅仅通过sock连接，这样可以防止绝大多数针对mysql的攻击；如果必须要开启mysql的网络连接，则最好设置此值，以防止穷举密码的攻击手段。

由于是内网使用，仅做Hive的元数据库使用，因此使用一下解决方案：

1、果断 set global max_connect_errors = 10000;

2、同时，在crontab增加一个定时任务，定时执行mysqladmin flush-hosts 命令清理一下hosts文件

修改完成后，问题解决，任务正常运行，后面是否还有问题，还需要继续观察！