Hive执行hql任务时，一直卡在mapreduce提交job处

问题描述：

Job Submission failed with exception 'java.net.ConnectException(Call From hadoop102/192.168.161.102 to hadoop103:8032 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

 

错误日志：
0: jdbc:hive2://hadoop102:10000> select deptno,MAX(sal) from db_hive_demo.emp group by deptno;
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
[3]+  Done                    bin/hive --service hiveserver2
[atguigu@hadoop102 hive-0.13.1-cdh5.3.6]$ java.net.ConnectException: Call From hadoop102/192.168.161.102 to hadoop103:8032 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
        at org.apache.hadoop.ipc.Client.call(Client.java:1479)
        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy22.getNewApplication(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:221)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy23.getNewApplication(Unknown Source)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:219)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:227)
        at org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:187)
        at org.apache.hadoop.mapred.YARNRunner.getNewJobID(YARNRunner.java:231)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:153)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
        at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:139)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:155)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1554)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1321)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1139)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:957)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:144)
        at org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:68)
        at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:199)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
        at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:212)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: 拒绝连接
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
        at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
        at org.apache.hadoop.ipc.Client.call(Client.java:1451)
        ... 50 more
Job Submission failed with exception 'java.net.ConnectException(Call From hadoop102/192.168.161.102 to hadoop103:8032 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

[1]-  Exit 143                bin/hive --service metastore
[atguigu@hadoop102 hive-0.13.1-cdh5.3.6]$ 
[atguigu@hadoop102 hive-0.13.1-cdh5.3.6]$ bin/hive --service hiveserver2 &
[3] 7208
[2]   Exit 143                bin/hive --service hiveserver2
[atguigu@hadoop102 hive-0.13.1-cdh5.3.6]$ Starting HiveServer2
19/02/01 19:27:34 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead

[atguigu@hadoop102 hive-0.13.1-cdh5.3.6]$ bin/beeline
Beeline version 0.13.1-cdh5.3.6 by Apache Hive
beeline> select * from db_hive_demo.emp;                              
No current connection
beeline> !connect jdbc:hive2://hadoop102:10000                        
scan complete in 3ms
Connecting to jdbc:hive2://hadoop102:10000
Enter username for jdbc:hive2://hadoop102:10000: 
Enter password for jdbc:hive2://hadoop102:10000: 
Connected to: Apache Hive (version 0.13.1-cdh5.3.6)
Driver: Hive JDBC (version 0.13.1-cdh5.3.6)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://hadoop102:10000> select * from db_hive_demo.emp;      
OK
+------------+------------+------------+----------+---------------+----------+-----------+-------------+--+
| emp.empno  | emp.ename  |  emp.job   | emp.mgr  | emp.hiredate  | emp.sal  | emp.comm  | emp.deptno  |
+------------+------------+------------+----------+---------------+----------+-----------+-------------+--+
| 7369       | SMITH      | CLERK      | 7902     | 1980-12-17    | 800.0    | NULL      | 20          |
| 7499       | ALLEN      | SALESMAN   | 7698     | 1981-2-20     | 1600.0   | 300.0     | 30          |
| 7521       | WARD       | SALESMAN   | 7698     | 1981-2-22     | 1250.0   | 500.0     | 30          |
| 7566       | JONES      | MANAGER    | 7839     | 1981-4-2      | 2975.0   | NULL      | 20          |
| 7654       | MARTIN     | SALESMAN   | 7698     | 1981-9-28     | 1250.0   | 1400.0    | 30          |
| 7698       | BLAKE      | MANAGER    | 7839     | 1981-5-1      | 2850.0   | NULL      | 30          |
| 7782       | CLARK      | MANAGER    | 7839     | 1981-6-9      | 2450.0   | NULL      | 10          |
| 7788       | SCOTT      | ANALYST    | 7566     | 1987-4-19     | 3000.0   | NULL      | 20          |
| 7839       | KING       | PRESIDENT  | NULL     | 1981-11-17    | 5000.0   | NULL      | 10          |
| 7844       | TURNER     | SALESMAN   | 7698     | 1981-9-8      | 1500.0   | 0.0       | 30          |
| 7876       | ADAMS      | CLERK      | 7788     | 1987-5-23     | 1100.0   | NULL      | 20          |
| 7900       | JAMES      | CLERK      | 7698     | 1981-12-3     | 950.0    | NULL      | 30          |
| 7902       | FORD       | ANALYST    | 7566     | 1981-12-3     | 3000.0   | NULL      | 20          |
| 7934       | MILLER     | CLERK      | 7782     | 1982-1-23     | 1300.0   | NULL      | 10          |
+------------+------------+------------+----------+---------------+----------+-----------+-------------+--+
14 rows selected (2.515 seconds)
0: jdbc:hive2://hadoop102:10000> select deptno,MAX(sal) from db_hive_demo.emp group by deptno;
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1549020275946_0001, Tracking URL = http://hadoop104:8088/proxy/application_1549020275946_0001/
Kill Command = /opt/module/hadoop-2.7.2/bin/hadoop job  -kill job_1549020275946_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2019-02-01 19:28:28,843 Stage-1 map = 0%,  reduce = 0%
2019-02-01 19:28:38,512 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.09 sec
2019-02-01 19:28:50,579 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 4.55 sec
MapReduce Total cumulative CPU time: 4 seconds 550 msec
Ended Job = job_1549020275946_0001
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 4.55 sec   HDFS Read: 879 HDFS Write: 30 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 550 msec
OK
+---------+---------+--+
| deptno  |   _c1   |
+---------+---------+--+
| 10      | 5000.0  |
| 20      | 3000.0  |
| 30      | 2850.0  |
+---------+---------+--+
3 rows selected (42.384 seconds)
0: jdbc:hive2://hadoop102:10000> 

错误原因：

首先自己写了一个集群自动化启动脚本，执行脚本启动datanode、namenode、resourcemanager、secondaryNamenode这些节点，
启动脚本的resourcemanager是在hadoop104启动
但是在yarn-site.xml中配置的是在hadoop103启动
脚本启动时resourcemanager其实没有启动成功，才会一直报ConnectException: 拒绝连接  的错误

 

解决方案：

将启动脚本和yarn-site.xml中的resourcemanager节点对应起来就会解决