Application state is completed.FinalApplicationStatus=SUCCEEDED.Redirecting to job history server的处理

EEEurekaaa！

于 2023-01-05 16:54:43 发布

阅读量322

点赞数

文章标签： java hadoop hdfs apache 服务器

本文链接：https://blog.csdn.net/m0_52940881/article/details/128566135

版权

今天在azkaban调度运行我的网站分析系统时，在运行数据预处理的jar包时出现了如下错误。

2023-01-05 16:15:21,554 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2023-01-05 16:15:21,554 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
05-01-2023 16:15:23 CST preprocess INFO - 2023-01-05 16:15:23,174 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:24 CST preprocess INFO - 2023-01-05 16:15:24,187 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:25 CST preprocess INFO - 2023-01-05 16:15:25,221 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:26 CST preprocess INFO - 2023-01-05 16:15:26,598 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:28 CST preprocess INFO - 2023-01-05 16:15:27,722 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:29 CST preprocess INFO - 2023-01-05 16:15:29,478 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:30 CST preprocess INFO - 2023-01-05 16:15:30,481 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:31 CST preprocess INFO - 2023-01-05 16:15:31,485 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:32 CST preprocess INFO - 2023-01-05 16:15:32,502 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:33 CST preprocess INFO - 2023-01-05 16:15:33,540 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:33 CST preprocess INFO - 2023-01-05 16:15:33,677 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
05-01-2023 16:15:34 CST preprocess INFO - 2023-01-05 16:15:34,679 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:35 CST preprocess INFO - 2023-01-05 16:15:35,682 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:36 CST preprocess INFO - 2023-01-05 16:15:36,685 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:37 CST preprocess INFO - 2023-01-05 16:15:37,686 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:38 CST preprocess INFO - 2023-01-05 16:15:38,688 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:39 CST preprocess INFO - 2023-01-05 16:15:39,690 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:40 CST preprocess INFO - 2023-01-05 16:15:40,692 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:41 CST preprocess INFO - 2023-01-05 16:15:41,694 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:42 CST preprocess INFO - 2023-01-05 16:15:42,695 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:43 CST preprocess INFO - 2023-01-05 16:15:43,697 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:44 CST preprocess INFO - 2023-01-05 16:15:44,041 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
05-01-2023 16:15:45 CST preprocess INFO - 2023-01-05 16:15:45,134 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:46 CST preprocess INFO - 2023-01-05 16:15:46,163 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:47 CST preprocess INFO - 2023-01-05 16:15:47,191 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:48 CST preprocess INFO - 2023-01-05 16:15:48,193 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:49 CST preprocess INFO - 2023-01-05 16:15:49,196 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:50 CST preprocess INFO - 2023-01-05 16:15:50,199 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:51 CST preprocess INFO - 2023-01-05 16:15:51,297 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:52 CST preprocess INFO - 2023-01-05 16:15:52,300 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:53 CST preprocess INFO - 2023-01-05 16:15:53,375 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:54 CST preprocess INFO - 2023-01-05 16:15:54,378 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
05-01-2023 16:15:54 CST preprocess INFO - Exception in thread "main" java.io.IOException: java.net.ConnectException: Your endpoint configuration is wrong; For more details see:  http://wiki.apache.org/hadoop/UnsetHostnameOrPort
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:345)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:430)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:870)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:331)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:328)
05-01-2023 16:15:55 CST preprocess INFO - 	at java.security.AccessController.doPrivileged(Native Method)
05-01-2023 16:15:55 CST preprocess INFO - 	at javax.security.auth.Subject.doAs(Subject.java:422)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:328)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapreduce.Job.isSuccessful(Job.java:624)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1603)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.example.LogCleaner.cleanDriver.main(cleanDriver.java:45)
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
05-01-2023 16:15:55 CST preprocess INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
05-01-2023 16:15:55 CST preprocess INFO - Caused by: java.net.ConnectException: Your endpoint configuration is wrong; For more details see:  http://wiki.apache.org/hadoop/UnsetHostnameOrPort
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
05-01-2023 16:15:55 CST preprocess INFO - 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.Client.call(Client.java:1508)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.Client.call(Client.java:1405)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
05-01-2023 16:15:55 CST preprocess INFO - 	at com.sun.proxy.$Proxy18.getJobReport(Unknown Source)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
05-01-2023 16:15:55 CST preprocess INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:326)
05-01-2023 16:15:55 CST preprocess INFO - 	... 17 more
05-01-2023 16:15:55 CST preprocess INFO - Caused by: java.net.ConnectException: 拒绝连接
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
05-01-2023 16:15:55 CST preprocess INFO - 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:699)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
05-01-2023 16:15:55 CST preprocess INFO - 	at org.apache.hadoop.ipc.Client.call(Client.java:1452)
05-01-2023 16:15:55 CST preprocess INFO - 	... 26 more
05-01-2023 16:16:26 CST preprocess INFO - 2023-01-05 16:16:26,101 WARN util.ShutdownHookManager: ShutdownHook '' timeout, java.util.concurrent.TimeoutException
05-01-2023 16:16:26 CST preprocess INFO - java.util.concurrent.TimeoutException
05-01-2023 16:16:26 CST preprocess INFO - 	at java.util.concurrent.FutureTask.get(FutureTask.java:205)
05-01-2023 16:16:26 CST preprocess INFO - 	at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:124)
05-01-2023 16:16:26 CST preprocess INFO - 	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95)
05-01-2023 16:16:32 CST preprocess INFO - Process with id 10225 completed unsuccessfully in 570 seconds.
05-01-2023 16:16:32 CST preprocess ERROR - Job run failed!
java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:312)
	at azkaban.execapp.JobRunner.runJob(JobRunner.java:947)
	at azkaban.execapp.JobRunner.doRun(JobRunner.java:697)
	at azkaban.execapp.JobRunner.run(JobRunner.java:653)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
	at azkaban.jobExecutor.utils.process.AzkabanProcess.run(AzkabanProcess.java:125)
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:304)
	... 8 more

之后在后续的多次尝试时而会出现报错，时而又正常运行。上面的错误提示是连接某个集群下的节点失败，尝试连接历史服务也失败，之所以连接历史服务失败是因为我根本没配置，奇怪的是不知道为什么会出现该错误，猜测可能是mapreduce任务执行过程中某台节点因为卡顿问题挂掉了，所以要通过历史服务对该节点进行重新连接，但是由于我没有配置所以报错。

配置方法：

在mapred-site.xml中加入以下配置	

<property>  
		<name>mapreduce.jobhistory.address</name>  
		<value>hadoop2:10020</value>  
	</property>
	<property>  
		<name>mapreduce.jobhistory.webapp.address</name>  
		<value>hadoop2:19888</value>  
	</property>

然后分发到节点的所以虚拟机上

ps：在你使用start-all.sh启动集群相关进程时，该进程默认不启动，需要手动启动

命令：
mapred --daemon start historyserver

EEEurekaaa！

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
Application state is completed.FinalApplicationStatus=SUCCEEDED.Redirecting to job history server的处理

上面的错误提示是连接某个集群下的节点失败，尝试连接历史服务也失败，之所以连接历史服务失败是因为我根本没配置，奇怪的是不知道为什么会出现该错误，猜测可能是mapreduce任务执行过程中某台节点因为卡顿问题挂掉了，所以要通过历史服务对该节点进行重新连接，但是由于我没有配置所以报错。今天在azkaban调度运行我的网站分析系统时，在运行数据预处理的jar包时出现了如下错误。ps：在你使用start-all.sh启动集群相关进程时，该进程默认不启动，需要手动启动。然后分发到节点的所以虚拟机上。
复制链接

扫一扫