HA运行WC程序踩的坑

HA运行WC程序踩的坑

报错信息

root@node01:/opt/bigdata/hadoop-3.3.0/share/hadoop/mapreduce# hadoop jar hadoop-mapreduce-examples-3.3.0.jar wordcount /data/wc/input /data/wc/output
2021-03-28 10:34:35,763 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
2021-03-28 10:34:36,252 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1616898424484_0001
2021-03-28 10:34:38,651 INFO input.FileInputFormat: Total input files to process : 1
2021-03-28 10:34:40,455 INFO mapreduce.JobSubmitter: number of splits:2
2021-03-28 10:34:41,196 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1616898424484_0001
2021-03-28 10:34:41,199 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-03-28 10:34:41,602 INFO conf.Configuration: resource-types.xml not found
2021-03-28 10:34:41,602 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-03-28 10:34:42,349 INFO impl.YarnClientImpl: Submitted application application_1616898424484_0001
2021-03-28 10:34:42,395 INFO mapreduce.Job: The url to track the job: http://node04:8088/proxy/application_1616898424484_0001/
2021-03-28 10:34:42,396 INFO mapreduce.Job: Running job: job_1616898424484_0001
2021-03-28 10:35:08,630 INFO mapreduce.Job: Job job_1616898424484_0001 running in uber mode : false
2021-03-28 10:35:08,704 INFO mapreduce.Job:  map 0% reduce 0%
2021-03-28 10:35:08,789 INFO mapreduce.Job: Job job_1616898424484_0001 failed with state FAILED due to: Application application_1616898424484_0001 failed 2 times due to AM Container for appattempt_1616898424484_0001_000002 exited with  exitCode: 1
Failing this attempt.Diagnostics: [2021-03-28 10:35:07.973]Exception from container-launch.
Container id: container_1616898424484_0001_02_000001
Exit code: 1

[2021-03-28 10:35:08.004]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.


[2021-03-28 10:35:08.004]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.


For more detailed output, check the application tracking page: http://node04:8088/cluster/app/application_1616898424484_0001 Then click on links to logs of each attempt.

解决方法

查看日志发现是空指针异常

由于用的是3.X版本，需要在yarn-site.xml中配置加上

<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>node03:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm2</name>
  <value>node04:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.address.rm1</name>
  <value>node03:8030</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.address.rm2</name>
  <value>node04:8030</value>
</property>

虽然不加是用默认的端口

但是回报空指针异常