Hadoop集群运行简单实例

集群配置信息

请看这篇文章Hadoop3.3.0完全分布式集群部署
这里我们简单在Hadoop运行一下实例。

1. 在一开始运行时，就遇到了一个报错信息.

hadoop 找不到或无法加载org.apache.hadoop.mapreduce.v2.app.MRAppMaster

2. 解决方案：我们到mapred-site.xml 配置文件添加配置

<property>
   <name>mapreduce.admin.user.env</name>
   <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
   <name>yarn.app.mapreduce.am.env</name>
   <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>

3. 首先，我们启动Hadoop和yarn.start-dfs.sh和start-yarn.sh，然后查看进程。

[root@master mapreduce]# jps
5954 ResourceManager
7458 Jps
5465 NameNode
5709 SecondaryNameNode

4. 我们进入目录cd/opt/hadoop/hadoop/share/hadoop/mapreduce

[root@master mapreduce]# ls
hadoop-mapreduce-client-app-3.3.0.jar              hadoop-mapreduce-client-nativetask-3.3.0.jar
hadoop-mapreduce-client-common-3.3.0.jar           hadoop-mapreduce-client-shuffle-3.3.0.jar
hadoop-mapreduce-client-core-3.3.0.jar             hadoop-mapreduce-client-uploader-3.3.0.jar
hadoop-mapreduce-client-hs-3.3.0.jar               hadoop-mapreduce-examples-3.3.0.jar
hadoop-mapreduce-client-hs-plugins-3.3.0.jar       jdiff
hadoop-mapreduce-client-jobclient-3.3.0.jar        lib-examples
hadoop-mapreduce-client-jobclient-3.3.0-tests.jar  sources

5. 然后我们执行pi 2 3

[root@master mapreduce]# hadoop jar hadoop-mapreduce-examples-3.3.0.jar pi 2 3

6.我们通过自己的浏览器打开查看日志，前提是虚拟机需要关闭防火墙。

1. 在浏览器输入ip地址和默认端口号8088即可
   
2. 我们回到终端即可查看运行结果。

#执行中
2021-04-27 15:07:14,997 INFO mapreduce.Job: Running job: job_1619506446012_0002
2021-04-27 15:07:22,274 INFO mapreduce.Job: Job job_1619506446012_0002 running in uber mode : false
2021-04-27 15:07:22,313 INFO mapreduce.Job:  map 0% reduce 0%
2021-04-27 15:07:31,623 INFO mapreduce.Job:  map 100% reduce 0%
2021-04-27 15:07:44,488 INFO mapreduce.Job:  map 100% reduce 100%
2021-04-27 15:08:20,051 INFO mapreduce.Job: Job job_1619506446012_0002 completed successfully
2021-04-27 15:08:20,988 INFO mapreduce.Job: Counters: 54
	File System Counters

#结果value值
Job Finished in 71.074 seconds
Estimated value of Pi is 4.00000000000000000000