目录
MapReduce的配置(主要是给开发用的,程序呢)
需要修改两个配置文件 mapred-site.xml和yarn-site.xml,在nn01节点安装之后,需要同步到其他的主机上.
[root@nn01 hadoop]# cd /usr/local/hadoop/etc/hadoop/
[root@nn01 hadoop]# cp mapred-site.xml.template mapred-site.xml
[root@nn01 hadoop]# vim mapred-site.xml
20 <property>
21 <name>mapreduce.framework.name</name> ---指定资源管理的的套件是什么,我们指定是yarn
22 <value>yarn</value>
23 </property>
解析:分布式计算框架mapred-site.xml只支持local和yarn两种,单机使用local,集群使用yaml
[root@nn01 hadoop]# vim yarn-site.xml
18 <property>
19 <name>yarn.resourcemanager.hostname</name> ----指定那台主机安装resourcemanager
20 <value>nn01</value>
21 </property>
22 <property>
23 <name>yarn.nodemanager.aux-services</name> ---使用哪种计算框架,是开发写的j计算框架的jar包
24 <value>mapreduce_shuffle</value>
25 </property>
[root@nn01 ~]# for i in {61..63}; do scp -r /usr/local/hadoop/ 192.168.1.$i:/usr/local/hadoop/ ; done
[root@nn01 hadoop]# ./sbin/start-yarn.sh ---启动resourcemanager和nodemanager两个节点
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-nn01.out
node3: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node3.out
node1: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node1.out
node2: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node2.out
[root@nn01 hadoop]# jps
2647 ResourceManager
1256 NameNode
2906 Jps
1439 SecondaryNameNode
[root@nn01 hadoop]# ssh node1 jps
1315 NodeManager
1411 Jps
1018 DataNode
[root@nn01 hadoop]# ssh node2 jps
983 DataNode
1279 NodeManager
1375 Jps
[root@nn01 hadoop]# ssh node3 jps
1280 NodeManager
1376 Jps
984 DataNode
验证
[root@nn01 hadoop]# /usr/local/hadoop/bin/yarn node -list
20/06/19 20:35:25 INFO client.RMProxy: Connecting to ResourceManager at nn01/192.168.1.60:8032
Total Nodes:3
Node-Id Node-State Node-Http-Address Number-of-Running-Containers
node1:43208 RUNNING node1:8042 0
node3:40913 RUNNING node3:8042 0
node2:36472 RUNNING node2:8042 0
hadoop统计词频
[root@nn01 hadoop]# /usr/local/hadoop/bin/hadoop --help #查看hadoop的一些操作
[root@nn01 hadoop]# /usr/local/hadoop/bin/hadoop fs #查看命令的一些参数
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]
[root@nn01 hadoop]# /usr/local/hadoop/bin/hadoop fs -ls /
[root@nn01 hadoop]# /usr/local/hadoop/bin/hadoop fs -mkdir /input
[root@nn01 hadoop]# /usr/local/hadoop/bin/hadoop