3.3配置yarn
3.3.1配置mapred-site.xml
cd $HADOOP_HOME/etc/hadoop #进入hadoop的配置文件目录
配置mapred-site.xml
vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.4</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.4</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.4</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
</configuration>
3.3.2配置yarn-site.xml
vi yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
3.3.3启动yarn
cd $HADOOP_HOME #切换到hadoop安装目录
sbin/start-yarn.sh #启动yarn
sbin/stop-yarn.sh #关闭yarn
#输入jps查看情况
46936 NodeManager
46762 ResourceManager
3.3.4查看可视化界面
查看hdfs运行情况
查看yarn运行情况
3.3.5使用hadoop的example跑wordcount
Sanxingdui Ruins,known as the "The Ninth Wonder of the World",are the largest ruins of the ancient Shu Kingdom discovered so far in Southwest China.Sanxingdui site has also unearthed more than 10,000 cultural relics,all dating back to 5,000 to 3,000 BC.These cultural relics include gold,pottery and ivory products with exquisite craftsmanship and peculiar shapes.They are of great value to the study of the process of early countries and the development of religious consciousness and occupy an important position in the history of human civilization.
vi /1.txt #将上面的原文存进来
hdfs dfs -put /1.txt /
hadoop jar hadoop-mapreduce-examples-3.2.4.jar wordcount /1.txt /result
#通过可视化,可以看到hdfs /目录下多了一个result目录
hdfs dfs -cat /result/part-r-00000
4.4简易集群搭建
4.4.1配置静态ip
vi /etc/sysconfig/network-scripts/ifcfg-ens33
将BOOTPROTO=dhcp改为=static
ONBOOT=no改为yes
在末尾添加
IPADDR=192.168.XX.200 #用自己的ip
NETMASK=255.255.255.0
GATEWAY=192.168.XX.2 #用自己的ip
DNS1=114.114.114.114
#重新加载网络设置
service network restart
4.4.2集群规划
#修改当前主机名称
hostnamectl set-hostname master
bash #重置
将原有的内容修改为master保存并重启虚拟机后显示
4.4.3修改hosts设置
vi /etc/hosts #文末添加
192.168.xx.200 master
192.168.xx.201 slave1
192.168.xx.202 slave2
192.168.xx.203 slave3
输入ping master测试是否连接
4.4.5解压hadoop与jdk并配置环境变量
解压
tar -zxf /opt/jdk-8u221-linux-x64.tar.gz -C /usr/local/
tar -zxf /opt/hadoop-3.2.4.tar.gz -C /usr/local/
配置环境变量
vi /etc/profile #文末添加以下内容
export JAVA_HOME=/usr/local/jdk1.8.0_221
export HADOOP_HOME=/usr/local/hadoop-3.2.4
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
source /etc/profile #使文件生效
4.4.6修改hadoop配置文件
cd /usr/local/hadoop-3.2.4/etc/hadoop #配置文件在此目录
在文件开头添加如下内容
HDFS_NAMENODE_USER=root
HDFS_DATANODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
YARN_RESOURCEMANAGER_USER=root
YARN_NODEMANAGER_USER=root
export JAVA_HOME=/usr/local/jdk1.8.0_221
保存并退出
vi core-site.xml #对hdfs的通信端口进行指定
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop</value>
</property>
</configuration>
vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
保存并退出
cd $HADOOP_HOME/etc/hadoop #进入hadoop的配置文件
配置mapred-site.xml
vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.4</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.4</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.4</value>
</property>
</configuration>
vi yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
vi workers #添加被管理的datanode节点
删除所有数据并添加代表着两个节点存数据
master
slave1
4.4.7删除运行痕迹
#从伪分布式搭建,需要删除伪分布式hadoop运行痕迹
rm -fr /tmp/*
rm -fr $HADOOP_HOME/logs/*
如出现问题,所有节点都需要删除,最后重新格式化
4.4.8关闭防火墙并实现免密
systemctl stop firewalld #关闭防火墙
systemctl disable firewalld #开机不开启防火墙
systemctl status firewalld #查看防火墙状态
ssh-keygen #生成密钥(多次回车)
cat ~/.ssh/id_rsa.pub >> ~./ssh/authorized_keys #追加密钥到钥匙包里面
如果后续启动集群显示免密失败,重新生成master的key,然后发送给slave1
ssh-keygen
cat ~/.ssh/id_rsa.pub >> ~./ssh/authorized_keys
ssh-copy-id -i root@slave1
同时在salve1上生成密钥,发送给自己,然后再发送给slave1
ssh-keygen
cat ~/.ssh/id_rsa.pub >> ~./ssh/authorized_keys
ssh-copy-id -i root@master
输入ssh locahost测试是否对自己免密远程登录
输入logout退出当前远程登录
4.4.9关机并克隆虚拟机
1.1修改slave1主机名
先将slave1开机 将克隆的主机名称修改为slave1
hostnamectl set-hostname slave1
bash #重置
1.2分别修改来两个主机的ip
需要修改slave1的ip
vi /etc/sysconfig/network-scripts/ifcfg-ens33
将BOOTPROTO=dhcp改为=static
ONBOOT=no改为yes
在末尾添加
IPADDR=192.168.XX.201 #用自己的ip
NETMASK=255.255.255.0
GATEWAY=192.168.XX.2 #用自己的ip
DNS1=114.114.114.114
#重新加载网络设置
service network restar
1.3格式化集群
将master开机-在主节点master上操作(不是slave节点)
hdfs namenode -format
#如果启动失败,有可能需要重新格式化
首先删除hadoop运行痕迹
rm -rf /tmp/*
再hdfs namenode -format
1.4启动集群
在master节点操作
cd $HADOOP_HOME
sbin/start-dfs.sh #启动hdfs集群
sbin/stop-dfs.sh #停止hdfs集群
sbin/start-yarn.sh #启动yarn集群
sbin/stop-yarn.sh #停止yarn集群