Centos7 hadoop完全分布式集群搭建
环境准备:centos7*4台
节点分布介绍:
node1 | node2 | node3 | node4 |
---|---|---|---|
192.168.7.33 | 192.168.7.34 | 192.168.7.35 | 192.168.7.36 |
Namenode(NN) | SecondaryNameNode(SNN) | ||
NodeManager(NM) | NodeManager(NM) | NodeManager(NM) | NodeManager(NM) |
DataNode(DN) | DataNode(DN) | DataNode(DN) | DataNode(DN) |
ResourceManager(RM) |
1.安装java1.8环境
2.修改4台主机hostname,并将一下内容添加/etc/hosts
192.168.7.33 node1
192.168.7.34 node2
192.168.7.35 node3
192.168.7.36 node4
3.同步四台服务器时间,误差不超过30s.(最好配置时间同步服务)
4.设置master的免密码登陆
———————————————————————————————————————————————
hadoop 集群是通过主节点的 rpc 调用来对整个集群进行统一的操作管理,如果不配置免密登录,在每次启动集群时需要输入每个从节点的机器密码,免密登录很好的解决此问题
———————————————————————————————————————————————
分别在node2,3,4上生成ssh链接的公钥和私钥,并将公钥追加到node1:root/.ssh/authorized_keys(我这里是用的root用户(生产环境建议单独建立一个hadoop的统一账户并赋予sudo权限)
5.关闭防火墙,关闭开机自启动
6.安装hadoop(所有节点)
sudo install -d -o hadoop -g hadoop /usr/local/hadoop
tar -zxvf hadoop-3.2.0.tar.gz -C /usr/local/hadoop
sudo vim /etc/profile
#添加以下环境变量
export HADOOP_HOME=/usr/local/hadoop/hadoop-3.2.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
# 创建文件夹
mkdir -p /usr/local/hadoop/tmp
mkdir -p /usr/local/hadoop/var
mkdir -p /usr/local/hadoop/dfs
mkdir -p /usr/local/hadoop/dfs/name
mkdir -p /usr/local/hadoop/dfs/data
7.修改配置文件(所有节点)
修改/usr/local/hadoop/hadoop-3.2.0/etc/hadoop/下的这些文件,(各个配置文件参数详解可参考官网https://hadoop.apache.org/docs/r3.2.1/)
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://node1:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
</configuration>
hadoop-env.sh
# 使用root启动则将user修改为root
export HADOOP_HOME=/usr/local/hadoop/hadoop-3.2.0
export HDFS_NAMENODE_USER=hadoop
export HDFS_DATANODE_USER=hadoop
export HDFS_SECONDARYNAMENODE_USER=hadoop
export YARN_RESOURCEMANAGER_USER=hadoop
export YARN_NODEMANAGER_USER=hadoop
yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>node1</value>
</property>
<property>
<name>yarn.resourcemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>node1:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>node2:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
mapred-site.xml
configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
workers
node1
node2
node3
node4
masters
node2
8.格式化namenode,并启动集群(node1)
hdfs namenode -format
start-dfs.sh
start-yarn.sh
# 检查启动状况
jps
#查看集群状态
Namenode:http://192.168.7.33:50070
ResourceManager:http://192.168.7.33:8088