使用三台虚拟机进行搭建,分别为node1,node2,node3
配置互信,免密钥登陆
添加java环境
首选在orcle网站下载jdk安装包,传送门
解压安装到/export/servers/
目录下,修改/etc/profile
文件,添加java路径
export JAVA_HOME=/export/servers/jdk1.8.0_261
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
执行java -verion
命令,
到此,java环境配置完成
解压hadoop-3.1.3.tar.gz
这里将解压内容放在`/export/servers/``目录下
tar -zxvf hadoop-3.1.3.tar.gz -C /export/servers/
将hadoop路径配置到/etc/profile
export HADOOP_HOME=/export/servers/hadoop
export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath):$CLASSPATH
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"
添加完执行source /etc/profile
创建目录
mkdir /export/servers/hadoop/dfs
mkdir /export/servers/hadoop/dfs/data
mkdir /export/servers/hadoop/dfs/name
mkdir /export/servers/hadoop/tmp
修改hadoop配置文件/export/servers/etc/hadoop/hadoop-env.sh
,添加Java路径
export JAVA_HOME=/export/servers/jdk1.8.0_261
修改core-site.xml文件,文件所在目录/export/servers/hadoop-3.1.3/etc/hadoop
,添加下面内容
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://node1:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/export/servers/hadoop/tmp</value>
</property>
</configuration>
修改hdfs-site.xml ,添加下面内容
<configuration>
<property>
<name>dfs.namenode.http-address</name>
<!-- Master为当前机器名或者IP地址 -->
<value>hdfs://node1:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<!-- 以下为存放节点命名的路径 -->
<value>/export/servers/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<!-- 以下为存放数据命名的路径 -->
<value>/export/servers/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<!-- 备份次数,因为有2台DataNode-->
<value>2</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<!-- Web HDFS-->
<value>true</value>
</property>
</configuration>
修改yarn-site.xml文件,添加下面内容
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<!-- Master为当前机器名或者ip号 -->
<value>node1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<!-- Node Manager辅助服务 -->
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<!-- Node Manager辅助服务类 -->
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<!-- CPU个数,需要根据当前计算机的CPU设置-->
<value>1</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<!-- Resource Manager管理地址 -->
<value>node1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<!-- Resource Manager Web地址 -->
<value>node1:8088</value>
</property>
</configuration>
修改 mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<!-- MapReduce Framework -->
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<!-- MapReduce JobHistory, 当前计算机的IP -->
<value>node1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<!-- MapReduce Web App JobHistory, 当前计算机的IP -->
<value>node1:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/export/servers/hadoop</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/export/servers/hadoop</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/export/servers/hadoop</value>
</property>
</configuration>
修改sbin/start-yarn.sh和sbin/stop-yarn.sh文件,添加
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
修改./sbin/start-dfs.sh和./sbin/stop-dfs.sh
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
在sbin/hadoop/workers
里删除localhost添加node1 node2 node3
将配置好的文件传递到其他两个节点上
scp -r ./hadoop-3.1.3 node2:$PWD
scp -r ./hadoop-3.1.3 node3:$PWD
scp /etc/profile node2:/etc/
scp /etc/profile node3:/etc/
分别在node2和node3上执行source /etc/profile
启动hadoop
最后在node1上执行hadoop namenode -format
格式化文件系统
执行成功后,在node1执行./sbin/start-all.sh
,执行结果如下
在node1执行jps
显示如下
在node2上执行jps
访问http://172.20.17.132:9001显示
访问http://172.20.17.132:8088/cluster