1.首先准备三台机器
centos1
centos2
centos3
2.设置从centos1到其他两台机器的免密登录
在centos1上执行命令:
ssh-keygen -t rsa
然后一路回车,接着执行如下命令:
ssh-copy-id -i ~/.ssh/id_rsa.pub root@centos2
ssh-copy-id -i ~/.ssh/id_rsa.pub root@centos3
3.下载jdk和hadoop安装包
jdk:https://www.oracle.com/cn/java/technologies/javase-downloads.html
hadoop:https://hadoop.apache.org/releases.html
下载之后,分别把两个安装包上传到centos1机器的/opt/目录下,再通过如下命令分别解压两个安装包:
tar -zxvf jdk-15.0.1
tar -zxvf hadoop-3.3.0.tar.gz
然后在三台机器上分别通过修改/etc/profile文件来设置环境变量:
export JAVA_HOME=/opt/jdk-15.0.1
export PATH=.:$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/opt/hadoop-3.3.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
然后执行命令使之生效:
source /etc/profile
4.配置hadoop
进入hadoop配置目录:
cd hadoop-3.3.0/etc/hadoop/
修改hadoop-env.sh文件:
export JAVA_HOME=/opt/jdk-15.0.1
修改core-site.xml文件:
<configuration>
<!--定义namenode地址 默认9000-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://centos1:9000</value>
</property>
</configuration>
修改yarn-site.xml文件:
<configuration>
<!-- 设置ResourceManager 域名 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>centos1</value>
</property>
<!-- 开启yarn.webapp.ui2 -->
<property>
<name>yarn.webapp.ui2.enable</name>
<value>true</value>
</property>
<!-- 默认为true, 当虚拟机内存不够多时,容易超出虚拟机内存 -->
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
</configuration>
修改workers文件:
centos2
centos3
然后再进入hadoop另一个目录:
cd /opt/hadoop-3.3.0/sbin
在start-dfs.sh和stop-dfs.sh文件开始处添加如下代码:
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
在start-yarn.sh和stop-yarn.sh文件开始处添加如下代码:
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
然后,把配置好的安装包复制到其他两台机器:
scp -r /opt/hadoop-3.3.0/ root@centos2:/opt/hadoop-3.3.0
scp -r /opt/hadoop-3.3.0/ root@centos3:/opt/hadoop-3.3.0
5.验证安装
格式化DFS:
hdfs namenode -format
启动hdfs和yarn服务:
start-all.sh
或者分别启动:
start-dfs.sh
start-yarn.sh
使用jps查看centos1的进程:
[root@centos1 sbin]# jps
14947 SecondaryNameNode
15220 ResourceManager
14695 NameNode
15755 Jps
使用jps查看centos2和centos3的进程:
[root@centos2 logs]# jps
5012 DataNode
5127 NodeManager
5289 Jps
然后,分别查看三台机器opt/hadoop-3.3.0/logs目录下的日志,保证没有错误出现
然后,查看centos1的端口号:
[root@centos1 sbin]# netstat -netlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:9870 0.0.0.0:* LISTEN 0 91670 14695/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 28170 915/sshd
tcp 0 0 192.168.106.128:8088 0.0.0.0:* LISTEN 0 94199 15220/java
tcp 0 0 192.168.106.128:8030 0.0.0.0:* LISTEN 0 94519 15220/java
tcp 0 0 192.168.106.128:8031 0.0.0.0:* LISTEN 0 94515 15220/java
tcp 0 0 192.168.106.128:8032 0.0.0.0:* LISTEN 0 94523 15220/java
tcp 0 0 192.168.106.128:8033 0.0.0.0:* LISTEN 0 94510 15220/java
tcp 0 0 192.168.106.128:9000 0.0.0.0:* LISTEN 0 92989 14695/java
tcp 0 0 0.0.0.0:9868 0.0.0.0:* LISTEN 0 94529 14947/java
tcp6 0 0 :::22 :::* LISTEN 0 28179 915/sshd
其中9870是hdfs服务管理台的端口,8088位yarn服务管理台的端口,浏览器访问如下:
yarn的ui2界面如下:
最后,感谢您看完此篇文章。