1.同步本地时间
安装chrony
# dnf install chrony #安装chrony # dnf install vim #安装vim # vim /etc/chrony.conf #打开文件
server cn.ntp.org.cn iburst server ntp.ntsc.ac.cn iburst
添加上面两行,并将第三行的pool。。。注释效果如图
# systemctl enable chronyd.service # systemctl restart chronyd.service # chronyc sources -v
三台机器同样的操作;
2.检查ssh免密登录
[root@cSlave1 joe]# rpm -qa | grep ssh
如图即可:
3.安装Java环境
输入命令下载
# dnf install java-1.8.0-openjdk java-1.8.0-openjdk-devel #查找java # which java /usr/bin/java # ls -lr /usr/bin/java lrwxrwxrwx. 1 root root 22 10月 6 12:17 /usr/bin/java -> /etc/alternatives/java # ls -lrt /etc/alternatives/java lrwxrwxrwx. 1 root root 73 10月 6 12:17 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64/jre/bin/java #所以jdk的安装路径位:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64
#配置java环境变量 # vim ~/.bashrc
#在文末添加 export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64 #按ESC 输入 :wq! 退出 #更新文件 # source ~/.bashrc
#查看java版本 # java -version openjdk version "1.8.0_302" # $JAVA_HOME/bin/java -version openjdk version "1.8.0_302" #比较上面两个。相同则成功
4.关闭防火墙
#查看防火墙状态 # systemctl status firewalld.service #关闭防火墙 # systemctl stop firewalld.service #关闭防火墙自启动 # systemctl disable firewalld.service #重启网络 # nmcli c reload
利用克隆即可创建其他的。
5.修改机器名
(可以在安装的时候修改,我选择在命令行修改)
# hostname localhost.localdomain # hostnamectl set-hostname cMaster # hostname cMaster
6.添加域名映射
查找ip地址
[joe@cMaster ~]$ ifconfig
[root@cMaster joe]# vim /etc/hosts
添加文本:
192.168.72.139 cMaster 192.168.72.136 cSlave1 192.168.72.140 cSlave2
检查虚拟机内是否实现网内互联
ping 主机名
7. ssh免密登录
在cMaster机器上操作
#生成密钥,一路回车 $ ssh-keygen -t rsa
#将密钥copy给其他机器,输入yes,然后输入密码 $ ssh-copy-id cSlave1 #测试是否能登陆 [root@cMaster joe]# ssh cslave1
重复上述步骤(!测试之后,记得退出即输入:exit)直达cMaster能够免密登录其他机器和本身
8.在cMaster上安装Hadoop
(1)安装
将安装包拷贝到虚拟机的主目录下
解压安装到./文件
$ tar -zxf /home/joe/hadoop-3.3.1.tar.gz -C ./
修改文件名为hadoop
修改用户权限
# sudo chown -R joe:joe ./hadoop
查看安装成功与否
$ cd ./hadoop
./bin/hadoop version
出现:
出现该问题:ERROR: JAVA_HOME is not set and could not be found.
解决方法:
# vim /home/joe/hadoop/etc/hadoop/hadoop-env.sh
写入:
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64
(2) 配置PATH变量
# vim hadoop-env.sh
#写入下面的文件: export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64 export HADOOP_CONF_DIR=/home/joe/hadoop/etc/hadoop
vim ~/.bashrc
#写入下面的文件 export PATH=$PATH:/home/joe/hadoop/bin:/home/joe/hadoop/sbin export JAVA_LIBRARY_PATH=/usr/local/lib/hadoop-native
(2)修改hadoop配置文件
修改slaves文件
# cd /home/joe/hadoop/etc/hadoop # vim ./slaves
添加下面两行
cSlave1 cSlave2
修改core-site.xml
# vim core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://cMaster:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file: <value>file:/home/joe/hadoop/tmp</value> <description>Abase for other temporary dorectories.</description> </property> </configuration>
修改hdfs-site.xml
# vim hdfs-site.xml
<configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>cMaster:50090</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/home/joe/hadoop/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/home/joe/hadoop/tmp/dfs/data</value> </property> </configuration>
修改mapred-site.xml
# vim mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>cMaster:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>cMaster:19888</value> </property> </configuration>
修改yarn-site.xml
# vim yarn-site.xml
<configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.resourcemanager.hostname</name> <value>cMaster</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
(3)将hadoop复制到其他结点
传输hadoop
# cd ~ #压缩文件 # tar -zcf /home/joe/hadoop-master.tar.gz /home/joe/hadoop #传输文件 # scp /home/joe/hadoop-master.tar.gz cSlave1:/home/joe
在cSlave1上
# cd ~ # tar -zxf /home/joe/hadoop-master.tar.gz -C /home/joe
其他机器同理;
9.启动Hadoop集群
开启: /home/joe/hadoop/sbin/start-dfs.sh /home/joe/hadoop/sbin/start-yarn.sh 关闭: /home/joe/hadoop/sbin/stop-dfs.sh /home/joe/hadoop/sbin/stop-yarn.sh
*出现问题:
问题(1)
ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation. Starting datanodes ERROR: Attempting to operate on hdfs datanode as root ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation. Starting secondary namenodes [cMaster] ERROR: Attempting to operate on hdfs secondarynamenode as root ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
解决:
# vim hadoop-env.sh
加入以下文本到最后一行:
export HDFS_NAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export YARN_RESOURCEMANAGER_USER=root export YARN_NODEMANAGER_USER=root
参考:
(26条消息) 【详细】VMware虚拟机配置Hadoop集群实验[CentOS8]_Coder_C9的博客-CSDN博客centos7:ssh免密登陆设置及常见错误 - 平复心态 - 博客园 (cnblogs.com)
Java - CentOS下JDK的安装教程(及JAVA_HOME配置、以jdk1.8为例) (hangge.com)Apache Hadoop 3.1.2 – Hadoop Cluster Setup
(26条消息) hadoop入门1:ERROR Cannot set priority of datanode process_小钢炮-CSDN博客(26条消息) CentOS 下搭建 Hadoop:Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).解决_haoweixl的博客-CSDN博客(26条消息) linux查看文件夹权限_manongxianfeng的博客-CSDN博客_查看文件夹权限