5.1HADOOP部署
hadoop官网:https://hadoop.apache.org/releases.html
5.1.1 配置
解压hadoop压缩包
[dxm@VM-4-15-centos hadoop-2.10.2]$ mkdir hadoop
[dxm@VM-4-15-centos hadoop-2.10.2]$ tar -zxvf /home/dxm/hadoop-2.10.2.tar.gz -C /home/dxm/hadoop/
jdk默认部署好了
配置hadoop环境变量
##1.进行编辑
[root@VM-4-15-centos ~]# sudo vim /etc/profile.d/my_env.sh
##2.增加如下环境变量
##JAVA_HOME
JAVA_HOME=/usr/java/jdk1.8.0_121
export JAVA_HOME
PATH=$PATH:$JAVA_HOME/bin
export PATH
##HADOOP_HOME
export HADOOP_HOME=/home/dxm/hadoop/hadoop-2.10.2
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
##FLINK需要
export HADOOP_CLASSPATH=`hadoop classpath`
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
##3.重新执行刚刚修改的文件
[root@VM-4-15-centos ~]# source /etc/profile.d/my_env.sh
5.1.1.1 单机环境部署
服务器配置:
1 * 腾讯云服务器2核2G
修改home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://127.0.0.1:9000</value>
</property>
</configuration>
修改home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/yarn-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
创建文件home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
修改home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/datanode</value>
</property>
</configuration>
创建目录
sudo mkdir -p /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/namenode
sudo mkdir -p /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/datanode
格式化HDFS
hadoop namenode -format
5.1.1.2 集群环境部署
服务器配置:
1 * 华为云耀云服务器2核4G fcs01
1 * 华为云耀云服务器1核2G fcs03
1 * 腾讯云服务器2核2G fcs02
host配置
vim /etc/hosts
##fcs01示例 fcs01填内网地址,fcs02、fcs03填外网地址
fcs01 192.168.0.25
fcs02 xxx.xxx.xx.xx
fcs03 xxx.xxx.xx.xx
##fcs02示例 fcs02填内网地址,fcs01、fcs03填外网地址
fcs01 xxx.xxx.xx.xx
fcs02 192.168.0.21
fcs03 xxx.xxx.xx.xx
##fcs03示例 fcs03填内网地址,fcs01、fcs02填外网地址
fcs01 xxx.xxx.xx.xx
fcs02 xxx.xxx.xx.xx
fcs03 192.168.0.27
##2.关闭防火墙
[root@hecs-37480 ~]# systemctl stop firewalld
[root@hecs-37480 ~]# systemctl disable firewalld
集群规划
fcs01 | fcs02 | fcs03 | |
---|---|---|---|
HDFS | DataNode | DataNode NameNode | DataNode SecondaryNameNode |
YARN | NodeManager ResourceManager | NodeManager | NodeManager |
(1)核心配置文件
配置core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- 指定NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://fcs02:8020</value>
</property>
<!-- 指定hadoop数据的存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/home/dxm/hadoop/hadoop-2.10.2/data</value>
</property>
<!-- 配置HDFS网页登录使用的静态用户为dxm -->(配置之后普通用户就可以在网页端有删除文件的权限了)
<property>
<name>hadoop.http.staticuser.user</name>
<value>dxm</value>
</property>
</configuration>
(2)HDFS配置文件
配置hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- NameNode web端访问地址-->
<property>
<name>dfs.namenode.http-address</name>
<value>fcs02:50070</value>
</property>
<!-- SecondaryNameNode web端访问地址-->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>fcs03:50080</value>
</property>
<!-- 配置nameNode元数据存储位置-->
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/namenode</value>
</property>
<!-- 配置dataNode元数据存储位置-->
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/datanode</value>
</property>
<!-- 指定DataNode的节点配置文件 -->
<property>
<name> dfs.hosts </name>
<value>/home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/slaves</value>
</property>
</configuration>
(3)YARN配置文件
配置yarn-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- 指定MapReduce走shuffle -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定ResourceManager的地址-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>fcs01</value>
</property>
</configuration>
(4)MapReduce配置
配置mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- 指定MapReduce程序运行在Yarn上-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
(5)修改slaves改成如下内容
fcs01
fcs02
fcs03
(6)将配置文件分发到各个节点
xsync配置可以参考文末xsync脚本
[dxm@hecs-37480 hadoop]$ xsync /home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/
[dxm@hecs-37480 hadoop-2.10.2]$ mkdir -p /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/namenode
[dxm@hecs-37480 hadoop-2.10.2]$ mkdir -p /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/datanode
[dxm@hecs-37480 hadoop-2.10.2]$ xsync /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/cd
如果不想配置xsync,也可以将上述配置复制到剩余节点
(7)格式化HDFS
hadoop namenode -format
5.1.2 启动
/home/dxm/hadoop/hadoop-2.10.2/sbin/start-dfs.sh
/home/dxm/hadoop/hadoop-2.10.2/sbin/start-yarn.sh
访问hadoop resourceManager:http://xx.xx.xx.xx:8088
看resourceManager部署在哪台机器上,文中的就是fcs01外网地址的8088端口
5.1.3 配置免密启动
至此,hdoop就算配置好了,但是因为hdfs启动需要输入密码,所以可以配置免密登陆
root用户配置免密
[root@VM-4-15-centos etc]# vim /etc/sshd_config
##1.加入以下内容
RSAAuthentication yes # 启用 RSA 认证
PubkeyAuthentication yes # 启用公钥私钥配对认证方式
AuthorizedKeysFile .ssh/authorized_keys # 公钥文件路径(和上面生成的文件同)
##2.重启服务
[root@VM-4-15-centos etc]# service sshd restart
Redirecting to /bin/systemctl restart sshd.service
复制密钥
[dxm@VM-4-15-centos ~]$ ssh-keygen -t rsa
##按照提示按下回车键,完成后会提示密钥生成位置
[dxm@VM-4-15-centos ~]$ cd ~/.ssh/
##创建文件
[dxm@VM-4-15-centos sbin]$ touch authorized_keys
##复制密钥
[dxm@VM-4-15-centos sbin]$ cp id_rsa.pub authorized_keys
##更改权限
[dxm@VM-4-15-centos .ssh]$ chmod 600 ~/.ssh/authorized_keys
[dxm@VM-4-15-centos .ssh]$ chmod 700 ~/.ssh
配置集群免密登陆
##fcs01安装rsync
[root@hecs-37480 ~]# yum install -y rsync
##首先确保 fcs02、fcs03已经配置好了密钥在home/dxm/.ssh/authorized_keys
##将fcs02、fcs03的密钥同步至fcs01
[dxm@hecs-37480 ~]$ rsync /home/dxm/.ssh/authorized_keys dxm@fcs02:/home/dxm/.ssh/
[dxm@hecs-37480 ~]$ rsync /home/dxm/.ssh/authorized_keys dxm@fcs03:/home/dxm/.ssh/
##测试,ssh不输入密码成功登陆fcs02
[dxm@hecs-37480 ~]$ ssh fcs02
Last failed login: Sun Jul 2 22:54:14 CST 2023 from 124.70.41.75 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Sun Jul 2 13:22:32 2023 from 220.112.121.52
[dxm@VM-4-15-centos ~]$ exit
logout
5.1.4 xsync脚本
cd /usr/local
mkdir shells
cd shells
vim xsync
加入脚本如下
#!/bin/bash
#1. 判断参数个数
if [ $# -lt 1 ]
then
echo Not Enough Arguement!
exit;
fi
#2. 遍历集群所有机器
for host in fcs02 fcs03
do
echo ==================== $host ====================
#3. 遍历所有目录,挨个发送
for file in $@
do
#4 判断文件是否存在
if [ -e $file ]
then
#5. 获取父目录
pdir=$(cd -P $(dirname $file); pwd)
#6. 获取当前文件的名称
fname=$(basename $file)
ssh $host "mkdir -p $pdir"
rsync -av $pdir/$fname $host:$pdir
else
echo $file does not exists!
fi
done
done
给文件加权限
chmod +x xsync
配置环境变量
vi /etc/profile.d/my_env.sh
export PATH=$PATH:/usr/local/shells
报错记录
使用脚本启动时会提示日志存储路径,可以去对应服务器取出日志
1.The authenticity of host ‘localhost (::1)’ can’t be established.
解决方案
ssh -o StrictHostKeyChecking=no localhost
2.NameNode启动失败java.net.BindException: Cannot assign requested address
##1.配置host
注意:云服务器中只有一块内网网卡,外网地址不是直接配置在云服务器中,程序无法绑定公网IP地址。因此服务器本身要改为 —— 内网IP+主机名。不然在可能会在NameNode启动时报错:Cannot assign requested address
##2.关闭防火墙
[root@hecs-37480 ~]# systemctl stop firewalld
[root@hecs-37480 ~]# systemctl disable firewalld