华为云耀云服务器部署Hadoop

dxm_!@

已于 2023-07-04 23:44:31 修改

阅读量189

点赞数

文章标签：华为云服务器 hadoop

于 2023-07-04 23:30:23 首次发布

本文链接：https://blog.csdn.net/weixin_43539436/article/details/131545132

版权

5.1HADOOP部署

hadoop官网：https://hadoop.apache.org/releases.html

5.1.1 配置

解压hadoop压缩包

[dxm@VM-4-15-centos hadoop-2.10.2]$ mkdir hadoop
[dxm@VM-4-15-centos hadoop-2.10.2]$ tar -zxvf /home/dxm/hadoop-2.10.2.tar.gz -C /home/dxm/hadoop/

jdk默认部署好了
配置hadoop环境变量

##1.进行编辑
[root@VM-4-15-centos ~]# sudo vim /etc/profile.d/my_env.sh

##2.增加如下环境变量
##JAVA_HOME
JAVA_HOME=/usr/java/jdk1.8.0_121
export JAVA_HOME
PATH=$PATH:$JAVA_HOME/bin
export PATH

##HADOOP_HOME
export HADOOP_HOME=/home/dxm/hadoop/hadoop-2.10.2
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

##FLINK需要
export HADOOP_CLASSPATH=`hadoop classpath`
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

##3.重新执行刚刚修改的文件
[root@VM-4-15-centos ~]# source /etc/profile.d/my_env.sh

5.1.1.1 单机环境部署

服务器配置：
1 * 腾讯云服务器2核2G

修改home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://127.0.0.1:9000</value>
        </property>
</configuration>

修改home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/yarn-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
 		<name>yarn.nodemanager.aux-services</name>
	 	<value>mapreduce_shuffle</value>
	</property>
	<property>
 		<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
		<value>org.apache.hadoop.mapred.ShuffleHandler</value>
	</property>
</configuration>

创建文件home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
		 	<name>mapreduce.framework.name</name>
	 		<value>yarn</value>
		</property>
</configuration>

修改home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
         <name>dfs.replication</name>
         <value>1</value>
        </property>
        <property>
         <name>dfs.namenode.name.dir</name>
         <value>file:/home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/namenode</value>
        </property>
        <property>
         <name>dfs.datanode.data.dir</name>
         <value>file:/home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/datanode</value>
        </property>
</configuration>

创建目录

sudo mkdir -p /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/namenode

sudo mkdir -p /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/datanode

格式化HDFS

hadoop namenode -format

5.1.1.2 集群环境部署

服务器配置：
1 * 华为云耀云服务器2核4G fcs01
1 * 华为云耀云服务器1核2G fcs03
1 * 腾讯云服务器2核2G fcs02
host配置

vim /etc/hosts

##fcs01示例 fcs01填内网地址，fcs02、fcs03填外网地址
fcs01 192.168.0.25
fcs02 xxx.xxx.xx.xx
fcs03 xxx.xxx.xx.xx

##fcs02示例 fcs02填内网地址，fcs01、fcs03填外网地址
fcs01 xxx.xxx.xx.xx
fcs02 192.168.0.21
fcs03 xxx.xxx.xx.xx

##fcs03示例 fcs03填内网地址，fcs01、fcs02填外网地址
fcs01 xxx.xxx.xx.xx
fcs02 xxx.xxx.xx.xx
fcs03 192.168.0.27

##2.关闭防火墙
[root@hecs-37480 ~]# systemctl stop firewalld
[root@hecs-37480 ~]# systemctl disable firewalld

集群规划

	fcs01	fcs02	fcs03
HDFS	DataNode	DataNode NameNode	DataNode SecondaryNameNode
YARN	NodeManager ResourceManager	NodeManager	NodeManager

(1)核心配置文件

配置core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <!-- 指定NameNode的地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://fcs02:8020</value>
    </property>
    <!-- 指定hadoop数据的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/dxm/hadoop/hadoop-2.10.2/data</value>
    </property>
    <!-- 配置HDFS网页登录使用的静态用户为dxm -->（配置之后普通用户就可以在网页端有删除文件的权限了）
    <property>
        <name>hadoop.http.staticuser.user</name>
        <value>dxm</value>
    </property>
</configuration>

(2)HDFS配置文件

配置hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
   <!-- NameNode web端访问地址-->
   <property>
        <name>dfs.namenode.http-address</name>
        <value>fcs02:50070</value>
    </property>
   <!-- SecondaryNameNode web端访问地址-->
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>fcs03:50080</value>
    </property>
    <!-- 配置nameNode元数据存储位置-->
    <property>
         <name>dfs.namenode.name.dir</name>
         <value>file:/home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/namenode</value>
    </property>
    <!-- 配置dataNode元数据存储位置-->
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/datanode</value>
    </property>
    <!-- 指定DataNode的节点配置文件 -->
    <property>
     <name> dfs.hosts </name>
     <value>/home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/slaves</value>
    </property>
</configuration>

(3)YARN配置文件

配置yarn-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <!-- 指定MapReduce走shuffle -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!-- 指定ResourceManager的地址-->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>fcs01</value>
    </property>
</configuration>

(4)MapReduce配置

配置mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    	<!-- 指定MapReduce程序运行在Yarn上-->
        <property>
		 	<name>mapreduce.framework.name</name>
	 		<value>yarn</value>
		</property>
</configuration>

(5)修改slaves改成如下内容

fcs01
fcs02
fcs03

(6)将配置文件分发到各个节点

xsync配置可以参考文末xsync脚本

[dxm@hecs-37480 hadoop]$ xsync /home/dxm/hadoop/hadoop-2.10.2/etc/hadoop/
[dxm@hecs-37480 hadoop-2.10.2]$ mkdir -p /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/namenode
[dxm@hecs-37480 hadoop-2.10.2]$ mkdir -p /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/hdfs/datanode
[dxm@hecs-37480 hadoop-2.10.2]$ xsync /home/dxm/hadoop/hadoop-2.10.2/hadoop_data/cd

如果不想配置xsync，也可以将上述配置复制到剩余节点

(7)格式化HDFS

hadoop namenode -format

5.1.2 启动

/home/dxm/hadoop/hadoop-2.10.2/sbin/start-dfs.sh
/home/dxm/hadoop/hadoop-2.10.2/sbin/start-yarn.sh

访问hadoop resourceManager:http://xx.xx.xx.xx:8088

看resourceManager部署在哪台机器上，文中的就是fcs01外网地址的8088端口

5.1.3 配置免密启动

至此，hdoop就算配置好了，但是因为hdfs启动需要输入密码，所以可以配置免密登陆

root用户配置免密

[root@VM-4-15-centos etc]# vim /etc/sshd_config
##1.加入以下内容
RSAAuthentication yes # 启用 RSA 认证

PubkeyAuthentication yes # 启用公钥私钥配对认证方式

AuthorizedKeysFile .ssh/authorized_keys # 公钥文件路径（和上面生成的文件同）
##2.重启服务
[root@VM-4-15-centos etc]# service sshd restart
Redirecting to /bin/systemctl restart sshd.service

复制密钥

[dxm@VM-4-15-centos ~]$ ssh-keygen -t rsa
##按照提示按下回车键,完成后会提示密钥生成位置
[dxm@VM-4-15-centos ~]$ cd ~/.ssh/
##创建文件
[dxm@VM-4-15-centos sbin]$ touch authorized_keys
##复制密钥
[dxm@VM-4-15-centos sbin]$ cp id_rsa.pub authorized_keys
##更改权限
[dxm@VM-4-15-centos .ssh]$ chmod 600 ~/.ssh/authorized_keys
[dxm@VM-4-15-centos .ssh]$ chmod 700 ~/.ssh

配置集群免密登陆

##fcs01安装rsync
[root@hecs-37480 ~]# yum install -y rsync

##首先确保 fcs02、fcs03已经配置好了密钥在home/dxm/.ssh/authorized_keys
##将fcs02、fcs03的密钥同步至fcs01
[dxm@hecs-37480 ~]$ rsync /home/dxm/.ssh/authorized_keys dxm@fcs02:/home/dxm/.ssh/
[dxm@hecs-37480 ~]$ rsync /home/dxm/.ssh/authorized_keys dxm@fcs03:/home/dxm/.ssh/

##测试，ssh不输入密码成功登陆fcs02
[dxm@hecs-37480 ~]$ ssh fcs02
Last failed login: Sun Jul  2 22:54:14 CST 2023 from 124.70.41.75 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Sun Jul  2 13:22:32 2023 from 220.112.121.52
[dxm@VM-4-15-centos ~]$ exit
logout

5.1.4 xsync脚本

cd /usr/local
mkdir shells
cd shells
vim xsync

加入脚本如下

#!/bin/bash
#1. 判断参数个数
if [ $# -lt 1 ]
then
  echo Not Enough Arguement!
  exit;
fi
#2. 遍历集群所有机器
for host in fcs02 fcs03
do
  echo ====================  $host  ====================
  #3. 遍历所有目录，挨个发送
  for file in $@
  do
    #4 判断文件是否存在
    if [ -e $file ]
    then
      #5. 获取父目录
      pdir=$(cd -P $(dirname $file); pwd)
      #6. 获取当前文件的名称
      fname=$(basename $file)
      ssh $host "mkdir -p $pdir"
      rsync -av $pdir/$fname $host:$pdir
    else
      echo $file does not exists!
    fi
  done
done

给文件加权限

chmod +x xsync

配置环境变量

vi /etc/profile.d/my_env.sh

export PATH=$PATH:/usr/local/shells

报错记录

使用脚本启动时会提示日志存储路径，可以去对应服务器取出日志

1.The authenticity of host ‘localhost (::1)’ can’t be established.

解决方案

ssh -o StrictHostKeyChecking=no localhost

2.NameNode启动失败java.net.BindException: Cannot assign requested address

##1.配置host
注意：云服务器中只有一块内网网卡，外网地址不是直接配置在云服务器中，程序无法绑定公网IP地址。因此服务器本身要改为 —— 内网IP+主机名。不然在可能会在NameNode启动时报错：Cannot assign requested address
##2.关闭防火墙
[root@hecs-37480 ~]# systemctl stop firewalld
[root@hecs-37480 ~]# systemctl disable firewalld