一、集群部署规划
环境Linux+centos8
注意：NameNode和SecondaryNameNode不要安装在同一台服务器
注意：ResourceManager也很消耗内存，不要和NameNode、SecondaryNameNode配置
因为环境不允许，所以有点重叠的部署组件
| 主机名 | namenode | datanode | JobHistoryServer | Resourcemanager | NodeManager | ZK | JournalNode | ZKFC |
|---|---|---|---|---|---|---|---|---|
| master | √ | √ | √ | √ | √ | |||
| salve1 | √ | √ | √ | √ | √ | |||
| salve2 | √ | √ | √ | √ | √ | |||
| salve3 | √ | √ | ||||||
| salve4 | √ | √ |
二、zookeeper安装部署以及脚本使用说明
2.0 将zookeeper安装包到/opt/software
[appview@master ~]$ cd /opt/software/ [appview@master software]$ [appview@master software]$ tar -zxvf apache-zookeeper-3.7.1-bin.tar.gz -C /opt/moudle/
2.1配置服务器编号
[appview@master software]$ cd /opt/moudle/zookeeper-3.7.1/ #### 创建zData目录 [appview@master software]$ mkdir zData [appview@master zData]$ vim myid ###### 添加 1 如下 [appview@master zData]$ cat myid 1 [appview@master zData]$
2.2配置zoo.cfg文件
[appview@master conf]$ pwd /opt/moudle/zookeeper-3.7.1/conf [appview@master conf]$ ll 总用量 12 -rw-r--r-- 1 appview appview 535 5月 7 2022 configuration.xsl -rw-r--r-- 1 appview appview 3435 5月 7 2022 log4j.properties -rw-rw-r-- 1 appview appview 1246 3月 29 08:45 zoo_sample.cfg [appview@master conf]$ [appview@master conf]$ mv zoo_sample.cfg zoo.cfg [appview@master conf]$ vim zoo.cfg
########## 修改数据路径 dataDir=/opt/moudle/zookeeper-3.7.1/zData ########## 添加以下内容 #######################cluster########################## server.1=master:2888:3888 server.2=salve1:2888:3888 server.3=salve2:2888:3888
2.3分发安装包
[appview@master conf]$ xsync-1-3 /opt/moudle/zookeeper-3.7.1
修改slave1、salve2路径/opt/moudle/zookeeper-3.7.1/zData/myid,分别改为对应的2、3
2.4添加环境变量
[appview@master software]$ vim ~/.bashrc
添加以下内容
#ZK_HOME export ZK_HOME=/opt/moudle/zookeeper-3.7.1 export PATH=$PATH:$ZK_HOME/bin
2.5 一键启动zookeeper脚本
[appview@master bin]$ vim zk.sh
脚本内容如下：
#!/bin/bash
case $1 in
"start"){
for i in master salve1 salve2
do
echo --------zookeeper-3.7.1 $i -----------------
ssh $i "/opt/moudle/zookeeper-3.7.1/bin/zkServer.sh start"
done
};;
"stop"){
for i in master salve1 salve2
do
echo --------zookeeper-3.7.1 $i -----------------
ssh $i "/opt/moudle/zookeeper-3.7.1/bin/zkServer.sh stop"
done
};;
"status"){
for i in master salve1 salve2
do
echo --------zookeeper-3.7.1 $i -----------------
ssh $i "/opt/moudle/zookeeper-3.7.1/bin/zkServer.sh status"
done
};;
esac
启动zookeeper服务
#####启动 [appview@master bin]$ zk.sh start #####停止 [appview@master bin]$ zk.sh stop #####状态 [appview@master bin]$ zk.sh status
2.6 脚本说明以及使用
rsync是一个强大的文件同步工具，可以在远程或本地之间同步文件和目录，并且在同步时可以实现增量传输，只传输变更的部分，从而节约带宽和时间
### 基本格式 rsync [选项] 源目录/文件 -a 归档模式（保留目录机构及权限） -v 显示详细输出信息 -r 递归复制整个目录 -u 仅在源文件比目标文件新或者不存在时进行复制 -z 传输时压缩文件 -e 指定远程shell程序 --delete 删除目标目录中不存在于源文件中的文件 --exclude = PATTERN:排除匹配指定模式的文件或目录
在家目录下创建bin目标，在里面放脚本可以在任意地方调用
分发脚本xsync如下
[appview@master ~]$ cd bin/ [appview@master bin]$ vim xsync #### 内容如下，根据自己需要分发的几个节点进行修改 #!/bin/bash # 判断参数个数 if [ $# -lt 1 ] then echo Not Enough Arguement! exit; fi # 遍历集群所有机器 for host in master salve1 salve2 salve3 salve4 do echo ================ $host ========================== # 遍历所有目录，挨个发送 for file in $@ do # 判断文件是否存在 if [ -e $file ] then # 获取父目录 pdir=$(cd -P $(dirname $file); pwd) # 获取当前文件的名称 fname=$(basename $file) ssh $host "mkdir -p $pdir" rsync -av $pdir/$fname $host:$pdir else echo $file does not exists!!! fi done done
xcall脚本如下
#!/bin/bash for host in master salve1 salve2 salve3 salve4 do echo ----------- $host ----------------- ssh $host "$*" done
xcall-1-3脚本如下
[appview@master bin]$ vim xcall-1-3 ########内容如下： #!/bin/bash for host in master salve1 salve2 do echo ----------- $host ----------------- ssh $host "$*" done
三、Hadoop安装部署
3.1将hadoop-3.3.4.tar.gz安装包到/opt/software/
[appview@master ~]$ cd /opt/software/ [appview@master software]$ [appview@master software]$ tar -zxvf hadoop-3.3.4.tar.gz -C /opt/moudle/
3.2将Hadoop添加到环境变量
[appview@master software]$ vim ~/.bashrc ### 添加以下内容 #HADOOP_HOME export HADOOP_HOME=/opt/moudle/hadoop-3.3.4 export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
保存后，将环境变量分发以下，或者每台执行以上操作也可，
### 环境变量分发 [appview@master software]$ xsync ~/.bashrc ### 执行以下命令使环境变量生效 [appview@master software]$ source ~/.bashrc
四、配置集群
1.核心配置文件
1.1配置core-site.xml
[appview@master software]$ cd $HADOOP_HOME/etc/hadoop [appview@master hadoop]$ vim core-site.xml
文件内容如下：
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <!-- 把多个 NameNode 的地址组装成一个集群 hdpcluster --> <property> <name>fs.defaultFS</name> <value>hdfs://hdpcluster</value> </property> <!-- 指定 hadoop 运行时产生文件的存储目录 --> <property> <name>hadoop.tmp.dir</name> <value>/opt/moudle/hadoop-3.3.4/data/tmp</value> </property> <!-- 指定 zkfc 要连接的 zkServer 地址 --> <property> <name>ha.zookeeper.quorum</name> <value>master:2181,salve1:2181,salve2:2181</value> </property> <!-- NN 连接 JN 重试次数，默认是 10 次 --> <property> <name>ipc.client.connect.max.retries</name> <value>20</value> </property> <!-- 重试时间间隔，默认 1s --> <property> <name>ipc.client.connect.retry.interval</name> <value>30000</value> </property> <!-- 配置HDFS网页登录使用的静态用户为appview --> <property> <name>hadoop.http.staticuser.user</name> <value>appview</value> </property> <!-- 配置该appview(superUser)允许通过代理访问的主机节点 --> <property> <name>hadoop.proxyuser.appview.hosts</name> <value>*</value> </property> <!-- 配置该appview(superUser)允许通过代理用户所属组 --> <property> <name>hadoop.proxyuser.appview.groups</name> <value>*</value> </property> <!-- 配置该appview(superUser)允许通过代理的用户--> <property> <name>hadoop.proxyuser.appview.users</name> <value>*</value> </property> </configuration>
2.HDFS配置文件
2.1配置hdfs-site.xml
[appview@master hadoop]$ vim hdfs-site.xml
文件内容如下：
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <!-- NameNode 数据存储目录 --> <property> <name>dfs.namenode.name.dir</name> <value>/opt/moudle/hadoop-3.3.4/data/dfs/namenode</value> </property> <!-- DataNode 数据存储目录 --> <property> <name>dfs.datanode.data.dir</name> <value>/opt/moudle/hadoop-3.3.4/data/dfs/datanode</value> </property> <!-- JournalNode 数据存储目录 --> <property> <name>dfs.journalnode.edits.dir</name> <value>/opt/moudle/hadoop-3.3.4/data/dfs/journaldata</value> </property> <!-- 完全分布式集群名称 --> <property> <name>dfs.nameservices</name> <value>hdpcluster</value> </property> <!-- 集群中 NameNode 节点都有哪些nn1,nn2,nn3 --> <property> <name>dfs.ha.namenodes.hdpcluster</name> <value>nn1,nn2</value> </property> <!-- NameNode 的 RPC 通信地址 --> <property> <name>dfs.namenode.rpc-address.hdpcluster.nn1</name> <value>master:9000</value> </property> <property> <name>dfs.namenode.rpc-address.hdpcluster.nn2</name> <value>salve1:9000</value> </property> <!-- NameNode 的 http 通信地址 --> <property> <name>dfs.namenode.http-address.hdpcluster.nn1</name> <value>master:9870</value> </property> <property> <name>dfs.namenode.http-address.hdpcluster.nn2</name> <value>salve1:9870</value> </property> <!-- 指定 NameNode 元数据在 JournalNode 上的存放位置 --> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://master:8485;salve1:8485;salve2:8485/hdpcluster</value> </property> <!-- 访问代理类：client 用于确定哪个 NameNode 为 Active --> <property> <name>dfs.client.failover.proxy.provider.hdpcluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!-- 配置隔离机制，即同一时刻只能有一台服务器对外响应 --> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence shell(/bin/true)</value> </property> <!-- 使用隔离机制时需要 ssh 秘钥登录--> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/appview/.ssh/id_rsa</value> </property> <!-- 启动 NN 故障自动切换--> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <!-- 指定副本数量 --> <property> <name>dfs.replication</name> <value>3</value> </property> <!-- yarn容器允许分配的最大最小内存 --> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>512</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>4096</value> </property> <!-- yarn容器允许管理的物理内存大小 --> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>4096</value> </property> <!-- 关闭yarn对虚拟内存的限制检查 --> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> </configuration>
3.YARN配置文件
3.1配置yarn-site.xml
[appview@master hadoop]$ vim yarn-site.xml
文件内容如下：
<?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <!-- Site specific YARN configuration properties --> <!-- 指定MR走shuffle--> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <!-- 启用 resourcemanager ha --> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <!-- 声明两台 resourcemanager 的地址 --> <property> <name>yarn.resourcemanager.cluster-id</name> <value>cluster-yarn</value> </property> <!--指定 resourcemanager 的逻辑列表--> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <!-- ========== rm1 的配置 ========== --> <!-- 指定 rm1 的主机名 --> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>salve1</value> </property> <!-- 指定 rm1 的 web 端地址 --> <property> <name>yarn.resourcemanager.webapp.address.rm1</name> <value>salve1:8088</value> </property> <!-- 指定 rm1 的内部通信地址 --> <property> <name>yarn.resourcemanager.address.rm1</name> <value>salve1:8032</value> </property> <!-- 指定 AM 向 rm1 申请资源的地址 --> <property> <name>yarn.resourcemanager.scheduler.address.rm1</name> <value>salve1:8030</value> </property> <!-- 指定供 NM 连接的地址 --> <property> <name>yarn.resourcemanager.resource-tracker.address.rm1</name> <value>salve1:8031</value> </property> <!-- 对管理员暴露的访问地址管理员通过该地址向RM发送管理命令 --> <property> <name>yarn.resourcemanager.admin.address.rm1</name> <value>salve1:8033</value> </property> <!-- RM1 configs end --> <!-- ========== rm2 的配置 ========== --> <!-- 指定 rm2 的主机名 --> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>salve2</value> </property> <!-- 指定 rm2 的 web 端地址 --> <property> <name>yarn.resourcemanager.webapp.address.rm2</name> <value>salve2:8088</value> </property> <!-- 指定 rm2 的内部通信地址 --> <property> <name>yarn.resourcemanager.address.rm2</name> <value>salve2:8032</value> </property> <!-- 指定 AM 向 rm2 申请资源的地址 --> <property> <name>yarn.resourcemanager.scheduler.address.rm2</name> <value>salve2:8030</value> </property> <!-- 指定供 NM 连接的地址 --> <property> <name>yarn.resourcemanager.resource-tracker.address.rm2</name> <value>salve2:8031</value> </property> <!-- 对管理员暴露的访问地址管理员通过该地址向RM发送管理命令 --> <property> <name>yarn.resourcemanager.admin.address.rm2</name> <value>salve2:8033</value> </property> <!-- RM2 configs end --> <!-- ========== rm3 的配置 ========== --> <!-- 指定 zookeeper 集群的地址 --> <property> <name>yarn.resourcemanager.zk-address</name> <value>master:2181,salve1:2181,salve2:2181</value> </property> <!-- ha状态切换为自动切换 --> <property> <name>yarn.resourcemanager.ha.automatic-failover.embedded</name> <value>true</value> </property> <!-- 指定 resourcemanager 的状态信息存储在 zookeeper 集群 --> <property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value> </property> <!-- 环境变量的继承 --> <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> </property> <!-- 开启日志聚集功能 --> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <!-- 设置日志聚集服务器地址 --> <property> <name>yarn.log.server.url</name> <value>http://master:19888/jobhistory/logs</value> </property> <!-- 设置日志保留时间为7天 --> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>604800</value> </property> <!--是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是 true --> <property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value> </property> <!--是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是 true --> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> </configuration>
4.MapReduce配置文件
4.1配置mapred-site.xml
[appview@master hadoop]$ vim mapred-site.xml
文件内容如下：
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>master:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>master:19888</value> </property> </configuration>
5.配置workers
[appview@master hadoop]$ vim workers
添加内容如下
salve2 salve3 salve4
6.配置env.sh
给以下三个文件添加java路径
[appview@master ~]$ cd $HADOOP_HOME/etc/hadoop [appview@master hadoop]$ pwd /opt/moudle/hadoop-3.3.4/etc/hadoop [appview@master hadoop]$ vim hadoop-env.sh [appview@master hadoop]$ vim yarn-env.sh [appview@master hadoop]$ vim mapred-env.sh
文件内容如下
export JAVA_HOME=/opt/moudle/openjdk-8u272
五、分发Hadoop并验证
[appview@master hadoop]$ xsync /opt/moudle/hadoop-3.3.4 [appview@master hadoop]$ hadoop version Hadoop 3.3.4 Source code repository https://github.com/apache/hadoop.git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb Compiled by stevel on 2022-07-29T12:32Z Compiled with protoc 3.7.1 From source with checksum fb9dd8918a7b8a5b430d61af858f6ec This command was run using /opt/moudle/hadoop-3.3.4/share/hadoop/common/hadoop-common-3.3.4.jar [appview@master hadoop]$
六、集群启动初始化
1.先启动journalnode在master、slave1、slave2
[appview@master ~]$ xcall hdfs --daemon start journalnode ##### 查看进程 [appview@master ~]$ xcall jps ----------- master ----------------- 27856 Jps 22881 QuorumPeerMain 23469 JournalNode ----------- salve1 ----------------- 24288 Jps 19649 QuorumPeerMain 19906 JournalNode ----------- salve2 ----------------- 12465 QuorumPeerMain 12679 JournalNode 16397 Jps
2.namenode格式化并启动namenode查看进程
[appview@master ~]$ hdfs namenode -format [appview@master ~]$ hdfs --daemon start namenode [appview@master ~]$ jps ----------- master ----------------- 27856 Jps 22881 QuorumPeerMain 23469 JournalNode 23161 NameNode
3.在slave1同步namenode信息
[appview@salve1 ~]$ hdfs namenode -bootstrapStandby
4.格式化ZKFC
[appview@master ~]$ hdfs zkfc -formatZK
5.停止所有的Hadoop的进程，然后重新启动
#### 停止hdfs进程 [appview@master ~]$ stop-dfs.sh ### 开启hdfs进程 [appview@master ~]$ start-dfs.sh #### 切换到安装有yarn的salve1或者salve2启动yarn进程 [appview@salve1 ~]$ start-yarn.sh
七、查看所有进程
[appview@master ~]$ xcall jps ----------- master ----------------- 22881 QuorumPeerMain 23670 DFSZKFailoverController 29142 Jps 23161 NameNode 23469 JournalNode 23902 JobHistoryServer ----------- salve1 ----------------- 25600 Jps 19649 QuorumPeerMain 19906 JournalNode 20020 DFSZKFailoverController 20174 ResourceManager 19775 NameNode ----------- salve2 ----------------- 12465 QuorumPeerMain 12580 DataNode 13220 NodeManager 12679 JournalNode 17400 Jps 13098 ResourceManager ----------- salve3 ----------------- 3057 Jps 1972 DataNode 2103 NodeManager ----------- salve4 ----------------- 2088 NodeManager 3002 Jps 1964 DataNode [appview@master ~]$
八、Web端查看HDFS的Web页面
遵从一主一备实现高可用，以上的配置文件以及包含了zookeeper的高可用了
HDFS的Web页面1：http://master:9870 HDFS的Web页面2：http://salve1:9870 YARN的Web页面：http://salve1:8088/cluster
451
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
