Hadoop3.3.1 ha 搭建

栋咚咚！

已于 2023-10-26 13:38:22 修改

阅读量162

点赞数

文章标签：大数据 hadoop

于 2023-07-31 17:06:01 首次发布

本文链接：https://blog.csdn.net/m0_61283095/article/details/132025295

版权

一、准备三台机器

1、首先关闭防火墙

2、配置hosts

3、配置免密

4、安装jdk三台机器都需要安装 jdk安装

二、安装zk 切记zk不要安装3.4.x 要安装3.6.x 不然会出现RM无法启动。版本兼容问题。 zk下载地址 zk 3.6.x安装

三、安装hadoop

1、下载hadoop3.3.1安装包 hadoop 3.3.1 包下载地址

2、配置环境变量所有的环境变量我都一块放在这了包括 jdk 、zk，只需要把安装地址更换即可。

JAVA_HOME=/opt/package/jdk1.8.0_181
CLASSPATH=.:$JAVA_HOME/lib.tools.jar
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH

export HIVE_HOME=/opt/package/apache-hive-3.1.2-bin
export PATH=$PATH:$HIVE_HOME/bin


export ZOOKEEPER_HOME=/opt/package/apache-zookeeper-3.6.4-bin
export PATH=$PATH:ZOOKEEPER_HOME/bin
export HADOOP_HOME=/opt/package/hadoop-3.3.1
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin


export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
export HDFS_ZKFC_USER=root
export HDFS_JOURNALNODE_USER=root


export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_CONF_DIR

export HADOOP_USER_NAME=root
export PATH=$PATH:$HADOOP_USER_NAME

export HADOOP_CLASSPATH=/opt/package/hadoop-3.3.1/share/hadoop
export PATH=$PATH:$HADOOP_CLASSPATH

export HADOOP_MAPRED_HOME=/opt/package/hadoop-3.3.1
export PATH=$PATH:$HADOOP_MAPRED_HOME/bin

3、接下来进入到hadoop-3.3.1/etc/hadoop 目录下，修改配置文件

1、配置hdfs-site.xml

<configuration>
 <property>
            <name>dfs.nameservices</name>
            <value>ns1</value>
    </property>
    <!-- ns1下面有两个NameNode，分别是nn1，nn2 -->
    <property>
            <name>dfs.ha.namenodes.ns1</name>
            <value>nn1,nn2</value>
    </property>
    <!-- nn1的RPC通信地址 -->
    <property>
            <name>dfs.namenode.rpc-address.ns1.nn1</name>
            <value>bigdata01:9000</value>
    </property>
    <!-- nn1的http通信地址 -->
    <property>
            <name>dfs.namenode.http-address.ns1.nn1</name>
            <value>bigdata01:50070</value>
    </property>
    <!-- nn2的RPC通信地址 -->
    <property>
            <name>dfs.namenode.rpc-address.ns1.nn2</name>
            <value>bigdata02:9000</value>
    </property>
    <!-- nn2的http通信地址 -->
    <property>
            <name>dfs.namenode.http-address.ns1.nn2</name>
            <value>bigdata02:50070</value>
    </property>


 <!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
    <property>
            <name>dfs.namenode.shared.edits.dir</name>
            <value>qjournal://bigdata01:8485;bigdata02:8485;bigdata03:8485/ns1</value>
    </property>
    <!-- 指定JournalNode在本地磁盘存放数据的位置 -->
    <property>
            <name>dfs.journalnode.edits.dir</name>
            <value>/opt/hadoop/zk/journaldata</value>
    </property>
    <!-- 开启NameNode失败自动切换 -->
    <property>
            <name>dfs.ha.automatic-failover.enabled</name>
            <value>true</value>
    </property>
    <!-- 配置失败自动切换实现方式 -->
    <property>
            <name>dfs.client.failover.proxy.provider.ns1</name>
            <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行-->
    <property>
            <name>dfs.ha.fencing.methods</name>
            <value>
                    sshfence
                    shell(/bin/true)
            </value>
    </property>
    <!-- 使用sshfence隔离机制时需要ssh免登陆 -->
    <property>
            <name>dfs.ha.fencing.ssh.private-key-files</name>
            <value>/home/hadoop/.ssh/id_rsa</value>
    </property>
    <!-- 配置sshfence隔离机制超时时间 -->
    <property>
            <name>dfs.ha.fencing.ssh.connect-timeout</name>
            <value>30000</value>
    </property>
    <!-- DataNode进程死亡或者网络故障造成DataNode无法与NameNode通信，NameNode不会
                      立即把该节点判定为死亡，要经过一段超时时间。HDFS默认的超时时间是10分钟+30秒，如果定
    义超时时间为timeout，则其计算公式为：
    timeout = 2 * heartbeat.recheck.interval + 10 * dfs.heartbeat.interval -->
    <property>
            <name>heartbeat.recheck.interval</name>
            <!-- 单位：毫秒 -->
            <value>2000</value>
    </property>
    <property>
            <name>dfs.heartbeat.interval</name>
            <!-- 单位：秒 -->
            <value>1</value>
    </property>
    <!-- 在日常维护hadoop集群过程中会发现这样一种现象：某个节点由于网络故障或者
                      DataNode进程死亡，被NameNode判定为死亡，HDFS马上自动开始数据块的容错拷贝，
    当该节点重新加入到集群中，由于该节点的数据并没有损坏，导致集群中某些block的
    备份数超过了设定数值。默认情况下要经过1个小时的时间才会对这些冗余block进行清理。
    而这个时长与数据块报告时间有关。DataNode会定期将该节点上的所有block信息报告给
    NameNode，默认间隔1小时。下面的参数可以修改报告时间 -->
    <property>
            <name>dfs.blockreport.intervalMsec</name>
            <value>10000</value>
            <description>Determines block reporting interval in milliseconds.</description>
    </property>
    <!--指定磁盘预留多少空间，防止磁盘被撑满用完，单位为bytes -->
    <property>
        <name>dfs.datanode.du.reserved</name>
        <value>10240000</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>bigdata01:50090</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/data/hadoop/dfs/name</value>
        <description>Path on the local filesystem where theNameNode stores the namespace and transactions logs
            persistently.
        </description>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/opt/hadoop/dfs/data</value>
        <description>Comma separated list of paths on the localfilesystem of a DataNode where it should store its
            blocks.
        </description>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>true</value>
        <description>need not permissions</description>
    </property>
    <!--NameNode有一个工作线程池用来处理客户端的远程过程调用及集群守护进程的调用。处理程序数量越多意味着要更大的池来处理来自不同DataNode的并发心跳以及客户端并发的元数据操作。对于大集群或者有大量客户端的集群来说，通常需要增大参数dfs.namenode.handler.count的默认值10。设置该值的一般原则是将其设置为集群大小的自然对数乘以20，即20logN，N为集群大小。
         如果该值设的太小，明显的状况就是DataNode在连接NameNode的时候总是超时或者连接被拒绝，但NameNode的远程过程调用队列很大时，远程过程调用延时就会加大。症状之间是相互影响的，很难说修改dfs.namenode.handler.count就能解决问题，但是在查找故障时，检查一下该值的设置是必要的-->
    <property>
        <name>dfs.datanode.handler.count</name>
        <value>35</value>
        <description>The number of server threads for the datanode.</description>
    </property>
    <!--读超时时间：dfs.client.socket-timeout。默认值1分钟。
             写超时时间：dfs.datanode.socket.write.timeout。默认8分钟。-->
    <property>
        <name>dfs.client.socket-timeout</name>
        <value>600000</value>
    </property>
    <property>
        <!--这里设置Hadoop允许打开最大文件数，默认4096，不设置的话会提示xcievers exceeded错误-->
        <name>dfs.datanode.max.transfer.threads</name>
        <value>409600</value>
    </property>
    <!---块大小-->

    <property>
        <name>dfs.blocksize</name>
        <value>134217728</value>
        <description>node2文件系统HDFS块大小为128M</description>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

2、配置core-site.xml

<configuration>
 <property>
        <name>fs.defaultFS</name>
        <value>hdfs://ns1</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
        <description>该属性值单位为KB，131072KB即为默认的 64M</description>
    </property>
    <!-- 指定zookeeper地址 -->
    <property>
            <name>ha.zookeeper.quorum</name>
#填写你的zookeeper的地址
            <value>bigdata01:2181,bigdata02:2181,bigdata03:2181</value>
    </property>
</configuration>

3、配置yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
 <!-- 开启RM高可用 -->
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>
    <!-- 指定RM的cluster id -->
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yrc</value>
    </property>

    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>
        <value>/yarn-leader-election</value>
    </property>
    <property>
        <name>yarn.client.failover-proxy-provider</name>
        <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
    </property>
    <!-- 指定RM的名字 -->
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>
    <!-- 分别指定RM的地址 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>bigdata01</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>bigdata02</value>
    </property>
    <!-- 指定zk集群地址 -->
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>bigdata01:2181,bigdata02:2181,bigdata03:2181</value>
    </property>

    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/opthadoop/yarn/nm</value>
    </property>
    <property>
        <description>The address of the applications manager interface in the RM.</description>
        <name>yarn.resourcemanager.address.rm1</name>
        <value>bigdata01:8032</value>
    </property>
    <property>
        <description>The address of the applications manager interface in the RM.</description>
        <name>yarn.resourcemanager.address.rm2</name>
        <value>bigdata02:8032</value>
    </property>


    <property>
        <description>The address of the scheduler interface.</description>
        <name>yarn.resourcemanager.scheduler.address.rm1</name>
        <value>bigdata01:8030</value>
    </property>
    <property>
        <description>The address of the scheduler interface.</description>
        <name>yarn.resourcemanager.scheduler.address.rm2</name>
        <value>bigdata02:8030</value>
    </property>


    <property>
        <description>The http address of the RM1 web application.</description>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>bigdata01:8088</value>
    </property>
    <property>
        <description>The http address of the RM2 web application.</description>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>bigdata02:8088</value>
    </property>


    <property>
        <description>The https adddress of the RM web application.</description>
        <name>yarn.resourcemanager.webapp.https.address.rm1</name>
        <value>bigdata01:8090</value>
    </property>
    <property>
        <description>The https adddress of the RM web application.</description>
        <name>yarn.resourcemanager.webapp.https.address.rm2</name>
        <value>bigdata02:8090</value>
    </property>


    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
        <value>bigdata01:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
        <value>bigdata02:8031</value>
    </property>


    <property>
        <description>The address of the RM admin interface.</description>
        <name>yarn.resourcemanager.admin.address.rm1</name>
        <value>bigdata01:8033</value>
    </property>
    <property>
        <description>The address of the RM admin interface.</description>
        <name>yarn.resourcemanager.admin.address.rm2</name>
        <value>bigdata02:8033</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>8192</value>
        <discription>每个节点可用内存,单位MB,默认8182MB</discription>
    </property>
    <property>
        <name>yarn.scheduler.minmum-allocation-mb</name>
        <value>1024</value>
        <discription>每个节点可用内存,单位MB,默认8182MB</discription>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>2.1</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>25600</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>true</value>
    </property>
    <property>
        <description>使用命令：[hadoop@node1 ~]$ hadoop classpath获取到</description>
        <name>yarn.application.classpath</name>
        <value>
            /home/hadoop/hadoop-3.3.1/etc/hadoop:/home/hadoop/hadoop-3.3.1/share/hadoop/common/lib/*:/home/hadoop/hadoop-3.3.1/share/hadoop/common/*:/home/hadoop/hadoop-3.3.1/share/hadoop/hdfs:/home/hadoop/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop-3.3.1/share/hadoop/hdfs/*:/home/hadoop/hadoop-3.3.1/share/hadoop/mapreduce/*:/home/hadoop/hadoop-3.3.1/share/hadoop/yarn:/home/hadoop/hadoop-3.3.1/share/hadoop/yarn/lib/*:/home/hadoop/hadoop-3.3.1/share/hadoop/yarn/*
        </value>
    </property>
</configuration>

4、配置mapred-site.xml

<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>bigdata02:10020</value>
        <description>MR JobHistory Server管理的日志的存放位置</description>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>bigdata02:19888</value>
        <description>查看历史服务器已经运行完的Mapreduce作业记录的web地址，需要启动该服务才行</description>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.staging-dir</name>
        <value>/data/hadoop/yarn/staging</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.done-dir</name>
        <value>${yarn.app.mapreduce.am.staging-dir}/done</value>
        <description>MR JobHistory Server管理的日志的存放位置,默认:/mr-history/done</description>
    </property>
    <property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>${yarn.app.mapreduce.am.staging-dir}/done_intermediate</value>
        <description>MapReduce作业产生的日志存放位置，默认值:/mr-history/tmp</description>
    </property>

<!--
         <property>
        <name>mapred.job.tracker</name>
        <value>master2:49001</value>
    </property>
-->
    <property>
        <name>mapred.local.dir</name>
        <value>/data/hadoop/var</value>
    </property>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=/opt/package/hadoop-3.3.1</value>
    </property>
    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=/opt/package/hadoop-3.3.1</value>
    </property>
    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=/opt/package/hadoop-3.3.1</value>
    </property>
   <property>
        <name>yarn.application.classpath</name>
 <value>/opt/package/hadoop-3.3.1/etc/hadoop:/opt/package/hadoop-3.3.1/share/hadoop/common/lib/*:/opt/package/hadoop-3.3.1/share/hadoop/common/*:/opt/package/hadoop-3.3.1/share/hadoop/hdfs:/opt/package/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/opt/package/hadoop-3.3.1/share/hadoop/hdfs/*:/opt/package/hadoop-3.3.1/share/hadoop/mapreduce/*:/opt/package/hadoop-3.3.1/share/hadoop/yarn:/opt/package/hadoop-3.3.1/share/hadoop/yarn/lib/*:/opt/package/hadoop-3.3.1/share/hadoop/yarn/*
</value>
   </property>
   <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/common/*,$HADOOP_MAPRED_HOME/share/hadoop/common/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/lib/*</value>
    </property>
</configuration>

5、配置 hadoop-env.sh

只需要添加jdk的环境变量即可

JAVA_HOME=/opt/package/jdk1.8.0_181

6、配置 yarn-env.sh

只需要添加jdk的环境变量即可

JAVA_HOME=/opt/package/jdk1.8.0_181

7、修改workers 把localhost删除，把你的机器的hostsname写入，有几台机器写几台。

bigdata01
bigdata02
bigdata03

8、把hadoop-3.3.1文件拷贝到其他机器

 scp -r hadoop-3.3.1 bigdata02:/opt/package/
 scp -r hadoop-3.3.1 bigdata03:/opt/package/

四、启动hadoop

参考 hadoop 启动

启动完成后进程如下

五、至此搭建完成，启动完成后可以跑一个wordcount样例。

栋咚咚！

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
Hadoop3.3.1 ha 搭建

六、备注：hadoop搭建文章有很多。但是按照有的文章搭建出来只可以把服务启动起来，但是跑mapreduce任务会有问题。问题也就是配置文件导致的，本篇文章的配置文件比较齐全，配置完成后跑mapreduce大概率不会有问题。切记zk不要安装3.4.x 要安装3.6.x 不然会出现RM无法启动。只需要添加jdk的环境变量即可。只需要添加jdk的环境变量即可。
复制链接

扫一扫