hadoop ha安装配置

最新推荐文章于 2022-04-21 15:38:35 发布

xiecj_2006

最新推荐文章于 2022-04-21 15:38:35 发布

阅读量330

点赞数

分类专栏：大数据文章标签： hadoop

本文链接：https://blog.csdn.net/xiecj_2006/article/details/103643046

版权

本文详细介绍了如何部署配置Hadoop HA集群，包括集群规划、防火墙设置、免密码登录、时间同步、JDK安装、Zookeeper部署、Hadoop环境配置、HDFS、MapReduce和YARN相关配置文件的修改、JournalNode启动、NameNode格式化、YARN启动、历史服务器启动等步骤，并进行了HDFS和YARN的切换测试以及Wordcount示例的运行。

摘要由CSDN通过智能技术生成

集群部署规划

	hadoop11	hadoop12	hadoop13	hadoop14
NameNode	✓	✓
DataNode	✓	✓	✓	✓
ResourceManager			✓	✓
NodeManager	✓	✓	✓	✓
ZooKeeper	✓	✓	✓
journalnode	✓	✓	✓
zkfc	✓	✓	✓
Hmaster	✓	✓
HRegionServer			✓	✓
scala	✓	✓	✓	✓
spark master	✓	✓
spark slaver			✓	✓
mysql master	✓	✓
mysql slave			✓	✓
hive	✓
flume	✓
kafka	✓	✓	✓
elasticsearch		✓	✓	✓
kibana				✓
logstash	✓	✓	✓
storm nimbus	✓	✓	✓
storm supervisor	✓	✓	✓
storm ui	✓	✓	✓
flink jobmanager	✓	✓
flink taskmanager			✓	✓
mongodb PRIMARY	✓
mongodb SECONDARY		✓	✓
mongodb ARBITER				✓
hue	✓
livy	✓
impala master	✓
impala slave		✓	✓	✓

防火墙

略

免密码登录

略

集群时间同步

略

JDK部署

略

zookeeper集群部署

略

解压缩

tar zxvf apache-zookeeper-3.5.5-bin.tar.gz -C /opt/module/

hadoop-env.sh增加JAVA_HOME

cat etc/hadoop/hadoop-env.sh
......
export JAVA_HOME=/opt/module/jdk1.8.0_211

设置/etc/profile

......
## JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_211
export PATH=$PATH:$JAVA_HOME/bin
## ZOOKEEPER_HOME
export ZOOKEEPER_HOME=/opt/module/apache-zookeeper-3.5.5-bin
export PATH=$PATH:$ZOOKEEPER_HOME/bin
## HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-3.2.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

生效

source /etc/profile

配置core-site.xml

cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<configuration>
    <!-- 指定hdfs的nameservice为xiechuan -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://xiechuan/</value>
    </property>

    <!-- 指定hadoop临时目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/module/hadoop-3.2.0/data/hadoopdata</value>
    </property>

    <!-- 指定zookeeper地址 -->
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>hadoop11:2181,hadoop12:2181,hadoop13:2181</value>
    </property>

    <!-- hadoop链接zookeeper的超时时长设置 -->
    <property>
        <name>ha.zookeeper.session-timeout.ms</name>
        <value>1000</value>
        <description>ms</description>
    </property>
</configuration>

配置hdfs-site.xml

cat hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

    <!-- 指定副本数 -->
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>

    <!-- 配置namenode和datanode的工作目录-数据存储目录 -->
    <pr