05_Hadoop集群配置

:::info
集群规划
:::

	hadoop102	hadoop103	hadoop104
HDFS	NameNode DataNode	DataNode	SecondaryNameNode DataNode
YARN	NodeManager	ResourceManger NodeManger	NodeManger

需要配置 四个配置文件 均存放在 $HADOOP_HOME/etc/hadoop 下

102 来到 $HADOOP_HOME/etc/hadoop

1、配置 core-site.xml

vim core-site.xml

<configuration>

    <!-- 指定 NameNode 的地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop102:8020</value>
    </property>
    <!-- 指定 hadoop 数据的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/module/hadoop-3.3.5/data</value>
    </property>

    <!-- 配置HDFS网页登录使用的静态用户为 ice -->
    <property>
        <name>hadoop.http.staticuser.user</name>
        <value>ice</value>
    </property>

</configuration>

2、 配置 hdfs-site.xml

vim hdfs-site.xml

<configuration>

    <!-- nn web端访问地址 -->
    <property>
        <name>dfs.namenode.http-address</name>
        <value>hadoop102:9870</value>
    </property>
    <!-- 2nn web端访问地址 -->
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop104:9868</value>
    </property>

</configuration>

3、 配置yarn-site.xml

vim yarn-site.xml

<configuration>

    <!-- 指定 MR 走 shuffle -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!-- 指定ResourceManager的地址 -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop103</value>
    </property>

    <!-- 运行mapreduce程序需要配置classpath -->
    <property>
        <name>yarn.application.classpath</name>
        <value>
            <!-- 执行 hadoop classpath -->
            <!-- 将路径填在此处 -->
            /opt/module/hadoop-3.3.4/etc/hadoop:/opt/module/hadoop-3.3.4/share/hadoop/common/lib/*:/opt/module/hadoop-3.3.4/share/hadoop/common/*:/opt/module/hadoop-3.3.4/share/hadoop/hdfs:/opt/module/hadoop-3.3.4/share/hadoop/hdfs/lib/*:/opt/module/hadoop-3.3.4/share/hadoop/hdfs/*:/opt/module/hadoop-3.3.4/share/hadoop/mapreduce/*:/opt/module/hadoop-3.3.4/share/hadoop/yarn:/opt/module/hadoop-3.3.4/share/hadoop/yarn/lib/*:/opt/module/hadoop-3.3.4/share/hadoop/yarn/*
        </value>
    </property>

</configuration>

4、 MapReduce 配置

vim mapred-site.xml

<configuration>

    <!-- 指定 MapReduce 程序 运行在 Yarn 上 -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

</configuration>

5、目前 只在 102 上做了 配置，分发 hadoop 配置文件

xsync /opt/module/hadoop-3.3.5/etc/hadoop

6、在103 104 查看分发 完成

cd /opt/module/hadoop-3.3.5/etc/hadoop

cat core-site.xml

7、配置 workers

vim /opt/module/hadoop-3.3.5/etc/hadoop/workers

集群 有几个 节点 就写几个
该文件末尾不允许有 空格
该文件全文不允许有 空行

hadoop102
hadoop103
hadoop104

8、分发配置

cd /opt/module/hadoop-3.3.5/etc/hadoop/
xsync workers