Hadoop集群配置文件

知识点：

hdfs---分布式文件系统   

namenode datanode

MapReduce---分布式计算框架

yarn---资源调度管理平台   

resourcemanager nodemanager

secondarynamenode

Master	slave1	slave2
namenode	datanode	datanode
datanode	rersourcemanager	nodemanager
nodemanager	nodemanager	secondarynamenode

1.查看module下文件

2.进入hadoop-3.1.4下查看

3.创建目录data并查看

4.进入etc下并查看

5.进入hadoop下并查看

6.修改core-site.xml文件

代码如下：

<property>
<name>fs.defaultFS</name>
<value>hdfs://Master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-3.1.4/data</value>
</property>
<property>
<name>hadoop.http.staticuser.user</name>
<value>root</value>
</property>

7.修改hdfs-site.xml文件

代码如下：

<property>
<name>dfs.namenode.http-address</name>
<value>Master:9870</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>slave2:9868</value>
</property>
<property>
    <name>dfs.permissions</name>    
    <value>false</value>
</property>

8.修改mapred-site.xml文件

代码如下：

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>Master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>Master:19888</value>
</property>

9.修改yarn-site.xml文件(classpath的value里用hadoop classpath查询）

代码如下：

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>slave1</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>/opt/module/hadoop-3.1.4/etc/hadoop:/opt/module/hadoop-3.1.4/share/hadoop/common/lib/*:/opt/module/hadoop-3.1.4/share/hadoop/common/*:/opt/module/hadoop-3.1.4/share/hadoop/hdfs:/opt/module/hadoop-3.1.4/share/hadoop/hdfs/lib/*:/opt/module/hadoop-3.1.4/share/hadoop/hdfs/*:/opt/module/hadoop-3.1.4/share/hadoop/mapreduce/lib/*:/opt/module/hadoop-3.1.4/share/hadoop/mapreduce/*:/opt/module/hadoop-3.1.4/share/hadoop/yarn:/opt/module/hadoop-3.1.4/share/hadoop/yarn/lib/*:/opt/module/hadoop-3.1.4/share/hadoop/yarn/*
</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>

10.修改workers文件

代码如下：

Master
slave1
slave2

11.添加修改环境变量相关设置

代码如下：

export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_JOURNALNODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root