hadoop（二）——集群配置、启动、添加新节点

一 hadoop配置

1. 进入hadoop配置文件夹cd /opt/module/hadoop-3.1.3/etc/hadoop/

2. 配置core-site.xml

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop101:8020</value>
    </property>
    <property>
        <name>hadoop.data.dir</name>
        <value>/opt/module/hadoop-3.1.3/data</value>
    </property>
    <property>
        <name>hadoop.proxyuser.hadoop.hosts</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.hadoop.groups</name>
        <value>*</value>
    </property>

</configuration>

3. 配置hdfs-site.xml

<configuration>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>file://${hadoop.data.dir}/name</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file://${hadoop.data.dir}/data</value>
  </property>
    <property>
    <name>dfs.namenode.checkpoint.dir</name>
    <value>file://${hadoop.data.dir}/namesecondary</value>
  </property>
    <property>
    <name>dfs.client.datanode-restart.timeout</name>
    <value>30</value>
  </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop103:9868</value>
  </property>
	<property>
    <name>dfs.permissions.enabled</name>
    <value>false</value>
  </property>

</configuration>

4. 配置yarn-site.xml

<configuration>

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop102</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>

<property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log.server.url</name>
        <value>http://hadoop101:19888/jobhistory/logs</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
</property>

</configuration>

5. 配置mapred-site.xml

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>


<!-- 历史服务器端地址 -->
<property>
    <name>mapreduce.jobhistory.address</name>
    <value>hadoop101:10020</value>
</property>

<!-- 历史服务器web端地址 -->
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop101:19888</value>
</property>

</configuration>

6. 配置workers

hadoop101
hadoop102
hadoop103

7. 记得照快照

二 集群启动

1. hadoop101操作

hdfs namenode -format	//格式化namenode
dfs-start.sh

2. hadoop102操作

yarn-start.sh

3. 查看hadoop101/102/103的启动的进程

jps	//各自执行，每个虚拟机都显示有4个进程（加上jps）

三 为集群添加新节点

1. 用hadoop101克隆新虚拟机，修改hostname和ip（上一篇）

2. 删除/opt/module/hadoop-3.1.3/data和log

3. hadoop101中修改/bin/xsync，添加hadoop104

4. hadoop101中修改workers，加入hadoop104

5. 用xsync同步workersxsync workers

6. 重启hadoop104reboot

7. 执行命令

hdfs --daemon start datanode
yarn --daemon start nodemanager
jps	//加上jps三个进程

8. 查看

登录http://hadoop101:9870/dfshealth.html#tab-datanode查看有新的节点hadoop104