Hadoop(四)

目录

Hadoop部署

一、配置core-site.xml。代理主机

二、HDFS配置文件

三、YARN配置

四、MapReduce配置文件

五、配置workers

配置历史服务器

一、配置mapred-site.xml

配置日志的聚集

---

Hadoop部署

一、配置core-site.xml。代理主机

[doudou@hadoop101 module]$ cd hadoop-3.1.3/

[doudou@hadoop101 hadoop-3.1.3]$ cd etc/hadoop/

[doudou@hadoop101 hadoop]$ cd core-site.xml

[doudou@hadoop101 hadoop]$ vim /core-site.xml

<configuration>
  <!-- 指定NameNode的地址 -->
  <property>
     <name>fs.defaultFS</name>
     <value>hdfs://hadoop102:8020</value>
  </property>
  <property>
     <name>hadoop.tmp.dir</name>
     <value>/opt/module/hadoop-3.1.3/data</value>
  </property>
  <property>
     <name>hadoop.http.staticuser.user</name>
     <value>doudou</value>
  </property>
  <property>
       <name>hadoop.proxyuser.doudou.hosts</name>
       <value>*</value>
  </property>
  <property>
     <name>hadoop.proxyuser.doudou.grups</name>
     <value>*</value>
  </property>
  <property>
       <name>hadoop.proxyuser.doudou.users</name>
         <value>*</value>
  </property>

</configuration>

二、HDFS配置文件

配置hdfs-site.xml

[doudou@hadoop101 hadoop]$ vim hdfs-site.xml 

<configuration>
  <property>
   <name>dfs.namenode.http-address</name>
   <value>hadoop101:9870</value>
  </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop103:9868</value>
  </property>
  <property>
     <name>dfs.replication</name>
     <value>1</value>
  </property>

</configuration>
~                         

分发

[doudou@hadoop101 hadoop]$ xsync hdfs-site.xml 

三、YARN配置

配置yarn-site.xml

[doudou@hadoop101 hadoop]$ vim yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
   <!-- 指定MR走shuffle -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <!-- 指定ResourceManager的地址-->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop103</value>
    </property>

    <!-- 环境变量的继承 -->
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
   <property>
       <name>yarn.scheduler.minimum-allocation-mb</name>
       <value>512</value>
   </property>
   <property>
       <name>yarn.scheduler.maximum-allocation-mb</name>
       <value>4096</value>
   </property>
   <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>4096</value>
   </property>
   <property>
      <name>yarn.nodemanager.vmem-check-enabled</name>
      <value>false</value>
   </property>
</configuration>

[doudou@hadoop101 hadoop]$ xsync yarn-site.xml

四、MapReduce配置文件

配置mapred-site.xml

[doudou@hadoop101 hadoop]$ vim mapred-site.xml 

<configuration>
	<!-- 指定MapReduce程序运行在Yarn上 -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

[doudou@hadoop101 hadoop]$ xsync mapred-site.xml 

五、配置workers

[doudou@hadoop101 ~]$ vim /opt/module/hadoop-3.1.3/etc/hadoop/workers

hadoop101
hadoop102
hadoop103

[doudou@hadoop101 hadoop]$ xsync workers 

配置历史服务器

一、配置mapred-site.xml

[doudou@hadoop101 hadoop]$ vim mapred-site.xml 

<!-- 历史服务器web端地址 -->
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop101:19888</value>
</property>
>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

[doudou@hadoop101 hadoop]$ xsync mapred-site.xml 

配置日志的聚集

一、配置yarn-site.xml

[doudou@hadoop101 hadoop]$ vim yarn-site.xml 

<!-- 开启日志聚集功能 -->
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>
<!-- 设置日志聚集服务器地址 -->
<property>  
    <name>yarn.log.server.url</name>  
    <value>http://hadoop101:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为7天 -->
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
</property>

[doudou@hadoop101 hadoop]$ xsync yarn-site.xml