hadoop集群配置

1.core-site.xml

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://192.168.0.111:9999</value>
    </property>

hadoop访问地址

2.hdfs-site.xml

<property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    
    <property>
       <name>dfs.namenode.name.dir</name>
       <value>/home/hadoop/bigdata/name</value>
     </property>
     
     <property>
       <name>dfs.datanode.data.dir</name>
       <value>/home/hadoop/bigdata/data</value>
     </property>
     
     
    <property>
       <name>dfs.webhdfs.enabled</name>
       <value>true</value>
     </property>

//  centeOS 不配这个导致50070无法访问
    <property> 
        <name>dfs.http.address</name> 
        <value>0.0.0.0:50070</value> 
    </property>

 

3.yarn-site.xml

 

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>node1:8032</value>
    </property>
    
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>node1:8030</value>
    </property>
    
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>node1:8031</value>
    </property>
    
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>node1:8033</value>
    </property>
    
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>node1:8088</value>
    </property>
    
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
    
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>

 

4.mapred-site.xml      

 <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
        <property>
            <name>mapred.job.tracker</name>
            <value>node3:9001</value>
        </property>

5.slave配置

node2
node3

 

6.配置hadoop-env.sh

 

export JAVA_HOME=/usr/local/jdk1.8.0_144

 

7.防火墙,改主机名

service iptables stop

hostname node1

 

8.环境变量

export HADOOP_HOME=/usr/local/hadoop-2.7.4
export JAVA_HOME=/usr/local/jdk1.8.0_144
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

 

9.时间同步

报错:

YarnException: Unauthorized request to start container. 
This token is expired. current time is 1537779423784 found 1535101599262
Note: System times on machines may be out of sync. Check system time and time zones.

yum -y install ntp ntpdate

ntpdate cn.pool.ntp.org

10.yarn配错

2018-09-24 17:02:03,211 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
2018-09-24 17:02:04,350 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-24 17:02:05,351 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

<property>
    <name>yarn.resourcemanager.address</name>
    <value>master:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:8031</value>
  </property>

11.启动命令

1.启停hdfs

    start-dfs.sh
    stop-dfs.sh

2.启tingyarn

    start-yarn.sh
    stop-yarn.sh

3.启动所有

    start-all.sh
    stop-all.sh

启动namenode
hadoop-daemon.sh start namenode
hadoop-daemon.sh start datanode

若发现有节点未启动
注意先清空 data与name中的文件