Hadoop集群搭建及开机自启动-学习笔记

一、Hadoop集群配置

1.解压并复制文件

tar -zxvf /home/software/hadoop-2.8.1.tar.gz
mkdir -p /home/opt/hadoop/
mv /home/software/hadoop-2.8.1 /home/opt/hadoop/

2.修改配置文件

cd /home/opt/hadoop/hadoop-2.8.1/etc/hadoop

1)hadoop-env.sh中配置JAVA_HOME

2)core-site.xml

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://master:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/hadoop/tmp</value>
  </property>
</configuration>

3)hdfs-site.xml

<configuration>  
  <property>  
    <name>dfs.replication</name>  
    <value>2</value>  
  </property>  
  <property>  
    <name>dfs.namenode.name.dir</name>  
    <value>file:/opt/hadoop/dfs/name</value>  
  </property>  
  <property>  
    <name>dfs.datanode.data.dir</name>  
    <value>file:/opt/hadoop/dfs/data</value>  
  </property>  
</configuration>

4)mapred-site.xml

<configuration>  
  <property>  
    <name>mapreduce.framework.name</name>  
    <value>yarn</value>  
  </property>  
  <property>  
    <name>mapreduce.jobhistory.address</name>  
    <value>master:10020</value>  
  </property>  
  <property>  
    <name>mapreduce.jobhistory.webapp.address</name>  
    <value>master:19888</value>  
  </property>  
</configuration>

5)yarn-site.xml

<configuration>  
  <property>  
    <name>yarn.nodemanager.aux-services</name>  
    <value>mapreduce_shuffle</value>  
  </property>  
  <property>  
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.address</name>  
    <value>master:8032</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.scheduler.address</name>  
    <value>master:8030</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.resource-tracker.address</name>  
    <value>master:8031</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.admin.address</name>  
    <value>master:8033</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.webapp.address</name>  
    <value>master:8088</value>  
  </property>  
</configuration>

3.修改slaves文件，加入
slave1
slave2
3.将所有配置文件复制到所有机器上
4.在master节点格式化

/home/opt/hadoop/hadoop-2.8.1/bin/hdfs namenode -format 

5.在master节点启动

/home/opt/hadoop/hadoop-2.8.1/sbin/start-all.sh
/home/opt/hadoop/hadoop-2.8.1/sbin/stop-all.sh

二、Hadoop开机自启

1.进入/etc/rc.d/init.d目录

cd /etc/rc.d/init.d

2.新建文件hadoop

touch hadoop

3.添加以下内容，只在master配置即可

#!/bin/bash
#chkconfig:2345 20 90
#description:hadoop
#processname:hadoop
export JAVA_HOME=/home/opt/java/jdk1.8.0_151
case $1 in
    start) su root /home/opt/hadoop/hadoop-2.8.1/sbin/start-all.sh;;
    stop) su root /home/opt/hadoop/hadoop-2.8.1/sbin/stop-all.sh;;
    *) echo "require start|stop" ;;
esac

4.给文件设置权限

chmod +x hadoop

5.加入开机自启列表

chkconfig --add hadoop

6.查看列表

chkconfig --list