Hadoop安装与配置

1.下载Hadoop压缩包

wget http://archive.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz

本地路径：
/home/liaohui/tools/hadoop/hadoop-2.7.2

2.解压
tar -zxvf hadoop-2.7.2.tar.gz

3.配置环境
gedit ~/.bashrc

最后添加配置项

#HADOOP VARIABLES START
export HADOOP_INSTALL=/home/liaohui/tools/hadoop/hadoop-2.7.2
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
#HADOOP VARIABLES END

4.保存配置
source ~/.bashrc

5.修改 /usr/hadoop/hadoop-2.7.2/etc/hadoop/hadoop-env.sh 在 export JAVA_HOME=${JAVA_HOME} 行前面插入如下内容
gedit hadoop-env.sh

export JAVA_HOME=/usr/local/jdk/jdk1.8.0_91
export HADOOP_CONF_DIR=/home/liaohui/tools/hadoop/hadoop-2.7.2/etc/hadoop

5、配置伪分布式
先在hadoop-2.7.2目录下新建一个文件夹tmp

cd /home/liaohui/tools/hadoop/hadoop-2.7.2
mkdir tmp

5.1、配置 core-site.xml
cd /home/liaohui/tools/hadoop/hadoop-2.7.2

gedit core-site.xml
在后面添加配置项：

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/home/liaohui/tools/hadoop/hadoop-2.7.2/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

5.2、配置 hdfs-site.xml
gedit hdfs-site.xml
在最后面添加配置项：

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/home/liaohui/tools/hadoop/hadoop-2.7.2/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/home/liaohui/tools/hadoop/hadoop-2.7.2/tmp/dfs/data</value>
    </property>
</configuration>

5.3、配置yarn-site.xml
gedit yarn-site.xml

<configuration>
  <property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
  </property>
  <property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

5.4、配置mapred-site.xml
从模板文件复制一个xml，
mv mapred-site.xml.template mapred-site.xml
gedit mapred-site.xml

<configuration>
 <property>
 <name>mapred.job.tracker</name>
 <value>localhost:9001</value>
 </property>
</configuration>

6、配置完成后，执行格式化：
hdfs namenode -format

7、启动hadoop
cd /home/liaohui/tools/hadoop/hadoop-2.7.2/sbin
start-all.sh

或者
start-dfs.sh
start-yarn.sh

查看各个进程是否正常启动，执行：jps。如果一切正常，将看到下列结果：

打开浏览器，输入如下内容
http://localhost:8088/ - Hadoop 管理介面

8.停止 hadoop
cd /home/liaohui/tools/hadoop/hadoop-2.7.2/sbin
stop-all.sh

9.设置日志打印信息
敲入命令：vi /etc/profile 在最后一行加入：export HADOOP_ROOT_LOGGER=DEBUG,console 保存退出，设置logger级别，
让其在控制台打印日志信息，再敲入命令：source /etc/profile，刷新设置。