centos Hadoop 3.1.2伪分布式配置 实践

正文：

hadoop的安装
http://dblab.xmu.edu.cn/blog/install-hadoop-in-centos/

hadoop相关环境变量配置：（这一步很关键！！！）
[hadoop@localhost hadoop-3.1.2]$ vim ~/.bashrc
注：.bashrc仅对当前用户生效，.profile对所有用户生效

# Hadoop Environment Variables
export HADOOP_HOME=/usr/local/hadoop-3.1.2
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

hadoop伪分布式 文件配置：（这一步也很关键！！！）
1.[hadoop@localhost hadoop-3.1.2]$ vim ./etc/hadoop/core-site.xml

<configuration>
<!-- 配置dataNode保存数据的位置  -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop-3.1.2/datanode</value>
        <description>Abase for other temporary directories.</description>
    </property>
<!-- 配置HDFS的主节点，nameNode  -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

2.[hadoop@localhost hadoop-3.1.2]$ vim ./etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>
                   

3.[hadoop@localhost hadoop-3.1.2]$ vim ./etc/hadoop/mapred-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>

         <property>
                 <name>mapreduce.application.classpath</name>
                 <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
         </property>
</configuration>

4.[hadoop@localhost hadoop-3.1.2]$ vim ./etc/hadoop/yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
        <property>
                <name>yarn.resourcemanger.hostname</name>
                <value>localhost</value>
        </property>


        <property>
                <name>yarme>yarn.nodemanager.env-whitelist</name>
                <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
         </property>

        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>

HDFS初始化
[hadoop@localhost hadoop-3.1.2]$ hdfs namenode -format

hadoop、yarn的启动和停止

[hadoop@localhost hadoop-3.1.2]$ start-dfs.sh
[hadoop@localhost hadoop-3.1.2]$ start-yarn.sh

[hadoop@localhost hadoop-3.1.2]$ stop-dfs.sh
[hadoop@localhost hadoop-3.1.2]$ stop-yarn.sh

=======================================================================

（详情可参看正文部分）
参考的博客：
1.厦门大学大数据实验室
http://dblab.xmu.edu.cn/blog/install-hadoop-in-centos/
注：hadoop安装部分讲的很好

2.剑行极锋的博客
https://blog.csdn.net/sven119/article/details/79930878
注：配置4个文件部分，及启动、关闭。

[hadoop@localhost hadoop-3.1.2]$ vim ./etc/hadoop/core-site.xml 
[hadoop@localhost hadoop-3.1.2]$ vim ./etc/hadoop/hdfs-site.xml 
[hadoop@localhost hadoop-3.1.2]$ vim ./etc/hadoop/mapred-site.xml 
[hadoop@localhost hadoop-3.1.2]$ vim ./etc/hadoop/yarn-site.xml 

问题解决：

1.针对在执行start-dfs.sh命令时出现
警告：WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
的解决方法参考的是（产生的原理我就不细说了，有兴趣的可以看一下链接）
https://stackoverflow.com/questions/38198857/warn-util-nativecodeloader-unable-to-load-native-hadoop-library-for-your-platfo/38199131

https://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning

操作：

在vim ~/.bashrc中添加

export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib/native"

=======================================================================