Hive2.1安装后运行mapreduce出错，换成hadoop2.6.4全程记录！

Hive2.1安装后运行mapreduce出错，换成hadoop2.6.4全程记录！

解压hadoop2.6.4到/opt目录后。


修改hadoop-env.sh
# The java implementation to use.
export JAVA_HOME=/usr/java/jdk
修改~/.bashrc

#HADOOP VARIABLES START
export JAVA_HOME=/usr/java/jdk
export HADOOP_INSTALL=/opt/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$JAVA_HOME/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export HIVE_HOME=/opt/hive
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib/native"
export CLASS_PATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export CLASS_PATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
#HADOOP VARIABLES END
export PATH=$PATH:$JAVA_HOME/bin:$HIVE_HOME/bin:$HADOOP_INSTALL/bin:$SPARK_HOME/bin

 如果把HADOOP_OPTS配置为：

HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib" 

 
 则会出现如下错误：       

[html] view plain copy 在CODE上查看代码片派生到我的代码片
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  


在hadoop根目录下执行Jar命令，运行自带的wordcount
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount input output
结果出来了：

16/11/23 12:46:36 INFO mapreduce.Job: Counters: 33
	File System Counters
		FILE: Number of bytes read=547468
		FILE: Number of bytes written=1054406
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework

说明Hadoop单机模式配置成功！下面，进入伪分布式模式配置。
在hadoop根目录下，新建tmp、hdfs、hdfs/data、hdfs/name目录。
脚本如下：

dyq@ubuntu:/opt/hadoop-2.6.4$ mkdir tmp
dyq@ubuntu:/opt/hadoop-2.6.4$ mkdir hdfs
dyq@ubuntu:/opt/hadoop-2.6.4$ mkdir hdfs/data
dyq@ubuntu:/opt/hadoop-2.6.4$ mkdir hdfs/name
dyq@ubuntu:/opt/hadoop-2.6.4$ ls
bin  hdfs     input  libexec      NOTICE.txt  README.txt  share
etc  include  lib    LICENSE.txt  output      sbin        tmp

core-site.xml配置如下：

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://192.168.0.10:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop-2.6.4/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
</configuration>

hdfs-site.xml配置：这里的/opt/hadoop-2.6.4/hdfs/name是我们刚才新建的几个目录。

<configuration>  
<property>  
        <name>dfs.replication</name>  
        <value>1</value>  
    </property>  
    <property>  
        <name>dfs.namenode.name.dir</name>  
        <value>file:/opt/hadoop-2.6.4/hdfs/name</value>  
    </property>  
    <property>  
        <name>dfs.datanode.data.dir</name>  
        <value>file:/opt/hadoop-2.6.4/hdfs/data</value>  
    </property>  
    <property>    //这个属性节点是为了防止后面eclopse存在拒绝读写设置的  
            <name>dfs.permissions</name>  
            <value>false</value>  
     </property>  
 </configuration> 

yarn-env.sh

# some Java parameters
export JAVA_HOME=/usr/java/jdk

mapred-env.sh

# some Java parameters
export JAVA_HOME=/usr/java/jdk

mapreduce-site.xml:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
    <!--指定JobTracker主机与端口号-->  
    <property>  
        <name>mapred.job.tracker</name>  
        <value>ubuntu:9001</value>  
    </property> 
</configuration>

修改yarn-site.xml：

<configuration>
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>ubuntu</value>
</property>
<!-- Site specific YARN configuration properties --> 
    <property>  
        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value>  
    </property>  
</configuration>

格式化namenode：
hadoop namenode -format

返回成功：

16/11/23 13:29:19 INFO namenode.FSImage: Allocated new BlockPoolId: BP-302189630-192.168.0.10-1479878959376
16/11/23 13:29:19 INFO common.Storage: Storage directory /opt/hadoop/hdfs/name has been successfully formatted.
16/11/23 13:29:19 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/11/23 13:29:19 INFO util.ExitUtil: Exiting with status 0
16/11/23 13:29:19 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu/192.168.0.10
************************************************************/

启动hadoop：

sbin/start-dfs.sh
dyq@ubuntu:/opt/hadoop$ sbin/start-dfs.sh
16/11/23 13:30:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [ubuntu]
ubuntu: starting namenode, logging to /opt/hadoop-2.6.4/logs/hadoop-dyq-namenode-ubuntu.out
localhost: starting datanode, logging to /opt/hadoop-2.6.4/logs/hadoop-dyq-datanode-ubuntu.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-2.6.4/logs/hadoop-dyq-secondarynamenode-ubuntu.out
16/11/23 13:30:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
dyq@ubuntu:/opt/hadoop$ jps
3016 SecondaryNameNode
3123 Jps
2826 DataNode
2721 NameNode

启动yarn试试：

dyq@ubuntu:/opt/hadoop$ sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop-2.6.4/logs/yarn-dyq-resourcemanager-ubuntu.out
localhost: starting nodemanager, logging to /opt/hadoop-2.6.4/logs/yarn-dyq-nodemanager-ubuntu.out
dyq@ubuntu:/opt/hadoop$ jps
3167 ResourceManager
3016 SecondaryNameNode
3497 Jps
2826 DataNode
2721 NameNode
3274 NodeManager
dyq@ubuntu:/opt/hadoop$ 

hadoop web控制台页面的端口整理：

50070：hdfs文件管理
8088：ResourceManager
8042：NodeManager

运行wordcount程序：
建立目录：hadoop dfs -mkdir  /input
hadoop dfs -put README.txt /input/


在hadoop主目录中运行：


hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount /input/README.txt /output/1124


如果出现错误：
1/1 local-dirs are bad: /opt/hadoop/tmp/nm-local-dir; 1/1 log-dirs are bad: /opt/hadoop/logs/userlogs 


则是临时文件夹设置有误，重新设置正确即可！