Hadoop Ubuntu安装过程_shuffle errors 都是0-CSDN博客

本文链接：https://blog.csdn.net/qq_26311233/article/details/50115989

1、单机版安装测试

1.1、安装jdk

官网下载安装包，解压装到指定的目录，修改profile或者bashrc，添加如下。

export JAVA_HOME=/usr/lib/jvm/java 安装路径

export JRE_HOME=${JAVA_HOME}/jre

export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib

export PATH=${JAVA_HOME}/bin:$PATH

更新环境变量 source /etc/profile

1.2、apache官网下载hadoop最新的安装包。

解压安装到指定的目录，修改profile或者bashrc，添加如下。

#HADOOP VARIABLES START

export JAVA_HOME=/usr/lib/jvm/java

export HADOOP_INSTALL=/usr/local/hadoop

export PATH=$PATH:$HADOOP_INSTALL/bin

export PATH=$PATH:$HADOOP_INSTALL/sbin

export HADOOP_MAPRED_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_HOME=$HADOOP_INSTALL

export HADOOP_HDFS_HOME=$HADOOP_INSTALL

export YARN_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"

#HADOOP VARIABLES END

更新环境变量 source/etc/profile然后修改hadoop/etc/hadoop/hadoop-env.sh 为export JAVA_HOME=/usr/lib/jvm/java

1.3、安装ssh

Apt-get installopenssh-server

打开服务 /etc/init.d/ssh start

查看下服务是否正确启动 ps -e | grep ssh

设置免密码登录，生成私钥和公钥 ssh-keygen -t rsa -P ""

在输入路径那儿笔者直接回车，这样在／home／hadoop/.ssh里面生成两个文件 id_rsa、id_rsa.pub，前者是私钥，后者是公钥。将公钥追加到authorized_keys里面，它保存所有允许以当前用户身份登录到ssh客户端用户的公钥内容。

cat ~/.ssh/id_rsa.pub>> ~/.ssh/authorized_keys

1.4、MapReduce测试

在hadoop下创建文件夹test，随后拷入文件到test。

bin/hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.7.1-sources.jarorg.apache.hadoop.examples.WordCount test output

结果显示如下：

File System Counters

FILE:Number of bytes read=562212

FILE:Number of bytes written=1108056

FILE:Number of read operations=0

FILE:Number of large read operations=0

FILE:Number of write operations=0

Map-ReduceFramework

Mapinput records=73

Mapoutput records=566

Mapoutput bytes=7737

Mapoutput materialized bytes=3411

Inputsplit bytes=107

Combineinput records=566

Combineoutput records=181

Reduceinput groups=181

Reduceshuffle bytes=3411

Reduceinput records=181

Reduceoutput records=181

SpilledRecords=362

ShuffledMaps =1

FailedShuffles=0

MergedMap outputs=1

GCtime elapsed (ms)=0

Totalcommitted heap usage (bytes)=403701760

ShuffleErrors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

FileInput Format Counters

BytesRead=5519

FileOutput Format Counters

BytesWritten=2719

2、分布式部署

在1的基础上，以下过程安装。

2.1、问题

启动系统会成功,但是会报错.如下:

1） hadoopchecknative –a

显示如下：

Unable to load native-hadoop library for your platform... usingbuiltin-java classes where applicable

解决：修改etc/hadoop-env.sh为

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native/"

2）hadoop checknative -a

15/11/1918:45:58 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2library system-native, will use pure-Java version

15/11/1918:45:58 INFO zlib.ZlibFactory: Successfully loaded & initializednative-zlib library

Nativelibrary checking:

hadoop: true/usr/local/hadoop/lib/native/libhadoop.so.1.0.0

zlib: true /lib/x86_64-linux-gnu/libz.so.1

snappy: false

lz4: true revision:99

bzip2: false

openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so

15/11/19 18:45:58 INFO util.ExitUtil:Exiting with status 1

解决：安装libbz2的库,重新编译库。

Apt-get install libbz2-dev

3）运行时，库为32位，与64位不能兼容，编译源文件。

2.2、编译hadoop

1）下载hadoop源码，按照BUILDING.txt提示安装。

* Oracle JDK 1.7 (preferred)

$sudo apt-get purge openjdk*

$sudo apt-get install software-properties-common

$sudo add-apt-repository ppa:webupd8team/java

$sudo apt-get update

$sudo apt-get install oracle-java7-installer

* Maven

$sudo apt-get -y install maven

* Native libraries

$sudo apt-get -y install build-essential autoconf automake libtool cmakezlib1g-dev pkg-config libssl-dev

* ProtocolBuffer 2.5.0 (required)

$sudo apt-get -y install libprotobuf-dev protobuf-compiler

提示：JDK要求1.7，protobuf版本2.5.0

（下载protobuf2.5.0源码，编译安装。

tar zxvf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

./configure --prefix=/usr/local

make && make install

将java设置成1.7的版本，版本过高出现兼容性编译出错。

update-alternatives--config java

填入提示数字，选择相应的版本java）

2）编译

mvn package -Pdist,native,docs -DskipTests –Dtar

编译报错，找不到protoc库。修改profile

export PATH=/usr/local/protoc/bin:$PATH

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

再接着编译，半小时后编译成功。（保持网络畅通，需要下载其他文件）

结果如下：

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main .................................SUCCESS [ 1.802 s]

[INFO] Apache Hadoop Project POM.......................... SUCCESS [ 1.794 s]

[INFO] Apache Hadoop Annotations.......................... SUCCESS [ 2.925 s]

[INFO] Apache Hadoop Assemblies........................... SUCCESS [ 0.143 s]

[INFO] Apache Hadoop Project Dist POM..................... SUCCESS [ 1.729 s]

[INFO] Apache Hadoop Maven Plugins........................ SUCCESS [ 2.945s]

[INFO] Apache Hadoop MiniKDC.............................. SUCCESS [ 3.023 s]

[INFO] Apache Hadoop Auth................................. SUCCESS [ 3.510 s]

[INFO] Apache Hadoop Auth Examples........................ SUCCESS [ 2.980s]

[INFO] Apache Hadoop Common............................... SUCCESS [02:05 min]

[INFO] Apache Hadoop NFS.................................. SUCCESS [ 9.609 s]

[INFO] Apache Hadoop KMS.................................. SUCCESS [ 28.863 s]

[INFO] Apache Hadoop Common Project....................... SUCCESS [ 0.023s]

[INFO] Apache Hadoop HDFS................................. SUCCESS [02:33 min]

[INFO] Apache Hadoop HttpFS............................... SUCCESS [ 48.291 s]

[INFO] Apache Hadoop HDFS BookKeeperJournal .............. SUCCESS [ 35.138 s]

[INFO] Apache Hadoop HDFS-NFS .............................SUCCESS [ 4.530 s]

[INFO] Apache Hadoop HDFS Project......................... SUCCESS [ 0.042 s]

[INFO] hadoop-yarn........................................ SUCCESS [ 0.044 s]

[INFO] hadoop-yarn-api.................................... SUCCESS [ 31.523 s]

[INFO] hadoop-yarn-common................................. SUCCESS [02:25 min]

[INFO] hadoop-yarn-server................................. SUCCESS [ 0.173 s]

[INFO] hadoop-yarn-server-common.......................... SUCCESS [ 8.642 s]

[INFO] hadoop-yarn-server-nodemanager..................... SUCCESS [ 18.605 s]

[INFO] hadoop-yarn-server-web-proxy....................... SUCCESS [ 3.025s]

[INFO]hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 6.607 s]

[INFO] hadoop-yarn-server-resourcemanager................. SUCCESS [ 18.473 s]

[INFO] hadoop-yarn-server-tests........................... SUCCESS [ 4.956 s]

[INFO] hadoop-yarn-client................................. SUCCESS [ 5.600 s]

[INFO] hadoop-yarn-server-sharedcachemanager.............. SUCCESS [ 3.075 s]

[INFO] hadoop-yarn-applications........................... SUCCESS [ 0.036 s]

[INFO]hadoop-yarn-applications-distributedshell .......... SUCCESS [ 2.193 s]

[INFO] hadoop-yarn-applications-unmanaged-am-launcher..... SUCCESS [ 1.522 s]

[INFO] hadoop-yarn-site................................... SUCCESS [ 0.045 s]

[INFO] hadoop-yarn-registry............................... SUCCESS [ 4.710 s]

[INFO] hadoop-yarn-project................................ SUCCESS [ 4.396 s]

[INFO] hadoop-mapreduce-client............................ SUCCESS [ 0.040 s]

[INFO] hadoop-mapreduce-client-core....................... SUCCESS [ 17.284 s]

[INFO] hadoop-mapreduce-client-common..................... SUCCESS [ 13.978 s]

[INFO] hadoop-mapreduce-client-shuffle.................... SUCCESS [ 3.510 s]

[INFO] hadoop-mapreduce-client-app........................ SUCCESS [ 7.370s]

[INFO] hadoop-mapreduce-client-hs......................... SUCCESS [ 4.945 s]

[INFO] hadoop-mapreduce-client-jobclient.................. SUCCESS [ 13.229 s]

[INFO] hadoop-mapreduce-client-hs-plugins................. SUCCESS [ 1.497 s]

[INFO] Apache Hadoop MapReduce Examples................... SUCCESS [ 4.689 s]

[INFO] hadoop-mapreduce................................... SUCCESS [ 2.875 s]

[INFO] Apache Hadoop MapReduce Streaming.................. SUCCESS [ 10.275 s]

[INFO] Apache Hadoop Distributed Copy..................... SUCCESS [ 23.982 s]

[INFO] Apache Hadoop Archives .............................SUCCESS [ 2.389 s]

[INFO] Apache Hadoop Rumen................................ SUCCESS [ 4.732 s]

[INFO] Apache Hadoop Gridmix.............................. SUCCESS [ 3.810 s]

[INFO] Apache Hadoop Data Join............................ SUCCESS [ 2.387 s]

[INFO] Apache Hadoop Ant Tasks............................ SUCCESS [ 2.065 s]

[INFO] Apache Hadoop Extras............................... SUCCESS [ 2.621 s]

[INFO] Apache Hadoop Pipes................................ SUCCESS [ 10.101 s]

[INFO] Apache Hadoop OpenStack support.................... SUCCESS [ 3.899 s]

[INFO] Apache Hadoop Amazon Web Servicessupport .......... SUCCESS [01:06 min]

[INFO] Apache Hadoop Azure support........................ SUCCESS [ 15.531 s]

[INFO] Apache Hadoop Client............................... SUCCESS [ 8.007 s]

[INFO] Apache Hadoop Mini-Cluster......................... SUCCESS [ 0.700 s]

[INFO] Apache Hadoop Scheduler LoadSimulator ............. SUCCESS [ 4.151s]

[INFO] Apache Hadoop Tools Dist........................... SUCCESS [ 10.364 s]

[INFO] Apache Hadoop Tools................................ SUCCESS [ 0.016 s]

[INFO] Apache Hadoop Distribution......................... SUCCESS [ 44.091 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO]------------------------------------------------------------------------

[INFO] Total time: 16:13 min

[INFO] Finished at:2015-11-18T22:38:15+08:00

[INFO] Final Memory: 202M/537M

[INFO]------------------------------------------------------------------------

3）库编译成功，将库从hadoop-dist/target/hadoop-2.7.1/lib/native中copy到安装目录的lib下，覆盖原有native。

测试native：成功

root@raid:/home/xiaobai/Desktop# hadoopchecknative

15/11/19 21:52:51 INFO bzip2.Bzip2Factory:Successfully loaded & initialized native-bzip2 library system-native

15/11/19 21:52:51 INFO zlib.ZlibFactory:Successfully loaded & initialized native-zlib library

Native library checking:

hadoop: true /usr/local/hadoop/lib/native/libhadoop.so.1.0.0

zlib: true /lib/x86_64-linux-gnu/libz.so.1

snappy: true /usr/local/lib/libsnappy.so.1

lz4: true revision:99

bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1

openssl: true/usr/lib/x86_64-linux-gnu/libcrypto.so

2.3、配置hadoop

1) 配置core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://localhost:9000</value>

</property>

2) 配置hdfs-site.xml

<name>hadoop.tmp.dir</name>

<value>file:/usr/local/hadoop/tmp</value>

</property>

<name>io.file.buffer.size</name>

</property>

<name>hadoop.native.lib</name>

<description>Should native hadooplibraries, if present, be used.</description>

</property>

3) 配置yarn-site.xml

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

4) 配置mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

2.4、系统测试

1）格式化系统

hdfs namenode -format

2）启动系统

root@raid:/home/xiaobai/Desktop#start-dfs.sh

Starting namenodes on [localhost]

localhost: starting namenode, loggingto /usr/local/hadoop/logs/hadoop-root-namenode-raid.out

localhost: starting datanode, loggingto /usr/local/hadoop/logs/hadoop-root-datanode-raid.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode,logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-raid.out

打开网页http://localhost:90000可查看系统节点信息。

3）关闭系统：

root@raid:/home/xiaobai/Desktop#stop-dfs.sh

Stopping namenodes on [localhost]

localhost: stopping namenode

localhost: stopping datanode

Stopping secondary namenodes [0.0.0.0]

0.0.0.0: stopping secondarynamenode