Hadoop 2.7.1的源码编译

2019独角兽企业重金招聘Python工程师标准>>>

Compile Hadoop from Source

Compile Hadoop 2.x help you fix error Unable to load native-hadoop library

1. Install Requirements

• Java Oracle: otherwise, while maven build you may be occur exception below if not using java oracle

  Failed to execute goal on project hadoop-annotations: Could not resolve dependencies for project org.apache.hadoop:hadoop-annotations:jar:2.7.1: Could not find artifact jdk.tools:jdk.tools:jar:1.7 at specified path /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/../lib/tools.jar

• Maven, libssl-dev, zlib-development, cmake

• Protocol Buffers: Version must be v2.5.0, otherwise while maven build you can occur exception below

  Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.1:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 2.6.1', expected version is '2.5.0'

  wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz tar -xzf protobuf-2.5.0.tar.gz cd protobuf-2.5.0 ./configure make make check make install protoc --version

  While install Protocol Buffers, if you occur exception below, this mean you using old gcc version

  error: uninitialized member 'google::protobuf::util::converter::DataPiece::

  Upgrade gcc

  cd /etc/yum.repos.d wget http://people.centos.org/tru/devtools-2/devtools-2.repo yum --enablerepo=testing-devtools-2-centos-6 install devtoolset-2-gcc devtoolset-2-gcc-c++ # export CC=/opt/rh/devtoolset-2/root/usr/bin/gcc # export CPP=/opt/rh/devtoolset-2/root/usr/bin/cpp # export CXX=/opt/rh/devtoolset-2/root/usr/bin/c++ # gcc --version # still old version, but CC, CPP, CXX used in protobuf build ln -s /opt/rh/devtoolset-2/root/usr/bin/* /usr/local/bin/ hash -r gcc --version

  While install Protocol Buffers, if you occur exception below, this mean you using old binutils version

  error: uninitialized member 'google::protobuf::util::converter::DataPiece::

  Upgrade binutils

  yum --enablerepo=testing-devtools-2-centos-6 install devtoolset-2-binutils ln -s /opt/rh/devtoolset-2/root/usr/bin/as /usr/local/bin/as hash -r as --version

Download Hadoop Source

wget http://mirrors.maychuviet.vn/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1-src.tar.gz tar -xzf hadoop-2.7.1-src.tar.gz

Compile Hadoop from Source

cd hadoop-2.7.1-src mvn package -Pdist,native -DskipTests -Dtar

转载于:https://my.oschina.net/u/2306127/blog/546616