CDH hadoop 编译

最新推荐文章于 2020-05-31 22:30:04 发布

wjl7813

最新推荐文章于 2020-05-31 22:30:04 发布

阅读量404

点赞数

分类专栏： hadoop

本文链接：https://blog.csdn.net/wjl7813/article/details/79101837

版权

hadoop 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

本文档详细介绍了如何在Linux环境下编译CDH Hadoop，包括安装必要的依赖，如svn、automake、libtool等，并下载Apache Maven、protobuf、snappy等组件进行编译和安装，最后成功运行hadoop checknative验证编译结果。

摘要由CSDN通过智能技术生成

最好是 163.yum epel.repo 的yum 资源

准备工作

yum install svn autoconf automake libtool cmake ncurses-devel openssl-devel gcc*
wget http://ftp.cuhk.edu.hk/pub/packages/apache.org/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz -O apache-maven-3.3.9-bin.tar.gz
wget ftp://ftp.netbsd.org/pub/pkgsrc/distfiles/protobuf-2.5.0.tar.gz -O protobuf-2.5.0.tar.gz
wget http://pkgs.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.1.tar.gz/8887e3b7253b22a31f5486bca3cbc1c2/snappy-1.1.1.tar.gz
tar xf apache-maven-3.3.9-bin.tar.gz -C /home/hadoop/app/

cat /home/hadoop/.bash_profile | grep MAVEN
export MAVEN_HOME=/home/hadoop/app/apache-maven-3.3.9
export PATH=$PATH:$MAVEN_HOME/bin

tar xf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure
make
make install
sudo make install
sudo ldconfig
protoc --version

tar xf snappy-1.1.1.tar.gz
cd ../snappy-1.1.1
./configure
make
make install
sudo make install
ls -lh /usr/local/lib |grep snappy

tar xf hadoop-2.6.0-cdh5.7.0-src.tar.gz
cd hadoop-2.6.0-cdh5.7.0

mvn clean package -Pdist,native -DskipTests -Dtar 如果报错的就用下面的尝试重新编译，且下面的编译效率更高些

mvn clean package -Pdist -Dtar -Dmaven.javadoc.skip=true -DskipTests -fail-at-end -Pnative

[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ bin/hadoop checknative
17/08/03 23:00:36 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
17/08/03 23:00:36 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so