hadoop 安装，hadoop 源码打包

最新推荐文章于 2022-06-26 19:52:42 发布

feng27156

最新推荐文章于 2022-06-26 19:52:42 发布

阅读量2.5k

点赞数 1

分类专栏： hadoop 文章标签： hadoop 安装 hadoop hadoop 配置 hadoop 源码编译 hadoop maven 报错

本文链接：https://blog.csdn.net/feng27156/article/details/40150667

版权

hadoop 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

hadoop 安装

本人的linux （centos6.5 x64）机器上已经装过，maven，jdk1.7 了。

maven 引用的是 oschina 的私服。http://maven.oschina.net/content/groups/public

开始着手下载hadoop版本。

http://www.eu.apache.org/dist/hadoop/

https://svn.apache.org/repos/asf/hadoop/common/tags/

本人下载的是 2.4.1 的。

从svn 上检出源码，通过

mvn clean install -DskipTests

编译.

在编译中出现

Apache Hadoop Common ............................... FAILURE [ 1.054 s]

Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.4.1:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1]

问了一下度娘，

编译需要protoc2.5.0的支持，所以还要下载protoc，下载地址:https://code.google.com/p/protobuf/downloads/list，要下载2.5.0版本噢

对protoc进行编译安装前先要装几个依赖包:gcc,gcc-c++,make 如果已经安装的可以忽略

yum install gcc  
yum install gcc-c++ 
yum install make

安装protoc

1 tar -xvf protobuf-2.5.0.tar.bz2  
2 cd protobuf-2.5.0  
3 ./configure --prefix=/opt/protoc/  
4 make && make install

其他大概没啥问题。

随时补充：

我测试导入的配置文件。

export JAVA_HOME=/usr/java/default
export HADOOP_INSTALL=/usr/hadoop/workspace/hadoop-2.4.1
export HADOOP_CONF_DIR=/usr/hadoop/workspace/hadoop-2.4.1/etc/hadoop
export YARN_CONF_DIR=/usr/hadoop/workspace/hadoop-2.4.1/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export HADOOP_YARN_HOME=$HADOOP_INSTALL
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/natiave
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"

export PATH=$PATH:$HADOOP_INSTALL/bin:$HADOOP_INSTALL/sbin

尼玛，源码编译，maven 3.1.1&3.2.2 ,jdk 1.7 64 位，centos 6.5 64 位，hadoop2.4.1。

通过官方说的mvn clean package -Pdist,native -DskipTests -Dtar根本不行，总是hadoop-hdfs 的javadoc 报错。

错误内容：

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-hdfs: MavenReportException: Error while creating arch
ive:
[ERROR] Exit code: 137 - /usr/hadoop/workspace/hadoop-2.4.1-src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/XmlEditsVisitor.java:32: 警告: OutputFormat是内部专用 API, 可能会在未来发行版中删除
[ERROR] import com.sun.org.apache.xml.internal.serialize.OutputFormat;
[ERROR] ^
[ERROR] /usr/hadoop/workspace/hadoop-2.4.1-src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/XmlEditsVisitor.java:33: 警告: XMLSerializer是内部专用 API, 可能会在未来发行版中删除
[ERROR] import com.sun.org.apache.xml.internal.serialize.XMLSerializer;
[ERROR] ^
[ERROR] /bin/sh: line 1: 26832 已杀死 /usr/java/jdk1.7.0_60/jre/../bin/javadoc -J-Xmx512m @options @packages
[ERROR]
[ERROR] Command line was: /usr/java/jdk1.7.0_60/jre/../bin/javadoc -J-Xmx512m @options @packages
[ERROR]
[ERROR] Refer to the generated Javadoc files in '/usr/hadoop/workspace/hadoop-2.4.1-src/hadoop-hdfs-project/hadoop-hdfs/target' dir.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-hdfs

经过一天的折腾，终于，用mvn clean package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true 过了。擦。

，尼玛，我都遇到这样的问题了。好吧。一定要修改linux的hostname，不然会有问题的。

hadoop jar workspace/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount /hadoop-hadoop-namenode-223.log /output，如果这个demo 运行成功的话，那么你的集群大致上没什么大的问题了。