hadoop2.4.1 spark1.0.1编译

spark编译用的是2.4.1版本，而hadoop用的则是2.5.0版本，这样在进行rpc通信的时候，会导致失败。于是将$SPARK_HOME下的pom.xml中的protobuf版本改为2.5.0,重新进行编译

重新编译spark

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

mvn -Pyarn -Phadoop-2.4.1 -Dhadoop.version=2.4.1 -DskipTests clean package

hadoop启动会报

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform.

因为hadoop32位编译所以64位需要重新编译

首先重新编译hadoop2.4.1

系统安装protobuf-2.5.0.tar.bz2

首先安装这个包不然编译会出错

https://code.google.com/p/protobuf/downloads/detail?name=protobuf-2.5.0.tar.bz2&can=2&q=

安装之前还需要安装

sudo apt-get install g++ zlib1g-dev libssl-dev

./configure --prefix=/usr

make 

make check

make install

重新编译hadoop

mvn package -Pdist,native -DskipTests -Dtar

编译eclipse hadoop plugin

https://github.com/winghc/hadoop2x-eclipse-plugin

进入src/contrib/eclipse-plugin

ant jar -Dversion=2.4.1 -Declipse.home=eclipse路径-Dhadoop.home=hadoop路径

将生成好的jar,复制到${eclipse.home}/plugins目录下

插件运行错误修改build.xml

ClassNotFoundException: org.apache.commons.collections.map.UnmodifiableMap


 <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-api-${slf4j-api.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
    <copy file="${hadoop.home}/share/hadoop/common/lib/guava-${guava.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
    <copy file="${hadoop.home}/share/hadoop/common/lib/hadoop-auth-2.4.1.jar"  todir="${build.dir}/lib" verbose="true"/>
    <copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
    <copy file="${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-3.2.1.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/hadoop-common-2.4.1.jar"  todir="${build.dir}/lib" verbose="true"/>


 lib/commons-configuration-1.6.jar,
lib/commons-collections-3.2.1.jar,


红色部分是修改build.xml的地方，错误就去除，该错需要查看eclipse的日志

ok最后祝大家一切顺利