蜗农的博客

总有一天,你会被现在的付出感动

Hadoop 2.4源码编译和导入eclipse



Hadoop源码编译


Linux 环境下编译源代码之前,需进行以下准备工作:

安装3.0.2及 以上版本的Maven

Protocol Buffers 安装版本为 2.5.0

安装findbugs

安装2.6及以上版本的CMake

确认可以连接互联网(Maven 要从代码库中下载依赖包)


Maven国内镜像配置
进入安装目录 ~/soft/apache-maven-3.0.5/conf,编辑 settings.xml 文件

修改settings.xml:

<mirror> 
 <id>nexus-osc</id> 
  <mirrorOf>*</mirrorOf> 
  <name>Nexus osc</name> 
  <url>http://maven.oschina.net/content/groups/public/</url> 
</mirror> 

修改<profiles>内容:

<profile> 
 <id>jdk-1.7</id> 
 <activation> 
     <jdk>1.7</jdk> 
 </activation> 
 <repositories> 
  <repository> 
   <id>nexus</id> 
   <name>local private nexus</name> 
   <url>http://maven.oschina.net/content/groups/public/</url> 
   <releases> 
    <enabled>true</enabled> 
   </releases> 
   <snapshots> 
    <enabled>false</enabled> 
   </snapshots> 
  </repository> 
 </repositories>
 <pluginRepositories> 
  <pluginRepository> 
   <id>nexus</id> 
   <name>local private nexus</name> 
   <url>http://maven.oschina.net/content/groups/public/</url> 
   <releases> 
    <enabled>true</enabled> 
   </releases> 
   <snapshots> 
    <enabled>false</enabled> 
   </snapshots> 
  </pluginRepository> 
 </pluginRepositories> 
</profile>

$ cp ~/soft/apache-maven-3.0.5/conf/settings.xml ~/.m2/

修改: vi /etc/resolv.conf     
nameserver 8.8.8.8
nameserver 8.8.4.4    

 

编译源码

进入源码目录:

cd ~/workspace/hadoop-2.4.0-src

tar –xzvf Hadoop-2.4.0-src.tar.gz

mvn clean

mvn package -Pdist,native,docs-DskipTests –Dtar

Hadoop导入eclipse

进入源码目录:

cd ~/workspace/hadoop-2.4.0-src

cd hadoop-maven-plugins/

mvn install

 

cd ~/workspace/hadoop-2.4.0-src

mvn eclipse:eclipse –DeskipTests


elipse-->import-->exist project from workspace导入源码 


eclipse设置Mavenwindow->preference->maven->{Installations...;userSettingsmaven\conf\settings.xml}

 

配置eclispe M2_REPO变量

Window/Preference/java/Build Path,M2_REPO默认路径修改为:

~/.m2/repository


导入eclipse报错的解决方法:

1.hadoop-streaming里面的build path有问题,显示/root/workspace/hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf(missing)

删除Link Source: hadoop-yarn-server-resourcemanager/conf
再添加Link Source:
源码根目录/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf,随便起个名字;
inclusion patterns:capacity-scheduler.xml;exclusion patters:**/*.java

2.AvroRecord cannot be resolved to a type TestAvroSerialization.java

解决方法如下:
首先,下载 avro-tools-1.7.4.jar(点击下载)。存放于 hadoop-2.4.0-src 目录中。
然后执行下述命令:
$ cd ~/workspace/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/src/test/avro
$ java -jar ~/workspace/hadoop-2.4.0-src/avro-tools-1.7.4.jar compile schema avroRecord.avsc ../java
最后,刷新Eclipse工程。

3. org.apache.hadoop.ipc.protobuf.× 无法解析
解决方法如下,依次执行下述命令。
$ cd ~/workspace/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/src/test/proto
//注释:进入该目录
$ protoc --java_out=../java *.proto

 

 

阅读更多
个人分类: hadoop
上一篇编程中遇到的SQL困惑总结
想对作者说点什么? 我来说一句

hadoop2.4.1的Eclipse插件

2014年08月07日 21.01MB 下载

没有更多推荐了,返回首页

关闭
关闭