用Maven编译Apache flume-ng 1.5.0源码及问题解决

最新推荐文章于 2023-05-05 14:12:02 发布

转载最新推荐文章于 2023-05-05 14:12:02 发布 · 4.9k 阅读

文章标签：

#flume

日志系统同时被 2 个专栏收录

12 篇文章

订阅专栏

flume

5 篇文章

订阅专栏

在今年的5月22号，Flume-ng 1.5.0版本正式发布，关于Flume-ng 1.5.0版本的新特性可以参见本博客的《Apache Flume-ng 1.5.0正式发布》进行了解。关于Apache flume-ng 1.4.0版本的编译可以参见本博客《基于Hadoop-2.2.0编译flume-ng 1.4.0及错误解决》。本文将讲述如何用Maven编译Apache flume-ng 1.5.0源码。

一、到官方网站下载相应版本的flume-ng源码

 
# wget http://mirror.bit.edu.cn/apache/flume/1.5.0/apache-flume-1.5.0-src.tar.gz
 
# tar -zxf apache-flume-1.5.0-src.tar.gz
 
# cd apache-flume-1.5.0-src

二、用maven在相应的Hadoop上编译Flume-ng 1.5.0源码

Flume-ng 1.5.0默认的Hadoop版本是1.0.1，Hbase是0.92.1。我们可以在编译的时候加上一些参数，指定Hadoop 2.2.0版本进行编译：

`1`	`-Phadoop-2`

三、Hadoop、Hbase对应的test不能下载

　　如果在编译的时候出现了类似如下的情况，请将相应pom.xml文件中的Hadoop-test和Hbase-test依赖包去掉：

 
<!--
 
 User: 过往记忆
 
 Date: 14-6-16
 
 Time: 23:45
 
 bolg: http://www.iteblog.com
 
 本文地址：http://www.iteblog.com/archives/1043
 
 过往记忆博客，专注于hadoop、hive、spark、shark、flume的技术博客，大量的干货
 
 过往记忆博客微信公共帐号：iteblog_hadoop
 
-->
 
<dependency>
 
        <groupId>org.apache.hbase</groupId>
 
        <artifactId>hbase</artifactId>
 
        <version>${hbase.version}</version>
 
        <classifier>tests</classifier>
 
        <scope>test</scope>
 
</dependency>
 
 
 
<dependency>
 
         <groupId>org.apache.hadoop</groupId>
 
         <artifactId>hadoop-test</artifactId>
 
         <version>${hadoop.version}</version>
 
</dependency>

四、elasticsearch-0.90.1.jar read错误

　　如果你在编译的过程中出现一下异常：

 
[ERROR] Failed to execute goal org.apache.maven.plugins:
 
maven-compiler-plugin:2.3.2:compile (default-compile) on
 
project flume-ng-elasticsearch-sink: Compilation failure
 
[ERROR] error: error reading
 
/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar;
 
error in opening zip file
 
[ERROR] -> [Help 1]

　　只需要到你maven本地仓库将elasticsearch-0.90.1.jar文件删除，然后重新下载即可：

 
# rm elasticsearch-0.90.1.jar
 
wget http://central.maven.org/maven2/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar

五、ua-parser-1.3.0下载失败

编译过程中，下载ua-parser-1.3.0.pom可能失败，出现如下错误：

 
[ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink:
 
 Could not resolve dependencies for  project
 
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0:
 
Failed to collect dependencies for  [org.apache.flume:flume-ng-core:jar:1.5.0
 
(compile), org.slf4j:slf4j-api:jar:1.6.1  (compile),
 
org.kitesdk:kite-morphlines-all:pom:0.12.0  (compile?),
 
 org.slf4j:jcl-over-slf4j:jar:1.6.1  (provided),
 
org.apache.solr:solr-test-framework:jar:4.3.0  (test),
 
org.kitesdk:kite-morphlines-solr-core:jar:tests:0.12.0  (test),
 
junit:junit:jar:4.10  (test)]: Failed to read artifact descriptor
 
for  ua_parser:ua-parser:jar:1.3.0: Could not transfer artifact
 
ua_parser:ua-parser:pom:1.3.0  from/to maven-twttr (http://maven.twttr.com):
 
 Connection to http://maven.twttr.com refused: Connection timed out -> [Help 1]

只需要加一个新的repository即可：

 
 <repository>
  
<id>p2.jfrog.org</id>
  
<url>http://p2.jfrog.org/libs-releases</url>
  
 </repository>

假如上面的repository仍然不能用，可以google “ua-parser-1.3.0.pom”获取其他源。

六、编译Flume-ng 1.5.0源码

`1`	`# mvn install -Phadoop-2` `-DskipTests -Dtar`

编译需要一段时间，这个需要看你编译时的网速。如果一切顺利的话，你将会看到以下输出，说明你的Flume-ng 1.5.0编译成功：

 
[INFO] ------------------------------------------------------------------------
 
[INFO] Reactor Summary:
 
[INFO]
 
[INFO] Apache Flume ...................................... SUCCESS [1.360s]
 
[INFO] Flume NG SDK ...................................... SUCCESS [2.903s]
 
[INFO] Flume NG Configuration ............................ SUCCESS [0.483s]
 
[INFO] Flume NG Core ..................................... SUCCESS [2.531s]
 
[INFO] Flume NG Sinks .................................... SUCCESS [0.110s]
 
[INFO] Flume NG HDFS Sink ................................ SUCCESS [1.668s]
 
[INFO] Flume NG IRC Sink ................................. SUCCESS [0.484s]
 
[INFO] Flume NG Channels ................................. SUCCESS [0.074s]
 
[INFO] Flume NG JDBC channel ............................. SUCCESS [0.579s]
 
[INFO] Flume NG file-based channel ....................... SUCCESS [0.712s]
 
[INFO] Flume NG Spillable Memory channel ................. SUCCESS [0.552s]
 
[INFO] Flume NG Node ..................................... SUCCESS [0.619s]
 
[INFO] Flume NG Embedded Agent ........................... SUCCESS [1.001s]
 
[INFO] Flume NG HBase Sink ............................... SUCCESS [2.373s]
 
[INFO] Flume NG ElasticSearch Sink ....................... SUCCESS [0.899s]
 
[INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [2.483s]
 
[INFO] Flume NG Kite Dataset Sink ........................ SUCCESS [35.925s]
 
[INFO] Flume Sources ..................................... SUCCESS [0.054s]
 
[INFO] Flume Scribe Source ............................... SUCCESS [1.231s]
 
[INFO] Flume JMS Source .................................. SUCCESS [2.133s]
 
[INFO] Flume Twitter Source .............................. SUCCESS [1.204s]
 
[INFO] Flume legacy Sources .............................. SUCCESS [0.043s]
 
[INFO] Flume legacy Avro source .......................... SUCCESS [1.234s]
 
[INFO] Flume legacy Thrift Source ........................ SUCCESS [1.320s]
 
[INFO] Flume NG Clients .................................. SUCCESS [0.038s]
 
[INFO] Flume NG Log4j Appender ........................... SUCCESS [6.175s]
 
[INFO] Flume NG Tools .................................... SUCCESS [1.559s]
 
[INFO] Flume NG distribution ............................. SUCCESS [1:51.993s]
 
[INFO] Flume NG Integration Tests ........................ SUCCESS [1.517s]
 
[INFO] ------------------------------------------------------------------------
 
[INFO] BUILD SUCCESS
 
[INFO] ------------------------------------------------------------------------
 
[INFO] Total time: 3:03.921s
 
[INFO] Finished at: Mon Jun 16  23:37:51  CST 2014
 
[INFO] Final Memory: 135M/745M
 
[INFO] ------------------------------------------------------------------------

最后将会在flume-ng-dist/target目录下产生以下的文件：

 
total 29208
 
drwxr-xr-x 3  wyp wyp     4096  Jun 16  23:36  apache-flume-1.5.0-bin
 
-rw-r--r-- 1  wyp wyp 27307850  Jun 16  23:36  apache-flume-1.5.0-bin.tar.gz
 
drwxr-xr-x 3  wyp wyp     4096  Jun 16  23:36  apache-flume-1.5.0-src
 
-rw-r--r-- 1  wyp wyp  2543174  Jun 16  23:37  apache-flume-1.5.0-src.tar.gz
 
drwxr-xr-x 2  wyp wyp     4096  Jun 16  23:36  archive-tmp
 
drwxr-xr-x 3  wyp wyp     4096  Jun 16  23:36  maven-shared-archive-resources
 
-rw-r--r-- 1  wyp wyp      990  Jun 16  23:37  rat.txt