用Maven编译Apache flume-ng 1.5.0源码及问题解决

在今年的5月22号，Flume-ng 1.5.0版本正式发布，关于Flume-ng 1.5.0版本的新特性可以参见本博客的《Apache Flume-ng 1.5.0正式发布》进行了解。关于Apache flume-ng 1.4.0版本的编译可以参见本博客《基于Hadoop-2.2.0编译flume-ng 1.4.0及错误解决》。本文将讲述如何用Maven编译Apache flume-ng 1.5.0源码。

一、到官方网站下载相应版本的flume-ng源码

1	# wget http://mirror.bit.edu.cn/apache/flume/1.5.0/apache-flume-1.5.0-src.tar.gz

2	# tar -zxf apache-flume-1.5.0-src.tar.gz

3	# cd apache-flume-1.5.0-src

二、用maven在相应的Hadoop上编译Flume-ng 1.5.0源码

Flume-ng 1.5.0默认的Hadoop版本是1.0.1，Hbase是0.92.1。我们可以在编译的时候加上一些参数，指定Hadoop 2.2.0版本进行编译：

1	-Phadoop-2

三、Hadoop、Hbase对应的test不能下载

　　如果在编译的时候出现了类似如下的情况，请将相应pom.xml文件中的Hadoop-test和Hbase-test依赖包去掉：

01

<!--

02	User: 过往记忆

03	Date: 14-6-16

04	Time: 23:45

05	bolg: http://www.iteblog.com

06	本文地址：http://www.iteblog.com/archives/1043

07	过往记忆博客，专注于hadoop、hive、spark、shark、flume的技术博客，大量的干货

08	过往记忆博客微信公共帐号：iteblog_hadoop

09

-->

10	<dependency>

11	<groupId>org.apache.hbase</groupId>

12	<artifactId>hbase</artifactId>

13	<version>${hbase.version}</version>

14	<classifier>tests</classifier>

15	<scope>test</scope>

16	</dependency>

17

18	<dependency>

19	<groupId>org.apache.hadoop</groupId>

20	<artifactId>hadoop-test</artifactId>

21	<version>${hadoop.version}</version>

22	</dependency>

四、elasticsearch-0.90.1.jar read错误

　　如果你在编译的过程中出现一下异常：

1	[ERROR] Failed to execute goal org.apache.maven.plugins:

2	maven-compiler-plugin:2.3.2:compile (default-compile) on

3	project flume-ng-elasticsearch-sink: Compilation failure

4	[ERROR] error: error reading

5	/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar;

6	error in opening zip file

7	[ERROR] -> [Help 1]

　　只需要到你maven本地仓库将elasticsearch-0.90.1.jar文件删除，然后重新下载即可：

1	# rm elasticsearch-0.90.1.jar

2	wget http://central.maven.org/maven2/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar

五、ua-parser-1.3.0下载失败

编译过程中，下载ua-parser-1.3.0.pom可能失败，出现如下错误：

01	[ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink:

02	Could not resolve dependencies for project

03	org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0:

04	Failed to collect dependencies for [org.apache.flume:flume-ng-core:jar:1.5.0

05	(compile), org.slf4j:slf4j-api:jar:1.6.1 (compile),

06	org.kitesdk:kite-morphlines-all:pom:0.12.0 (compile?),

07	org.slf4j:jcl-over-slf4j:jar:1.6.1 (provided),

08	org.apache.solr:solr-test-framework:jar:4.3.0 (test),

09	org.kitesdk:kite-morphlines-solr-core:jar:tests:0.12.0 (test),

10	junit:junit:jar:4.10 (test)]: Failed to read artifact descriptor

11	for ua_parser:ua-parser:jar:1.3.0: Could not transfer artifact

12	ua_parser:ua-parser:pom:1.3.0 from/to maven-twttr (http://maven.twttr.com):

13	Connection to http://maven.twttr.com refused: Connection timed out -> [Help 1]

只需要加一个新的repository即可：

<repository>
  <id>p2.jfrog.org</id>
  <url>http://p2.jfrog.org/libs-releases</url>
</repository>

假如上面的repository仍然不能用，可以google “ua-parser-1.3.0.pom”获取其他源。

六、编译Flume-ng 1.5.0源码

1	# mvn install -Phadoop-2 -DskipTests -Dtar

编译需要一段时间，这个需要看你编译时的网速。如果一切顺利的话，你将会看到以下输出，说明你的Flume-ng 1.5.0编译成功：

01	[INFO] ------------------------------------------------------------------------

02	[INFO] Reactor Summary:

03

[INFO]

04	[INFO] Apache Flume ...................................... SUCCESS [1.360s]

05	[INFO] Flume NG SDK ...................................... SUCCESS [2.903s]

06	[INFO] Flume NG Configuration ............................ SUCCESS [0.483s]

07	[INFO] Flume NG Core ..................................... SUCCESS [2.531s]

08	[INFO] Flume NG Sinks .................................... SUCCESS [0.110s]

09	[INFO] Flume NG HDFS Sink ................................ SUCCESS [1.668s]

10	[INFO] Flume NG IRC Sink ................................. SUCCESS [0.484s]

11	[INFO] Flume NG Channels ................................. SUCCESS [0.074s]

12	[INFO] Flume NG JDBC channel ............................. SUCCESS [0.579s]

13	[INFO] Flume NG file-based channel ....................... SUCCESS [0.712s]

14	[INFO] Flume NG Spillable Memory channel ................. SUCCESS [0.552s]

15	[INFO] Flume NG Node ..................................... SUCCESS [0.619s]

16	[INFO] Flume NG Embedded Agent ........................... SUCCESS [1.001s]

17	[INFO] Flume NG HBase Sink ............................... SUCCESS [2.373s]

18	[INFO] Flume NG ElasticSearch Sink ....................... SUCCESS [0.899s]

19	[INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [2.483s]

20	[INFO] Flume NG Kite Dataset Sink ........................ SUCCESS [35.925s]

21	[INFO] Flume Sources ..................................... SUCCESS [0.054s]

22	[INFO] Flume Scribe Source ............................... SUCCESS [1.231s]

23	[INFO] Flume JMS Source .................................. SUCCESS [2.133s]

24	[INFO] Flume Twitter Source .............................. SUCCESS [1.204s]

25	[INFO] Flume legacy Sources .............................. SUCCESS [0.043s]

26	[INFO] Flume legacy Avro source .......................... SUCCESS [1.234s]

27	[INFO] Flume legacy Thrift Source ........................ SUCCESS [1.320s]

28	[INFO] Flume NG Clients .................................. SUCCESS [0.038s]

29	[INFO] Flume NG Log4j Appender ........................... SUCCESS [6.175s]

30	[INFO] Flume NG Tools .................................... SUCCESS [1.559s]

31	[INFO] Flume NG distribution ............................. SUCCESS [1:51.993s]

32	[INFO] Flume NG Integration Tests ........................ SUCCESS [1.517s]

33	[INFO] ------------------------------------------------------------------------

34	[INFO] BUILD SUCCESS

35	[INFO] ------------------------------------------------------------------------

36	[INFO] Total time: 3:03.921s

37	[INFO] Finished at: Mon Jun 16 23:37:51 CST 2014

38	[INFO] Final Memory: 135M/745M

39	[INFO] ------------------------------------------------------------------------

最后将会在flume-ng-dist/target目录下产生以下的文件：

查看源代码

打印 帮助

1	total 29208

2	drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 apache-flume-1.5.0-bin

3	-rw-r--r-- 1 wyp wyp 27307850 Jun 16 23:36 apache-flume-1.5.0-bin.tar.gz

4	drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 apache-flume-1.5.0-src

5	-rw-r--r-- 1 wyp wyp  2543174 Jun 16 23:37 apache-flume-1.5.0-src.tar.gz

6	drwxr-xr-x 2 wyp wyp     4096 Jun 16 23:36 archive-tmp

7	drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 maven-shared-archive-resources

8	-rw-r--r-- 1 wyp wyp      990 Jun 16 23:37 rat.txt

apache-flume-1.5.0-bin这个文件夹里面的东西可以直接拿来用。