在今年的5月22号,Flume-ng 1.5.0版本正式发布,关于Flume-ng 1.5.0版本的新特性可以参见本博客的《Apache Flume-ng 1.5.0正式发布》进行了解。关于Apache flume-ng 1.4.0版本的编译可以参见本博客《基于Hadoop-2.2.0编译flume-ng 1.4.0及错误解决》。本文将讲述如何用Maven编译Apache flume-ng 1.5.0源码。
一、到官方网站下载相应版本的flume-ng源码
2 | # tar -zxf apache-flume- 1.5 . 0 -src.tar.gz |
3 | # cd apache-flume- 1.5 . 0 -src |
二、用maven在相应的Hadoop上编译Flume-ng 1.5.0源码
Flume-ng 1.5.0默认的Hadoop版本是1.0.1,Hbase是0.92.1。我们可以在编译的时候加上一些参数,指定Hadoop 2.2.0版本进行编译:
三、Hadoop、Hbase对应的test不能下载
如果在编译的时候出现了类似如下的情况,请将相应pom.xml文件中的Hadoop-test和Hbase-test依赖包去掉:
07 | 过往记忆博客,专注于hadoop、hive、spark、shark、flume的技术博客,大量的干货 |
08 | 过往记忆博客微信公共帐号:iteblog_hadoop |
11 | <groupId>org.apache.hbase</groupId> |
12 | <artifactId>hbase</artifactId> |
13 | <version>${hbase.version}</version> |
14 | <classifier>tests</classifier> |
19 | <groupId>org.apache.hadoop</groupId> |
20 | <artifactId>hadoop-test</artifactId> |
21 | <version>${hadoop.version}</version> |
四、elasticsearch-0.90.1.jar read错误
如果你在编译的过程中出现一下异常:
1 | [ERROR] Failed to execute goal org.apache.maven.plugins: |
2 | maven-compiler-plugin: 2.3 . 2 :compile ( default -compile) on |
3 | project flume-ng-elasticsearch-sink: Compilation failure |
4 | [ERROR] error: error reading |
5 | /org/elasticsearch/elasticsearch/ 0.90 . 1 /elasticsearch- 0.90 . 1 .jar; |
6 | error in opening zip file |
只需要到你maven本地仓库将elasticsearch-0.90.1.jar文件删除,然后重新下载即可:
1 | # rm elasticsearch- 0.90 . 1 .jar |
五、ua-parser-1.3.0下载失败
编译过程中,下载ua-parser-1.3.0.pom可能失败,出现如下错误:
01 | [ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink: |
02 | Could not resolve dependencies for project |
03 | org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar: 1.5 . 0 : |
04 | Failed to collect dependencies for [org.apache.flume:flume-ng-core:jar: 1.5 . 0 |
05 | (compile), org.slf4j:slf4j-api:jar: 1.6 . 1 (compile), |
06 | org.kitesdk:kite-morphlines-all:pom: 0.12 . 0 (compile?), |
07 | org.slf4j:jcl-over-slf4j:jar: 1.6 . 1 (provided), |
08 | org.apache.solr:solr-test-framework:jar: 4.3 . 0 (test), |
09 | org.kitesdk:kite-morphlines-solr-core:jar:tests: 0.12 . 0 (test), |
10 | junit:junit:jar: 4.10 (test)]: Failed to read artifact descriptor |
11 | for ua_parser:ua-parser:jar: 1.3 . 0 : Could not transfer artifact |
12 | ua_parser:ua-parser:pom: 1.3 . 0 from/to maven-twttr (http: |
只需要加一个新的repository即可:
<repository>
<id>p2.jfrog.org</id>
<url>http://p2.jfrog.org/libs-releases</url>
</repository>
假如上面的repository仍然不能用,可以google “ua-parser-1.3.0.pom”获取其他源。
六、编译Flume-ng 1.5.0源码
1 | # mvn install -Phadoop- 2 -DskipTests -Dtar |
编译需要一段时间,这个需要看你编译时的网速。如果一切顺利的话,你将会看到以下输出,说明你的Flume-ng 1.5.0编译成功:
01 | [INFO] ------------------------------------------------------------------------ |
02 | [INFO] Reactor Summary: |
04 | [INFO] Apache Flume ...................................... SUCCESS [ 1 .360s] |
05 | [INFO] Flume NG SDK ...................................... SUCCESS [ 2 .903s] |
06 | [INFO] Flume NG Configuration ............................ SUCCESS [ 0 .483s] |
07 | [INFO] Flume NG Core ..................................... SUCCESS [ 2 .531s] |
08 | [INFO] Flume NG Sinks .................................... SUCCESS [ 0 .110s] |
09 | [INFO] Flume NG HDFS Sink ................................ SUCCESS [ 1 .668s] |
10 | [INFO] Flume NG IRC Sink ................................. SUCCESS [ 0 .484s] |
11 | [INFO] Flume NG Channels ................................. SUCCESS [ 0 .074s] |
12 | [INFO] Flume NG JDBC channel ............................. SUCCESS [ 0 .579s] |
13 | [INFO] Flume NG file-based channel ....................... SUCCESS [ 0 .712s] |
14 | [INFO] Flume NG Spillable Memory channel ................. SUCCESS [ 0 .552s] |
15 | [INFO] Flume NG Node ..................................... SUCCESS [ 0 .619s] |
16 | [INFO] Flume NG Embedded Agent ........................... SUCCESS [ 1 .001s] |
17 | [INFO] Flume NG HBase Sink ............................... SUCCESS [ 2 .373s] |
18 | [INFO] Flume NG ElasticSearch Sink ....................... SUCCESS [ 0 .899s] |
19 | [INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [ 2 .483s] |
20 | [INFO] Flume NG Kite Dataset Sink ........................ SUCCESS [ 35 .925s] |
21 | [INFO] Flume Sources ..................................... SUCCESS [ 0 .054s] |
22 | [INFO] Flume Scribe Source ............................... SUCCESS [ 1 .231s] |
23 | [INFO] Flume JMS Source .................................. SUCCESS [ 2 .133s] |
24 | [INFO] Flume Twitter Source .............................. SUCCESS [ 1 .204s] |
25 | [INFO] Flume legacy Sources .............................. SUCCESS [ 0 .043s] |
26 | [INFO] Flume legacy Avro source .......................... SUCCESS [ 1 .234s] |
27 | [INFO] Flume legacy Thrift Source ........................ SUCCESS [ 1 .320s] |
28 | [INFO] Flume NG Clients .................................. SUCCESS [ 0 .038s] |
29 | [INFO] Flume NG Log4j Appender ........................... SUCCESS [ 6 .175s] |
30 | [INFO] Flume NG Tools .................................... SUCCESS [ 1 .559s] |
31 | [INFO] Flume NG distribution ............................. SUCCESS [ 1 : 51 .993s] |
32 | [INFO] Flume NG Integration Tests ........................ SUCCESS [ 1 .517s] |
33 | [INFO] ------------------------------------------------------------------------ |
35 | [INFO] ------------------------------------------------------------------------ |
36 | [INFO] Total time: 3 : 03 .921s |
37 | [INFO] Finished at: Mon Jun 16 23 : 37 : 51 CST 2014 |
38 | [INFO] Final Memory: 135M/745M |
39 | [INFO] ------------------------------------------------------------------------ |
最后将会在flume-ng-dist/target目录下产生以下的文件:
2 | drwxr-xr-x 3 wyp wyp 4096 Jun 16 23 : 36 apache-flume- 1.5 . 0 -bin |
3 | -rw-r--r-- 1 wyp wyp 27307850 Jun 16 23 : 36 apache-flume- 1.5 . 0 -bin.tar.gz |
4 | drwxr-xr-x 3 wyp wyp 4096 Jun 16 23 : 36 apache-flume- 1.5 . 0 -src |
5 | -rw-r--r-- 1 wyp wyp 2543174 Jun 16 23 : 37 apache-flume- 1.5 . 0 -src.tar.gz |
6 | drwxr-xr-x 2 wyp wyp 4096 Jun 16 23 : 36 archive-tmp |
7 | drwxr-xr-x 3 wyp wyp 4096 Jun 16 23 : 36 maven-shared-archive-resources |
8 | -rw-r--r-- 1 wyp wyp 990 Jun 16 23 : 37 rat.txt |
apache-flume-1.5.0-bin这个文件夹里面的东西可以直接拿来用。