Flume简介&源码编译

Flume简介

Flume OG:Flume original generation， 即Flume 0.9.x版本
Flume NG:Flume next generation ，即Flume 1.x版本

现在一般叫flume-ng，也是使用最为广泛的

0.9.x的架构

Agent用于采集数据，agent是flume中产生数据流的地方，同时，agent会将产生的数据流传输到collector。对应的，collector用于对数据进行聚合，往往会产生一个更大的流。

1.x架构

Flume NG，它摒弃了Master和zookeeper，collector也没有了，web配置台也没有了，只剩下source，sink和channel，此时一个Agent的概念包括source,channel和sink，完全由一个分布式系统变成了传输工具。不同机器之间的数据传输不再是OG那样由agent->collector，而是由一个Agent端的sink流向另一个agent的source

由于支持多种多样的输入和输出，因此被广泛应用

源码编译

下载


git clone https://git-wip-us.apache.org/repos/asf/flume.git
mvn clean compile


maven.twttr.com


<repository>
<id>nexus.axiomalaska.com</id>
<url>http://nexus.axiomalaska.com/nexus/content/repositories/public</url>
</repository>

或者
使用vpn，我这里用的就是它，简单快速

执行mvn clean compile


[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ flume-ng-tests ---
[INFO] Compiling 1 source file to D:\todoRead\flume\flume-ng-tests\target\classes
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Flume checkstyle project ........................... SUCCESS [ 1.473 s]
[INFO] Apache Flume ....................................... SUCCESS [ 0.111 s]
[INFO] Flume NG SDK ....................................... SUCCESS [ 4.854 s]
[INFO] Flume NG Configuration ............................. SUCCESS [ 2.234 s]
[INFO] Flume Auth ......................................... SUCCESS [ 5.890 s]
[INFO] Flume NG Core ...................................... SUCCESS [ 7.526 s]
[INFO] Flume NG Sinks ..................................... SUCCESS [ 0.027 s]
[INFO] Flume NG HDFS Sink ................................. SUCCESS [ 4.838 s]
[INFO] Flume NG IRC Sink .................................. SUCCESS [ 1.286 s]
[INFO] Flume NG Channels .................................. SUCCESS [ 0.057 s]
[INFO] Flume NG JDBC channel .............................. SUCCESS [ 2.067 s]
[INFO] Flume NG file-based channel ........................ SUCCESS [ 4.602 s]
[INFO] Flume NG Spillable Memory channel .................. SUCCESS [ 1.595 s]
[INFO] Flume NG Node ...................................... SUCCESS [ 2.352 s]
[INFO] Flume NG Embedded Agent ............................ SUCCESS [ 1.780 s]
[INFO] Flume NG HBase Sink ................................ SUCCESS [ 5.930 s]
[INFO] Flume NG ElasticSearch Sink ........................ SUCCESS [ 3.078 s]
[INFO] Flume NG Morphline Solr Sink ....................... SUCCESS [02:08 min]
[INFO] Flume Shared Utils ................................. SUCCESS [ 0.022 s]
[INFO] Flume Shared Kafka Test Utils ...................... SUCCESS [ 1.041 s]
[INFO] Flume Kafka Sink ................................... SUCCESS [ 1.311 s]
[INFO] Flume HTTP/S Sink .................................. SUCCESS [ 1.620 s]
[INFO] Flume NG Kite Dataset Sink ......................... SUCCESS [ 8.023 s]
[INFO] Flume NG Hive Sink ................................. SUCCESS [ 3.037 s]
[INFO] Flume Sources ...................................... SUCCESS [ 0.016 s]
[INFO] Flume Scribe Source ................................ SUCCESS [ 1.806 s]
[INFO] Flume JMS Source ................................... SUCCESS [ 1.603 s]
[INFO] Flume Twitter Source ............................... SUCCESS [ 1.363 s]
[INFO] Flume Kafka Source ................................. SUCCESS [ 2.197 s]
[INFO] Flume Taildir Source ............................... SUCCESS [ 3.198 s]
[INFO] flume-kafka-channel ................................ SUCCESS [ 4.038 s]
[INFO] Flume legacy Sources ............................... SUCCESS [ 0.016 s]
[INFO] Flume legacy Avro source ........................... SUCCESS [ 1.246 s]
[INFO] Flume legacy Thrift Source ......................... SUCCESS [ 1.582 s]
[INFO] Flume NG Clients ................................... SUCCESS [ 0.018 s]
[INFO] Flume NG Log4j Appender ............................ SUCCESS [ 1.342 s]
[INFO] Flume NG Tools ..................................... SUCCESS [ 1.148 s]
[INFO] Flume NG distribution .............................. SUCCESS [ 1.460 s]
[INFO] Flume NG Integration Tests ......................... SUCCESS [ 1.083 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:36 min
[INFO] Finished at: 2017-08-24T13:34:38+08:00
[INFO] Final Memory: 297M/1182M
[INFO] ------------------------------------------------------------------------