新手入门
-
- Flume NG是什么？
- 有什么改变？
- 获得Flume NG
- Flume NG是什么？
-
-
- 从源码构建
- 配置
-
- flume-ng全局选项
- flume-ng agent选项
- flume-ng avro-client 选项
- 提供反馈
-
Flume NG是什么？
Flume NG的目标是比Flume OG在简单性，大小和容易部署上有显著性地提高。为了实现这个目标，Flume NG将不会兼容Flume OG.我们目前在征求那些对测试Flume NG正确性、是否易用以及与其他系统的融合度有兴趣的用户的反馈信息。
有什么改变？
Flume NG（下一代）是在与Flume OG基础理念相同的情况下的一种大相径庭的实现。如果你已经对Flume很熟悉了，那么下面就是你需要知道内容。
- 你仍然需要有sources和sinks组件来做相同的事，只是现在用channels来连接。
- Channels是可插拔的和持久的。Flume NG中in-memory channel处理速度快但不支持持久化而file-based channel支持event持久化。
- There's no more logical or physical nodes. We call all physical nodes agents and agents can run zero or more sources and sinks.现在没有逻辑节点或者物理节点。我们称物理节点为agents而agents中可以运行0至多个sources和sinks。
- 没有master和不再依赖ZooKeeper。现在，Flume是依赖于一个简单的基于文件配置配置系统。
- 所有的都是插件，一些面向用户，一些针对工具和系统开发者。插件化组件包括channels、sources、sinks、interceptors、sink processors和event serializeres。
请自行阅读JIRAs文件寻找你认为重要的特性。
获得Flume NG
你可以在Flume官网中 Downloads 下载源码。如果你不打算为Flume打补丁的话，那么使用二进制文件将是简单的方式。
用源码构建
为了用源码搭建Flume NG，你必须要有git、Sun JDK1.6，Apache Maven3.x，大概90MB的磁盘空间和网络连接。
1. 检查资源
$ git clone https://git-wip-us.apache.org/repos/asf/flume.git flume
$ cd flume
$ git checkout trunk2.编译项目
Apache Flume搭建时需要比默认配置更多的内存，我们建议你做以下的Maven 选项：
export MAVEN_OPTS="-Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"
# Build the code and run the tests (note: use mvn install, not mvn package, since we deploy Jenkins SNAPSHOT jars daily, and Flume is a multi-module project)
$ mvn install
# ...or build the code without running the tests
$ mvn install -DskipTests（需要说明的是Flume需要在构建路径下放置Google Protocol Buffers compiler来保证成功率。你可以根据here里面的介绍来下载和安装。）
这将在flume-ng-dist/target目录下生成两种类型的包，它们是
- apache-flume-ng-dist-1.4.0-SNAPSHOT-bin.tar.gz - A binary distribution of Flume, ready to run.
- apache-flume-ng-dist-1.4.0-SNAPSHOT-src.tar.gz - A source-only distribution of Flume.
如果你只是一个想要运行Flume的用户，你大概只需要-bin 版本。将它复制并解压，你就可以使用Flume了。
$ cp flume-ng-dist/target/apache-flume-1.4.0-SNAPSHOT-bin.tar.gz .
$ tar -zxvf apache-flume-1.4.0-SNAPSHOT-bin.tar.gz
$ cd apache-flume-1.4.0-SNAPSHOT-bin3.按照工作模板创建一个你自己的属性文档（或者从头开始创建一个）
$ cp conf/flume-conf.properties.template conf/flume.conf4.（可选）按照模板创建一个你自己的flume-env.sh文档（或者从头开始创建一个）。如果命令行中通过 –conf/-c指定了conf目录的话，那么fluem-ng会在该路径下搜寻“flume-env.sh”文档。一个使用flume-env.sh的情况是当你使用你自定义的Flume NG组件进行开发时通过JAVA_OPTS来指定调试或者评测选项。
$ cp conf/flume-env.sh.template conf/flume-env.sh5.配置和运行Flume NG
在你完成 Flume NG的配置之后，你可以通过 bin/flume-ng 可执行文件来运行它。这个脚本拥有一系列参数和模式。
配置
Flume使用基于Java属性文件格式的配置文档。运行agent时需要通过 –f<file>选项来告诉Flume。这个文件可以放在任意位置，但从历史和未来的角度来看，这个conf目录将会是配置文件的正确位置。
让我们从一个基础的例子开始吧。将下面复制并粘贴到conf/flume.conf中：
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414
# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.log-sink1.channel = ch1
agent1.sinks.log-sink1.type = logger
# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = log-sink1这个例子创建了内存channel（一个不可靠和高效传输）将一个Avro RPC source,和一个logger sink连接在一起。Avro source接收到的任何event都会按照规划的路线传到ch1 channel中然后传递给logger sink。需要重点说明的是定义组件只是配置Flume的前半部分工作；它们必须配置在<agent>中被激活。多个source、channel和sink是可以被配置的，用空格隔开每个组件就可以。
至于所有的鞋机，请自行查看Javadoc中的org.apache.flume.conf.properties.PropertiesFileConfigurationProvider类。
这是当前已经实现的channels、sinks和sources的列表。每个插件都有其自己的选项和需要配置的属性，请自行阅读javadoc。
| Component | Type | Description | Implementation Class |
| Channel | memory | In-memory, fast, non-durable event transport 一个将event存储在内容中，快速传输但无法持久化的channel。 | MemoryChannel |
| Channel | file | A channel for reading, writing, mapping, and manipulating a file 一个对文件进行读、写、映射和操作的channel | FileChannel |
| Channel | jdbc | JDBC-based, durable event transport (Derby-based) 基于JDBC，支持持久化的channel | JDBCChannel |
| Channel | recoverablememory | A durable channel implementation that uses the local file system for its storage 一个使用本地文件系统实现持久化的channel | RecoverableMemoryChannel |
| Channel | org.apache.flume.channel.PseudoTxnMemoryChannel | Mainly for testing purposes. Not meant for production use. 用于测试，不用于生产 | PseudoTxnMemoryChannel |
| Channel | (custom type as FQCN) | Your own Channel impl. 自定义channel | (custom FQCN) |
| Source | avro | Avro Netty RPC event source | AvroSource |
| Source | exec | Execute a long-lived Unix process and read from stdout 执行一个长连接Unix进程并从标准输出设备读取数据 | ExecSource |
| Source | netcat | Netcat style TCP event source | NetcatSource |
| Source | seq | Monotonically incrementing sequence generator event source 单调递增序列发生器的事件source | SequenceGeneratorSource |
| Source | org.apache.flume.source.StressSource | Mainly for testing purposes. Not meant for production use. Serves as a continuous source of events where each event has the same payload. The payload consists of some number of bytes (specified by size property, defaults to 500) where each byte has the signed value Byte.MAX_VALUE (0x7F, or 127). 主要用于测试，不适合用于生产。用于接收每个拥有相同的有效负载的event。那有效负载包含一组字节（通过 size属性指定，默认为500）每个字节都是最大值（Byte.MAX_VALUE(0X7F或者127)） | org.apache.flume.source.StressSource |
| Source | syslogtcp |
| SyslogTcpSource |
| Source | syslogudp |
| SyslogUDPSource |
| Source | org.apache.flume.source.avroLegacy.AvroLegacySource |
| AvroLegacySource |
| Source | org.apache.flume.source.thriftLegacy.ThriftLegacySource |
| ThriftLegacySource |
| Source | org.apache.flume.source.scribe.ScribeSource |
| ScribeSource |
| Source | (custom type as FQCN) | Your own Source impl. 自定义Source | (custom FQCN) |
| Sink | hdfs | Writes all events received to HDFS (with support for rolling, bucketing, HDFS-200 append, and more) 将所有接收到events写到HDFS（支持回滚，桶装和追加以及其他） | HDFSEventSink |
| Sink | org.apache.flume.sink.hbase.HBaseSink | A simple sink that reads events from a channel and writes them to HBase. 一个简单的sink用于将从channel读到的数据写到HBase | org.apache.flume.sink.hbase.HBaseSink |
| Sink | org.apache.flume.sink.hbase.AsyncHBaseSink |
| org.apache.flume.sink.hbase.AsyncHBaseSink |
| Sink | logger | Log events at INFO level via configured logging subsystem (log4j by default) 通过配置日志子系统将INFO级别的events打印出来。 | LoggerSink |
| Sink | avro | Sink that invokes a pre-defined Avro protocol method for all events it receives (when paired with an avro source, forms tiered collection) 一个调用预先定义好的Avro protocol方法来处理接收的所有event的sink（与avro source配对，形成分层收集） | AvroSink |
| Sink | file_roll |
| RollingFileSink |
| Sink | irc |
| IRCSink |
| Sink | null | /dev/null for Flume - blackhole all events received event黑洞，有来无回 | NullSink |
| Sink | (custom type as FQCN) | Your own Sink impl. 自定义sink | (custom FQCN) |
| ChannelSelector | replicating |
| ReplicatingChannelSelector |
| ChannelSelector | multiplexing |
| MultiplexingChannelSelector |
| ChannelSelector | (custom type) | Your own ChannelSelector impl. | (custom FQCN) |
| SinkProcessor | default |
| DefaultSinkProcessor |
| SinkProcessor | failover |
| FailoverSinkProcessor |
| SinkProcessor | load_balance | Provides the ability to load-balance flow over multiple sinks. 当存在多个sink时实现负载均衡 | LoadBalancingSinkProcessor |
| SinkProcessor | (custom type as FQCN) | Your own SinkProcessor impl. | (custom FQCN) |
| Interceptor$Builder | host |
| HostInterceptor$Builder |
| Interceptor$Builder | timestamp | TimestampInterceptor | TimestampInterceptor$Builder |
| Interceptor$Builder | static |
| StaticInterceptor$Builder |
| Interceptor$Builder | regex_filter |
| RegexFilteringInterceptor$Builder |
| Interceptor$Builder | (custom type as FQCN) | Your own Interceptor$Builder impl. | (custom FQCN) |
| EventSerializer$Builder | text |
| BodyTextEventSerializer$Builder |
| EventSerializer$Builder | avro_event |
| FlumeEventAvroEventSerializer$Builder |
| EventSerializer | org.apache.flume.sink.hbase.SimpleHbaseEventSerializer |
| SimpleHbaseEventSerializer |
| EventSerializer | org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer |
| SimpleAsyncHbaseEventSerializer |
| EventSerializer | org.apache.flume.sink.hbase.RegexHbaseEventSerializer |
| RegexHbaseEventSerializer |
| HbaseEventSerializer | Custom implementation of serializer for HBaseSink. | Your own HbaseEventSerializer impl. | (custom FQCN) |
| AsyncHbaseEventSerializer | Custom implementation of serializer for AsyncHbase sink. | Your own AsyncHbaseEventSerializer impl. | (custom FQCN) |
| EventSerializer$Builder | Custom implementation of serializer for all sinks except for HBaseSink and AsyncHBaseSink. | Your own EventSerializer$Builder impl. | (custom FQCN) |
flume-ng可执行可以让你运行一个Flume NG agent或者一个 Avro 客户端用于测试和实验。无论怎样，你必须指定一个命令（例如 agent或者avro-client）和一个conf目录（--conf<conf dir>）。所有其他的选项都可以用命令行指定。
使用上面的fiume.conf来启动flume服务器
bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1需要注意的是agent 的名字是通过 –n agent1来指定的并且必须和-conf/flume.conf中给定的名字相匹配。
你的输出应该是这样的：
$ bin/flume-ng agent --conf conf/ -f conf/flume.conf -n agent1
2012-03-16 16:36:11,918 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:58)] Starting lifecycle supervisor 1
2012-03-16 16:36:11,921 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent1
2012-03-16 16:36:11,926 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:110)] Node manager starting
2012-03-16 16:36:11,928 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:58)] Starting lifecycle supervisor 10
2012-03-16 16:36:11,929 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:114)] Node manager started
2012-03-16 16:36:11,926 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-03-16 16:36:11,930 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:87)] Configuration provider started
2012-03-16 16:36:11,930 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:189)] Checking file:conf/flume.conf for changes
2012-03-16 16:36:11,931 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:196)] Reloading configuration file:conf/flume.conf
2012-03-16 16:36:11,936 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.properties.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:225)] Starting validation of configuration for agent: agent1, initial-configuration: AgentConfiguration[agent1]
SOURCES: {avro-source1=ComponentConfiguration[avro-source1]
CONFIG: {port=41414, channels=ch1, type=avro, bind=0.0.0.0}
RUNNER: ComponentConfiguration[runner]
CONFIG: {}
}
CHANNELS: {ch1=ComponentConfiguration[ch1]
CONFIG: {type=memory}
}
SINKS: {log-sink1=ComponentConfiguration[log-sink1]
CONFIG: {type=logger, channel=ch1}
RUNNER: ComponentConfiguration[runner]
CONFIG: {}
}
2012-03-16 16:36:11,936 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] Post-validation flume configuration contains configuation for agents: [agent1]
2012-03-16 16:36:11,937 (conf-file-poller-0) [DEBUG - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:67)] Creating instance of channel ch1 type memory
2012-03-16 16:36:11,944 (conf-file-poller-0) [DEBUG - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:73)] Creating instance of source avro-source1, type avro
2012-03-16 16:36:11,957 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:69)] Creating instance of sink log-sink1 typelogger
2012-03-16 16:36:11,963 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.onNodeConfigurationChanged(DefaultLogicalNodeManager.java:52)] Node configuration change:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:AvroSource: { bindAddress:0.0.0.0 port:41414 } }} sinkRunners:{log-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@79f6f296 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel@43b09468} }
2012-03-16 16:36:11,974 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:122)] Avro source starting:AvroSource: { bindAddress:0.0.0.0 port:41414 }
2012-03-16 16:36:11,975 (Thread-1) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:123)] Polling sink runner starting
2012-03-16 16:36:12,352 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.source.AvroSource.start(AvroSource.java:132)] Avro source started
flume-ng 全局选项
| Option | Description |
| --conf,-c <conf> | Use configs in <conf> directory |
| --classpath,-C <cp> | Append to the classpath |
| --dryrun,-d | Do not actually start Flume, just print the command |
| -Dproperty=value | Sets a JDK system property value |
flume-ng agent选项
当给一个agent命令时，Flume NG agent将会根据一个给定的配置文件进行启动。
| Option | Description |
| --conf-file,-f <file> | Indicates which configuration file you want to run with (required) |
| --name,-n <agentname> | Indicates the name of agent on which we're running (required) |
flume-ng avro-client 选项
运行一个Avro client从标准输入发送数据或文件到一个Flume NG Avro Source所监听的主机和端口上。
| Option | Description |
| --host,-H <hostname> | Specifies the hostname of the Flume agent (may be localhost) |
| --port,-p <port> | Specifies the port on which the Avro source is listening |
| --filename,-F <filename> | Sends each line of <filename> to Flume (optional) |
| --headerFile,-F <file> | Header file containing headers as key/value pairs on each new line |
Avroclient将每一行以\n, \r, or \r\n结尾的数据当做一个event。把avro-clinet 命令当做Flume中cat命令。例如，下面的命令创建了一个event并将它发送到Flume’avro source所监听的端口41414。
在一个新的窗口输入以下内容：
$ bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F /etc/passwd -Dflume.root.logger=DEBUG,console你能看到以下信息：
2012-03-16 16:39:17,124 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:175)] Finished
2012-03-16 16:39:17,127 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:178)] Closing reader
2012-03-16 16:39:17,127 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:183)] Closing transceiver
2012-03-16 16:39:17,129 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:73)] Exiting服务器所运行的第一个窗口会打印以下内容：
2012-03-16 16:39:16,738 (New I/O server boss #1 ([id: 0x49e808ca, /0:0:0:0:0:0:0:0:41414])) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /1
27.0.0.1:39577 => /127.0.0.1:41414] OPEN
2012-03-16 16:39:16,742 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 => /127.0.0.1:41414] BOU
ND: /127.0.0.1:41414
2012-03-16 16:39:16,742 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 => /127.0.0.1:41414] CON
NECTED: /127.0.0.1:39577
2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] DISCONNECTED
2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] UNBOUND
2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] CLOSED
2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@5c1ae90c }
2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@6aba4211 }
2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@6a47a0d4 }
2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@48ff4cf }
...恭喜！你的Apache Flume已经成功运行了。
提供反馈
对于帮助构建、配置和运行Flume来说，最好的地方就是用户的邮件列表。发送邮件到user-subscribe@flume.apache.org 进行订阅和一旦你订阅之后user@flume.apache.org 会发送订阅信息给你。档案信息可以在 http://mail-archives.apache.org/mod_mbox/incubator-flume-user/ (2012/7建立) and http://mail-archives.apache.org/mod_mbox/incubator-flume-user/http://mail-archives.apache.org/mod_mbox/flume-user/ 得到。
如果你确认你发现一个bug或者需要一个特性或者提升，请不要害羞。去这个 https://issues.apache.org/jira/browse/FLUME 网站为该版本的Flume提一个JIRA.对于NG版本，请为合适的里程碑/发布留下“影响版本”的标识。可以只留下你对于未达之处的无法确定的任何想法。当我们需要的时候会像你征求细节。需要说明的是你必须创建一个Apache JIRA账户以致你可以提出问题。
下面为原文
Getting Started
What is Flume NG?
Flume NG aims to be significantly simpler, smaller, and easier to deploy than Flume OG. In doing so, we do not commit to maintaining backward compatibility of Flume NG with Flume OG. We're currently soliciting feedback from those who are interested in testing Flume NG for correctness, ease of use, and potential integration with other systems.
What's Changed?
Flume NG (Next Generation) is a huge departure from Flume OG (Original Generation) in its implementation although many of the original concepts are the same. If you're already familiar with Flume, here's what you need to know.
- You still have sources and sinks and they still do the same thing. They are now connected by channels.
- Channels are pluggable and dictate durability. Flume NG ships with an in-memory channel for fast, but non-durable event delivery and a file-based channel for durable event delivery.
- There's no more logical or physical nodes. We call all physical nodes agents and agents can run zero or more sources and sinks.
- There's no master and no ZooKeeper dependency anymore. At this time, Flume runs with a simple file-based configuration system.
- Just about everything is a plugin, some end user facing, some for tool and system developers. Pluggable components include channels, sources, sinks, interceptors, sink processors, and event serializers.
Please file JIRAs and/or vote for features you feel are important.
Getting Flume NG
Flume is available as a source tarball and binary on the Downloads section of the Flume Website. If you are not planning on creating patches for Flume, the binary is likely the easiest way to get started.
Building From Source
To build Flume NG from source, you'll need git, the Sun JDK 1.6, Apache Maven 3.x, about 90MB of local disk space and an Internet connection.
1. Check out the source
$ git clone https://git-wip-us.apache.org/repos/asf/flume.git flume
$ cd flume
$ git checkout trunk
2. Compile the project
The Apache Flume build requires more memory than the default configuration. We recommend you set the following Maven options:
export MAVEN_OPTS="-Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"
# Build the code and run the tests (note: use mvn install, not mvn package, since we deploy Jenkins SNAPSHOT jars daily, and Flume is a multi-module project)
$ mvn install
# ...or build the code without running the tests
$ mvn install -DskipTests
(Please note that Flume requires that Google Protocol Buffers compiler be in the path for the build to be successful. You download and install it by following the instructions here.)
This produces two types of packages in flume-ng-dist/target. They are:
- apache-flume-ng-dist-1.4.0-SNAPSHOT-bin.tar.gz - A binary distribution of Flume, ready to run.
- apache-flume-ng-dist-1.4.0-SNAPSHOT-src.tar.gz - A source-only distribution of Flume.
If you're a user and you just want to run Flume, you probably want the -bin version. Copy one out, decompress it, and you're ready to go.
$ cp flume-ng-dist/target/apache-flume-1.4.0-SNAPSHOT-bin.tar.gz .
$ tar -zxvf apache-flume-1.4.0-SNAPSHOT-bin.tar.gz
$ cd apache-flume-1.4.0-SNAPSHOT-bin
3. Create your own properties file based on the working template (or create one from scratch)
$ cp conf/flume-conf.properties.template conf/flume.conf
4. (Optional) Create your flume-env.sh file based on the template (or create one from scratch). The flume-ng executable looks for and sources a file named "flume-env.sh" in the conf directory specified by the --conf/-c commandline option. One use case for using flume-env.sh would be to specify debugging or profiling options via JAVA_OPTS when developing your own custom Flume NG components such as sources and sinks.
$ cp conf/flume-env.sh.template conf/flume-env.sh
5. Configure and Run Flume NG
After you've configured Flume NG (see below), you can run it with the bin/flume-ng executable. This script has a number of arguments and modes.
Configuration
Flume uses a Java property file based configuration format. It is required that you tell Flume which file to use by way of the -f <file> option (see above) when running an agent. The file can live anywhere, but historically - and in the future - the conf directory will be the correct place for config files.
Let's start with a basic example. Copy and paste this into conf/flume.conf:
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414
# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.log-sink1.channel = ch1
agent1.sinks.log-sink1.type = logger
# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = log-sink1
This example creates a memory channel (i.e. an unreliable or "best effort" transport), an Avro RPC source, and a logger sink and connects them together. Any events received by the Avro source are routed to the channel ch1 and delivered to the logger sink. It's important to note that defining components is the first half of configuring Flume; they must be activated by listing them in the <agent>.channels, <agent>.sources, and sections. Multiple sources, sinks, and channels may be listed, separated by a space.
For full details, please see the javadoc for the org.apache.flume.conf.properties.PropertiesFileConfigurationProvider class.
This is a listing of the implemented sources, sinks, and channels at this time. Each plugin has its own optional and required configuration properties so please see the javadocs (for now).
| Component | Type | Description | Implementation Class |
| Channel | memory | In-memory, fast, non-durable event transport 一个将event存储在内容中，快速传输但无法持久化的channel。 | MemoryChannel |
| Channel | file | A channel for reading, writing, mapping, and manipulating a file 一个对文件进行读、写、映射和操作的channel | FileChannel |
| Channel | jdbc | JDBC-based, durable event transport (Derby-based) 基于JDBC，支持持久化的channel | JDBCChannel |
| Channel | recoverablememory | A durable channel implementation that uses the local file system for its storage 一个使用本地文件系统实现持久化的channel | RecoverableMemoryChannel |
| Channel | org.apache.flume.channel.PseudoTxnMemoryChannel | Mainly for testing purposes. Not meant for production use. 用于测试，不用于生产 | PseudoTxnMemoryChannel |
| Channel | (custom type as FQCN) | Your own Channel impl. 自定义channel | (custom FQCN) |
| Source | avro | Avro Netty RPC event source | AvroSource |
| Source | exec | Execute a long-lived Unix process and read from stdout 执行一个长连接Unix进程并从标准输出设备读取数据 | ExecSource |
| Source | netcat | Netcat style TCP event source | NetcatSource |
| Source | seq | Monotonically incrementing sequence generator event source 单调递增序列发生器的事件source | SequenceGeneratorSource |
| Source | org.apache.flume.source.StressSource | Mainly for testing purposes. Not meant for production use. Serves as a continuous source of events where each event has the same payload. The payload consists of some number of bytes (specified by size property, defaults to 500) where each byte has the signed value Byte.MAX_VALUE (0x7F, or 127). 主要用于测试，不适合用于生产。用于接收每个拥有相同的有效负载的event。那有效负载包含一组字节（通过 size属性指定，默认为500）每个字节都是最大值（Byte.MAX_VALUE(0X7F或者127)） | org.apache.flume.source.StressSource |
| Source | syslogtcp |
| SyslogTcpSource |
| Source | syslogudp |
| SyslogUDPSource |
| Source | org.apache.flume.source.avroLegacy.AvroLegacySource |
| AvroLegacySource |
| Source | org.apache.flume.source.thriftLegacy.ThriftLegacySource |
| ThriftLegacySource |
| Source | org.apache.flume.source.scribe.ScribeSource |
| ScribeSource |
| Source | (custom type as FQCN) | Your own Source impl. 自定义Source | (custom FQCN) |
| Sink | hdfs | Writes all events received to HDFS (with support for rolling, bucketing, HDFS-200 append, and more) 将所有接收到events写到HDFS（支持回滚，桶装和追加以及其他） | HDFSEventSink |
| Sink | org.apache.flume.sink.hbase.HBaseSink | A simple sink that reads events from a channel and writes them to HBase. 一个简单的sink用于将从channel读到的数据写到HBase | org.apache.flume.sink.hbase.HBaseSink |
| Sink | org.apache.flume.sink.hbase.AsyncHBaseSink |
| org.apache.flume.sink.hbase.AsyncHBaseSink |
| Sink | logger | Log events at INFO level via configured logging subsystem (log4j by default) 通过配置日志子系统将INFO级别的events打印出来。 | LoggerSink |
| Sink | avro | Sink that invokes a pre-defined Avro protocol method for all events it receives (when paired with an avro source, forms tiered collection) 一个调用预先定义好的Avro protocol方法来处理接收的所有event的sink（与avro source配对，形成分层收集） | AvroSink |
| Sink | file_roll |
| RollingFileSink |
| Sink | irc |
| IRCSink |
| Sink | null | /dev/null for Flume - blackhole all events received event黑洞，有来无回 | NullSink |
| Sink | (custom type as FQCN) | Your own Sink impl. 自定义sink | (custom FQCN) |
| ChannelSelector | replicating |
| ReplicatingChannelSelector |
| ChannelSelector | multiplexing |
| MultiplexingChannelSelector |
| ChannelSelector | (custom type) | Your own ChannelSelector impl. | (custom FQCN) |
| SinkProcessor | default |
| DefaultSinkProcessor |
| SinkProcessor | failover |
| FailoverSinkProcessor |
| SinkProcessor | load_balance | Provides the ability to load-balance flow over multiple sinks. 当存在多个sink时实现负载均衡 | LoadBalancingSinkProcessor |
| SinkProcessor | (custom type as FQCN) | Your own SinkProcessor impl. | (custom FQCN) |
| Interceptor$Builder | host |
| HostInterceptor$Builder |
| Interceptor$Builder | timestamp | TimestampInterceptor | TimestampInterceptor$Builder |
| Interceptor$Builder | static |
| StaticInterceptor$Builder |
| Interceptor$Builder | regex_filter |
| RegexFilteringInterceptor$Builder |
| Interceptor$Builder | (custom type as FQCN) | Your own Interceptor$Builder impl. | (custom FQCN) |
| EventSerializer$Builder | text |
| BodyTextEventSerializer$Builder |
| EventSerializer$Builder | avro_event |
| FlumeEventAvroEventSerializer$Builder |
| EventSerializer | org.apache.flume.sink.hbase.SimpleHbaseEventSerializer |
| SimpleHbaseEventSerializer |
| EventSerializer | org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer |
| SimpleAsyncHbaseEventSerializer |
| EventSerializer | org.apache.flume.sink.hbase.RegexHbaseEventSerializer |
| RegexHbaseEventSerializer |
| HbaseEventSerializer | Custom implementation of serializer for HBaseSink. | Your own HbaseEventSerializer impl. | (custom FQCN) |
| AsyncHbaseEventSerializer | Custom implementation of serializer for AsyncHbase sink. | Your own AsyncHbaseEventSerializer impl. | (custom FQCN) |
| EventSerializer$Builder | Custom implementation of serializer for all sinks except for HBaseSink and AsyncHBaseSink. | Your own EventSerializer$Builder impl. | (custom FQCN) |
The flume-ng executable lets you run a Flume NG agent or an Avro client which is useful for testing and experiments. No matter what, you'll need to specify a command (e.g. agent or avro-client) and a conf directory (--conf <conf dir>). All other options are command-specific.
To start the flume server using the flume.conf above:
bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1
Notice that the agent name is specified by -n agent1 and must match a agent name given in -f conf/flume.conf
Your output should look something like this:
$ bin/flume-ng agent --conf conf/ -f conf/flume.conf -n agent1
2012-03-16 16:36:11,918 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:58)] Starting lifecycle supervisor 1
2012-03-16 16:36:11,921 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent1
2012-03-16 16:36:11,926 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:110)] Node manager starting
2012-03-16 16:36:11,928 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:58)] Starting lifecycle supervisor 10
2012-03-16 16:36:11,929 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:114)] Node manager started
2012-03-16 16:36:11,926 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-03-16 16:36:11,930 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:87)] Configuration provider started
2012-03-16 16:36:11,930 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:189)] Checking file:conf/flume.conf for changes
2012-03-16 16:36:11,931 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:196)] Reloading configuration file:conf/flume.conf
2012-03-16 16:36:11,936 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.properties.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:225)] Starting validation of configuration for agent: agent1, initial-configuration: AgentConfiguration[agent1]
SOURCES: {avro-source1=ComponentConfiguration[avro-source1]
CONFIG: {port=41414, channels=ch1, type=avro, bind=0.0.0.0}
RUNNER: ComponentConfiguration[runner]
CONFIG: {}
}
CHANNELS: {ch1=ComponentConfiguration[ch1]
CONFIG: {type=memory}
}
SINKS: {log-sink1=ComponentConfiguration[log-sink1]
CONFIG: {type=logger, channel=ch1}
RUNNER: ComponentConfiguration[runner]
CONFIG: {}
}
2012-03-16 16:36:11,936 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] Post-validation flume configuration contains configuation for agents: [agent1]
2012-03-16 16:36:11,937 (conf-file-poller-0) [DEBUG - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:67)] Creating instance of channel ch1 type memory
2012-03-16 16:36:11,944 (conf-file-poller-0) [DEBUG - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:73)] Creating instance of source avro-source1, type avro
2012-03-16 16:36:11,957 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:69)] Creating instance of sink log-sink1 typelogger
2012-03-16 16:36:11,963 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.onNodeConfigurationChanged(DefaultLogicalNodeManager.java:52)] Node configuration change:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:AvroSource: { bindAddress:0.0.0.0 port:41414 } }} sinkRunners:{log-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@79f6f296 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel@43b09468} }
2012-03-16 16:36:11,974 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:122)] Avro source starting:AvroSource: { bindAddress:0.0.0.0 port:41414 }
2012-03-16 16:36:11,975 (Thread-1) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:123)] Polling sink runner starting
2012-03-16 16:36:12,352 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.source.AvroSource.start(AvroSource.java:132)] Avro source started
flume-ng global options
| Option | Description |
| --conf,-c <conf> | Use configs in <conf> directory |
| --classpath,-C <cp> | Append to the classpath |
| --dryrun,-d | Do not actually start Flume, just print the command |
| -Dproperty=value | Sets a JDK system property value |
flume-ng agent options
When given the agent command, a Flume NG agent will be started with a given configuration file (required).
| Option | Description |
| --conf-file,-f <file> | Indicates which configuration file you want to run with (required) |
| --name,-n <agentname> | Indicates the name of agent on which we're running (required) |
flume-ng avro-client options（flume-ng avro-client 选项）
Run an Avro client that sends either a file or data from stdin to a specified host and port where a Flume NG Avro Source is listening.
| Option | Description |
| --host,-H <hostname> | Specifies the hostname of the Flume agent (may be localhost) |
| --port,-p <port> | Specifies the port on which the Avro source is listening |
| --filename,-F <filename> | Sends each line of <filename> to Flume (optional) |
| --headerFile,-F <file> | Header file containing headers as key/value pairs on each new line |
The Avro client treats each line (terminated by \n, \r, or \r\n) as an event. Think of the avro-client command as cat for Flume. For instance, the following creates one event per Linux user and sends it to Flume's avro source on localhost:41414.
In a new window type the following:
$ bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F /etc/passwd -Dflume.root.logger=DEBUG,console
You should see something like this:
2012-03-16 16:39:17,124 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:175)] Finished
2012-03-16 16:39:17,127 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:178)] Closing reader
2012-03-16 16:39:17,127 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:183)] Closing transceiver
2012-03-16 16:39:17,129 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:73)] Exiting
And in your first window, where the server is running:
2012-03-16 16:39:16,738 (New I/O server boss #1 ([id: 0x49e808ca, /0:0:0:0:0:0:0:0:41414])) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /1
27.0.0.1:39577 => /127.0.0.1:41414] OPEN
2012-03-16 16:39:16,742 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 => /127.0.0.1:41414] BOU
ND: /127.0.0.1:41414
2012-03-16 16:39:16,742 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 => /127.0.0.1:41414] CON
NECTED: /127.0.0.1:39577
2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] DISCONNECTED
2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] UNBOUND
2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] CLOSED
2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@5c1ae90c }
2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@6aba4211 }
2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@6a47a0d4 }
2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@48ff4cf }
...
Congratulations! You have Apache Flume running!
Providing Feedback
For help building, configuring, and running Flume (NG or otherwise), the best place is always the user mailing list. Send an email to user-subscribe@flume.apache.org to subscribe and user@flume.apache.org to post once you've subscribed. The archives are available at http://mail-archives.apache.org/mod_mbox/incubator-flume-user/ (up through part of July 2012) and http://mail-archives.apache.org/mod_mbox/incubator-flume-user/http://mail-archives.apache.org/mod_mbox/flume-user/ (starting through part of July 2012 onwards).
If you believe you've found a bug or wish to file a feature request or improvement, don't be shy. Go to https://issues.apache.org/jira/browse/FLUME and file a JIRA for the version of Flume. For NG, please set the "Affects Version" to the appropriate milestone / release. Just leave any field you're not sure about blank. We'll bug you for details if we need them. Note that you must create an Apache JIRA account and log in before you can file issues.
因个人能力实在有限，难免会出现这样那样的话，希望大家不吝指教。
601
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
