Flume2

Flume流程：
client ------------------> source -------------------> channel --------------------> sink 
         Interceptors               Channel Selectors           Sink Processors
Flume官网（用户指南）（具体视版本情况而定）：
http://archive.cloudera.com/cdh5/cdh/5/flume-ng-1.6.0-cdh5.7.0/FlumeUserGuide.html

选择器介绍（详见官网）：

• Flume Channel Selectors
  • Replicating Channel Selector (default)
  • Multiplexing Channel Selector
  • Custom Channel Selector
• Flume Sink Processors
  • Default Sink Processor
  • Failover Sink Processor
  • Load balancing Sink Processor
  • Custom Sink Processor
• Flume Interceptors
  • Timestamp Interceptor
  • Host Interceptor
  • Static Interceptor
  • UUID Interceptor
  • Morphline Interceptor
  • Search and Replace Interceptor
  • Regex Filtering Interceptor
  • Regex Extractor Interceptor

 需掌握的Flume分层模式：

Setting multi-agent flow：
avro-sink的hostname以及ip是要和avro-source的相对应

In order to flow the data across multiple agents or hops, the sink of the previous agent and source of the current hop need to be avro type with the sink pointing to the hostname (or IP address) and port of the source.

Consolidation：

A very common scenario in log collection is a large number of log producing clients sending data to a few consumer agents that are attached to the storage subsystem. For example, logs collected from hundreds of web servers sent to a dozen of agents that write to HDFS cluster.

This can be achieved in Flume by configuring a number of first tier agents with an avro sink, all pointing to an avro source of single agent (Again you could use the thrift sources/sinks/clients in such a scenario). This source on the second tier agent consolidates the received events into a single channel which is consumed by a sink to its final destination.

Multiplexing the flow：

Flume supports multiplexing the event flow to one or more destinations. This is achieved by defining a flow multiplexer that can replicate or selectively route an event to one or more channels.

The above example shows a source from agent “foo” fanning out the flow to three different channels. This fan out can be replicating or multiplexing. In case of replicating flow, each event is sent to all three channels. For the multiplexing case, an event is delivered to a subset of available channels when an event’s attribute matches a preconfigured value. For example, if an event attribute called “txnType” is set to “customer”, then it should go to channel1 and channel3, if it’s “vendor” then it should go to channel2, otherwise channel3. The mapping can be set in the agent’s configuration file.