Flume结合场景讲解

flume 使用

简单使用flume 
场景1： 
1、通过netcat发布消息 
2、用flume接收netcat发布的消息，最终显示在终端 
3、flume的channels是用内存存储的

先定义flume-conf.properties.log 这样的文件

#定义agent的配置 定义sources 定义channels 定义sinks
a1.sources = r1
a1.sinks = k1
a1.channels = c1
#定义sources的来源
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
#定义sinks的方式
a1.sinks.k1.type = logger
#定义channels的方式
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
#将sinks channels sources 连接起来
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动telnet来做agent的sources

[root@SZB-L0038787 ~]# telnet localhost 44444
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
xlucas
OK

启动flume agent

[root@SZB-L0038787 apache-flume-1.8.0-bin]# ./bin/flume-ng agent -name a1 -c conf -f conf/flume-conf.properties.log -Dflume.root.logger=INFO,console
2017-12-29 10:23:24,268 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting
2017-12-29 10:23:24,279 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)] Reloading configuration file:conf/flume-conf.properties.log
2017-12-29 10:23:24,297 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k1 Agent: a1
2017-12-29 10:23:24,297 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2017-12-29 10:23:24,298 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2017-12-29 10:23:24,323 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [a1]
2017-12-29 10:23:24,323 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)] Creating channels
2017-12-29 10:23:24,333 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel c1 type memory
2017-12-29 10:23:24,339 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)] Created channel c1
2017-12-29 10:23:24,340 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source r1, type netcat
2017-12-29 10:23:24,354 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: k1, type: logger
2017-12-29 10:23:24,358 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)] Channel c1 connected to [r1, k1]
2017-12-29 10:23:24,382 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4e4bc79 counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
2017-12-29 10:23:24,394 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:144)] Starting Channel c1
2017-12-29 10:23:24,455 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2017-12-29 10:23:24,456 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
2017-12-29 10:23:24,458 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)] Starting Sink k1
2017-12-29 10:23:24,459 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)] Starting Source r1
2017-12-29 10:23:24,460 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:155)] Source starting
2017-12-29 10:23:24,474 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:166)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
2017-12-29 10:23:39,724 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 78 6C 75 63 61 73 0D                            xlucas. }

如果有些参数不是固定的，是通过参数获取的可以用 
例如: 
这个sources的定义中port是可以不指定的端口，通过运行的时候在指定端口

a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = ${port}

运行指定端口 
-DpropertiesImplementation=org.apache.flume.node.EnvVarResolverProperties 
这个参数值需要加入

[root@SZB-L0038787 apache-flume-1.8.0-bin]# port=44444 ./bin/flume-ng agent -name a1 -c conf -f conf/flume-conf.properties.log -Dflume.root.logger=INFO,console  -DpropertiesImplementation=org.apache.flume.node.EnvVarResolverProperties

使用zookeeper来做conf文件 
zookeeper的地址是10.20.23.29 agent放在/flume/a1里面

[root@SZB-L0038787 apache-flume-1.8.0-bin]# ./bin/flume-ng agent --conf conf -z 10.20.23.29:2181 -p /flume -name a1 -Dflume.root.logger=INFO,console -DpropertiesImplementation=org.apache.flume.node.EnvVarResolverProperties2017-12-29 16:54:27,830 (lifecycleSupervisor-1-0-EventThread) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:144)] Starting Channel c1
2017-12-29 16:54:27,901 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2017-12-29 16:54:27,902 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
2017-12-29 16:54:27,902 (lifecycleSupervisor-1-0-EventThread) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)] Starting Sink k1
2017-12-29 16:54:27,904 (lifecycleSupervisor-1-0-EventThread) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)] Starting Source r1
2017-12-29 16:54:27,904 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:155)] Source starting
2017-12-29 16:54:27,908 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:166)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
2017-12-29 16:54:38,952 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 78 6C 75 63 61 73 0D                            xlucas. }

发送消息

[root@SZB-L0038787 ~]# telnet localhost 44444
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
xlucas
OK