hello flume (Ubuntu 下 flume1.5单机版安装以及简单入门示例)

最新推荐文章于 2023-10-25 14:12:03 发布

追寻北极

最新推荐文章于 2023-10-25 14:12:03 发布

阅读量1.9k

点赞数

分类专栏： cloud

本文链接：https://blog.csdn.net/joeyon1985/article/details/42967655

版权

cloud 专栏收录该内容

234 篇文章 0 订阅

订阅专栏

1,下载最新的flume安装包：

wget http://www.apache.org/dist/flume/stable/apache-flume-1.5.2-bin.tar.gz

2，在安装目录解压：

 tar -zxvf apache-flume-1.5.2-bin.tar.gz

3,设置环境变量

export JAVA_HOME=/usr ;

export FLUME_HOME=/home/joeyon/apache-flume-1.5.2-bin;

export PATH=$PATH:FLUME_HOME/bin

4,配置一个简单单机的测试案例

</pre>4.1在conf目录下新建一个文件，example-conf.properties，内容如下：<pre name="code" class="html"># Describe the source 
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
# 将数据输出至日志中
a1.sinks.k1.type = logger

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

4.2在conf目录下新建一个源文件file01并向其中写入数据

# Use a channel which buffers events in memory
   touch file01;

   echo "hello world" > file01

4.3 启动agent代理

flume-ng agent -c . -f /home/joeyon/apache-flume-1.5.2-bin/conf/example-conf.properties  -n a1 -Dflume.root.logger=INFO,console

启动成功日志如下：

+ exec /usr/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/data/projects/apache-flume-1.5.2-bin/conf:/data/projects/apache-flume-1.5.2-bin/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /data/projects/apache-flume-1.5.2-bin/conf/example-conf.properties -n a1
2015-01-21 13:44:43,231 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2015-01-21 13:44:43,239 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:/data/projects/apache-flume-1.5.2-bin/conf/example-conf.properties
2015-01-21 13:44:43,248 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k1 Agent: a1
2015-01-21 13:44:43,249 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2015-01-21 13:44:43,249 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2015-01-21 13:44:43,261 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [a1]
2015-01-21 13:44:43,262 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2015-01-21 13:44:43,271 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel c1 type memory
2015-01-21 13:44:43,276 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel c1
2015-01-21 13:44:43,277 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source r1, type avro
2015-01-21 13:44:43,297 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: k1, type: logger
2015-01-21 13:44:43,301 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel c1 connected to [r1, k1]
2015-01-21 13:44:43,309 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: { bindAddress: localhost, port: 44444 } }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@27b6a3ac counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
2015-01-21 13:44:43,320 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel c1
2015-01-21 13:44:43,383 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2015-01-21 13:44:43,383 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
2015-01-21 13:44:43,387 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink k1
2015-01-21 13:44:43,388 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source r1
2015-01-21 13:44:43,388 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:228)] Starting Avro source r1: { bindAddress: localhost, port: 44444 }...
2015-01-21 13:44:43,771 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
2015-01-21 13:44:43,771 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: r1 started
2015-01-21 13:44:43,772 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:253)] Avro source r1 started.

4.4.启动avro-client客户端向agent代理发送数据：

flume-ng avro-client -c . -H localhost -p 44444 -F /home/joeyon/apache-flume-1.5.2-bin/conf/file01

日志如下：

+ exec /usr/bin/java -Xmx20m -cp '/data/projects/apache-flume-1.5.2-bin/conf:/data/projects/apache-flume-1.5.2-bin/lib/*' -Djava.library.path= org.apache.flume.client.avro.AvroCLIClient -H localhost -p 44444 -F /home/joeyon/apache-flume-1.5.2-bin/conf/file01

4.5查看agen控制台输出代理端输出如下则收集成功：

2015-01-21 13:50:13,276 (New I/O server boss #1 ([id: 0xc0b7fffb, /127.0.0.1:44444])) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 => /127.0.0.1:44444] OPEN
2015-01-21 13:50:13,277 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 => /127.0.0.1:44444] BOUND: /127.0.0.1:44444
2015-01-21 13:50:13,278 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 => /127.0.0.1:44444] CONNECTED: /127.0.0.1:3809
2015-01-21 13:50:13,703 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 :> /127.0.0.1:44444] DISCONNECTED
2015-01-21 13:50:13,703 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 :> /127.0.0.1:44444] UNBOUND
2015-01-21 13:50:13,703 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 :> /127.0.0.1:44444] CLOSED
2015-01-21 13:50:13,704 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed(NettyServer.java:209)] Connection to /127.0.0.1:3809 disconnected.
2015-01-21 13:50:17,411 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 3E 3E 3E 3E 3E hello world>>>>> }
2015-01-21 13:50:17,411 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 6D 79 20 6E 61 6D 65 20 69 73 20 6A 6F 65 79 6F my name is joeyo }
2015-01-21 13:50:17,412 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 77 65 6C 63 6F 6D 20 74 6F 20 63 68 69 6E 61 3E welcom to china> }