Flume安装配置笔记

1. 下载并解压

wget http://mirrors.cnnic.cn/apache/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz

tar -xzvf apache-flume-1.7.0-bin.tar.gz

mv setup/apache-flume-1.7.0-bin flume

2. 修改配置

cd flume/

cp flume-conf.properties.template flume-conf.properties

vi flume-conf.properties

agent.sources = r1

agent.channels = c1

agent.sinks = w1

# For each one of the sources, the type is defined

agent.sources.r1.type = avro

agent.sources.r1.bind = 0.0.0.0

agent.sources.r1.port = 44444

agent.sources.r1.channels = c1

# Each sink's type must be defined

agent.sinks.w1.type = logger

agent.sinks.w1.channel = c1

# Each channel's type is defined.

agent.channels.c1.type = memory

agent.channels.c1.capacity = 100

3. $vi /etc/profile #添加环境配置 export FLUME_HOME=/home/dingyongqiang/flume/apache-flume-1.7.0-bin export PATH=.: $PATH :: $FLUME_HOME /bin

使环境变量立即生效：source /etc/profile

4. 启动flume

参数	作用	举例
–conf 或 -c	指定配置文件夹，包含flume-env.sh和log4j的配置文件	–conf conf
–conf-file 或 -f	配置文件地址	–conf-file conf/flume.conf
–name 或 -n	agent名称	–name a1
-z	zookeeper连接字符串	-z zkhost:2181,zkhost1:2181
-p	zookeeper中的存储路径前缀	-p /flume

bin/flume-ng agent --conf-file flume-conf.properties --name agent1 -Dflume.root.logger=INFO,console --conf=conf

报错

Info: Including Hive libraries found via () for Hive access

+ exec /usr/lib/jvm/java-8-oracle/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '=:/home/dingyongqiang/flume/apache-flume-1.7.0-bin/lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file flume-conf.properties --name agent1 conf

log4j:WARN No appenders could be found for logger (org.apache.flume.node.Application).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

原来是网上copy而来的命令有问题，坑。。。

正确的命令如下：

bin/flume-ng agent --conf-file conf/flume-conf.properties --name agent1 -Dflume.root.logger=INFO,console --conf conf

貌似又出错了

Info: Including Hive libraries found via () for Hive access

+ exec /usr/lib/jvm/java-8-oracle/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/home/dingyongqiang/flume/apache-flume-1.7.0-bin/conf:/home/dingyongqiang/flume/apache-flume-1.7.0-bin/lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file conf/flume-conf.properties --name agent1

2017-01-22 16:10:37,217 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting

2017-01-22 16:10:37,221 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)] Reloading configuration file:conf/flume-conf.properties

2017-01-22 16:10:37,242 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:w1

2017-01-22 16:10:37,242 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:w1

2017-01-22 16:10:37,242 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: w1 Agent: agent

2017-01-22 16:10:37,251 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [agent]

2017-01-22 16:10:37,252 (conf-file-poller-0) [WARN - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:135)] No configuration found for this host:agent1

2017-01-22 16:10:37,258 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

又是抄下来的一个坑。。。agent名称必须和配置文件中的保持一致

启动命令应改为：

bin/flume-ng agent --conf-file conf/flume-conf.properties --name agent -Dflume.root.logger=INFO,console --conf conf

输出如下，启动成功

Info: Including Hive libraries found via () for Hive access

+ exec /usr/lib/jvm/java-8-oracle/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/home/dingyongqiang/flume/apache-flume-1.7.0-bin/conf:/home/dingyongqiang/flume/apache-flume-1.7.0-bin/lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file conf/flume-conf.properties --name agent

2017-01-22 16:19:23,456 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting

2017-01-22 16:19:23,462 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)] Reloading configuration file:conf/flume-conf.properties

2017-01-22 16:19:23,471 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:w1

2017-01-22 16:19:23,471 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:w1

2017-01-22 16:19:23,471 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: w1 Agent: agent

2017-01-22 16:19:23,480 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [agent]

2017-01-22 16:19:23,480 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)] Creating channels

2017-01-22 16:19:23,485 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel c1 type memory

2017-01-22 16:19:23,599 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)] Created channel c1

2017-01-22 16:19:23,600 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source r1, type avro

2017-01-22 16:19:23,615 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: w1, type: logger

2017-01-22 16:19:23,617 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)] Channel c1 connected to [r1, w1]

2017-01-22 16:19:23,623 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: { bindAddress: 0.0.0.0, port: 44444 } }} sinkRunners:{w1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4972a622 counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }

2017-01-22 16:19:23,629 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:144)] Starting Channel c1

2017-01-22 16:19:23,817 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.

2017-01-22 16:19:23,818 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started

2017-01-22 16:19:23,820 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)] Starting Sink w1

2017-01-22 16:19:23,820 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)] Starting Source r1

2017-01-22 16:19:23,821 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:234)] Starting Avro source r1: { bindAddress: 0.0.0.0, port: 44444 }...

2017-01-22 16:19:24,101 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.

2017-01-22 16:19:24,102 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: r1 started

2017-01-22 16:19:24,102 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:259)] Avro source r1 started.

5. 测试flume

我们打开一个新的console，新建一个data/flumetemp目录，并写入内容

dingyongqiang@vm-20160906-08:~/flume/data/flumetemp$ echo 123456 >>a

执行如下命令

dingyongqiang@vm-20160906-08:~/flume/apache-flume-1.7.0-bin$ bin/flume-ng avro-client --conf conf -H localhost -p 44444 -F ../data/flumetmp/a

输出

2017-01-22 16:31:13,708 (New I/O server boss #17) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x537b6f68, /127.0.0.1:38823 => /127.0.0.1:44444] OPEN

2017-01-22 16:31:13,710 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x537b6f68, /127.0.0.1:38823 => /127.0.0.1:44444] BOUND: /127.0.0.1:44444

2017-01-22 16:31:13,710 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x537b6f68, /127.0.0.1:38823 => /127.0.0.1:44444] CONNECTED: /127.0.0.1:38823

2017-01-22 16:31:13,905 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x537b6f68, /127.0.0.1:38823 :> /127.0.0.1:44444] DISCONNECTED

2017-01-22 16:31:13,905 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x537b6f68, /127.0.0.1:38823 :> /127.0.0.1:44444] UNBOUND

2017-01-22 16:31:13,905 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x537b6f68, /127.0.0.1:38823 :> /127.0.0.1:44444] CLOSED

2017-01-22 16:31:13,905 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed(NettyServer.java:209)] Connection to /127.0.0.1:38823 disconnected.

似乎没有输出收集到的日志内容

6. 换一个source-spooldir试试

修改配置如下（仅sources部分）：

agent.sources.r1.type = spooldir

agent.sources.r1.spoolDir = /home/dingyongqiang/log

agent.sources.r1.channels = c1

启动后再次测试

vi 1.log

写入Hello Flume! 作为测试内容。然后拷贝到flume监听路径。

cp 1.log /home/dingyongqiang/log

接着就可以在前一个终端看到刚刚采集的内容了,如下：

2017-01-22 17:00:39,916 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:324)] Last read took us just up to a file boundary. Rolling to the next file, if there is one.

2017-01-22 17:00:39,917 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:433)] Preparing to move file /home/dingyongqiang/log/1.log to /home/dingyongqiang/log/1.log.COMPLETED

2017-01-22 17:00:43,962 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 21 Hello Flume! }