flume中的monitor

http://flume.apache.org/FlumeUserGuide.html#monitoring

如果没有监控器，那么flume采集了多少event，sink了多少event，都没法直接知道(可以统计日志，但是比较麻烦),如果使用flume的监控，就比较简单了。一共有4种Reporting方式，JMX Reporting、Ganglia Reporting、JSON Reporting、Custom Reporting， 这里以最简单的JSON Reporting为例:

json方式配置

a1.sources = src-1
a1.channels = ch-1
a1.sinks = k1

a1.sources.src-1.type = spooldir
a1.sources.src-1.channels = ch-1
#往这个目录写入文本文件，flume就会自动采集
a1.sources.src-1.spoolDir = /Users/waixingren/software/flume/flume/data/input 
a1.sources.src-1.fileHeader = true

a1.channels.ch-1.type=memory

a1.sinks.k1.type = file_roll
a1.sinks.k1.channel = ch-1
#flume的输出目录
a1.sinks.k1.sink.directory = /Users/waixingren/software/flume/flume/data/output

命名为：conf/agent.conf

启动

flume-ng agent -n a1 –conf conf -f conf/agent.conf -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

启动后打开：http://localhost:34545/metrics

生成数据

echo "aaa" >> /Users/waixingren/software/flume/flume/data/input/test.txt

显示：

{"CHANNEL.ch-1":{"ChannelCapacity":"100","ChannelFillPercentage":"0.0","Type":"CHANNEL","ChannelSize":"0","EventTakeSuccessCount":"1","StartTime":"1520496058842","EventTakeAttemptCount":"9","EventPutAttemptCount":"1","EventPutSuccessCount":"1","StopTime":"0"},"SINK.k1":{"ConnectionCreatedCount":"2","Type":"SINK","ConnectionClosedCount":"1","BatchCompleteCount":"0","BatchEmptyCount":"0","EventDrainAttemptCount":"1","StartTime":"1520496058843","EventDrainSuccessCount":"1","BatchUnderflowCount":"0","ConnectionFailedCount":"0","StopTime":"0"},"SOURCE.src-1":{"EventReceivedCount":"1","Type":"SOURCE","AppendBatchAcceptedCount":"1","EventAcceptedCount":"1","AppendReceivedCount":"0","StartTime":"1520496058905","AppendAcceptedCount":"0","OpenConnectionCount":"0","AppendBatchReceivedCount":"1","StopTime":"0"}}

JMX方式

配置

打开flume-env.sh
增加：

export JAVA_OPTS=”-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false”

启动

./bin/flume-ng agent -n a1 –conf conf -f conf/agent.conf

打开jconsole

echo "aaa" >> /Users/waixingren/software/flume/flume/data/input/test.txt