flume-ng 使用系列

对变名日志的处理，对于java工程可以简单修改log4j的配置文件，增加flume配置即可

对于非java程序，采取规避的方案，当然如果你足够强大可以重写编译flume代码，自定义一个source type支持，将spoolDir 类型的source加强

监控变化的文件。

采用tail -F收集日志，在空闲时刻将进程重启，收集指定日志即可

样例：

1：client-agent配置

a1.sinks = remotesink
a1.channels = c1
a1.sources = r1

a1.sources.r1.type = exec
a1.sources.r1.shell = /bin/bash -c
a1.sources.r1.command = /usr/local/flume/conf/flume.sh
a1.sources.r1.channels = c1
a1.sources.r1.batchSize = 1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = ip
a1.sources.r1.interceptors.i1.value = 192.168.254.162
a1.sources.r1.interceptors.i2.type = static
a1.sources.r1.interceptors.i2.key = NodeName
a1.sources.r1.interceptors.i2.value = 1mynode
a1.sources.r1.interceptors.i3.type = static
a1.sources.r1.interceptors.i3.key = ModuleType
a1.sources.r1.interceptors.i3.value = miner

a1.channels.c1.type = memory
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 100

a1.sinks.remotesink.type = avro
a1.sinks.remotesink.hostname = 192.168.252.167
a1.sinks.remotesink.port = 44446
a1.sinks.remotesink.channel = c1

2：flume.sh

#!/bin/bash
tail -F /usr/local/flume/$(date '+%Y%m%d').log

3：定时任务

#!/bin/bash
ps aux|grep flume|grep -v grep|awk '{print $2}'|xargs kill -9

/usr/local/flume/bin/flume-ng agent --conf-file /usr/local/flume/conf/client.conf --name a1 -Dflume.root.logger=INFO,LOGFILE -c /usr/local/flume/conf/ -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

4：计划任务

0 0 * * * sh /usr/local/flume/conf/kill.sh >/dev/null
1 0 * * * sh /usr/local/flume/conf/restart.sh >/dev/null