from http://blog.csdn.net/wuguangxue255/article/details/11710225
对变名日志的处理,对于java工程可以简单修改log4j的配置文件,增加flume配置即可
对于非java程序,采取规避的方案,当然如果你足够强大可以重写编译flume代码,自定义一个source type支持,将spoolDir 类型的source加强
监控变化的文件。
采用tail -F收集日志,在空闲时刻将进程重启,收集指定日志即可
样例:
1:client-agent配置
a1.sinks = remotesink
a1.channels = c1
a1.sources = r1
a1.sources.r1.type = exec
a1.sources.r1.shell = /bin/bash -c
a1.sources.r1.command = /usr/local/flume/conf/flume.sh
a1.sources.r1.channels = c1
a1.sources.r1.batchSize = 1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = ip
a1.sources.r1.interceptors.i1.value = 192.168.254.162
a1.sources.r1.interceptors.i2.type = static
a1.sources.r1.interceptors.i2.key = NodeName
a1.sources.r1.interceptors.i2.value = 1mynode
a1.sources.r1.interceptors.i3.type = static
a1.sources.r1.interceptors.i3.key = ModuleType
a1.sources.r1.interceptors.i3.value = miner
a1.channels.c1.type = memory
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 100
a1.sinks.remotesink.type = avro
a1.sinks.remotesink.hostname = 192.168.252.167
a1.sinks.remotesink.port = 44446
a1.sinks.remotesink.channel = c1
2:flume.sh
#!/bin/bash
tail -F /usr/local/flume/$(date '+%Y%m%d').log
3:定时任务
#!/bin/bash
ps aux|grep flume|grep -v grep|awk '{print $2}'|xargs kill -9
/usr/local/flume/bin/flume-ng agent --conf-file /usr/local/flume/conf/client.conf --name a1 -Dflume.root.logger=INFO,LOGFILE -c /usr/local/flume/conf/ -Dflume.monitoring.type=http -Dflume.monitoring.port=34545
4:计划任务
0 0 * * * sh /usr/local/flume/conf/kill.sh >/dev/null
1 0 * * * sh /usr/local/flume/conf/restart.sh >/dev/null