1 Flume安装和配置
1.在10台日志服务器和1台日志收集服务器上分别上传apache-flume-1.6.0-bin.tar安装包。
2.解压apache-flume-1.6.0-bin.tar安装包。
tar -zxvf apache-flume-1.6.0-bin.tar
3.修改配置文件
1) 进入解压后flume的conf目录,使用vim命令编辑flume-env.sh.template的文件,修改下面的配置,然后保存退出。
export JAVA_HOME=/home/super/software/jdk/jdk1.7.0_80/ |
注意:
1. 去掉开头 #的注释
2. Jdk为服务器中jdk的路径
2) 把名为flume-env.sh.template的文件去掉.template后缀
mv flume-env.sh.template flume-env.sh
2 日志服务器相关配置
1、编写flume采集日志配置
vim logserver.conf
#gent的名称为"logserver" logserver.sources = r1 logserver.sinks = k1 logserver.channels = c1
#source配置信息 logserver.sources.r1.type = spooldir #日志服务器IP地址(需要修改) logserver.sources.r1.bind = 10.34.1.116 logserver.sources.r1.port = 55555 #日志服务器日志存放目录(需要修改) logserver.sources.r1.spoolDir = /home/super/logs/
#sink配置信息 logserver.sinks.k1.type = avro #出口审计检测引擎服务器IP地址 logserver.sinks.k1.hostname = 10.34.1.118 logserver.sinks.k1.port = 44444
#channel配置信息 logserver.channels.c1.type = memory logserver.channels.c1.capacity = 1000 logserver.channels.c1.transactionCapacity = 100
#将source和sink绑定至该channel上 logserver.sources.r1.channels = c1 logserver.sinks.k1.channel = c1 |
2、编写flume启动脚本
1)编写logserver_startflume.sh脚本
#!/bin/bash #JAVA HOME路径 export JAVA_HOME=/home/super/software/jdk/jdk1.7.0_80 #日志服务器序号id(需要修改) id=0 #日志服务器总数量 servertotal=10 #Flume路径(需要修改) FLUME_HOME=/home/super/software/apache-flume-1.6.0-bin #按天轮询启动Flume a=`date +%d` b=$(($a%$servertotal)) echo $b if [ $b = $id ] ; then nohup ${FLUME_HOME}/bin/flume-ng agent -n logserver -f ${FLUME_HOME}/logserver.conf & sleep 10 count=`ps -ef |grep "Application" |grep -v "grep" |wc -l` if [ 0 == $count ] ; then sleep 60 nohup ${FLUME_HOME}/bin/flume-ng agent -n logserver -f ${FLUME_HOME}/logserver.conf & else echo "Flume Start Succses" fi else echo "No Need To Start" fi |
注意:
1. 脚本中10台机器的id分别为0 到 9 ,不能重复!
2. 脚本中JAVA_HOME和FLUME_HOME路径根据环境进行配置
3. 编写flume停止脚本
1)编写logserver_stopflume.sh脚本
#!/bin/sh
#FLUME进程名 NAME=Application echo $NAME ID=`ps -ef | grep "$NAME" | grep -v "$0" | grep -v "grep" | awk '{print $2}'` echo $ID for id in $ID do kill -9 $id echo "Kill Flume Process $id" done sleep 10 count=`ps -ef |grep "$NAME" |grep -v "grep" |wc -l` #echo $count if [ 0 != $count ];then ID=`ps -ef | grep "$NAME" | grep -v "$0" | grep -v "grep" | awk '{print $2}'` echo $ID for id in $ID do kill -9 $id echo "Kill Flume Process $id" done fi |
4.配置定时启动脚本任务
1) 执行crontab -e命令,进入crontab定时配置文件中,添加内容
50 15* * * /home/super/software/apache-flume-1.6.0-bin/logserver_startflume.sh 57 15* * * /home/super/software/apache-flume-1.6.0-bin/logserver_stopflume.sh |
注意:
1. 基本格式为:* * * * * command
分时 日月 周命令
2. 脚本的路径应为绝对路径!
3 收集服务器相关配置
1、编写flume收集日志配置
vim logreceiver.conf
logreceiver.sources = r1 logreceiver.sinks = k1 logreceiver.channels = c1
#source配置信息 logreceiver.sources.r1.type = avro #出口审计检测引擎服务器IP地址 logreceiver.sources.r1.bind = 10.34.1.118 logreceiver.sources.r1.port = 44444
#sink配置信息 #file_roll表示将数据存入本地文件系统 logreceiver.sinks.k1.type = file_roll #出口审计检测引擎服务器日志存放目录 logreceiver.sinks.k1.sink.directory =/home/audit/logs/ logreceiver.sinks.k1.sink.rollInterval = 0
#channel配置信息 logreceiver.channels.c1.type = memory logreceiver.channels.c1.capacity = 1000 logreceiver.channels.c1.transactionCapacity = 100
#将source和sink绑定至该channel上 logreceiver.sources.r1.channels = c1 logreceiver.sinks.k1.channel = c1 |
2、编写flume启动脚本
使用vim编辑logreceiver_startflume.sh脚本。
#!/bin/bash #JAVA HOME路径(根据情况需要修改) export JAVA_HOME=/home/super/software/jdk/jdk1.7.0_80 #Flume路径(根据情况需要修改) FLUME_HOME=/home/super/software/apache-flume-1.6.0-bin #按天轮询启动Flume nohup ${FLUME_HOME}/bin/flume-ng agent -n logserver -f ${FLUME_HOME}/logserver.conf & sleep 10 count=`ps -ef |grep "Application" |grep -v "grep" |wc -l` if [ 0 == $count ] ; then sleep 60 nohup ${FLUME_HOME}/bin/flume-ng agent -n logserver -f ${FLUME_HOME}/logserver.conf & else echo "Flume Start Succses" fi |
注意:
1. 脚本中JAVA_HOME和FLUME_HOME路径根据环境进行配置
3. 编写flume关闭脚本
使用vim编辑logreceiver_stopflume.sh脚本。
#!/bin/sh #FLUME进程名 NAME=Application echo $NAME ID=`ps -ef | grep "$NAME" | grep -v "$0" | grep -v "grep" | awk '{print $2}'` echo $ID for id in $ID do kill -9 $id echo "Kill Flume Process $id" done sleep 10 count=`ps -ef |grep "$NAME" |grep -v "grep" |wc -l` #echo $count if [ 0 != $count ];then ID=`ps -ef | grep "$NAME" | grep -v "$0" | grep -v "grep" | awk '{print $2}'` echo $ID for id in $ID do kill -9 $id echo "Kill Flume Process $id" done fi |
4、配置定时启动脚本任务
2) 执行crontab -e命令,进入crontab定时配置文件中,添加内容
50 15 * * * /home/super/software/apache-flume-1.6.0-bin/logreceiver_startflume.sh 57 15 * * * /home/super/software/apache-flume-1.6.0-bin/logreceiver_stopflume.sh |
注意:
1.基本格式为:* * * * * command
分时 日月 周命令
2.脚本的路径应为绝对路径!