flume定时采集日志的配置

Flume安装和配置

1.10台日志服务器和1台日志收集服务器上分别上传apache-flume-1.6.0-bin.tar安装包。

2.解压apache-flume-1.6.0-bin.tar安装包。

tar -zxvf apache-flume-1.6.0-bin.tar

3.修改配置文件

1) 进入解压后flumeconf目录,使用vim命令编辑flume-env.sh.template的文件,修改下面的配置,然后保存退出。

export JAVA_HOME=/home/super/software/jdk/jdk1.7.0_80/

注意:

1. 去掉开头 #的注释

2. Jdk为服务器中jdk的路径  

2) 把名为flume-env.sh.template的文件去掉.template后缀

mv flume-env.sh.template flume-env.sh

2 日志服务器相关配置

1、编写flume采集日志配置

vim logserver.conf

#gent的名称为"logserver"

logserver.sources = r1

logserver.sinks = k1

logserver.channels = c1

 

#source配置信息

logserver.sources.r1.type = spooldir

#日志服务器IP地址(需要修改)

logserver.sources.r1.bind = 10.34.1.116

logserver.sources.r1.port = 55555

#日志服务器日志存放目录(需要修改)

logserver.sources.r1.spoolDir = /home/super/logs/

 

#sink配置信息

logserver.sinks.k1.type = avro

#出口审计检测引擎服务器IP地址

logserver.sinks.k1.hostname = 10.34.1.118

logserver.sinks.k1.port = 44444

 

#channel配置信息

logserver.channels.c1.type = memory

logserver.channels.c1.capacity = 1000

logserver.channels.c1.transactionCapacity = 100

 

#sourcesink绑定至该channel

logserver.sources.r1.channels = c1

logserver.sinks.k1.channel = c1

 

2、编写flume启动脚本

1编写logserver_startflume.sh脚本

#!/bin/bash

#JAVA HOME路径

export JAVA_HOME=/home/super/software/jdk/jdk1.7.0_80

#日志服务器序号id(需要修改)

id=0

#日志服务器总数量

servertotal=10

#Flume路径(需要修改)

FLUME_HOME=/home/super/software/apache-flume-1.6.0-bin

#按天轮询启动Flume

a=`date +%d`

b=$(($a%$servertotal))

echo $b

if [ $b = $id ] ; then

   nohup ${FLUME_HOME}/bin/flume-ng agent -n logserver -f ${FLUME_HOME}/logserver.conf &

   sleep 10

   count=`ps -ef |grep "Application" |grep -v "grep" |wc -l`

   if [ 0 == $count ] ; then

          sleep 60

          nohup ${FLUME_HOME}/bin/flume-ng agent -n logserver -f ${FLUME_HOME}/logserver.conf &

   else

          echo "Flume Start Succses"

   fi

else

   echo "No Need To Start"

fi

注意:

1. 脚本中10台机器的id分别为0 9 ,不能重复!

2. 脚本中JAVA_HOMEFLUME_HOME路径根据环境进行配置

 

3. 编写flume停止脚本

1编写logserver_stopflume.sh脚本

#!/bin/sh

 

#FLUME进程名

NAME=Application

echo $NAME

ID=`ps -ef | grep "$NAME" | grep -v "$0" | grep -v "grep" | awk '{print $2}'`

   echo $ID

   for id in $ID

   do

     kill -9 $id

     echo "Kill Flume Process $id"

   done

   sleep 10

      count=`ps -ef |grep "$NAME" |grep -v "grep" |wc -l`

      #echo $count

      if [ 0 != $count ];then

        ID=`ps -ef | grep "$NAME" | grep -v "$0" | grep -v "grep" | awk '{print $2}'`

        echo $ID

        for id in $ID

        do

            kill -9 $id

            echo "Kill Flume Process $id"

        done

      fi

 

 

4.配置定时启动脚本任务

1) 执行crontab -e命令,进入crontab定时配置文件中,添加内容

50 15* * * /home/super/software/apache-flume-1.6.0-bin/logserver_startflume.sh

57 15* * * /home/super/software/apache-flume-1.6.0-bin/logserver_stopflume.sh

注意:

1. 基本格式为:* * * * * command

命令 

2. 脚本的路径应为绝对路径!

 

3 收集服务器相关配置

1、编写flume收集日志配置

vim logreceiver.conf

logreceiver.sources = r1

logreceiver.sinks = k1

logreceiver.channels = c1

 

#source配置信息

logreceiver.sources.r1.type = avro

#出口审计检测引擎服务器IP地址

logreceiver.sources.r1.bind = 10.34.1.118

logreceiver.sources.r1.port = 44444

 

#sink配置信息

#file_roll表示将数据存入本地文件系统

logreceiver.sinks.k1.type = file_roll

#出口审计检测引擎服务器日志存放目录

logreceiver.sinks.k1.sink.directory =/home/audit/logs/

logreceiver.sinks.k1.sink.rollInterval = 0

 

#channel配置信息

logreceiver.channels.c1.type = memory

logreceiver.channels.c1.capacity = 1000

logreceiver.channels.c1.transactionCapacity = 100

 

#sourcesink绑定至该channel

logreceiver.sources.r1.channels = c1

logreceiver.sinks.k1.channel = c1

 

2、编写flume启动脚本

使用vim编辑logreceiver_startflume.sh脚本。

#!/bin/bash

#JAVA HOME路径(根据情况需要修改)

export JAVA_HOME=/home/super/software/jdk/jdk1.7.0_80

#Flume路径(根据情况需要修改)

FLUME_HOME=/home/super/software/apache-flume-1.6.0-bin

#按天轮询启动Flume

nohup ${FLUME_HOME}/bin/flume-ng agent -n logserver -f ${FLUME_HOME}/logserver.conf &

sleep 10

count=`ps -ef |grep "Application" |grep -v "grep" |wc -l`

if [ 0 == $count ] ; then

   sleep 60

   nohup ${FLUME_HOME}/bin/flume-ng agent -n logserver -f ${FLUME_HOME}/logserver.conf &

else

   echo "Flume Start Succses"

fi

注意:

1. 脚本中JAVA_HOMEFLUME_HOME路径根据环境进行配置

 

3. 编写flume关闭脚本

使用vim编辑logreceiver_stopflume.sh脚本。

#!/bin/sh

#FLUME进程名

NAME=Application

echo $NAME

ID=`ps -ef | grep "$NAME" | grep -v "$0" | grep -v "grep" | awk '{print $2}'`

   echo $ID

   for id in $ID

   do

     kill -9 $id

     echo "Kill Flume Process $id"

   done

   sleep 10

      count=`ps -ef |grep "$NAME" |grep -v "grep" |wc -l`

      #echo $count

      if [ 0 != $count ];then

        ID=`ps -ef | grep "$NAME" | grep -v "$0" | grep -v "grep" | awk '{print $2}'`

        echo $ID

        for id in $ID

        do

            kill -9 $id

            echo "Kill Flume Process $id"

        done

      fi

 

4、配置定时启动脚本任务

2) 执行crontab -e命令,进入crontab定时配置文件中,添加内容

50 15 * * * /home/super/software/apache-flume-1.6.0-bin/logreceiver_startflume.sh

57 15 * * * /home/super/software/apache-flume-1.6.0-bin/logreceiver_stopflume.sh

注意:

1.基本格式为:* * * * * command

命令 

2.脚本的路径应为绝对路径!

  • 2
    点赞
  • 10
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值