Flume是一个分布式、可靠、和高可用的海量日志采集、聚合和传输的系统
逻辑架构
环境准备
hostname IP flume安装目录
owl5 agent /opt/flume
owl6 agent /opt/flume
owl8 10.1.1.8 collector /opt/flume 监听端口4545
目标: 把owl5 owl6 服务器tomcat日志合并到 owl8 /bak/log/owl/tomcat/目录
一. 服务器端 owl8 安装 flume
1. 下载安装
下载地址 http://mirrors.cnnic.cn/apache/flume/
wget http://mirrors.cnnic.cn/apache/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz
tar zxf apache-flume-1.6.0-bin.tar.gz
mv apache-flume-1.6.0-bin /opt/flume/
2. 配置环境变量
vim /etc/profile
export FLUME_HOME=/opt/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export PATH=$JAVA_HOME/bin:/opt/nginx/sbin:$FLUME_HOME/bin:$PATH
使配置生效 . /etc/profile
验证是否安装成功 flume-ng version
3. 配置启动内存
cp /opt/flume/conf/flume-env.ps1.template /opt/flume/conf/flume-env.sh
vim /opt/flume/conf/flume-env.sh 增加
JAVA_HOME=/usr/java/jdk1.7.0_80
JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"
4. 创建配置文件 /opt/flume/conf/avro_tomcat.conf
创建日志存放目录
mkdir -p /bak/logs/owl/tomcat
创建配置文件
vim /opt/flume/conf/avro_tomcat.conf
collector1.sources = AvroIn
collector1.sources.AvroIn.type = avro
collector1.sources.AvroIn.bind = 0.0.0.0
collector1.sources.AvroIn.port = 4545 # 定义服务端端口 4545
collector1.sources.AvroIn.channels = mc1
collector1.channels = mc1
collector1.channels.mc1.type = memory
collector1.channels.mc1.capacity = 100
collector1.channels.mc1.transactionCapacity = 100
collector1.sinks = LocalOut
collector1.sinks.LocalOut.type = file_roll
collector1.sinks.LocalOut.sink.directory = /bak/logs/owl/tomcat
collector1.sinks.LocalOut.sink.rollInterval = 0
collector1.sinks.LocalOut.channel = mc1
二. owl5 owl6 安装flume
创建配置文件
vim /opt/flume/conf/tomcat.conf
agent1.sources = ngrinder
agent1.sources.ngrinder.type = exec
agent1.sources.ngrinder.command = tail -F /bak/logs/tomcat/catalina.out # Tomcat 日志路径
agent1.sources.ngrinder.channels = mc1
agent1.channels = mc1
agent1.channels.mc1.type = memory
agent1.channels.mc1.capacity = 100
agent1.sinks = avro-sink
agent1.sinks.avro-sink.type = avro
agent1.sinks.avro-sink.channel = mc1
agent1.sinks.avro-sink.hostname = 10.1.1.8
agent1.sinks.avro-sink.port = 4545
三. 启动flume
1. owl8 启动 flume
nohup flume-ng agent -c /opt/flume/conf -f /opt/flume/conf/avro_tomcat.conf -n collector1 -Dflume.root.logger=INFO,console >/dev/null 2>&1 &
注意 -n collector1 需要与配置文件中对应
2. owl5 owl6 启动 flume
nohup flume-ng agent -c /opt/flume/conf -f /opt/flume/conf/tomcat.conf -n agent1 >/dev/null 2>&1 &
#命令参数说明
-c /opt/flume/conf 指定配置目录为 /opt/flume/conf
-f /opt/flume/conf/tomcat.conf 指定配置文件为 /opt/flume/conf/tomcat.conf
-n agent1 指定agent名字为agent1,需要与avro.conf中的一致
-Dflume.root.logger=INFO,console 指定DEBUF模式在console输出INFO信息
官网解释 http://flume.apache.org/FlumeUserGuide.html
Property Name Default Description
capacity 100 指定channet存放的Event最大数量,根据日志量设置一个合适的值,如果采用file-channel,且磁盘充足,可以设置得大些
capacity 100 The maximum number of events stored in the channel
transactionCapacity 100 The maximum number of events the channel will take from a source or give to a sink per transaction
transactionCapacity 不要大于 capacity
3. 测试
启动flume后,会在owl8 日志目录生成类似的文件
ls -l /bak/logs/owl/tomcat
-rw-r--r-- 1 root root 193012 Mar 1 14:26 1456810019222-1
在owl5 catalina.out 执行插入一行日志,并在owl8 查看,延时大概15秒
echo owl5555555555555555555 >> /bak/logs/tomcat-v2/catalina.out