一. flume安装配置
1. flume下载:
http://archive.apache.org/dist/flume/
2. fulme部署:上传rz,解压:tar zxvf
apache-flume-1.7.0-bin.tar.gz
二. flume的kafka配置
1. 配置Kafka信息
新建kafka-conf.properties,上传至flume的apache-flume-1.7.0-bin/conf目录下,配置内容如下:
#client
agent.channels=ch1
agent.sources=src1
agent.sinks=sk1
#define source monitor a logfile
agent.sources.src1.type=exec
agent.sources.src1.command=
tail -F /data/jsp/log_producer/logs/producer.log
agent.sources.src1.channels=ch1
agent.channels.ch1.type=memory
agent.channels.ch1.capacity=10000
agent.channels.ch1.transactionCapacity=100
#define kafka receiver
agent.sinks.sk1.type=org.apache.flume.sink.kafka.KafkaSink
agent.sinks.sk1.brokerList=
ip1:9092,ip2:9092,ip3:9092
agent.sinks.sk1.topic=
kafkatest
agent.sinks.sk1.serializer.class=kafka.serializer.StringEncoder
agent.sinks.sk1.channel=ch1
agent.sinks.sk1.batchSize=20
注意:信息字段名称配置需仔细
需要注意以下三个重要配置:
agent.sources.src1.command=tail -F /data/jsp/log_producer/logs/producer.log
tail监听日志文件
agent.sinks.sk1.brokerList=ip1:9092,ip2:9092,ip3:9092
kafka集群配置
agent.sinks.sk1.topic=kafkatest
flume向kafkatest主题push数据
2. 配置JAVA
修改cong目录下的flume-env.sh,主要配置参数JAVA_HOME和JAVA_OPTS,配置内容如下:
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.
# Enviroment variables can be set here.
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
# export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
# Let Flume write raw event data and configuration information to its log files for debugging
# purposes. Enabling these flags is not recommended in production,
# as it may result in logging sensitive user information or encryption secrets.
export JAVA_OPTS="-Xms1024m -Xmx1024m -Xss256k -Xmn512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"
# Note that the Flume conf directory is always included in the classpath.
#FLUME_CLASSPATH=""
#set JAVA_HOME
至此flume的kafka配置完毕!
3. flume启动
flume启动命令: ./flume-ng agent --conf-file
/opt/apache-flume-1.7.0-bin/conf/kafka-conf.properties -c
/opt/apache-flume-1.7.0-bin/conf/ --name agent -Dflume.root.logger=DEBUG,console
注意:命令中绿色表示部分用的是绝对路径
4. flume测试脚本
(1) 测试脚本,模拟生产日志记录,保存为producer_log.sh,用chmod赋执行权限,执行命令./producer_log.sh,会向producer.log写日志。脚本编码如下:
---------------------------------------------------------------------------------------------------------------
for
((i=0;i<=1000;i++));
do
echo
"kafka_test-"
+$i>>
/data/jsp/log_producer/logs/producer.log
;
done
--------------------------------------------------------------------------------------------------------------
日志内容样式:
kafka_fulume_test-+1
5. flume与kafka互动测试
启动ZK和Kafka,在主题kafkatest,启动消费者服务,详见《Kafka-安装部署》
创建主题: ./kafka-topics.sh --create --zookeeper ip:2181 --replication-factor 1 --partitions 1 --topic
kafkatest
启动消费者:./kafka-console-consumer.sh --
zookeeper ip1:2181,ip2:2181,ip3:2181 --topic
kafkatest --from-beginning
消费者服务端console会打印日志内容!