flume与kafka整合

原创 2018年04月17日 10:06:31

一. flume安装配置

2. fulme部署:上传rz,解压:tar zxvfapache-flume-1.7.0-bin.tar.gz

二. flume的kafka配置

1. 配置Kafka信息

   新建kafka-conf.properties,上传至flume的apache-flume-1.7.0-bin/conf目录下,配置内容如下:

#client
agent.channels=ch1
agent.sources=src1
agent.sinks=sk1

#define source monitor a logfile
agent.sources.src1.type=exec
agent.sources.src1.command=tail -F /data/jsp/log_producer/logs/producer.log
agent.sources.src1.channels=ch1

agent.channels.ch1.type=memory
agent.channels.ch1.capacity=10000
agent.channels.ch1.transactionCapacity=100

#define kafka receiver
agent.sinks.sk1.type=org.apache.flume.sink.kafka.KafkaSink
agent.sinks.sk1.brokerList=ip1:9092,ip2:9092,ip3:9092
agent.sinks.sk1.topic=kafkatest
agent.sinks.sk1.serializer.class=kafka.serializer.StringEncoder
agent.sinks.sk1.channel=ch1
agent.sinks.sk1.batchSize=20注意:信息字段名称配置需仔细

需要注意以下三个重要配置:
agent.sources.src1.command=tail -F /data/jsp/log_producer/logs/producer.log tail监听日志文件
agent.sinks.sk1.brokerList=ip1:9092,ip2:9092,ip3:9092 kafka集群配置
agent.sinks.sk1.topic=kafkatest flume向kafkatest主题push数据

2. 配置JAVA

    修改cong目录下的flume-env.sh,主要配置参数JAVA_HOME和JAVA_OPTS,配置内容如下:

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.

# Enviroment variables can be set here.
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
# export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

# Let Flume write raw event data and configuration information to its log files for debugging
# purposes. Enabling these flags is not recommended in production,
# as it may result in logging sensitive user information or encryption secrets.
export JAVA_OPTS="-Xms1024m -Xmx1024m -Xss256k -Xmn512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"

# Note that the Flume conf directory is always included in the classpath.
#FLUME_CLASSPATH=""
#set JAVA_HOME

至此flume的kafka配置完毕!

3. flume启动

flume启动命令: ./flume-ng agent --conf-file/opt/apache-flume-1.7.0-bin/conf/kafka-conf.properties -c /opt/apache-flume-1.7.0-bin/conf/ --name agent -Dflume.root.logger=DEBUG,console注意:命令中绿色表示部分用的是绝对路径

4. flume测试脚本

(1) 测试脚本,模拟生产日志记录,保存为producer_log.sh,用chmod赋执行权限,执行命令./producer_log.sh,会向producer.log写日志。脚本编码如下:
---------------------------------------------------------------------------------------------------------------
for((i=0;i<=1000;i++));
do echo "kafka_test-"+$i>>/data/jsp/log_producer/logs/producer.log;
done
--------------------------------------------------------------------------------------------------------------
日志内容样式:kafka_fulume_test-+1

5. flume与kafka互动测试

    启动ZK和Kafka,在主题kafkatest,启动消费者服务,详见《Kafka-安装部署》
创建主题: ./kafka-topics.sh --create --zookeeper ip:2181 --replication-factor 1 --partitions 1 --topic kafkatest
启动消费者:./kafka-console-consumer.sh --zookeeper ip1:2181,ip2:2181,ip3:2181 --topic kafkatest --from-beginning
消费者服务端console会打印日志内容!


Flume与Kafka整合案例详解

环境配置 名称 版本 下载地址 Centos 7.0 64x 百度 Zookeeper 3.4.5 Flume 1.6.0 Kafka 2.1...
  • qq_26840065
  • qq_26840065
  • 2017-04-14 16:52:41
  • 8143

大数据———Flume与Kafka整合

环境配置 名称 版本 下载地址 Centos 7.0 64x 百度 Flume 1.8.0 http://flume.apache.org/dow...
  • qq_37518574
  • qq_37518574
  • 2018-01-18 15:31:21
  • 202

kafka和flume整合

Kafka作为source:配置文件:**#定义各个模块** a1.sources = kafka a1.sinks = log a1.channels = c1#配置kafka source #s...
  • Anonymous_cx
  • Anonymous_cx
  • 2016-05-19 20:47:09
  • 805

Flume和Kafka整合安装

版本号: RedHat6.5   JDK1.8    flume-1.6.0   kafka_2.11-0.8.2.1 1.flume安装 RedHat6.5安装单机flume1.6...
  • chongxin1
  • chongxin1
  • 2017-07-26 09:23:58
  • 868

Kafka flume 整合

前提 前提是要先把flume和kafka独立的部分先搭建好。 下载插件包 下载flume-kafka-plus:https://github.com/beyondj2ee/flumeng-kaf...
  • czw698
  • czw698
  • 2015-04-16 17:00:30
  • 5143

Flume和Kafka整合

采集单元选用模块化程度极高的 Flume-ng,传输单元选择高吞吐率的 Kafka,将两者结合共同构成分布式计算集群的基础数据输入组件。 材料准备: Zookeeper - 3.4.6 Flum...
  • u012689336
  • u012689336
  • 2016-10-06 22:37:06
  • 1532

flume整合kafka之kafka接收flume数据

1.创建配置文件 基本配置如下: #source a1.sources = sysSrc a1.sources.sysSrc.type=avro a1.sources.sysSrc.b...
  • beExcellentOne
  • beExcellentOne
  • 2016-11-10 13:46:01
  • 1839

flume整合kafka和hdfs

flume版本:1.7.0 kafka版本:2.11-0.10.1.0 hadoop 版本:2.6.0 最近在玩这个flume和kafka这两个东西,网上有很多这方面的简介,我就不多说了,我的理解为啥...
  • whoami_zy
  • whoami_zy
  • 2016-12-20 13:26:16
  • 2443

log4j+flume+kafka+strom整合

log4j+flume+kafka+strom整合   整合流程图。 第一步  整合log4j和flume(主机名为hadoop0) (1) 安装flume在/usr/so...
  • liuxiangke0210
  • liuxiangke0210
  • 2015-12-31 09:56:24
  • 1276

flume+kafka+storm整合00

一、安装 flume,kafka, storm 的安装在下面三篇文章: flume:1.6.0 kafka:注意这里最好下载scala2.10版本的kafka,因为scala2.10版本的兼容性...
  • wuxintdrh
  • wuxintdrh
  • 2017-03-19 11:40:46
  • 690
收藏助手
不良信息举报
您举报文章:flume与kafka整合
举报原因:
原因补充:

(最多只允许输入30个字)