Flume+Kafka整合

1.下载flume-kafka-plus:https://github.com/beyondj2ee/flumeng-kafka-plugin

在该项目中提取以下jar包放入环境中flume的lib下

还有package目录的flumeng-kafka-plugin.jar包一并放到flume的lib目录下

flume配置文件：

 

#agent
producer.sources = s
producer.channels = c
producer.sinks = r

#source
producer.sources.s.type = spooldir
producer.sources.s.spoolDir = /data/logs/data-schedule
producer.sources.s.channels = c

producer.sinks.r.type = org.apache.flume.plugins.KafkaSink
producer.sinks.r.metadata.broker.list=127.0.0.1:9092
producer.sinks.r.partition.key=0
producer.sinks.r.partitioner.class=org.apache.flume.plugins.SinglePartition
producer.sinks.r.serializer.class=kafka.serializer.StringEncoder
producer.sinks.r.request.required.acks=0
producer.sinks.r.max.message.size=1000000
producer.sinks.r.producer.type=sync
producer.sinks.r.custom.encoding=UTF-8
producer.sinks.r.custom.topic.name=test

#Specify the channel for the sink
producer.sinks.r.channel = c

producer.channels.c.type = memory
producer.channels.c.capacity = 1000

 

启动flume：

./flume-ng agent --conf ../conf --conf-file ../conf/flume2.conf --name producer -Dflume.root.logger=INFO,console