Flume和Kafka的整合完成实时数据采集

1.Flume和Kafka整合框架

2.写FLUME的agent配置文件

exec-memory-avro.conf

exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel


exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /home/bigdata/data/data.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c


exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = bigdata
exec-memory-avro.sinks.avro-sink.port = 44444


exec-memory-avro.channels.memory-channel.type = memory


exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

avro-memory-kafka.conf

avro-memory-kafka.sources = avro-source
avro-memory-kafka.sinks = kafka-sink
avro-memory-kafka.channels = memory-channel


avro-memory-kafka.sources.avro-source.type = avro
avro-memory-kafka.sources.avro-source.bind = hadoop000
avro-memory-kafka.sources.avro-source.port = 44444


avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.brokerList = hadoop000:9092
avro-memory-kafka.sinks.kafka-sink.topic = hello_topic
avro-memory-kafka.sinks.kafka-sink.batchSize = 5
avro-memory-kafka.sinks.kafka-sink.requiredAcks =1 


avro-memory-kafka.channels.memory-channel.type = memory


avro-memory-kafka.sources.avro-source.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel

3.启动框架

（1）启动zookeeper

（2）启动kafka

（3）先启动FLUME的avro-memory-kafka.conf，再启动exec-memory-avro.conf

（4）启动kafkaconsumer

4.测试

echo hello world >>data.log

在consumer控制台上输出，搭建成功