flume集成kafka

2019独角兽企业重金招聘Python工程师标准>>>

1.kafka的配置参照

https://my.oschina.net/u/1591525/blog/2251910

2.flum配置 

在flume的conf目录下新建 kafka.properties

agent.sources = s1                                                                                                                   

agent.channels = c1                                                                                                                  

agent.sinks = k1                                                                                                                                                                                                                                            

agent.sources.s1.type=exec                                                                                                           

agent.sources.s1.command=tail -F /tmp/logs/kafka.log                                                                                 

agent.sources.s1.channels=c1                                                                                                         

agent.channels.c1.type=memory                                                                                                        

agent.channels.c1.capacity=10000                                                                                                     

agent.channels.c1.transactionCapacity=100                                                                                                                                                                                                                           

#设置Kafka接收器                                                                                                                     

agent.sinks.k1.type= org.apache.flume.sink.kafka.KafkaSink                                                                           

#设置Kafka的broker地址和端口号                                                                                                       

agent.sinks.k1.brokerList=master:9092                                                                                                

#设置Kafka的Topic                                                                                                                    

agent.sinks.k1.topic=kafkatest                                                                                                       

#设置序列化方式                                                                                                                      

agent.sinks.k1.serializer.class=kafka.serializer.StringEncoder                                                                                                                                                                                    

agent.sinks.k1.channel=c1 

3.连接测试

启动kafka和zk

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties &

bin/kafka-server-start.sh -daemon config/server.properties &

启动flume

bin/flume-ng agent -n agent  -c conf  -f  conf/kafka.properties

kafka生产者和消费者参照springboot和kafka集成配置

 

转载于:https://my.oschina.net/u/1591525/blog/2251989