flume整合kafka+zookeeper

1、启动flume

 bin/flume-ng agent --conf conf/ -f conf/flume.conf -n producer

下面试flume.conf的配置文件

#memory channel called ch1 on agent1

producer.channels.channel1.type = memory

 

# Define an Avro source called avro-source1 on agent1 and tell it

# to bind to 0.0.0.0:41414. Connect it to channel ch1.

producer.sources.source1.channels = channel1

producer.sources.source1.type = syslogudp

producer.sources.source1.bind = 127.0.0.1

producer.sources.source1.port = 41414

 

# Define a logger sink that simply logs all events it receives

# and connect it to the other end of the same channel.

producer.sinks.sink1.channel = channel1

producer.sinks.sink1.type = org.apache.flume.plugins.KafkaSink

producer.sinks.sink1.metadata.broker.list=127.0.0.1:9092

producer.sinks.sink1.custom.partition.key=kafkaPartition

producer.sinks.sink1.custom.topic.name=kafkaTopic

producer.sinks.sink1.serializer.class=kafka.serializer.StringEncoder

 

# Finally, now that we've defined all of our components, tell

# agent1 which ones we want to activate.

producer.channels = channel1

producer.sources = source1

producer.sinks = sink1

2、启动kafka

bin/zookeeper-server-start.sh config/zookeeper.properties--启动zookeeper

bin/kafka-server-start.sh config/server.properties--启动kafkaServer

3、执行echo 'wo l g q .' |nc -u localhost 41414 把信息输出到41414端口

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning   --启动consumer查看是否接受到信息

项目核心代码结构截图

   项目模块依赖

特别提醒：开发人员在开发的时候可以将自己的业务REST服务化或者Dubbo服务化

2.    项目依赖介绍

   2.1 后台管理系统、Rest服务系统、Scheculer定时调度系统依赖如下图：

 

       2.2 Dubbo独立服务项目依赖如下图：

 

3.  项目功能部分截图：

 

 

 

 

 

 

 

zookeeper、dubbo服务启动 

 

 

dubbo管控台 

 

 

 

 

 

 

 REST服务平台

 

 

 

 

 

转载于:https://my.oschina.net/qiuwenshuo/blog/735840