Flume
Flume作用:收集日志
组件下载地址:http://archive.apache.org/dist/flume/1.6.0/
版本:1.6.0以上集成Kafka
Flume配置
a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type = syslogudp
a1.sources.r1.port = 514
a1.sources.r1.host = 0.0.0.0
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.brokerList=10.2.4.12:9092,10.2.4.13:9092,10.2.4.14:9092
a1.sinks.k1.topic=test_rce_yjd
a1.sinks.k1.batchSize=20
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
启动命令
/flume/apache-flume-1.6.0-bin/bin/flume-ng agent --conf conf --conf-file /flume/apache-flume-1.6.0-bin/conf/syslog.conf --name a1 -Dflume.root.logger=INFO,console
Kafka
Kafka作用:缓存日志
监听是否发送至集群中
(1)shell 传送门:http://blog.csdn.net/yl3395017/article/details/73650250
(2)消费 传送门:http://blog.csdn.net/yl3395017/article/details/77547881
Storm
Strom作用:流式处理
(1)消费Kafka集群信息 传送门:http://blog.csdn.net/yl3395017/article/details/77477716
效果