收集access.log数据到kafka集群
如果必要
清除原来的日志信息
xcall.sh "echo > /usr/local/openresty/nginx/logs/access.log"
开zk
开redis
![](https://img-blog.csdnimg.cn/2022010701423095343.png)
开kafka
![](https://img-blog.csdnimg.cn/2022010701423134571.png)
1 .kafka 主题
创建主题
kafka-topics.sh --zookeeper s102:2181 --partitions 4 --replication-factor 3 --create --topic big12-umeng-raw-logs
![](https://img-blog.csdnimg.cn/2022010701423128679.png)
查看主题
kafka-topics.sh --zookeeper s102:2181 --list
![](https://img-blog.csdnimg.cn/2022010701423193667.png)
创建kafka消费者 ----此步可以省略
kafka-console-consumer.sh --zookeeper s102:2181 --topic big12-umeng-raw-logs
![](https://img-blog.csdnimg.cn/2022010701423152478.png)
2 配置flume,收集access.log文件到kafka集群
![](https://img-blog.csdnimg.cn/2022010701423155818.png)
/soft/flume/conf sudo nano umeng_nginx_to_kafka.conf
flume配置文件
![](https://img-blog.csdnimg.cn/2022010701423250900.png)
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /usr/local/openresty/nginx/logs/access.log
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = big12-umeng-raw-logs
a1.sinks.k1.kafka.bootstrap.servers = s102:9092
a1.sinks.k1.kafka.flumeBatchSize = 20
#acks 0:不需要回执 1:leader写入磁盘后回执 -1:所有节点写入磁盘后回执
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.linger.ms = 0
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1
发送配置文件到s102
![](https://img-blog.csdnimg.cn/2022010701423269201.png)
![](https://img-blog.csdnimg.cn/2022010701423240272.png)
3 .启动flume
s101-s102启动flume
flume-ng agent -f /soft/flume/conf/umeng_nginx_to_kafka.conf -n a1 &
![](https://img-blog.csdnimg.cn/2022010701423229140.png)