flume 实时读取本地文件到kafka

本文写的比较简单，因为都大同小异，如果忘记flume的使用和语法等，可以看下如下文章，回顾下
flume 实时读取本地文件到hdfs
https://blog.csdn.net/qq_41712271/article/details/103939224

#Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /opt/test.log
a1.sources.r1.shell = /bin/bash -c
 
# kafka sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = topic-1
a1.sinks.k1.kafka.bootstrap.servers = cdh001:9092,cdh002:9092,cdh003:9092
# 一批中要处理多少条消息。较大的批次可提高吞吐量，同时增加延迟。
a1.sinks.k1.kafka.flumeBatchSize = 6000
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.linger.ms = 1
a1.sinks.ki.kafka.producer.compression.type = snappy
 
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
 
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

运行flume
./bin/flume-ng agent -c conf/ -f /usr/local/txt_kafka.conf -n a1

测试，往文件中追加记录，运行kafka的消费端，查看消费的情况
echo $(date "+%Y-%m-%d %H:%M:%S") >> /opt/test.log
./kafka-console-consumer.sh --bootstrap-server cdh001:9092,cdh002:9092,cdh003:9092 --from-beginning --topic topic-1