一、准备要传输的文件
二、配置对应的conf文件
每个配置conf文件配对相应的文件,这里拿events.conf举例
events.sources=eventsSource
events.channels=eventsChannel
events.sinks=eventsSink
events.sources.eventsSource.type=spooldir
//文件要传输到的目录,在此目录上读取
events.sources.eventsSource.spoolDir=/opt/flumelogfile/events
events.sources.eventsSource.deserializer=LINE
events.sources.eventsSource.deserializer.maxLineLength=320000
//此行表示传入的文件必须满足这种格式才会读取
events.sources.eventsSource.includePattern=events_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
events.sources.eventsSource.interceptors=head_filter
events.sources.eventsSource.interceptors.head_filter.type=regex_filter
//不包含event_id开头的一行,去除首行
events.sources.eventsSource.interceptors.head_filter.regex=^event_id*
events.sources.eventsSource.interceptors.head_filter.excludeEvents=true
events.channels.eventsChannel.type=file
events.channels.eventsChannel.checkpointDir=/opt/flumelogfile/checkpoint/events
events.channels.eventsChannel.dataDirs=/opt/flumelogfile/data/events
events.sinks.eventsSink.type=org.apache.flume.sink.kafka.KafkaSink
events.sinks.eventsSink.batchSize=640
//kafka对应的broker地址
events.sinks.eventsSink.brokerList=192.168.91.180:9092
//kafka对应的topic主题
events.sinks.eventsSink.topic=events
events.sources.eventsSource.channels=eventsChannel
events.sinks.eventsSink.channel=eventsChannel
三、启动flume
在/opt/soft/flume190目录下启动以下命令
./bin/flume-ng agent --name userfriends --conf ./conf/ --conf-file ./conf/events/uf.conf -Dflume.root.logger=INFO,console
出现以下图片则启动成功
四、创建topic主题,启动kafka消费者
创建主题
注意:副本数不能大于broker数
kafka-topics.sh --create --zookeeper 192.168.91.180:2181 --topic events --partitions 1 --replication-factor 1
启动消费者
kafka-console-consumer.sh --bootstrap-server 192.168.91.180:9092 --topic events
五、将传输的文件cp到conf配置文件指定的目录
cp ./events.csv /opt/flumelogfile/ea/events_2023-04-02.csv
传输到指定文件后,kafka消费者就会读取到对应的文件