【Kafka】（六）Flume读取数据连接到Kafka，利用Kafka Stream实现数据清洗

需求：
Flume将数据加载进入Kafka的流输入topic后，利用Kafka Stream将数据清洗后写入流输入topic。

案例一：处理user_friends.csv数据

一、Flume读取数据连接到Kafka

创建Topic

[root@hadoop001 ~]# kafka-topics.sh --zookeeper 192.168.247.201:2181 --create --topic user_friends_raw  --partitions 1 --replication-factor 1

编写 userFriend-flume-kafka.conf

vi /opt/flume/conf/jobkb09/userFriend-flume-kafka.conf

内容如下：

user_friend.sources=userFriendSource
user_friend.channels=userFriendChannel
user_friend.sinks=userFriendSink

user_friend.sources.userFriendSource.type=spooldir
user_friend.sources.userFriendSource.spoolDir=/opt/flume/conf/jobkb09/dataSourceFile/userFriend
user_friend.sources.userFriendSource.deserializer=LINE
user_friend.sources.userFriendSource.deserializer.maxLineLength=320000
user_friend.sources.userFriendSource.includePattern=userFriend_[0-9]{
   4}-[0-9]{
   2}-[0-9]{
   2}.csv
user_friend.sources.userFriendSource.interceptors=head_filter
user_friend.sources.userFriendSource.interceptors.head_filter.type=regex_filter
user_friend.sources.userFriendSource.interceptors.head_filter.regex=^user,friends*
user_friend.sources.userFriendSource.interceptors.head_filter.excludeEvents=true
          
user_friend.channels.userFriendChannel.type=file
user_friend.channels.userFriendChannel.checkpointDir=/opt/flume/conf/jobkb09/checkPointFile/userFriend
user_friend.channels.userFriendChannel.dataDirs=/opt/flume/conf/jobkb09/dataChannelFile/userFriend

user_friend.sinks.userFriendSink.type=org.apache.flume.sink.kafka.KafkaSink
user_friend.sinks.userFriendSink.batchSize=640
user_friend.sinks.userFriendSink.brokerList=192.168.247.201:9092
user_friend.sinks.userFriendSink.topic=user_friends_raw

user_friend.sources.userFriendSource.channels=userFriendChannel
user_friend.sinks.userFriendSink.channel=userFriendChannel

启动flume agent

cd /opt/flume
./bin/flume-ng agent --name user_friend --conf ./conf/ --conf-file ./conf/jobkb09/userFriend-flume-kafka.conf -Dflume.root.logger=INFO,console

复制user_friends.csv

cd opt/flume/conf/jobkb09/
cp ./tmp/user_friends.csv /opt/flume/conf/jobkb09/dataSourceFile/userFriend/userFriend_2020-12-08.csv

查看消息队列中每个分区中的数量

kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.247.201:9092 --topic user_friends_raw --time -1 --offsets 1

消费消息

kafka-console-consumer.sh --topic user_friends_raw --bootstrap-server 192.168.247.201:9092 --</