flume读取kafka消息并存储到HDFS上出现异常Attempt to heart beat failed since member id is not valid, reset it and t

flume读取kafka消息并存储到HDFS上出现异常Attempt to heart beat failed since member id is not valid, reset it and try to re-jo

1 报错情景:

flume根据kafka不同的topic进行收集信息,但是启动flume发现flume日志报错: Attempt to heart beat failed since member id is not valid, reset it and try to re-jo

在这里插入图片描述：

2 原因:

由于f1,f2监听kafak时，需要接收kafka订阅的topic里面的信息,所以需要配置group.id,则f1,f2默认groupId都是"flume"
,f1,f2属于同一个消费者组(consumerGroup),在我们这里是订阅不同的topic，所以需要指定groupId为不同的topic。

3 解决办法:

为两个flume配置不同的group.id即可:

a1.sources.r1.type=org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.kafka.bootstrap.servers=Hadoop102:9092，Hadoop103:9092，Hadoop104:9092
a1.sources.r1.kafka.topics=topic_start
a1.sources.r1.kafka.consumer.group.id=group_start

a2.sources.r2.type=org.apache.flume.source.kafka.KafkaSource
a2.sources.r2.kafka.bootstrap.servers=Hadoop102:9092，Hadoop103:9092，Hadoop104:9092
a2.sources.r2.kafka.topics=topic_event
a2.sources.r2.kafka.consumer.group.id=group_event