flume读取kafka消息并存储到HDFS上出现异常Attempt to heart beat failed since member id is not valid, reset it and try to re-jo
1 报错情景:
flume根据kafka不同的topic进行收集信息,但是启动flume发现flume日志报错: Attempt to heart beat failed since member id is not valid, reset it and try to re-jo
在这里插入图片描述:
2 原因:
由于f1,f2监听kafak时,需要接收kafka订阅的topic里面的信息,所以需要配置group.id,则f1,f2默认groupId都是"flume"
,f1,f2属于同一个消费者组(consumerGroup),在我们这里是订阅不同的topic,所以需要指定groupId为不同的topic。
3 解决办法:
为两个flume配置不同的group.id即可:
a1.sources.r1.type=org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.kafka.bootstrap.servers=Hadoop102:9092,Hadoop103:9092,Hadoop104:9092
a1.sources.r1.kafka.topics=topic_start
a1.sources.r1.kafka.consumer.group.id=group_start
a2.sources.r2.type=org.apache.flume.source.kafka.KafkaSource
a2.sources.r2.kafka.bootstrap.servers=Hadoop102:9092,Hadoop103:9092,Hadoop104:9092
a2.sources.r2.kafka.topics=topic_event
a2.sources.r2.kafka.consumer.group.id=group_event