超级会员免费看
本文主要讨论了Storm在消费Kafka主题时遇到的问题，包括如何设置从Kafka的最早或最新消息开始消费，以及在故障恢复时如何依赖ZooKeeper中的offset信息。强调了重新部署时保持SpoutConfig.id不变以避免数据丢失或意外行为的重要性，并介绍了SpoutConfig中的关键配置参数及其作用。
问题描述：
kafka是之前早就搭建好的，新建的storm集群要消费kafka的主题，由于kafka中已经记录了很多消息，storm消费时从最开始消费
问题解决：
下面是摘自官网的一段话：
How KafkaSpout stores offsets of a Kafka topic and recovers in case of failures
As shown in the above KafkaConfig properties, you can control from where in the Kafka topic the spout begins to read by setting KafkaConfig.startOffsetTime as follows:
kafka.api.OffsetRequest.EarliestTime(): read from the beginning of the topic (i.e. from the oldest messages onwards)
kafka.api.OffsetRequest.LatestTime(): read from the end of the topic (i.e. any new messsages that are being written to the topic)
A Unix timestamp aka seconds since the epoch (e.g. via System.currentTimeMil
kafka是之前早就搭建好的，新建的storm集群要消费kafka的主题，由于kafka中已经记录了很多消息，storm消费时从最开始消费
问题解决：
下面是摘自官网的一段话：
How KafkaSpout stores offsets of a Kafka topic and recovers in case of failures
As shown in the above KafkaConfig properties, you can control from where in the Kafka topic the spout begins to read by setting KafkaConfig.startOffsetTime as follows:
kafka.api.OffsetRequest.EarliestTime(): read from the beginning of the topic (i.e. from the oldest messages onwards)
kafka.api.OffsetRequest.LatestTime(): read from the end of the topic (i.e. any new messsages that are being written to the topic)
A Unix timestamp aka seconds since the epoch (e.g. via System.currentTimeMil
订阅专栏 解锁全文
扫一扫
6万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
