在使用Spark Streaming消费kafka中的数据时, 官方推荐将offset信息存储在kafka中
stream.foreachRDD { rdd =>
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
// some time later, after outputs have completed
stream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)
}
但是kafka不支持事务,这样可能会出现问题, 若spark当前批次开始处理时, 上一批次的offset没有更新完或更新失败, 就会发生重复消费的状况。
因此可以利用数据库的事务, 将业务逻辑及更新offset一起放在事务中(数据存放到哪里, offset就存放在哪里), 利用了事务的原子性、幂等性保证避免了数据重复消费、没有消费的情况。
val fromOffsets = selectOffsetsFromYourDatabase.map { resultSet =>
new TopicPartition(resultSet.string("topic"), resultSet.int("partition")) -> resultSet.long("offset")
}.toMap
val stream = KafkaUtils.createDirectStream[String, String](
streamingContext,
PreferConsistent,
Assign[String, String](fromOffsets.keys.toList, kafkaParams, fromOffsets)
)
stream.foreachRDD { rdd =>
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
val results = yourCalculation(rdd)
// begin your transaction
// update results
// update offsets where the end of existing offsets matches the beginning of this batch of offsets
// assert that offsets were updated correctly
// end your transaction
}