1、释疑 kafka streams api和普通 consumer producer区别
https://stackoverflow.com/questions/44014975/kafka-consumer-api-vs-streams-api
2、kafka streams foreach操作如何进行数据库读写
3、kafka streams 如何指定分区 offset进行消费
4、kafka streams 如何获取topic-partition offset
5、kafka streams 如何在重新执行之前获取偏移量限制,并且如何取消限制
1、streams api 的疑惑
1、释疑 kafka streams api和普通 consumer producer区别
https://stackoverflow.com/questions/44014975/kafka-consumer-api-vs-streams-api
2、kafka streams foreach操作如何进行数据库读写
3、kafka streams 如何指定分区 offset进行消费
这位哥说的意思没有现成方法可用,在kafka命令行操作 这不太现实啊
4、kafka streams 如何获取topic-partition offset
https://stackoverflow.com/questions/40807346/how-can-i-get-the-offset-value-in-kstream
https://stackoverflow.com/questions/48333706/how-to-get-current-kafka-topic-inside-kafka-stream?rq=1
ok 获取倒是能获取 问题是无法指定位置消费
public static void main(final String[] args) {
try {
final Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streamProcessor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.STATE_DIR_CONFIG, "state-store");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10 * 1000);
props.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
StreamsBuilder streamsBuilder = new StreamsBuilder();
final KStream<String, String> textLines = streamsBuilder.stream(inputTopicList);
final KStream<String, String> textLines = builder.stream(inputTopiclist);
textLines.transform(getTopicDetailsTransformer::new)
.foreach(new ForeachAction<String, String>() {
public void apply(String key, String value) {
System.out.println(key + ": " + value);
}
});
textLines.to(outputTopic);
} catch (Exception e) {
System.out.println(e);
}
}
private static class getTopicDetailsTransformer implements Transformer<String, String, KeyValue<String, String>> {
private ProcessorContext context;
@Override
public void init(final ProcessorContext context) {
this.context = context;
}
public KeyValue<String, String> transform(final String recordKey, final String recordValue) {
//here i am returning key as topic name.
return KeyValue.pair(context.topic(), recordValue);
}
@Override
public void close() {
// Not needed.
}
}
5、kafka streams 如何在重新执行之前获取偏移量限制,并且如何取消限制
和3类似 也是没有解决方案
感觉kafka 流处理问题很多 一点也不友好
6、kafka 如何处理过期消息,对于消费者的影响
7、Kafka consumer 消费了一些已经消费的消息