保证storm和kafka上的消息有序
- 整体流程:kafka生产者->kafka消费者->spout->bolt:
- kafka的生产者按制定字段发送到指定主题上,保证一类消息的先后次序(如果想要所有的消息都有序,可以只用一个分区,但是这样就只能有一个消费者消费)
producer = kafkaClientService.< String, String > newKafkaProducer( new Properties() );
//指定主题名称和字段
producer.send( new ProducerRecord < String, String >( topicName, key, jsonStr ) );
- kafka的消费者(在storm的spout中)也同样按序拉取消息,无论一个消费者消费几个分区都能保证消息有序
consumer = kafkaClientService.< String, String > newKafkaConsumer( groupId, properties );
List < String > topics = new ArrayList < String >();
topics.add( topicName );
consumer.subscribe( topics );
ConsumerRecords < String, String > records = consumer.poll( 30000 );
for ( ConsumerRecord < String, String > record : records ) {
//这里的key是特定字段,与kafka的分区字段一致
collector.emit( new Values( record.key(), record.value() ) );
}
topologyBuilder.setBolt( ConstData.PROCESSOR_BOLT, new ProcessorBolt(), processorBoltParallel )
.fieldsGrouping( ConstData.SPOUT, filedName )