spring 集成 kafka

spring集成kafka例子链接地址：http://download.csdn.net/detail/qinyanbin123/8959339

使用spring-integration-kafka发送消息

Outbound Channel Adapter用来发送消息到Kafka。 消息从Spring Integration Channel中读取。 你可以在Spring application context指定这个channel。
一旦配置好这个Channel，就可以利用这个Channel往Kafka发消息。 明显地，Spring Integration特定的消息发送给这个Adaptor，然后发送前在内部被转为Kafka消息。当前的版本要求你必须指定消息key和topic作为头部数据 (header)，消息作为有载荷(payload)。
例如

1 2 3 4 5 6

final MessageChannel channel = ctx.getBean( "inputToKafka", MessageChannel.class); channel.send( MessageBuilder.withPayload(payload) //设置有效载荷 .setHeader( "messageKey", "key") //指定key .setHeader( "topic", "test").build()); /指定topic/

实际代码如下：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

import java.util.Random; import org.springframework.context.support.ClassPathXmlApplicationContext; import org.springframework.integration.support.MessageBuilder; import org.springframework.messaging.MessageChannel; public class Producer { private static final String CONFIG = "/context.xml"; private static Random rand = new Random(); public static void main(String[] args) { final ClassPathXmlApplicationContext ctx = new ClassPathXmlApplicationContext(CONFIG, Producer.class); ctx.start(); final MessageChannel channel = ctx.getBean( "inputToKafka", MessageChannel.class); for ( int i = 0; i < 100; i++) { channel.send(MessageBuilder.withPayload( "Message-" + rand.nextInt()).setHeader( "messageKey", String.valueOf(i)).setHeader( "topic", "test").build()); } try { Thread.sleep( 100000); } catch (InterruptedException e) { e.printStackTrace(); } ctx.close(); } }

Spring 配置文件：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi= "http://www.w3.org/2001/XMLSchema-instance" xmlns:int= "http://www.springframework.org/schema/integration" xmlns:int-kafka= "http://www.springframework.org/schema/integration/kafka" xmlns:task= "http://www.springframework.org/schema/task" xsi:schemaLocation= "http://www.springframework.org/schema/integration/kafka http://www.springframework.org/schema/integration/kafka/spring-integration-kafka.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd"> <int:channel id="inputToKafka"> <int:queue/> </int:channel> <int-kafka:outbound-channel-adapter id="kafkaOutboundChannelAdapter" kafka-producer-context-ref= "kafkaProducerContext" auto-startup= "false" channel= "inputToKafka" order= "3" > <int:poller fixed-delay="1000" time-unit="MILLISECONDS" receive-timeout="0" task-executor="taskExecutor"/> </int-kafka:outbound-channel-adapter> <task:executor id="taskExecutor" pool-size="5" keep-alive="120" queue-capacity="500"/> <bean id="producerProperties" class= "org.springframework.beans.factory.config.PropertiesFactoryBean"> <property name="properties"> <props> <prop key="topic.metadata.refresh.interval.ms">3600000 </prop> <prop key="message.send.max.retries">5 </prop> <prop key="serializer.class">kafka.serializer.StringEncoder </prop> <prop key="request.required.acks">1 </prop> </props> </property> </bean> <int-kafka:producer-context id="kafkaProducerContext" producer-properties= "producerProperties"> <int-kafka:producer-configurations> <int-kafka:producer-configuration broker-list="localhost:9092" topic= "test" compression-codec= "default"/> </int-kafka:producer-configurations> </int-kafka:producer-context> </beans>

int:channel是配置Spring Integration Channel, 此channel基于queue。
int-kafka:outbound-channel-adapter是outbound-channel-adapter对象， 内部使用一个线程池处理消息。关键是kafka-producer-context-ref。
int-kafka:producer-context配置producer列表，要处理的topic，这些Producer最终要转换成Kafka的Producer。

producer的配置参数如下：

1 2 3 4 5 6 7 8 9 10 11 12

broker-list List of comma separated brokers that this producer connects to topic Topic name or Java regex pattern of topic name compression-codec Compression method to be used. Default is no compression. Supported compression codec are gzip and snappy. Anything else would result in no compression value-encoder Serializer to be used for encoding messages. key-encoder Serializer to be used for encoding the partition key key- class-type Type of the key class. This will be ignored if no key-encoder is provided value- class-type Type of the value class. This will be ignored if no value-encoder is provided. partitioner Custom implementation of a Kafka Partitioner interface. async True/ False - default is false. Setting this to true would make the Kafka producer to use an async producer batch-num-messages Number of messages to batch at the producer. If async is false, then this has no effect.

value-encoder 和key-encoder可以是其它实现了Kafka Encoder接口的Bean。同样partitioner也是实现了Kafka的Partitioner接口的Bean。
一个Encoder的例子：

1 2 3

<bean id="kafkaEncoder" class="org.springframework.integration.kafka.serializer.avro.AvroSpecificDatumBackedKafkaEncoder"> <constructor-arg value="com.company.AvroGeneratedSpecificRecord" /> </bean>

Spring Integration Kafka 也提供了个基于Avro的Encoder。 Avro也是Apache的一个项目， 在大数据处理时也是一个常用的序列化框架。
不指定Encoder将使用Kafka缺省的Encoder (kafka.serializer.DefaultEncoder, byte[] --> same byte[])。

producerProperties可以用来设置配置属性进行调优。配置属性列表请参考http://kafka.apache.org/documentation.html#producerconfigs

使用spring-integration-kafka接收消息

同样的原理实现一个消费者：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61

package com.colobu.spring_kafka_demo; import java.util.Collection; import java.util.HashMap; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Random; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import org.slf4j.LoggerFactory; import org.springframework.context.support.ClassPathXmlApplicationContext; import org.springframework.integration.channel.QueueChannel; import org.springframework.messaging.Message; import ch.qos.logback.classic.Level; public class Consumer { private static final String CONFIG = "/consumer_context.xml"; private static Random rand = new Random(); @SuppressWarnings({ "unchecked", "unchecked", "rawtypes" }) public static void main(String[] args) { ch.qos.logback.classic.Logger rootLogger = (ch.qos.logback.classic.Logger)LoggerFactory.getLogger(ch.qos.logback.classic.Logger.ROOT_LOGGER_NAME); rootLogger.setLevel(Level.toLevel( "info")); final ClassPathXmlApplicationContext ctx = new ClassPathXmlApplicationContext(CONFIG, Consumer.class); ctx.start(); final QueueChannel channel = ctx.getBean( "inputFromKafka", QueueChannel.class); Message msg; while((msg = channel.receive()) != null) { HashMap map = (HashMap)msg.getPayload(); Set<Map.Entry> set = map.entrySet(); for (Map.Entry entry : set) { String topic = (String)entry.getKey(); System.out.println( "Topic:" + topic); ConcurrentHashMap<Integer,List< byte[]>> messages = (ConcurrentHashMap<Integer,List< byte[]>>)entry.getValue(); Collection<List< byte[]>> values = messages.values(); for (Iterator<List< byte[]>> iterator = values.iterator(); iterator.hasNext();) { List< byte[]> list = iterator.next(); for ( byte[] object : list) { String message = new String(object); System.out.println( "\tMessage: " + message); } } } } try { Thread.sleep( 100000); } catch (InterruptedException e) { e.printStackTrace(); } ctx.close(); } }

Spring的配置文件如下：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi= "http://www.w3.org/2001/XMLSchema-instance" xmlns:int= "http://www.springframework.org/schema/integration" xmlns:int-kafka= "http://www.springframework.org/schema/integration/kafka" xmlns:task= "http://www.springframework.org/schema/task" xsi:schemaLocation= "http://www.springframework.org/schema/integration/kafka http://www.springframework.org/schema/integration/kafka/spring-integration-kafka.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd"> <int:channel id="inputFromKafka"> <int:queue/> </int:channel> <int-kafka:inbound-channel-adapter id="kafkaInboundChannelAdapter" kafka-consumer-context-ref= "consumerContext" auto-startup= "false" channel= "inputFromKafka"> <int:poller fixed-delay="10" time-unit="MILLISECONDS" max-messages-per-poll= "5" /> </int-kafka:inbound-channel-adapter> <bean id="consumerProperties" class= "org.springframework.beans.factory.config.PropertiesFactoryBean"> <property name="properties"> <props> <prop key="auto.offset.reset">smallest </prop> <prop key="socket.receive.buffer.bytes">10485760 </prop> <!-- 10M --> <prop key="fetch.message.max.bytes">5242880 </prop> <prop key="auto.commit.interval.ms">1000 </prop> </props> </property> </bean> <int-kafka:consumer-context id="consumerContext" consumer-timeout= "4000" zookeeper-connect= "zookeeperConnect" consumer-properties= "consumerProperties"> <int-kafka:consumer-configurations> <int-kafka:consumer-configuration group-id="mygroup" max-messages= "5000"> <int-kafka:topic id="test" streams="4" /> </int-kafka:consumer-configuration> <!-- <int-kafka:consumer-configuration group-id="default3" value-decoder="kafkaSpecificDecoder" key-decoder="kafkaReflectionDecoder" max-messages="10"> <int-kafka:topic-filter pattern="regextopic.*" streams="4" exclude="false" /> </int-kafka:consumer-configuration> --> </int-kafka:consumer-configurations> </int-kafka:consumer-context> <int-kafka:zookeeper-connect id="zookeeperConnect" zk-connect= "localhost:2181" zk-connection-timeout= "6000" zk-session-timeout= "400" zk-sync-time= "200" /> </beans>

 

如果运行有timeout错误 改成10000。

 

这个配置和Producer类似， 同样声明一个channel, 定义inbound-channel-adapter, 它引用Bean kafka-consumer-context,
kafka-consumer-context定义了消费者的列表。 consumer-configuration还提供了topic-filter，使用正则表达式建立白名单或者黑名单(exclude属性)。

消费者上下文还需要zookeeper-connect。

由于spring-integration-kafka只实现了high level Consumer API,这也就意味着你不可能回滚重新查看以前的消息， 因为high level API不提供offset管理。

注意Channel中得到的有效负载的类型是：
Map<String, Map<Integer, List<Object>>>,
这个Map的key是topic, 值还是另外的Map对象。
这个值Map的key值是分区号，value值是消息列表。 在本例中由于消息是字符串， 转换成了byte[]数组。

这种复杂的结构是由于Kafka的设计造成的。 Kafka保证对于同一个topic的同一个分区的消息是严格有序的。所有这种数据结构可以提供有序的消息。