由于项目中Storm或者Spark的实时数据源来自Kafka,本文章具体说明了Spring和Kafka的整合
maven中依赖如下:
<dependency> <groupId>org.springframework.integration</groupId> <artifactId>spring-integration-kafka</artifactId> <version>2.1.0.RELEASE</version> </dependency>
由于spring-integration-kafka只实现了high level Consumer API,这也就意味着你不可能回滚重新查看以前的消息, 因为high level API不提供offset管理。因此Spring和Kafka的集成中比较适用往Kafka中生产数据,Storm(storm-kafka适用low level 的API定时向ZK写offset)或者Spark(实现了createDircetDstream 自己管理offset)来消费数据
context_adapt.xml 配置文件<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:int="http://www.springframework.org/schema/integration" xmlns:int-kafka="http://www.springframework.org/schema/integration/kafka" xmlns:task="http://www.springframework.org/schema/task" xsi:schemaLocation="http://www.springframework.org/schema/integration/kafka http://www.springframework.org/schema/integration/kafka/spring-integration-kafka.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd"> <int:channel id="inputToKafka"></int:channel> <int-kafka:outbound-channel-adapter id="kafkaOutboundChannelAdapter" kafka-template="template" auto-startup="true" channel="inputToKafka" order="3" topic="testPartion" message-key-expression="'bar'" partition-id-expression="1"> <int-kafka:request-handler-advice-chain> <bean class="org.springframework.integration.handler.advice.RequestHandlerCircuitBreakerAdvice"/> </int-kafka:request-handler-advice-chain> </int-kafka:outbound-channel-adapter> <bean id="template" class="org.springframework.kafka.core.KafkaTemplate"> <constructor-arg> <bean class="org.springframework.kafka.core.DefaultKafkaProducerFactory"> <constructor-arg> <map> <entry key="bootstrap.servers" value="localhost:9092"/> <entry key="key.serializer" value="org.apache.kafka.common.serialization.StringSerializer"></entry> <entry key="value.serializer" value="org.apache.kafka.common.serialization.StringSerializer"></entry> </map> </constructor-arg> </bean> </constructor-arg> </bean>public class SpringKafkaProducer { private static final String CONFIG = "/context_adapt.xml"; private static Random rand = new Random(); private static ClassPathXmlApplicationContext ctx; static {ClassPathXmlApplicationContext实际上就已经包含了BeanFactory所提供的功能 ctx = new ClassPathXmlApplicationContext(CONFIG, Producer.class); ctx.start(); } public static void sendMessage2Kafka(String message) { final MessageChannel channel = ctx.getBean("inputToKafka", MessageChannel.class); channel.send(MessageBuilder.withPayload(message).build()); } public static void main(String[] args) { sendMessage2Kafka("springKafka"); } }
在 Kafka端消费数据,执行如下命令:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testPartion --from-beginning
已经测试过可以正常的消费数据,spring比较适合向Kafka向数据 bean的配置和加载比较成熟