spring集成kfka实现消息队列

1.搭建kafka运行环境 ，下载地址：http://kafka.apache.org/downloads

• 下载完修改安装文件config里面的server.properties文件，将listeners=PLAINTEXT://IP:9092改成服务器的IP，修改zookeeper注册地址
• .\bin\windows\kafka-server-start.bat .\config\server.properties使用该命令启动kafka
• 测试一下是否可以发送消息
  bin\windows\kafka-console-producer.bat –broker-list IP9:092 –topic pipi
  bin\windows\kafka-console-consumer.bat –bootstrap-server IP:9092 –topic
  pipi –from- beginning

2.搭建生产者环境

• 配置producer.xml文件

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<!-- 定义producer属性-->
<bean id="kafakaproducer"  class="java.util.HashMap">
 <constructor-arg>
 <map>
     <entry key="bootstrap.servers"  value="10.168.114.91:9092"/>
     <entry key="group.id"  value="0"/>
     <entry key="retries"  value="10"/>
     <entry key="batch.size"  value="16384"/>
     <entry key="linger.ms"  value="1"/>
     <entry key="key.serializer"  value="org.apache.kafka.common.serialization.IntegerSerializer"/>
     <entry key="value.serializer"  value="org.apache.kafka.common.serialization.StringSerializer"/>
 </map>
 </constructor-arg>
</bean>
<!--配置kafka的工厂 -->
<bean id="producerFactory" class="org.springframework.kafka.core.DefaultKafkaProducerFactory">
<constructor-arg>
     <ref bean="kafakaproducer"/>
</constructor-arg>
</bean>
<bean id="kafkaTemplate" class="org.springframework.kafka.core.KafkaTemplate">
<constructor-arg>
<ref bean="producerFactory"/>
</constructor-arg>
    <property name="defaultTopic" value="mhb-test"/>
    </bean>
</beans>

• 测试类

public class SpringKafkaTest {
    public static void main(String[] args) {
        ApplicationContext ctx = new ClassPathXmlApplicationContext("classpath:config/spring-kafka.xml");
        KafkaTemplate kafkaTemplate = ctx.getBean("kafkaTemplate", KafkaTemplate.class);
        for(int i=0;i<1000000000;i++){
            String message="第"+i+"条信息";
            kafkaTemplate.send("pipi",message);
            System.out.println("message="+message);
        }

    }
}

3 . 搭建消费者环境

• 配置consumer.xml文件

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
    <!-- 定义消费者属性 -->
    <bean id="kafkaConsumerProperites" class="java.util.HashMap">
        <constructor-arg>
            <map>
                <entry key="bootstrap.servers" value="10.168.114.91:9092" />
                <entry key="group.id" value="0"/>
                <entry key="enable.auto.commit" value="true"/>
                <entry key="auto.commit.interval.ms" value="1000"/>
                <entry key="session.timeout.ms" value="15000"/>
                <entry key="key.deserializer" value="org.apache.kafka.common.serialization.IntegerDeserializer"/>
                <entry key="value.deserializer" value="org.apache.kafka.common.serialization.StringDeserializer"/>
            </map>
        </constructor-arg>
    </bean>
    <!--配置kafka的工厂 -->
    <bean id="kafkaConsumerFactory"
        class="org.springframework.kafka.core.DefaultKafkaConsumerFactory">
        <constructor-arg>
            <ref bean="kafkaConsumerProperites" />
        </constructor-arg>
    </bean>
<!--消费者具体实体类-->
    <bean id="KafkaConsumerListener" class="service.KafkaConsumerListener" />
    <!--配置容器属性-->
    <bean id="consumerContainerProperties"
        class="org.springframework.kafka.listener.config.ContainerProperties">
        <constructor-arg value="pipi" />
        <property name="messageListener" ref="KafkaConsumerListener" />
    </bean>
<!--配置消费者容器-->
    <bean id="conusmerContainer"class="org.springframework.kafka.listener.KafkaMessageListenerContainer " init-method="doStart">
        <constructor-arg ref="kafkaConsumerFactory" />
        <constructor-arg ref="consumerContainerProperties" />
    </bean>
</beans>

• 配置监听器

@Component("kafkaConsumerListener")
public class KafkaConsumerListener implements MessageListener<String, String>{
    @Override
    public void onMessage(ConsumerRecord<String, String> record) {
        System.out.printf("接收到消息="+ record.value());        
    }
}

• 消费者是web项目，启动后监听器就开始工作了，然后跑生产者的测试类就可以在消费者中控制台中看到日志了、

一个kafka消费者的管理工具kafkamonitor 在界面中就能看到消息模式和消费者

• kafkamonitor是一个jar，只需把它放在kafka安装目录windows下，编写一个脚本并运行即可启动项目

java -cp KafkaOffsetMonitor-assembly-0.2.0.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk 10.168.114.91:2181 --port 8089 --refresh 10.seconds --retain 1.days