前言。好多网上的整合都不能使用了。现更新最新的。
一、安装和配置kafka.参考http://kafka.apache.org/quickstart
tar -xzf kafka_2.11-0.10.2.0.tgz
cd kafka_2.11-0.10.2.0
修改config/server.properties. 打开listeners. 根据实际情况修改zookeeper.connect
listeners=PLAINTEXT://192.168.1.9:9092
zookeeper.connect=ser9:2181,ser40:2181
如果增加为集群模式:broker.id为集群中不同的编号,要唯一。参考http://kafka.apache.org/documentation/#prodconfig
broker.id=1
listeners=PLAINTEXT://192.168.1.40:9092
log.dirs=/var/log/kafka-logs
num.partitions=8
default.replication.factor=3
min.insync.replicas=2
auto.create.topics.enable=true
开启服务
bin/kafka-server-start.sh config/server.properties &
创建主题
bin/kafka-topics.sh --create --zookeeper ser9:2181 --replication-factor 1 --partitions 1 --topic test
二、修改pom.xml
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-log4j-appender</artifactId>
<version>0.10.2.0</version>
</dependency>
log4j.properties
log4j.logger.org.apache = DEBUG,kafka
# appender kafka
log4j.appender.kafka=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.kafka.topic=test
# multiple brokers are separated by comma ",".
log4j.appender.kafka.brokerList=192.168.1.9:9092
log4j.appender.kafka.compressionType=none
log4j.appender.kafka.syncSend=true
log4j.appender.kafka.layout=org.apache.log4j.PatternLayout
log4j.appender.kafka.layout.ConversionPattern=%-d{yyyy-MM-dd HH\:mm\:ss,SSS} [%5p] (%F\:%L) - %m%n
三、打开consumer
bin/kafka-console-consumer.sh --bootstrap-server ser9:9092 --topic test --from-beginning