kafka+log4j配置

2019独角兽企业重金招聘Python工程师标准>>>

前言。好多网上的整合都不能使用了。现更新最新的。

一、安装和配置kafka.参考http://kafka.apache.org/quickstart 

tar -xzf kafka_2.11-0.10.2.0.tgz
cd kafka_2.11-0.10.2.0

修改config/server.properties. 打开listeners. 根据实际情况修改zookeeper.connect

listeners=PLAINTEXT://192.168.1.9:9092
zookeeper.connect=ser9:2181,ser40:2181

如果增加为集群模式：broker.id为集群中不同的编号，要唯一。参考http://kafka.apache.org/documentation/#prodconfig

broker.id=1
listeners=PLAINTEXT://192.168.1.40:9092
log.dirs=/var/log/kafka-logs
num.partitions=8
default.replication.factor=3
min.insync.replicas=2
auto.create.topics.enable=true

开启服务

bin/kafka-server-start.sh config/server.properties &

创建主题

bin/kafka-topics.sh --create --zookeeper ser9:2181 --replication-factor 1 --partitions 1 --topic test

二、修改pom.xml

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.10.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-log4j-appender</artifactId>
            <version>0.10.2.0</version>
        </dependency>

log4j.properties

log4j.logger.org.apache = DEBUG,kafka

# appender kafka
log4j.appender.kafka=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.kafka.topic=test
# multiple brokers are separated by comma ",".
log4j.appender.kafka.brokerList=192.168.1.9:9092
log4j.appender.kafka.compressionType=none
log4j.appender.kafka.syncSend=true
log4j.appender.kafka.layout=org.apache.log4j.PatternLayout
log4j.appender.kafka.layout.ConversionPattern=%-d{yyyy-MM-dd HH\:mm\:ss,SSS} [%5p] (%F\:%L) - %m%n

三、打开consumer

bin/kafka-console-consumer.sh --bootstrap-server ser9:9092 --topic test --from-beginning

 

转载于:https://my.oschina.net/u/160697/blog/854275