kafka

集群安装，先配置好 zk 集群，启动。

解压 kafka，之后修改配置文件  修改 config 下的 server.properties

修改 log 存储位置

修改broker的 id

配置好外部的 zk 位置

启动 kafka 集群   ./bin/kafka-server-start.sh ./config/server.properties 1>/dev/null 2>&1  &

 

[root@hadoop03 kafka]# [2019-04-29 21:01:31,176] INFO Verifying properties (kafka.utils.VerifiableProperties)  验证配置属性
[2019-04-29 21:01:31,214] INFO Property broker.id is overridden to 2 (kafka.utils.VerifiableProperties)  broker.id 是 2 

以下是一些配置信息
INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
INFO Property log.dirs is overridden to /home/lishuo/app/kafka/logs (kafka.utils.VerifiableProperties)
INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)
INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)
INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)
INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)
INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
INFO Property zookeeper.connect is overridden to hadoop05:2181,hadoop06:2181,hadoop07:2181 (kafka.utils.VerifiableProperties)
INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
INFO [Kafka Server 2], starting (kafka.server.KafkaServer)
INFO [Kafka Server 2], Connecting to zookeeper on hadoop05:2181,hadoop06:2181,hadoop07:2181 (kafka.server.KafkaServer)
INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
INFO Client environment:host.name=node2 (org.apache.zookeeper.ZooKeeper)
INFO Client environment:java.version=1.8.0_181 (org.apache.zookeeper.ZooKeeper)
INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
INFO Client environment:java.home=/home/lishuo/app/jdk1.8.0_181/jre (org.apache.zookeeper.ZooKeeper)
INFO Client environment:java.class.path=.:/home/lishuo/app/jdk1.8.0_181/lib/dt.jar:/home/lishuo/app/jdk1.8.0_181/lib/tools.jar:/home/lishuo/app/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/home/lishuo/app/kafka/bin/../examples/build/libs//kafka-examples*.jar:/home/lishuo/app/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/lishuo/app/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/lishuo/app/kafka/bin/../clients/build/libs/kafka-clients*.jar:/home/lishuo/app/kafka/bin/../libs/jopt-simple-3.2.jar:/home/lishuo/app/kafka/bin/../libs/kafka_2.10-0.8.2.0.jar:/home/lishuo/app/kafka/bin/../libs/kafka_2.10-0.8.2.0-javadoc.jar:/home/lishuo/app/kafka/bin/../libs/kafka_2.10-0.8.2.0-scaladoc.jar:/home/lishuo/app/kafka/bin/../libs/kafka_2.10-0.8.2.0-sources.jar:/home/lishuo/app/kafka/bin/../libs/kafka_2.10-0.8.2.0-test.jar:/home/lishuo/app/kafka/bin/../libs/kafka-clients-0.8.2.0.jar:/home/lishuo/app/kafka/bin/../libs/log4j-1.2.16.jar:/home/lishuo/app/kafka/bin/../libs/lz4-1.2.0.jar:/home/lishuo/app/kafka/bin/../libs/metrics-core-2.2.0.jar:/home/lishuo/app/kafka/bin/../libs/scala-library-2.10.4.jar:/home/lishuo/app/kafka/bin/../libs/slf4j-api-1.7.6.jar:/home/lishuo/app/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/home/lishuo/app/kafka/bin/../libs/snappy-java-1.1.1.6.jar:/home/lishuo/app/kafka/bin/../libs/zkclient-0.3.jar:/home/lishuo/app/kafka/bin/../libs/zookeeper-3.4.6.jar:/home/lishuo/app/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)
INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
INFO Client environment:os.version=3.10.0-862.14.4.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
INFO Client environment:user.dir=/home/lishuo/app/kafka (org.apache.zookeeper.ZooKeeper)
初始化 zk连接
INFO Initiating client connection, connectString=hadoop05:2181,hadoop06:2181,hadoop07:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@5b0abc94 (org.apache.zookeeper.ZooKeeper)
INFO Opening socket connection to server hadoop07/192.168.93.136:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
INFO Socket connection established to hadoop07/192.168.93.136:2181, initiating session (org.apache.zookeeper.ClientCnxn)
INFO Session establishment complete on server hadoop07/192.168.93.136:2181, sessionid = 0x36a68f96e9d0001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
INFO Loading logs. (kafka.log.LogManager)
INFO Logs loading complete. (kafka.log.LogManager)
INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
INFO [Socket Server on Broker 2], Started (kafka.network.SocketServer)
INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
INFO Registered broker 2 at path /brokers/ids/2 with address node2:9092. (kafka.utils.ZkUtils$)  端口9092
INFO [Kafka Server 2], started (kafka.server.KafkaServer)  启动成功