Kafka 集群环境搭建

最新推荐文章于 2023-12-19 16:13:54 发布
最新推荐文章于 2023-12-19 16:13:54 发布
阅读量485 收藏
点赞数
分类专栏： Kafka 文章标签： kafka
Kafka目录结构：[url]http://donald-draper.iteye.com/blog/2396760[/url]
Kafka配置文件：[url]http://donald-draper.iteye.com/blog/2397000[/url]
Kafka Standy模式、创建主题，生产消费消息：[url]http://donald-draper.iteye.com/blog/2397170[/url]
上一篇文章，我们启动kafka的standy模式，并使用相关命令创建topic，生产消息和消费消息，我们只是运行一个broker,，没太多的意思。今天我们搭建一个kafka集群。对于Kafka,一个broker仅仅只是规模为1的集群,现在我们扩展至三个节点。我们在单机，只修改broker的监听端口，id及日志目录，其实多机版集群道理是一样的，只不过ip地址不一样。

[b]首先为每个broker创建一个配置文件[/b]: 

Last login: Sun Oct 22 21:43:13 2017
[donald@Donald_Draper ~]$ ls
Desktop  Documents  Downloads  kafka_2.11-0.11.0.1  kafka_2.11-0.11.0.1.tgz  Music  Pictures  Public  Templates  Videos
[donald@Donald_Draper ~]$ cd kafka_2.11-0.11.0.1/
[donald@Donald_Draper kafka_2.11-0.11.0.1]$ ls
bin  config  libs  LICENSE  logs  NOTICE  site-docs
[donald@Donald_Draper kafka_2.11-0.11.0.1]$ cd config/
[donald@Donald_Draper config]$ ls
connect-console-sink.properties    connect-file-sink.properties    connect-standalone.properties  producer.properties     zookeeper.properties
connect-console-source.properties  connect-file-source.properties  consumer.properties            server.properties
connect-distributed.properties     connect-log4j.properties        log4j.properties               tools-log4j.properties
[donald@Donald_Draper config]$ cp server.properties server1.properties 
[donald@Donald_Draper config]$ cp server.properties server2.properties 
[donald@Donald_Draper config]$ ls
connect-console-sink.properties    connect-file-sink.properties    connect-standalone.properties  producer.properties  server.properties
connect-console-source.properties  connect-file-source.properties  consumer.properties            server1.properties   tools-log4j.properties
connect-distributed.properties     connect-log4j.properties        log4j.properties               server2.properties   zookeeper.properties

现在已经有三个broker配置文件，现在来修改broker1和broker2配置文件，即server1.properties与server2.properties 

[donald@Donald_Draper config]$ vim server1.properties 
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs1
[donald@Donald_Draper config]$ vim server2.properties 
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs2

[donald@Donald_Draper config]$ vim server.properties 
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
[donald@Donald_Draper config]$ ^C

broker.id是集群中每个节点的唯一且永久的名称，我们修改端口和日志分区是因为我们现在在同一台机器上运行，我们要防止broker在同一端口上注册和覆盖对方的数据。需要注意的是日志目录由于我们测试用的是临时目录/tmp,在生产环境中最好换个目录，因为当机器宕机时，日志文件可能丢失，这个在hadoop集群中会出现，我不知道在kafka集群中会不会出现，个人认为应该会出现，最好换个目录。同时，我们并没有使用Zookeeper集群，Zookeeper是standy模式，其实在生产环境中Zookeeper应该是高可用的，必须是集群。我们就不使用Zookeeper集群了，这个不是重点，我们在可以下下面这篇文章中找到Zookeeper集群的搭建：
Hadoop2.7.1高可用环境搭建：[url]http://donald-draper.iteye.com/blog/2302217[/url]
主要修改Zookeeper的配置文件zoo.cfg 
server.1=192.168.126.126:2888:3888  
server.2=192.168.126.127:2888:3888  
server.3=192.168.126.128:2888:3888

并创建zookeeper数据文件夹zdata，并在zdata中创建myid文件，在三台机上内容分别为上面server后的id
然后再修改broker server配置文件的一下选项 
############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=192.168.126.126:2181,192.168.126.127:2181,192.168.126.128:2181


[b]现在来看启动集群，先启动zookeeper，再启动3个broker[/b] 
[donald@Donald_Draper bin]$ ./zookeeper-server-start.sh ../config/zookeeper.properties  &
[1] 5334
[donald@Donald_Draper bin]$ 
...
[2017-10-22 22:25:12,828] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-10-22 22:25:12,828] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-10-22 22:25:12,879] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

[donald@Donald_Draper bin]$ ./kafka-server-start.sh ../config/server.properties &
[1] 5334
[donald@Donald_Draper bin]$ 
...
[2017-10-22 22:27:04,624] INFO Registered broker 0 at path /brokers/ids/0 with addresses: EndPoint(Donald_Draper.server.com,9093,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2017-10-22 22:27:04,625] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2017-10-22 22:27:05,047] INFO Kafka version : 0.11.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2017-10-22 22:27:05,047] INFO Kafka commitId : c2a0d5f9b1f45bf5 (org.apache.kafka.common.utils.AppInfoParser)
[2017-10-22 22:27:05,048] INFO [Kafka Server 0], started (kafka.server.KafkaServer)

[donald@Donald_Draper bin]$ ./kafka-server-start.sh ../config/server1.properties &
[1] 5900
[donald@Donald_Draper bin]$ 
...
[2017-10-22 22:27:04,624] INFO Registered broker 1 at path /brokers/ids/1 with addresses: EndPoint(Donald_Draper.server.com,9093,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2017-10-22 22:27:04,625] WARN No meta.properties file under dir /tmp/kafka-logs1/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2017-10-22 22:27:05,047] INFO Kafka version : 0.11.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2017-10-22 22:27:05,047] INFO Kafka commitId : c2a0d5f9b1f45bf5 (org.apache.kafka.common.utils.AppInfoParser)
[2017-10-22 22:27:05,048] INFO [Kafka Server 1], started (kafka.server.KafkaServer)

[donald@Donald_Draper bin]$ ./kafka-server-start.sh ../config/server2.properties &
[1] 6611
[donald@Donald_Draper bin]$ 
...
[2017-10-22 22:31:18,442] INFO Creating /brokers/ids/2 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-10-22 22:31:18,496] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-10-22 22:31:18,497] INFO Registered broker 2 at path /brokers/ids/2 with addresses: EndPoint(Donald_Draper.server.com,9094,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2017-10-22 22:31:18,550] WARN No meta.properties file under dir /tmp/kafka-logs2/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2017-10-22 22:31:19,288] INFO Kafka version : 0.11.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2017-10-22 22:31:19,288] INFO Kafka commitId : c2a0d5f9b1f45bf5 (org.apache.kafka.common.utils.AppInfoParser)
[2017-10-22 22:31:19,288] INFO [Kafka Server 2], started (kafka.server.KafkaServer)


[donald@Donald_Draper bin]$ netstat -ntlp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::3306                 :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 :::36055                :::*                    LISTEN      5597/java           
tcp6       0      0 ::1:631                 :::*                    LISTEN      -                   
tcp6       0      0 :::45464                :::*                    LISTEN      5334/java           
tcp6       0      0 :::45979                :::*                    LISTEN      5900/java           
tcp6       0      0 :::34402                :::*                    LISTEN      6611/java           
tcp6       0      0 :::9092                 :::*                    LISTEN      5597/java           
tcp6       0      0 :::9093                 :::*                    LISTEN      5900/java           
tcp6       0      0 :::2181                 :::*                    LISTEN      5334/java           
tcp6       0      0 :::9094                 :::*                    LISTEN      6611/java           
[donald@Donald_Draper bin]$


[b]现在集群已经启动了，我们在创建一个复制因子为3的主题：[/b] 

[donald@Donald_Draper bin]$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
[2017-10-22 22:36:23,175] INFO Accepted socket connection from /127.0.0.1:41964 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-10-22 22:36:23,214] INFO Client attempting to establish new session at /127.0.0.1:41964 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-10-22 22:36:23,236] INFO Established session 0x15f44792df10005 with negotiated timeout 30000 for client /127.0.0.1:41964 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-10-22 22:36:23,754] INFO Got user-level KeeperException when processing sessionid:0x15f44792df10005 type:setData cxid:0x6 zxid:0x122 txntype:-1 reqpath:n/a Error Path:/config/topics/my-replicated-topic Error:KeeperErrorCode = NoNode for /config/topics/my-replicated-topic (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-10-22 22:36:23,764] INFO Got user-level KeeperException when processing sessionid:0x15f44792df10005 type:create cxid:0x8 zxid:0x123 txntype:-1 reqpath:n/a Error Path:/config/topics Error:KeeperErrorCode = NodeExists for /config/topics (org.apache.zookeeper.server.PrepRequestProcessor)
Created topic "my-replicated-topic".
[2017-10-22 22:36:23,834] INFO Processed session termination for sessionid: 0x15f44792df10005 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-10-22 22:36:23,838] INFO Closed socket connection for client /127.0.0.1:41964 which had sessionid 0x15f44792df10005 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-10-22 22:36:23,887] INFO Got user-level KeeperException when processing sessionid:0x15f44792df10000 type:create cxid:0x186 zxid:0x127 txntype:-1 reqpath:n/a Error Path:/brokers/topics/my-replicated-topic/partitions/0 Error:KeeperErrorCode = NoNode for /brokers/topics/my-replicated-topic/partitions/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-10-22 22:36:23,889] INFO Got user-level KeeperException when processing sessionid:0x15f44792df10000 type:create cxid:0x187 zxid:0x128 txntype:-1 reqpath:n/a Error Path:/brokers/topics/my-replicated-topic/partitions Error:KeeperErrorCode = NoNode for /brokers/topics/my-replicated-topic/partitions (org.apache.zookeeper.server.PrepRequestProcessor)

[b]浏览主题列表[/b] 
[donald@Donald_Draper bin]$  ./kafka-topics.sh --list --zookeeper localhost:2181  
[2017-10-22 22:36:34,017] INFO Accepted socket connection from /127.0.0.1:41970 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-10-22 22:36:34,021] INFO Client attempting to establish new session at /127.0.0.1:41970 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-10-22 22:36:34,024] INFO Established session 0x15f44792df10006 with negotiated timeout 30000 for client /127.0.0.1:41970 (org.apache.zookeeper.server.ZooKeeperServer)
__consumer_offsets
my-replicated-topic
test
[2017-10-22 22:36:34,065] INFO Processed session termination for sessionid: 0x15f44792df10006 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-10-22 22:36:34,067] INFO Closed socket connection for client /127.0.0.1:41970 which had sessionid 0x15f44792df10006 (org.apache.zookeeper.server.NIOServerCnxn)
[donald@Donald_Draper bin]$

从输出来看我们的my-replicated-topic主题创建成功，test是我们standy模式的测试主题。
好了，现在我们已经有了一个集群了，我们怎么知道每个集群在做什么呢？运行命令“describe topics” 
	
[donald@Donald_Draper bin]$ ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
[2017-10-22 22:39:34,232] INFO Accepted socket connection from /127.0.0.1:41972 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-10-22 22:39:34,233] INFO Client attempting to establish new session at /127.0.0.1:41972 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-10-22 22:39:34,237] INFO Established session 0x15f44792df10007 with negotiated timeout 30000 for client /127.0.0.1:41972 (org.apache.zookeeper.server.ZooKeeperServer)
Topic:my-replicated-topic       PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: my-replicated-topic      Partition: 0    Leader: 2       Replicas: 2,0,1 Isr: 2,0,1
[2017-10-22 22:39:35,852] INFO Processed session termination for sessionid: 0x15f44792df10007 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-10-22 22:39:35,860] INFO Closed socket connection for client /127.0.0.1:41972 which had sessionid 0x15f44792df10007 (org.apache.zookeeper.server.NIOServerCnxn)
[donald@Donald_Draper bin]$

关键在这两行： 
Topic:my-replicated-topic       PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: my-replicated-topic      Partition: 0    Leader: 2       Replicas: 2,0,1 Isr: 2,0,1


下面来解释一下上面的输出，第一行是所有分区的摘要，每一个线提供一个分区信息，
因为我们只有一个分区，所有只有一条行介绍分区的信息。

1."leader"：该节点负责所有指定分区的读和写，每个节点的领导都是随机选择的。
2."replicas":备份的节点，无论该节点是否是leader或者目前是否还活着，只是显示。
3."isr"：备份节点的集合，也就是活着的节点集合。

从上面可看出，leader为broker2。
我们再次运行上面的这个命令，看看一开始我们创建的test主题的相关信息： 


[donald@Donald_Draper bin]$ ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
[2017-10-22 22:46:54,560] INFO Accepted socket connection from /127.0.0.1:41974 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-10-22 22:46:54,579] INFO Client attempting to establish new session at /127.0.0.1:41974 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-10-22 22:46:54,619] INFO Established session 0x15f44792df10008 with negotiated timeout 30000 for client /127.0.0.1:41974 (org.apache.zookeeper.server.ZooKeeperServer)
Topic:test      PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: test     Partition: 0    Leader: 0       Replicas: 0     Isr: 0
[2017-10-22 22:46:55,069] INFO Processed session termination for sessionid: 0x15f44792df10008 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-10-22 22:46:55,071] INFO Closed socket connection for client /127.0.0.1:41974 which had sessionid 0x15f44792df10008 (org.apache.zookeeper.server.NIOServerCnxn)
[donald@Donald_Draper bin]$


关键在这两行： 
Topic:test      PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: test     Partition: 0    Leader: 0       Replicas: 0     Isr: 0



从上面可看出，由于上面我们运行的是standy模式，，所以test主题没有Replicas所以是0，同时isr为0，leader也是broker0。

现在我们来启动一个两个终端一个生产消息，一个消费消息：
生产者： 
[donald@Donald_Draper bin]$ ./kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
>

消费者： 
[donald@Donald_Draper bin]$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic


在生产者端，生产消息： 
[donald@Donald_Draper bin]$ ./kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
>hello everyone!   
>how about ?
>


消费者中断输出为： 
[donald@Donald_Draper bin]$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
hello everyone!
how about ?



[b]现在我们来，测试集群的容错，kill掉leader，Broker2作为当前的leader，也就是kill掉Broker2。[/b] 


[donald@Donald_Draper bin]$ ps aux | grep server2.properties
donald     6611  3.3 15.9 3676844 297348 pts/3  Sl   22:31   0:49 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/home/donald=/kafka_2.11-0.11.0.1/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/home/donald=/kafka_2.11-0.11.0.1/bin/../logs -Dlog4j.configuration=file:./../config/log4j.properties -cp :/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/aopalliance-repackaged-2.5.0-b05.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/argparse4j-0.7.0.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/commons-lang3-3.5.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/connect-api-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/connect-file-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/connect-json-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/connect-runtime-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/connect-transforms-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/guava-20.0.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/hk2-api-2.5.0-b05.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/hk2-locator-2.5.0-b05.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/hk2-utils-2.5.0-b05.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jackson-annotations-2.8.5.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jackson-core-2.8.5.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jackson-databind-2.8.5.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jackson-module-jaxb-annotations-2.8.5.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/javassist-3.21.0-GA.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/javax.annotation-api-1.2.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/javax.inject-1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/javax.inject-2.5.0-b05.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/javax.servlet-api-3.1.0.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jersey-client-2.24.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jersey-common-2.24.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jersey-container-servlet-2.24.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jersey-container-servlet-core-2.24.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jersey-guava-2.24.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jersey-media-jaxb-2.24.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jersey-server-2.24.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/jopt-simple-5.0.3.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/kafka_2.11-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/kafka_2.11-0.11.0.1-sources.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/kafka_2.11-0.11.0.1-test-sources.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/kafka-clients-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/kafka-log4j-appender-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/kafka-streams-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/kafka-streams-examples-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/kafka-tools-0.11.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/log4j-1.2.17.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/lz4-1.3.0.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/maven-artifact-3.5.0.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/metrics-core-2.2.0.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/plexus-utils-3.0.24.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/reflections-0.9.11.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/rocksdbjni-5.0.1.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/scala-library-2.11.11.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/slf4j-api-1.7.25.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/slf4j-log4j12-1.7.25.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/snappy-java-1.1.2.6.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/validation-api-1.1.0.Final.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/zkclient-0.10.jar:/home/donald=/kafka_2.11-0.11.0.1/bin/../libs/zookeeper-3.4.10.jar kafka.Kafka ../config/server2.properties
donald     9171  0.0  0.0 112784   988 pts/0    S+   22:55   0:00 grep --color=auto server2.properties
[donald@Donald_Draper bin]$ kill -9 6611

在windows上： 

> wmic process get processid,caption,commandline | find "java.exe" | find "server2.properties"
java.exe    java  -Xmx1G -Xms1G -server -XX:+UseG1GC ... build\libs\kafka_2.11-0.11.0.1.jar"  kafka.Kafka config\server2.properties    6611
> taskkill /pid 6611 /f


再次查看my-replicated-topic主题 
[donald@Donald_Draper bin]$ ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
[2017-10-22 22:59:55,370] INFO Accepted socket connection from /127.0.0.1:42250 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-10-22 22:59:55,375] INFO Client attempting to establish new session at /127.0.0.1:42250 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-10-22 22:59:55,378] INFO Established session 0x15f44792df10009 with negotiated timeout 30000 for client /127.0.0.1:42250 (org.apache.zookeeper.server.ZooKeeperServer)
Topic:my-replicated-topic       PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: my-replicated-topic      Partition: 0    Leader: 0       Replicas: 2,0,1 Isr: 0,1
[2017-10-22 22:59:55,630] INFO Processed session termination for sessionid: 0x15f44792df10009 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-10-22 22:59:55,643] INFO Closed socket connection for client /127.0.0.1:42250 which had sessionid 0x15f44792df10009 (org.apache.zookeeper.server.NIOServerCnxn)
[donald@Donald_Draper bin]$


从输出来看，备份节点broker0成为新的leader，而broker2已经不在同步备份集合里了。

重新启动一个消息者终端2： 
[donald@Donald_Draper bin]$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
hello everyone!
how about ?

从输出来看消息并没有丢失。

重新生产消息： 
[donald@Donald_Draper bin]$ ./kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
>hello everyone!   
>how about ?
>[2017-10-22 22:58:33,385] WARN Connection to node 2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

>who is living ? 
>

消费者终端输出为：
消息者终端1: 
[donald@Donald_Draper bin]$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
hello everyone!
how about ?
...
[2017-10-22 22:58:39,207] WARN Connection to node 2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

who is living ?


消息者终端2 
[donald@Donald_Draper bin]$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
hello everyone!
how about ?

who is living ?


从生产者和消息者1的输出来看，它们已经知道broker2已经不可用。
到目前为止kafka集群搭建及测试完毕，关闭集群： 
[donald@Donald_Draper bin]$ ./kafka-server-stop.sh ../config/server.properties 
[donald@Donald_Draper bin]$ ./kafka-server-stop.sh ../config/server1.properties 
[donald@Donald_Draper bin]$ ./kafka-server-stop.sh ../config/server2.properties 
[donald@Donald_Draper bin]$ ./zookeeper-server-stop.sh  ../config/zookeeper.properties 
[donald@Donald_Draper bin]$ netstat -ntlp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::3306                 :::*                    LISTEN      -                   
tcp6       0      0 :::35219                :::*                    LISTEN      9858/java           
tcp6       0      0 :::35286                :::*                    LISTEN      8893/java           
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 ::1:631                 :::*                    LISTEN      -                   
[donald@Donald_Draper bin]$


下一篇我们来讲kafka connect。
kafka集群环境搭建
li_xuemin的博客
08-11 246
kafka环境搭建创建docker-compose搭建zk集群搭建kafka集群 创建docker-compose 执行docker-compose --version如果出现以下报错，则需要配置docker-compose，如果显示版本则不需要配置。 bash: docker-compose: command not found... 执行以下命令拉取docker-compose sudo curl -L "https://github.com/docker/compose/releases/d
Kafka集群环境搭建
Shinlikr的博客
11-27 308
（此篇默认Kafka普通环境已经搭建好） 一、Jdk的安装 二、Zookeeper的安装配置 1、在之前搭建好的zookeeper环境基础上进行集群环境的配置。修改zoo.cfg文件，在文件末尾添加如下内容： server.1=127.0.0.1:2888:3888 server.2=127.0.0.1:2889:3889 server.3=127.0.0.1:2890:3890 2、复制粘贴完整...
zookeeper 链接报错 KeeperErrorCode = NodeExists for /
bb123116的博客
11-17 2460
原因是刚刚运行过一次该语句 文件已经创建 百度说删除缓存的version-2文件夹 但是也不知道去哪删 发现只要在这个语句外套一层if语句即可
zookeeper报错：KeeperErrorCode = NodeExists for /**
T_tang_的博客
01-28 5253
KeeperErrorCode = NodeExists for /** 关于这个问题我找了很久 有以下问题可能你zookeeper中已经有了该路径。 若是没有，运行后添加成功了，那就有可能是你的maven运行时先编译了一遍，添加过了，所以要在maven编译环境时去除test 还可以在pom.xml中配置 <plugin> <groupId>org.apache.maven.plugins</groupId>
Kafka学习理解-listeners配置
Data & Analysis
12-03 4663
listeners# broker 服务器要监听的地址及端口 . 默认是 localhost:9092 ,0.0.0.0的话 ,表示监听本机的所有ip地址. 本机配置: localhost: 只监听本机的地址请求, 客户端也只能用localhost来请求 127.0.0.1: 同localhost, 在请求上可能有与区分 , 看client的请求吧 . 客户端也只能用127.0.0.1来请求 192.168.0.1: 有点脑子的都不要用这个 , 你的局域网不一定是192.168段的....
zookeeper Error:KeeperErr orCode = NodeExists
MyNoteBlog的博客
02-21 2万+
项目中采用dubbo远程调用框架,本地启动加载dubbo.xml配置文件时,zookeeper打印错误日志,如下： ath:/dubbo/com.mk.base.api..UserLoginService/providers Error: KeeperErrorCode = NodeExists for /dubbo/com.mk.base.api..UserLoginS ervice/prov
【kafka集群重启服务器后启动报错详解】No `meta.properties` found in /tmp/kraft-combined-logs（两种解决方法，已解决）
最新发布
写好每一行代码
12-19 501
原因在 /kraft/server.properties 中设置的日志目录是 /tmp/kraft-combined-logs，虚拟机重启后目录 /tmp 就被清空了，没有修改 log.dirs 日志文件存贮目录，导致启动报错找不到 meta.properties 文件在 /tmp/kraft-combined-logs 目录下。
kafka集群搭建文档
05-31
本篇文档将详细介绍如何在Linux环境中搭建Kafka集群，同时结合Hadoop和Spark的分布式安装，构建一个完整的数据处理平台。 首先，搭建Kafka集群的基础条件包括： 1. 至少一台Linux服务器，但为了高可用性，推荐多台...
docker容器中搭建kafka集群环境.doc
11-07
docker容器中搭建kafka集群环境，kafka集群配置注意事项与优化
ERROR: KeeperErrorCode = NoNode for /hbase/master与Path does not exist on HDFS or WebHDFS is disabled
qq_50869885的博客
10-28 2060
ERROR: KeeperErrorCode = NoNode for /hbase/master与Path does not exist on HDFS or WebHDFS is disabled可能的解决方法，小白向
注册不上zookeeper无报错_zookeeper使用过程的错误
weixin_42488997的博客
12-24 2101
一、zookeeper启动成功，dubbo服务也注册成功，但是服务消费者调用失败报错如下：[myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x16ae75366b60004 with negotiated timeout 40000 for client /127.0.0.1:50942019-05-2...
windows下zookeeper和kafka的启动和关闭顺序及注意事项
wow_awsl_qwq的博客
05-03 3887
首先：一定要先启动ZooKeeper 再启动Kafka 顺序不可以改变。 先关闭kafka ，再关闭zookeeper。 启动ZooKeeper后不要暴力关闭命令行，否则接下来启动kafka时会连接不上，显示Timeout错误 两个都不能暴力关闭 kafka关闭时： 现象：在windows的命令行里启动kafka之后，当关闭命令行窗口时，就会强制关闭kafka。这种关闭方式为暴力关闭，很可能会导致kafka无法完成对日志文件的解锁。届时，再次启动kafka的时候，就会提示日志文件被锁，无法成功启动。 方案：
zookeeper频繁异常问题分析
热门推荐
09-02 10万+
1.1            操作系统信息 1.1.1          CPU信息 从上述可以看出在问题点CPU使用率并不高，故排除CPU问题。 1.1.2          内存信息   CM监控界面并无内存信息，无法提供图片。 但问题期间，监控机器内存，发现还有剩余，故排除内存问题。 1.2            ZKFAILOVER日志信息   2017-09-0
Zookeeper的事务性请求之后台日志分析
qq_36816062的博客
03-04 578
Zookeeper的事务性请求之后台日志分析 客户端发送事务性请求到Zookeeper集群，集群中任何一台接收到，都会将该事务性请求转发给Leader，由Leader通过原子广播给其他服务器进件事务持久化操作，超过半数以上的服务器返回事务执行成功，则该事务请求操作完成后。自己在本地测试了下，发现日志如下 1.Leader节点 2021-02-25 15:05:39,007 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServe
ZooKeeper运行报错KeeperErrorCode = NoNode for .....
椰子真的是只猫
08-22 9万+
最近遇到一个很尴尬的报错，org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode KeeperErrorCode = NoNode for
kafka创建主题报错：KeeperErrorCode = NoNode for /brokers/ids
Jay_Weirdo的博客
08-15 5212
解决办法之一： 根据自己安装的kafka路径，进入kafka目录下，然后创建zookeeper目录，进入zookeeper目录，创建myid文件，并将0写入该文件中。 cd /user/local/kafka mkdir zookeeper cd zookeeper touch myid echo 0 > myid ...
kafka错误
kongkong的专栏
11-05 2167
1. Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. [2019-11-05 08:56:44,341] ERROR java.util.concurrent.Execution...
CentOS环境下Kafka分布式集群详细搭建指南
这个过程涵盖了Kafka集群搭建的基础步骤，但实际部署中还需要考虑更多的因素，如安全性、监控、数据备份与恢复策略、高可用性和性能优化等。此外，随着Kafka版本的更新，配置细节可能会有所变化，因此建议参考最新的...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值