zookeeper集合和kafka集合简单搭建使用

搭建之前默认jdk环境已经配置完毕，下面开始
1、准备三台linux物理机，这里以ubuntu为例，我这里准备了三台，如下：
hadoop102:192.168.238.139
hadoop103:192.168.238.140
hadoop104:192.168.238.141

分别将zookeeper和kafka安装包解压到一个目录下，我的是在下面目录

root@hadoop102:/opt/module# ll
total 143856
drwxr-xr-x  6 huipanxing huipanxing      4096 9月   7 19:15 ./
drwxrwxr-x  5 root       root            4096 5月  30 23:27 ../
drwxr-xr-x 15 root       root            4096 5月  18 14:37 hadoop-2.7.2/
drwxr-xr-x  9 root       root            4096 9月   7 19:53 kafka/
drwxr-xr-x 15 huipanxing huipanxing      4096 5月  27 21:47 spark-2.4.3-bin-hadoop2.7/
-rw-r--r--  1 root       root       147282706 5月  30 23:17 wordcount-jar-with-dependencies.jar
drwxr-xr-x 16 root       root            4096 9月   6 22:49 zookeeper/

2、设置环境变量

export ZOOKEEPER_HOME=/opt/module/zookeeper
export PATH=$PATH:$ZOOKEEPER_HOME/bin
export KAFKA_HOME=/opt/module/kafka
PATH=${KAFKA_HOME}/bin:$PATH

将上面的内容加到 /etc/profile文件中，然后执行source /etc/profile 命令使其生效。
3、修改zookeeper中conf目录下的zoo_sample.cfg为zoo.cfg

mv zoo_sample.cfg zoo.cfg

修改三个zookeeper节点中的zoo.cfg文件，修改dataDir，添加server.0、server.1、server.2

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/opt/module/zookeeper/log
dataLogDir=/opt/module/zookeeper/log
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.0=192.168.238.139:2888:3888
server.1=192.168.238.140:2888:3888
server.2=192.168.238.141:2888:3888

在3个zookeeper节点中上面配置的dataDir目录下分别创建myid文件，并分别添加内容0、1、2
4、启动zookeeper
在三个节点zookeeper的bin目录启动服务 如果没有可执行权限，记得先给权限哦

root@hadoop102:/opt/module/zookeeper/bin# zkServer.sh start

三台全部启动后查询启动成功没有,三台的时候一般是第三个启动的为leader，其他的两台为follwer：

root@hadoop102:/opt/module/zookeeper/bin# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper/bin/../conf/zoo.cfg
Mode: follower

至此，zookeeper的集群已经搭建完毕。下面是kafka集群
5、kafka集群搭建
打开配置文件

root@hadoop102:/opt/module/kafka# vim config/server.properties 

三个节点分别添加如下内容

broker.id=0
listeners=PLAINTEXT://192.168.238.139:9092
zookeeper.connect=192.168.238.139:2181,192.168.238.140:2181,192.168.238.141:2181

broker.id=1
listeners=PLAINTEXT://192.168.238.140:9092
zookeeper.connect=192.168.238.139:2181,192.168.238.140:2181,192.168.238.141:2181

broker.id=2
listeners=PLAINTEXT://192.168.238.141:9092
zookeeper.connect=192.168.238.139:2181,192.168.238.140:2181,192.168.238.141:2181

对应自己的ip设置，并且broker.id一定要跟zookeeper的myid文件中的数字一致。
6、启动kafka
确保zookeeper已启动,启动三个节点上的kafka：

root@hadoop102:/opt/module/kafka# ./bin/kafka-server-start.sh config/server.properties

在其中一台虚拟机(192.168.238.139)创建topic

kafka-topics.sh --create --zookeeper 192.168.238.139:2181 --replication-factor 3 --partitions 1 --topic test-topic

启动生产者

root@hadoop102:/opt/module/kafka# bin/kafka-console-producer.sh --topic test_topic --broker-list 192.168.238.139:9092

启动其他两个节点的消费者服务

root@hadoop103:/opt/module/kafka# bin/kafka-console-consumer.sh --bootstrap-server 192.168.238.141:9092 --topic test_topic --from-beginning --group default_g

这里补充一点：同一个topic 的一个parttiton消息，只能被consumergroup 的一个consumer 消费，如果 consumer大于partition的数量，则有一部分consumser 不能消费消息

然后在192.168.238.139节点上生产，查询其他两个节点的消费信息

root@hadoop102:/opt/module/kafka# bin/kafka-console-producer.sh --topic test_topic --broker-list 192.168.238.139:9092
>hello
>huipanxing
>nihao
>test success
>haha  

root@hadoop103:/opt/module/kafka# bin/kafka-console-consumer.sh --bootstrap-server 192.168.238.141:9092 --topic test_topic --from-beginning --group default_g
hello
huipanxing
nihao
test success
haha

root@hadoop104:/opt/module/kafka# bin/kafka-console-consumer.sh --bootstrap-server 192.168.238.141:9092 --topic test_topic --from-beginning --group default_g
roup
hello
huipanxing
nihao
test success
haha

到此kakfa的集群服务生产消费已经实现。

7、zookeeper节点查看
启动zookeeper客户端

root@hadoop102:/opt/module/zookeeper/bin# zkCli.sh

[zk: localhost:2181(CONNECTED) 18] ls /brokers/ids
[0, 1, 2]

查看到kafka的三个节点id，而且在idea工具中也可以看到Topic