kafka-完全分布式集群安装及简单使用

kafka完全分布式集群安装：

要求：前置zookeeper集群

版本：

kafka2.12.1

zookeeper3.4.5

1.解压安装包：

tar -zxvf /opt/software/kafka_2.12-1.0.0.tgz -C /usr/local/kafka

2.更名为kafka

mv /usr/local/kafka/kafka_2.12-1.0.0 kafka

3.修改环境变量

export $KAFKA_HOME=/usr/local/kafka/kafka

export PATH=$PATH:$KAFKA_HOME/bin:$PATH

source /etc/profile 所有用户

source ~/.bashrc 当前用户

4.配置文件

文件路径：$KAFKA_HOME/config/server.properties

修改：

broker.id=1 //相应的slave1,slave2即为2，3

listeners=PLAINTEXT://master:9092

advertised.listeners=PLAINTEXT://master:9092

log.dirs=/usr/local/kafka/kafka-logs-1 //修改日志位置

zookeeper.connect=master:2181,slave1:2181,slave2:2181 //即集群主机名：2181 连接zookeeper

5.分发kafka到slave1,slave2

scp -r /usr/local/kafka root@slave1:/usr/local
scp -r /usr/local/kafka root@slave2:/usr/local

6.从节点配置

修改：

broker.id=2 | broker.id=3

log.dirs=/usr/local/kafka/kafka-logs-2 | log.dirs=/usr/local/kafka/kafka-logs-3

listeners=PLAINTEXT://slave1:9092 | listeners=PLAINTEXT://slave2:9092

advertised.listeners=PLAINTEXT://slave1:9092 | advertised.listeners=PLAINTEXT://slave2:9092

环境变量参见第三步

7.启动zookeeper

三个节点：zkServer.sh start

启动后zkServer.sh status查看状态 确保无误

8.启动kafka

三个节点:

kafka-server-start.sh /usr/local/kafka/kafka/config/server.properties > /home/kafka.log &

//在后台运行kafka，且将启动输出存放到/home/kafka.log

jps查看进程：52063 Kafka

9.使用kafka

创建指定topic：

kafka-topics.sh -zookeeper master:2181 -topic testtopic -replication-factor 2 -partitions 5 -create

Created topic “testtopic”.

查看topic：

kafka-topics.sh -zookeeper master:2181 -list

testtopic

启动指定生产者（producer）：

kafka-console-producer.sh -broker-list slave1:9092,slave2:9092 -topic testtopic

启动消费者（consumer）:另一终端

kafka-console-consumer.sh -zookeeper master:2181 -from-beginning -topic testtopic

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

测试生产者（producer）:

输入

wo shi xi xi hou
ni shi ha ha hou
wo men dou shi hou
wo shi ni de hou
;

测试消费者（consumer）:另一终端

显示

wo shi xi xi hou
ni shi ha ha hou
wo men dou shi hou
wo shi ni de hou
;