kafka安装

1、将kafka压缩包解压

2、修改配置文件

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

 

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://hdp1:9092

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=hdp1:2181,hdp2:2181,hdp3:2181

 

############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/root/kafkadata/kafka-logs

3、拷贝到hdp2和hdp3上，

scp -r apps/kafka_2.12-2.2.0 hdp-2:$PWD
scp -r apps/kafka_2.12-2.2.0 hdp-3:$PWD

4、启动zookeeper集群，启动命令为

./zkmananger.sh

5、启动kafka集群，在bin路径下

./kafka-server-start.sh ../config/server.properties

6、出现下面代码说明启动成功

在hdp2和hdp3做同样操作

7、创建主题

./kafka-topics.sh --create --zookeeper hdp1:2181,hdp2:2181,hdp3:2181 --replication-factor 1 --partitions 1 --topic test

8、可以进入zookeeper客户端查看主题有没有创建成功

zkCli.sh

9、在一台CRT上写出生产者

./kafka-console-producer.sh --broker-list hdp1:9092,hdp2:9092,hdp3:9092 --topic test

在另一台CRT上接收数据，消费者信息

./kafka-console-consumer.sh --bootstrap-server hdp1:9092,hdp2:9092,hdp3:9092 --topic test --from-beginning

当生产者发送消息时，消费者会接收到消息。