kafka集群部署

官网下载kafka 3.0.0版本

wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.12-3.0.0.tgz

eg:两台服务器(使用外部zookeeper hadoop-zookeeper-01,hadoop-zookeeper-02,hadoop-zookeeper-03)

     hadoop-slave-01

     hadoop-slave-02

当前目录/home/bigdata/

tar -zxvf kafka_2.12-3.0.0.tgz

mv kafka_2.12-3.0.0 kafka

cd kafka

mkdir logs

cd config

修改server.properties文件

broker.id=0    (以此类推 hadoop-slave-02填写1)

advertised.listeners=PLAINTEXT://hadoop-slave-01:9092

log.dirs=/home/bigdata/kafka/logs

num.partitions=2

zookeeper.connect=hadoop-zookeeper-01:2181,hadoop-zookeeper-02:2181,hadoop-zookeeper-03:2181

启动服务：

cd /home/bigdata/kafka/bin

./kafka-server-start.sh -daemon ../config/server.properties

测试是否安装完成

创建topic为test2

./kafka-topics.sh --bootstrap-server hadoop-slave-01:9092,hadoop-slave-02:9092 --create --topic test2  --partitions 2 --replication-factor 2

topic为test2的消息producer

./kafka-console-producer.sh --broker-list hadoop-slave-01:9092,hadoop-slave-02:9092 --topic test2

topic为test2的消息consumer

./kafka-console-consumer.sh --bootstrap-server hadoop-slave-01:9092,hadoop-slave-02:9092 --from-beginning --topic test2