kafka和zk集群安装，以及kafka密码认证配置

最新推荐文章于 2024-08-09 17:39:05 发布

新手上路丶大家小心

最新推荐文章于 2024-08-09 17:39:05 发布

阅读量1.4w

点赞数 1

本文链接：https://blog.csdn.net/weixin_43999932/article/details/105658694

版权

准备三台机器

10.10.9.203

10.10.9.204

10.10.9.205

1.下载kafka安装包

mkdir /data

cd /data

wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.4.1/kafka_2.12-2.4.1.tgz

新版kafka都自带zookeeper

tar xf kafka_2.12-2.4.1.tgz

mv kafka_2.12-2.4.1 kafka

2.搭建zk集群

zk配置文件，三台一样

cd kafka

[root@localhost kafka]# egrep -v "(^#|^$)" config/zookeeper.properties

dataDir=/tmp/zookeeper

tickTime=2000

initLimit=10

syncLimit=5

clientPort=2181

maxClientCnxns=0

admin.enableServer=false

server.1=10.10.9.203:2888:3888

server.2=10.10.9.204:2888:3888

server.3=10.10.9.205:2888:3888

10.10.9.203：

echo 1 > /tmp/zookeeper/myid

10.10.9.204：

echo 2 > /tmp/zookeeper/myid

10.10.9.205：

echo 3 > /tmp/zookeeper/myid

三台机器上操作

cd bin/

./zookeeper-server-start.sh  -daemon ../config/zookeeper.properties

这样zk集群就搭建成功了

3.搭建kafka集群

三台机器操作 需要注意id，和listeners不同

[root@localhost kafka]# egrep -v "(^#|^$)" config/server.properties 
broker.id=0

listeners=SASL_PLAINTEXT://10.10.9.203:9095

security.inter.broker.protocol=SASL_PLAINTEXT  

sasl.enabled.mechanisms=PLAIN  

sasl.mechanism.inter.broker.protocol=PLAIN  

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1

transaction.state.log.replication.factor=1

transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

zookeeper.connect=10.10.9.203:2181,10.10.9.204:2181,10.10.9.205:2181

zookeeper.connection.timeout.ms=6000

group.initial.rebalance.delay.ms=0

[root@localhost kafka]# cat config/kafka_server_jaas.conf 

KafkaServer {

    org.apache.kafka.common.security.plain.PlainLoginModule required
    
    username="kafka"
    
    password="kafkapswd"
    
    user_kafka="kafkapswd"
    
    user_mooc="moocpswd";
};

[root@localhost kafka]# cat config/kafka_client_jaas.conf 

KafkaClient {
        
        org.apache.kafka.common.security.plain.PlainLoginModule required
        
        username="mooc"
        
        password="moocpswd";
};
在bin/kafka-server-start.sh添加

if [ "x$KAFKA_OPTS"  ]; then
    export KAFKA_OPTS="-Djava.security.auth.login.config=/data/kafka/config/kafka_server_jaas.conf"
fi

在kafka-console-producer.sh 和 kafka-console-consumer.sh 文件添加

if [ "x$KAFKA_OPTS"  ]; then
    export KAFKA_OPTS="-Djava.security.auth.login.config=/data/kafka/config/kafka_client_jaas.conf"
fi

./kafka-server-start.sh  -daemon ../config/server.properties
都是三台机器操作
kafka集群也搭建成功了

4.测试生产消费账号验证

cd /data/kafka
#集群中创建topic
bin/kafka-topics.sh --create --zookeeper 10.10.9.203:2181,10.10.9.204:2181,10.10.9.205:2181 --replication-factor 3 --partitions 3 --topic topicTest
#集群中查看topic
bin/kafka-topics.sh --list --zookeeper 10.10.9.203:2181,10.10.9.204:2181,10.10.9.205:2181
#生产者
bin/kafka-console-producer.sh --broker-list 10.10.9.203:9095,10.10.9.204:9095,10.10.9.205:9095 --topic topicTest --producer-property security.protocol=SASL_PLAINTEXT --producer-property sasl.mechanism=PLAIN
#消费者
bin/kafka-console-consumer.sh --bootstrap-server 10.10.9.203:9095,10.10.9.204:9095,10.10.9.205:9095 --topic topicTest --consumer-property security.protocol=SASL_PLAINTEXT --consumer-property sasl.mechanism=PLAIN --from-beginning