Kafka SASL/PLAIN加密 及Kafka-Python整合

SASL/PLAIN

SASL/PLAIN是kafka中一种使用用户名/密码的身份验证机制，本文使用Kafka-Python2.02 及kafka3.2.0进行简单的整合操作。

1. kafka增加认证信息：

1、在Kafka每个节点的安装目录下的config下新建一个kafka_server_jaas.conf文件

并写入以下内容

JAAS文件定义了链接Kafka Broker时所需要的用户名密码及broker各个节点之间相互通信的用户名密码

KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin"
user_admin="admin"
user_producer="producer"
user_consumer="consumer";
};

2、config下修改server.properties，添加如下内容
kafka3.0之后版本弃用了SimpleAclAuthorizer验证，改为kafka.security.authorizer.AclAuthorizer

##当前主机名称
listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://192.168.29.128:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
#authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
# 修改验证机制
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
allow.everyone.if.no.acl.found=true

2. 修改kafka脚本-添加相关配置文件内容：

1、修改启动脚本

找到bin目录下kafka-server-start.sh的最后一行并在中间添加如下内容:
以实际路径为准

-Djava.security.auth.login.config=config/kafka_server_jaas.conf

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS -Djava.security.auth.login.config=config/kafka_server_jaas.conf kafka.Kafka "$@"

3. 启动kafka并进行测试：

启动kafka(先启动Zookeeper)

bin/kafka-server-start.sh config/server.properties

另外说一下，开启kafka sasl加密认证后 无法查看主题的情况需要添加一个配置文件并在使用​ kafka-topics.sh指向该配置文件即可

例：

在config目录下新建config.properties文件，并添加如下内容

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=“admin” password=“admin”;
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

查看主题情况

bin/kafka-topics.sh --list --bootstrap-server localhost:9092 --command-config config.properties

Kafka-Python整合

生产者示例

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    key_serializer=lambda k: json.dumps(k).encode(),
    bootstrap_servers='192.168.29.128:9092',
    security_protocol = 'SASL_PLAINTEXT',
    sasl_mechanism = 'PLAIN',
    sasl_plain_username = 'admin',
    sasl_plain_password = 'admin'
)  # 假设生产的消息为json字符串

data = {'Data': '我是一条json消息'}

producer.send("Test", data)
producer.close()

消费者示例

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'Test',
    bootstrap_servers='192.168.29.128:9092',
    group_id = 'Test_Group',
    auto_offset_reset = 'latest',
    security_protocol = 'SASL_PLAINTEXT',
    sasl_mechanism = 'PLAIN',
    sasl_plain_username = 'admin',
    sasl_plain_password = 'admin',
    enable_auto_commit = 'False'
)
#
for msg in consumer:
    print(msg)
    consumer.commit()

消费者打印信息如下：

ConsumerRecord(topic=‘Test’, partition=0, offset=0, timestamp=1666523826729, timestamp_type=0, key=b’null’, value=b’{“Data”: “\u6211\u662f\u4e00\u6761json\u6d88\u606f”}‘, headers=[], checksum=None, serialized_key_size=4, serialized_value_size=52, serialized_header_size=-1)
ConsumerRecord(topic=‘Test’, partition=0, offset=1, timestamp=1666523865688, timestamp_type=0, key=b’null’, value=b’{“Data”: “\u6211\u662f\u4e00\u6761json\u6d88\u606f”}', headers=[], checksum=None, serialized_key_size=4, serialized_value_size=52, serialized_header_size=-1)