python连接kerberos验证的kafka

刚开始看python的kafka接口和java的有点不一样,没有那个keytab的参数设置,还以为不支持kerberos,后面发现原来是不用的,手动配一下就行了。

先配置好kerberos

yum install krb5-server krb5-libs krb5-auth-dialog krb5-workstation

配置下 /var/kerberos/krb5kdc/kdc.conf和/etc/krb5.conf

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 DUNI.COM = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_realm = DUNI.COM
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 DUNI.COM = {
  kdc = 192.168.0.102
  admin_server = 192.168.0.102
 }

[domain_realm]
 .duni.com = DUNI.COM
 duni.com = DUNI.COM

创建Kerberos database

kdb5_util create -s

修改/var/kerberos/krb5kdc/kadm5.acl文件用来设置权限,具体参考

https://docs.oracle.com/cd/E23823_01/html/816-5174/kadm5.acl-4.html#REFMAN4kadm5.acl-4

开启服务

service krb5kdc start

service kadmin start

chkconfig krb5kdc on

chkconfig kadmin on

添加principal

add_principal zookeeper/localhost@DUNI.COM

add_principal kafka/192.168.0.102@DUNI.COM

zookeeper和kafka是必须创建的,验证的时候会用到,启动zookeeper或者kafka server时可以从/var/log/krb5kdc.log看到验证的过程

导出keytab

ktadd -k /etc/security/keytabs/kafka_server.keytab kafka/192.168.0.102@DUNI.COM

切换到kafka用户启动zookeeper和kafka server,配置如下

config/zookeeper.properties

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

config/zookeeper_jaas.conf

Server{
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    useTicketCache=false
    keyTab="/etc/security/keytabs/kafka_server.keytab"
    principal="zookeeper/localhost@DUNI.COM";
};

config/server.properties

advertised.host.name=192.168.0.102
advertised.listeners=SASL_PLAINTEXT://192.168.0.102:9092
listeners=SASL_PLAINTEXT://192.168.0.102:9092
authorizer.class.name = kafka.security.auth.SimpleAclAuthorizer
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka
super.users=User:kafka

 config/kafka_server_jaas.conf

KafkaServer {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      serviceName="kafka"
      keyTab="/etc/security/keytabs/kafka_server.keytab"
      principal="kafka/192.168.0.102@DUNI.COM";
  };

  // Zookeeper client authentication
Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  serviceName="zookeeper"
  keyTab="/etc/security/keytabs/kafka_server.keytab"
  principal="kafka/192.168.0.102@DUNI.COM";
};

启动前要设置环境变量,可以写到脚本里面

export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/kafka/kafka_2.11-2.0.0/config/zookeeper_jaas.conf'
bin/zookeeper-server-start.sh config/zookeeper.properties

export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/kafka/kafka_2.11-2.0.0/config/kafka_server_jaas.conf'
bin/kafka-server-start.sh config/server.properties

启动过程中可以根据/var/log/krb5kdc.log调试

 config/producer.properties config/consumer.properties

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

config/kafka_client_jass.conf

KafkaClient {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      keyTab="/etc/security/keytabs/kafka_server.keytab"
      serviceName="kafka"
      principal="kafka/192.168.0.102@DUNI.COM";
};
  // Zookeeper client authentication
Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  serviceName="zookeeper"
  keyTab="/etc/security/keytabs/kafka_server.keytab"
  principal="kafka/192.168.0.102@DUNI.COM";
};

启动producer和consumer也是要设置环境变量,可以写到脚本

export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/kafka/kafka_2.11-2.0.0/config/kafka_client_jass.conf"

如果要在其他机器用python连接kafka的话,首先把/etc/krb5.conf复制过来,keytab文件也要拿过来,然后kinit -kt keytab principal kinit成功之后你获取的票据就会缓存到本地,可以用klist查看,那个python除了装kafka-python外还有gssapi,也可以yum install python-gssapi,一个简单的连接例子如下

#coding=utf8

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers=["xxxx:9200"],
                         security_protocol="SASL_PLAINTEXT",
                         sasl_mechanism="GSSAPI",
                         sasl_kerberos_service_name="xxxx")
print "connect success."
future = producer.send("xxxx", "test")
result = future.get(timeout=60)
print "send success."

如果出现超时情况,可以检查下topic的权限问题

  • 1
    点赞
  • 11
    收藏
    觉得还不错? 一键收藏
  • 7
    评论
Kafka可以通过Kerberos进行身份验证和安全连接。下面是通过Kerberos连接Kafka的步骤: 1. 配置Kafka服务器: - 在Kafka服务器上安装Kerberos客户端和服务端软件。 - 配置Kafka服务器的`server.properties`文件,设置以下属性: ``` listeners=SASL_PLAINTEXT://<kafka_server>:<kafka_port> security.inter.broker.protocol=SASL_PLAINTEXT sasl.mechanism.inter.broker.protocol=GSSAPI sasl.enabled.mechanisms=GSSAPI sasl.kerberos.service.name=kafka ``` - 启动Kafka服务器。 2. 创建Kerberos主体和Keytab文件: - 在Kerberos服务器上创建一个服务主体(例如:`kafka/kafka_server_hostname@REALM`)。 - 为该服务主体生成Keytab文件,并将其分发到Kafka服务器。 3. 配置Kafka客户端: - 在Kafka客户端上安装Kerberos客户端软件。 - 配置Kafka客户端的`client.properties`文件,设置以下属性: ``` bootstrap.servers=<kafka_server>:<kafka_port> security.protocol=SASL_PLAINTEXT sasl.mechanism=GSSAPI sasl.kerberos.service.name=kafka sasl.kerberos.principal=<kafka_client_principal> sasl.kerberos.keytab=<kafka_client_keytab> ``` - 将Kerberos客户端的`krb5.conf`文件配置为正确的Kerberos Realm和KDC信息。 4. 使用Kafka客户端: - 使用Kafka提供的Kerberos认证库进行身份验证连接。 - 例如,使用Kafka命令行工具发送和接收消息: ``` kafka-console-producer.sh --broker-list <kafka_server>:<kafka_port> --topic <topic> --producer.config client.properties kafka-console-consumer.sh --bootstrap-server <kafka_server>:<kafka_port> --topic <topic> --consumer.config client.properties ``` 通过以上步骤,你可以通过Kerberos安全连接Kafka。请确保正确配置KerberosKafka的相关参数,并提供正确的主体和Keytab文件路径。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 7
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值