python连接kerberos验证的kafka

刚开始看python的kafka接口和java的有点不一样，没有那个keytab的参数设置，还以为不支持kerberos，后面发现原来是不用的，手动配一下就行了。

先配置好kerberos

yum install krb5-server krb5-libs krb5-auth-dialog krb5-workstation

配置下 /var/kerberos/krb5kdc/kdc.conf和/etc/krb5.conf

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 DUNI.COM = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_realm = DUNI.COM
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 DUNI.COM = {
  kdc = 192.168.0.102
  admin_server = 192.168.0.102
 }

[domain_realm]
 .duni.com = DUNI.COM
 duni.com = DUNI.COM

创建Kerberos database

kdb5_util create -s

修改/var/kerberos/krb5kdc/kadm5.acl文件用来设置权限，具体参考

https://docs.oracle.com/cd/E23823_01/html/816-5174/kadm5.acl-4.html#REFMAN4kadm5.acl-4

开启服务

service krb5kdc start

service kadmin start

chkconfig krb5kdc on

chkconfig kadmin on

添加principal

add_principal zookeeper/localhost@DUNI.COM

add_principal kafka/192.168.0.102@DUNI.COM

zookeeper和kafka是必须创建的，验证的时候会用到，启动zookeeper或者kafka server时可以从/var/log/krb5kdc.log看到验证的过程

导出keytab

ktadd -k /etc/security/keytabs/kafka_server.keytab kafka/192.168.0.102@DUNI.COM

切换到kafka用户启动zookeeper和kafka server，配置如下

config/zookeeper.properties

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

config/zookeeper_jaas.conf

Server{
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    useTicketCache=false
    keyTab="/etc/security/keytabs/kafka_server.keytab"
    principal="zookeeper/localhost@DUNI.COM";
};

config/server.properties

advertised.host.name=192.168.0.102
advertised.listeners=SASL_PLAINTEXT://192.168.0.102:9092
listeners=SASL_PLAINTEXT://192.168.0.102:9092
authorizer.class.name = kafka.security.auth.SimpleAclAuthorizer
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka
super.users=User:kafka

 config/kafka_server_jaas.conf

KafkaServer {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      serviceName="kafka"
      keyTab="/etc/security/keytabs/kafka_server.keytab"
      principal="kafka/192.168.0.102@DUNI.COM";
  };

  // Zookeeper client authentication
Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  serviceName="zookeeper"
  keyTab="/etc/security/keytabs/kafka_server.keytab"
  principal="kafka/192.168.0.102@DUNI.COM";
};

启动前要设置环境变量，可以写到脚本里面

export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/kafka/kafka_2.11-2.0.0/config/zookeeper_jaas.conf'
bin/zookeeper-server-start.sh config/zookeeper.properties

export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/kafka/kafka_2.11-2.0.0/config/kafka_server_jaas.conf'
bin/kafka-server-start.sh config/server.properties

启动过程中可以根据/var/log/krb5kdc.log调试

 config/producer.properties config/consumer.properties

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

config/kafka_client_jass.conf

KafkaClient {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      keyTab="/etc/security/keytabs/kafka_server.keytab"
      serviceName="kafka"
      principal="kafka/192.168.0.102@DUNI.COM";
};
  // Zookeeper client authentication
Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  serviceName="zookeeper"
  keyTab="/etc/security/keytabs/kafka_server.keytab"
  principal="kafka/192.168.0.102@DUNI.COM";
};

启动producer和consumer也是要设置环境变量，可以写到脚本

export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/kafka/kafka_2.11-2.0.0/config/kafka_client_jass.conf"

如果要在其他机器用python连接kafka的话，首先把/etc/krb5.conf复制过来，keytab文件也要拿过来，然后kinit -kt keytab principal kinit成功之后你获取的票据就会缓存到本地，可以用klist查看，那个python除了装kafka-python外还有gssapi，也可以yum install python-gssapi，一个简单的连接例子如下

＃coding=utf8

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers=["xxxx:9200"],
                         security_protocol="SASL_PLAINTEXT",
                         sasl_mechanism="GSSAPI",
                         sasl_kerberos_service_name="xxxx")
print "connect success."
future = producer.send("xxxx", "test")
result = future.get(timeout=60)
print "send success."

如果出现超时情况，可以检查下topic的权限问题