kafka acl配置

概览

kafka附带一个可插拔的Authorizer和out-of-box authorizer实现，并使用zookeeper来存储所有acl。默认情况下，如果资源R没有关联acl，除了超级用户，没有用户允许访问。如果你想改变这种行为，你可以在broker.properties配置：

acl的格式定义 "Principal P is [Allowed/Denied] Operation O From Host H On Resource R”，你可以在KIP-11上阅读更多关于acl的结构。为了添加，删除或列出acl，你可以使用Kafka authorizer CLI 。下面表格将列出operations,、resources 和APIs之间的关系。

Operation	Resource	API
ALTER	Topic	AlterTopics (Will be introduced in a future release)
CLUSTER_ACTION	Cluster	LeaderAndIsr
CLUSTER_ACTION	Cluster	StopReplica
CLUSTER_ACTION	Cluster	UpdateMetadata
CLUSTER_ACTION	Cluster	ControlledShutdown
CREATE	Cluster	CreateTopics (Will be introduced in a future release)
CREATE	Cluster	Metadata if auto.create.topics.enable
DELETE	Topic	DeleteTopics (Will be introduced in a future release)
DESCRIBE	Topic	Offsets
DESCRIBE	Topic	Metadata
DESCRIBE	Cluster	ListGroups
DESCRIBE	Group	DescribeGroup
READ	Group	GroupCoordinator
READ	Group	Heartbeat
READ	Group	JoinGroup
READ	Group	LeaveGroup
READ	Group	OffsetCommit
READ	Group	OffsetFetch
READ	Group	SyncGroup
READ	Topic	Fetch
READ	Topic	GroupCoordinator
READ	Topic	OffsetCommit
READ	Topic	OffsetFetch
WRITE	Topic	Produce

上面的Operation适用于所有客户端（producers, consumers, admin）和集群内部Broker之间的Operation。在一个安全环境下的kafka集群，客户端和集群内部Broker之间的operation都需要被授权。集群内部Broker之间的operation拆分为cluster 和 topic两方面。Cluster 更倾向与集群内部之间的管理，类似于broker的升级、partition metadata、leader 之间的切换、partition的in-sync副本的设置、集群shutdown的控制。

由于topic partitions内部采用副本机制，因此为每个topic授予和所有集群broker通讯的权限显得非常重要。集群内部broker之间复制一个topic partion的副本需要授予READ和DESCRIBE权限，READ权限默认包含了DESCRIBE权限。

有两种方法可以避免你为每个topic配置集群之间的acl：
1、配置一个超级用户，超级用户用于访问所有资源和管理集群的权限（下面将进行单独介绍）
2、使用通配符的方式单独设置你的acl信息
Producers和consumers需要被授予操作topic的权限，但是他们需要设置不同的principals。Producers需要被授予执行WRITE 和READ的权限。我们还需要记住，管理员用户可以执行命令行工具，也需要授权。管理员需要被授予DELETE、CREATE、ALTER （暂时还不支持）。
常用场景：
创建一个topic，客户端的principal需要对一个topic有CREATE 、DESCRIBE 操作权限
produce 客户端的principal需要对一个topic有WRITE 操作权限
consume客户端的principal需要对一个topic、group 有READ 操作权限
注意：服务器端需要授予更新metadata（CLUSTER_ACTION）的权限，并且要授予读取topic副本的权限。

配置

启用kafka ACLS你需要配置授权。Kafka本身自带了简单的授权实现，为了使用它你需要在server.properties下配
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
默认情况下，如果资源R没有关联acl，除了超级用户，没有用户允许访问。如果你想改变这种方式你可以做如下配置
allow.everyone.if.no.acl.found=true
配置超级用户（server.properties）
super.users=User:Bob;User:Alice
默认情况下，SSL的用户名称的形式是"CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown".可修改，在broker.properties设置自定义的PrincipalBuilder，如下。
principal.builder.class=CustomizedPrincipalBuilderClass
默认情况下，SASL用户名是Kerberos principal的主要组成部分。可修改，通过在broker.properteis中的sasl.kerberos.principal.to.local.rules来自定义规则。

在SSL启用但是客户端没有被授权的情况下，客户端通过SSL端口连接集群，服务器端日志将会出现用客户端用ANONYMOUS的用户名连接集群。这种配置提供加密和服务器身份验证,但是客户会匿名连接。另外一种出现客户端ANONYMOUS用户名连接的情况是服务器端采用PLAINTEXT加密通道。通过给匿名用户读/写权限，意味着你运行任何人无需进行授权便可以连接服务器集群。

授权

1、环境查看

确认环境无授权信息
[root@hadoop001 bin]# ./kafka-acls.sh --list --authorizer-properties zookeeper.connect=localhost:2181/kafka 
[root@hadoop001 bin]# 

2、授权用户集群管理权限

[root@hadoop001 bin]# ./kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181/kafka --allow-principal User:ANONYMOUS  --cluster --add
Adding ACLs for resource `Cluster:kafka-cluster`: 
        User:ANONYMOUS has Allow permission for operations: All from hosts: * 

Current ACLs for resource `Cluster:kafka-cluster`: 
        User:ANONYMOUS has Allow permission for operations: All from hosts: * 
验证（生产）
[root@hadoop001 ~]# kafka-console-producer --broker-list hadoop001:9092 --topic test
23r123
[2017-01-06 15:22:45,804] WARN Error while fetching metadata with correlation id 0 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2017-01-06 15:22:45,886] WARN Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
验证（消费）
[root@hadoop001 ~]# kafka-consolconsumer --bootstrap-server hadoop001:9092 --from-beginning --topic test --new-consumer   
[2017-01-06 15:25:24,289] WARN Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2017-01-06 15:25:24,292] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: console-consumer-47668

3、授权用户生产权限

未授予前进行producer数据
[root@hadoop001 ~]# kafka-console-producer --broker-list hadoop001:9092 --topic test
2525
235235
325t235
kafka server端log日志报错如下：
Topic and partition to exceptions: test-1 -> org.apache.kafka.common.errors.TopicAuthorizationException
授权并验证授权结果
[root@hadoop001 bin]# ./kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181/kafka --allow-principal User:ANONYMOUS  --producer --topic=*  --add
Adding ACLs for resource `Topic:*`: 
        User:ANONYMOUS has Allow permission for operations: Describe from hosts: *
        User:ANONYMOUS has Allow permission for operations: Write from hosts: * 

Adding ACLs for resource `Cluster:kafka-cluster`: 
        User:ANONYMOUS has Allow permission for operations: Create from hosts: * 

Current ACLs for resource `Topic:*`: 
        User:ANONYMOUS has Allow permission for operations: Describe from hosts: *
        User:ANONYMOUS has Allow permission for operations: Write from hosts: *
        User:ANONYMOUS has Allow permission for operations: Read from hosts: * 

[root@hadoop001 bin]# ./kafka-acls.sh --list --authorizer-properties zookeeper.connect=localhost:2181/kafka 
Current ACLs for resource `Topic:*`: 
        User:ANONYMOUS has Allow permission for operations: Describe from hosts: *
        User:ANONYMOUS has Allow permission for operations: Write from hosts: *
        User:ANONYMOUS has Allow permission for operations: Read from hosts: * 

Current ACLs for resource `Cluster:kafka-cluster`: 
        User:ANONYMOUS has Allow permission for operations: Create from hosts: *
        User:ANONYMOUS has Allow permission for operations: All from hosts: *

验证（生产），说明leader
[root@hadoop001 ~]# kafka-console-producer --broker-list hadoop001:9092 --topic test
2rt2
[2017-01-06 15:27:16,236] WARN Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-01-06 15:27:16,323] WARN Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

4、授予消费权限   

未授予消费权限，消费数据报错
[root@hadoop001 bin]# kafka-console-consumer --bootstrap-server hadoop001:9092 --new-consumer --topic test --from-beginning
[2017-01-06 16:03:14,746] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: console-consumer-5669

授权并验证授权结果
[root@hadoop001 bin]# ./kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181/kafka --allow-principal User:ANONYMOUS  --consumer --topic=* --group=*  --add
Adding ACLs for resource `Topic:*`: 
        User:ANONYMOUS has Allow permission for operations: Describe from hosts: *
        User:ANONYMOUS has Allow permission for operations: Read from hosts: * 

Adding ACLs for resource `Group:*`: 
        User:ANONYMOUS has Allow permission for operations: Read from hosts: * 

Current ACLs for resource `Topic:*`: 
        User:ANONYMOUS has Allow permission for operations: Describe from hosts: *
        User:ANONYMOUS has Allow permission for operations: Write from hosts: *
        User:ANONYMOUS has Allow permission for operations: Read from hosts: * 

Current ACLs for resource `Group:*`: 
        User:ANONYMOUS has Allow permission for operations: Read from hosts: * 


[root@hadoop001 bin]# ./kafka-acls.sh --list --authorizer-properties zookeeper.connect=localhost:2181/kafka 
Current ACLs for resource `Group:*`: 
        User:ANONYMOUS has Allow permission for operations: Read from hosts: * 

Current ACLs for resource `Topic:*`: 
        User:ANONYMOUS has Allow permission for operations: Describe from hosts: *
        User:ANONYMOUS has Allow permission for operations: Write from hosts: *
        User:ANONYMOUS has Allow permission for operations: Read from hosts: * 

Current ACLs for resource `Cluster:kafka-cluster`: 
        User:ANONYMOUS has Allow permission for operations: Create from hosts: *
        User:ANONYMOUS has Allow permission for operations: All from hosts: * 

注意：

1、由于kafka副本策略，需要给所有topic赋予Read权限到BrokerList，不然会报如下错误
2017-01-06 15:39:21,575 ERROR kafka.server.ReplicaFetcherThread: [ReplicaFetcherThread-0-10000], Error for partition [__consumer_offsets,1] to broker 10000:org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [Topic authorization failed.]


2017-01-06 15:43:13,255 ERROR kafka.server.ReplicaFetcherThread: [ReplicaFetcherThread-0-10000], Error for partition [test,0] to broker 10000:org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [Topic authorization failed.]