kafka集群安装和加密

前提
公司要求kafka和zookeeper加密传输，让我自己去了解kafka的加密方法，经过多天的研究失败了，最终还是联系了上一届kafka加密成功的开发协助并且得到了方法，此方法来自于https://www.cnblogs.com/chenandy/p/11846802.html，我做了相关的补充和一些修改。
认证所需要的jar
kafka-clients-0.10.0.1.jar
lz4-1.3.0.jar
slf4j-api-1.7.21.jar
slf4j-log4j12-1.7.21.jar
snappy-java-1.1.2.6.jar
版本：
OS: centos 7.3
Java: jdk1.8.0_162
zookeeper: zookeeper-3.4.10.tar.gz
kafka: kafka_2.11-1.0.2.tgz
集群主机：
192.168.1.86 dphd-192-168-1-86
192.168.1.87 dphd-192-168-1-87
192.168.1.88 dphd-192-168-1-88

1. 安装jdk1.8
   1.1 ) vim /etc/profile #环境变量配置
   export JAVA_HOME=/usr/local/jdk1.8.0_162
   export JRE_HOME=/usr/local/jdk1.8.0_162/jre
   export PATH=$JAV{A}_{H}OME/bin:$JRE_HOME/bin:$PATHexportCLASSPATH=::$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

   [root@host-10-200-86-163 ~]# sh /nas/nas_log_pbs/auto_install/centos_7/tomcat_install.sh

   参数:
        1 ) sh tomcat_install.sh install_jdk7     (---  安装jdk7 ---)
        2 ) sh tomcat_install.sh install_jdk8     (---  安装jdk8 ----)
        3 ) sh tomcat_install.sh install_tomcat7  (--- 安装tomcat7 --- )
        4 ) sh tomcat_install.sh install_tomcat8  ( --- 安装tomcat8 --  )
   

   [root@host-10-200-86-163 ~]# sh /nas/nas_log_pbs/auto_install/centos_7/tomcat_install.sh install_jdk8

2. 安装zookeeper
   复制代码
   #安装指定目录
   [root@dphd-192-168-1-86 src]# cd /usr/local/src
   [root@dphd-192-168-1-86 src]# tar zxpf zookeeper-3.4.10.tar.gz
   [root@dphd-192-168-1-86 src]# mv zookeeper-3.4.10 /usr/local/zookeeper
   #配置文件
   [root@dphd-192-168-1-86 src]# mkdir -p /zk_data/zk1
   [root@dphd-192-168-1-86 src]# echo “1” >>/zk_data/zk1/myid
   [root@dphd-192-168-1-86 src]# mkdir -p /usr/local/zookeeper/logs
   [root@dphd-192-168-1-86 src]# cat /usr/local/zookeeper/conf/zoo.cfg
   tickTime=2000
   initLimit=10
   syncLimit=5
   dataDir=/zk_data/zk1
   dataLogDir=/usr/local/zookeeper/logs
   clientPort=2181
   server.1=192.168.1.86:3181:4181
   server.2=192.168.1.87:3182:4182
   server.3=192.168.1.88:3183:4183
   authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
   requireClientAuthScheme=sasl
   jaasLoginRenew=3600000
   [root@dphd-192-168-1-86 src]#

3. 安装kafka

sudo mkdir -p /opt/kafka-logs
[migu@sx-wx-yulan-ott-10-186-180-10 kafka]$ sudo chmod -R 777 !$

[root@dphd-192-168-1-86 src]# cd /usr/local/src/
[root@dphd-192-168-1-86 src]# tar zxpf kafka_2.11-1.0.2.tgz
[root@dphd-192-168-1-86 src]# mv kafka_2.11-1.0.2 …/kafka
[root@dphd-192-168-1-86 src]# mkdir -p /opt/kafka-logs
#配置文件
[root@dphd-192-168-1-86 src]# cat /kafka/config/server.properties

broker.id=56
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
port=9092
host.name=10.186.74.56
listeners = SASL_PLAINTEXT://10.186.74.56:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
log.dirs=/data/logs/kafka
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.186.74.55:2181,10.186.74,54:2181,10.186.74.56:2181
zookeeper.connection.timeout.ms=60000
group.initial.rebalance.delay.ms=0
authorizer.class.name = kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin

[root@dphd-192-168-1-86 src]#
4) zookeeper SASL_PLAINTEXT认证
4.1 zookeeper集群配置SASL（三台都要改）
在/usr/locla/zookeeper/conf/zoo.cfg 配置文件加上下面参数，上面已经操作完
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000
4.2 编写JAAS文件（三台都要改）
这个文件定义需要链接到Zookeeper服务器的用户名和密码

[root@dphd-192-168-1-86 conf]# cat /zookeeper/conf/jaas.conf
Server {
org.apache.kafka.common.security.plain.PlainLoginModule required
username=“admin”
password=“admin”
user_admin=“admin”;
};

配置文件我命名为/zookeeper/conf/jaas.conf，并放在部署目录的conf/下。文件中定义了身份认证类（org.apache.kafka.common.security.plain.PlainLoginModule），可以看到这个认证类是kafka命名空间，也就是需要加入kafka的插件。

4.3、 向zookeeper添加Kafka认证插件
由于Zookeeper的认证机制是使用插件，这个插件只要支持JAAS即可。Kafka需要链接到Zookeeper，直接使用Kafka的认证插件。这个插件类也包含在kafka-clients中（Maven项目）。将依赖的几个jar加入Zookeeper启动的classpath即可。如下是kafka-clients-0.10.0.1相关jar，包括其依赖：
mkdir -p zookeeper/conf/jar
cp kafka/libs/{kafka-clients-0.10.0.1.jar,lz4-1.3.0.jar,slf4j-api-1.7.21.jar,slf4j-log4j12-1.7.21.jar,snappy-java-1.1.2.6.jar} zookeeper/conf/jar/
4.4、 zookeeper在启动的时候要加载配置文件和jar需要如下配置：
加载jar包: /usr/local/zookeeper/conf/jar/*.jar
加载认证文件:/usr/local/zookeeper/conf/jaas.conf

[root@dphd-192-168-1-86 conf]# cat /usr/local/zookeeper/bin/zkEnv.sh
#上面的启动文件加上下面参数
for i in /usr/local/zookeeper/conf/jar/*.jar; do
CLASSPATH=“$i:$CLASSPATH”
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=/usr/local/zookeeper/conf/jaas.conf"
4.5、启动所有节点 将所有zookeeper节点的Quorum进程开启，查看zookeeper日志，看看之后所有节点是否都能稳定运行，再试试bin/zkCli.sh链接所有节点。是否都能通。
启动命令
/data/soft/zookeeper/bin/zkServer.sh start /data/soft/zookeeper/conf/zoo.cfg
/data/soft/zookeeper/bin/zkServer.sh status /data/soft/zookeeper/conf/zoo.cfg
/data/soft/kafka/bin/secured-kafka-server-start.sh /data/soft/kafka/config/server.properties &
sudo nohup /data/soft/kafka/bin/secured-kafka-server-start.sh /data/soft/kafka/config/server.properties &
cat /data/soft/kafka/bin/secured-kafka-server-start.sh

#!/bin/bash

Licensed to the Apache Software Foundation (ASF) under one or more

contributor license agreements. See the NOTICE file distributed with

this work for additional information regarding copyright ownership.

The ASF licenses this file to You under the Apache License, Version 2.0

(the “License”); you may not use this file except in compliance with

the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an “AS IS” BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

if [ $# -lt 1 ];
then
echo "USAGE: 0 [ − d a e m o n ] s e r v e r . p r o p e r t i e s [ − − o v e r r i d e p r o p e r t y = v a l u e ] ∗ " e x i t