kafka SSL 部署资料整理

kafka如果不加权限控制，就很简单了，网上教程很多。关键是，如果不加任何权限控制，那么随便来一个broker就能加入网络，比较麻烦。添加权限部分参考资料如下
http://kafka.apache.org/documentation/#security_overview 关于kafka安全的官方指引，按照上面操作一步步来就行。

噢，如果要看懂上面这篇东东，需要有一些RSA/SSL的背景知识，还需要知道keytool/openssl这两个工具的操作，推荐一篇不错的文章 http://www.cnblogs.com/pixy/p/4722381.html

说明一下，
keystore用来存放key，truststore用来存放受信任的证书，jks是常见证书的一种格式
证书生成的一般过程是：生成本地的key-》生成证书申请-》最后找证书签发单位签发证书（如果不是公开用途，可以自己生成CA来签发）。

还有个偷懒的方式哈，就是用脚本：

#!/usr/bin/env bash

set -e

KEYSTORE_FILENAME="kafka.keystore.jks"
VALIDITY_IN_DAYS=3650
DEFAULT_TRUSTSTORE_FILENAME="kafka.truststore.jks"
TRUSTSTORE_WORKING_DIRECTORY="truststore"
KEYSTORE_WORKING_DIRECTORY="keystore"
CA_CERT_FILE="ca-cert"
KEYSTORE_SIGN_REQUEST="cert-file"
KEYSTORE_SIGN_REQUEST_SRL="ca-cert.srl"
KEYSTORE_SIGNED_CERT="cert-signed"

function file_exists_and_exit() {
  echo "'$1' cannot exist. Move or delete it before"
  echo "re-running this script."
  exit 1
}

if [ -e "$KEYSTORE_WORKING_DIRECTORY" ]; then
  file_exists_and_exit $KEYSTORE_WORKING_DIRECTORY
fi

if [ -e "$CA_CERT_FILE" ]; then
  file_exists_and_exit $CA_CERT_FILE
fi

if [ -e "$KEYSTORE_SIGN_REQUEST" ]; then
  file_exists_and_exit $KEYSTORE_SIGN_REQUEST
fi

if [ -e "$KEYSTORE_SIGN_REQUEST_SRL" ]; then
  file_exists_and_exit $KEYSTORE_SIGN_REQUEST_SRL
fi

if [ -e "$KEYSTORE_SIGNED_CERT" ]; then
  file_exists_and_exit $KEYSTORE_SIGNED_CERT
fi

echo
echo "Welcome to the Kafka SSL keystore and trusttore generator script."

echo
echo "First, do you need to generate a trust store and associated private key,"
echo "or do you already have a trust store file and private key?"
echo
echo -n "Do you need to generate a trust store and associated private key? [yn] "
read generate_trust_store

trust_store_file=""
trust_store_private_key_file=""

if [ "$generate_trust_store" == "y" ]; then
  if [ -e "$TRUSTSTORE_WORKING_DIRECTORY" ]; then
    file_exists_and_exit $TRUSTSTORE_WORKING_DIRECTORY
  fi

  mkdir $TRUSTSTORE_WORKING_DIRECTORY
  echo
  echo "OK, we'll generate a trust store and associated private key."
  echo
  echo "First, the private key."
  echo
  echo "You will be prompted for:"
  echo " - A password for the private key. Remember this."
  echo " - Information about you and your company."
  echo " - NOTE that the Common Name (CN) is currently not important."

  openssl req -new -x509 -keyout $TRUSTSTORE_WORKING_DIRECTORY/ca-key \
    -out $TRUSTSTORE_WORKING_DIRECTORY/ca-cert -days $VALIDITY_IN_DAYS

  trust_store_private_key_file="$TRUSTSTORE_WORKING_DIRECTORY/ca-key"

  echo
  echo "Two files were created:"
  echo " - $TRUSTSTORE_WORKING_DIRECTORY/ca-key -- the private key used later to"
  echo "   sign certificates"
  echo " - $TRUSTSTORE_WORKING_DIRECTORY/ca-cert -- the certificate that will be"
  echo "   stored in the trust store in a moment and serve as the certificate"
  echo "   authority (CA). Once this certificate has been stored in the trust"
  echo "   store, it will be deleted. It can be retrieved from the trust store via:"
  echo "   $ keytool -keystore <trust-store-file> -export -alias CARoot -rfc"

  echo
  echo "Now the trust store will be generated from the certificate."
  echo
  echo "You will be prompted for:"
  echo " - the trust store's password (labeled 'keystore'). Remember this"
  echo " - a confirmation that you want to import the certificate"

  keytool -keystore $TRUSTSTORE_WORKING_DIRECTORY/$DEFAULT_TRUSTSTORE_FILENAME \
    -alias CARoot -import -file $TRUSTSTORE_WORKING_DIRECTORY/ca-cert

  trust_store_file="$TRUSTSTORE_WORKING_DIRECTORY/$DEFAULT_TRUSTSTORE_FILENAME"

  echo
  echo "$TRUSTSTORE_WORKING_DIRECTORY/$DEFAULT_TRUSTSTORE_FILENAME was created."

  # don't need the cert because it's in the trust store.
  rm $TRUSTSTORE_WORKING_DIRECTORY/$CA_CERT_FILE
else
  echo
  echo -n "Enter the path of the trust store file. "
  read -e trust_store_file

  if ! [ -f $trust_store_file ]; then
    echo "$trust_store_file isn't a file. Exiting."
    exit 1
  fi

  echo -n "Enter the path of the trust store's private key. "
  read -e trust_store_private_key_file

  if ! [ -f $trust_store_private_key_file ]; then
    echo "$trust_store_private_key_file isn't a file. Exiting."
    exit 1
  fi
fi

echo
echo "Continuing with:"
echo " - trust store file:        $trust_store_file"
echo " - trust store private key: $trust_store_private_key_file"

mkdir $KEYSTORE_WORKING_DIRECTORY

echo
echo "Now, a keystore will be generated. Each broker and logical client needs its own"
echo "keystore. This script will create only one keystore. Run this script multiple"
echo "times for multiple keystores."
echo
echo "You will be prompted for the following:"
echo " - A keystore password. Remember it."
echo " - Personal information, such as your name."
echo "     NOTE: currently in Kafka, the Common Name (CN) does not need to be the FQDN of"
echo "           this host. However, at some point, this may change. As such, make the CN"
echo "           the FQDN. Some operating systems call the CN prompt 'first / last name'"
echo " - A key password, for the key being generated within the keystore. Remember this."

# To learn more about CNs and FQDNs, read:
# https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/X509ExtendedTrustManager.html

keytool -keystore $KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME \
  -alias localhost -validity $VALIDITY_IN_DAYS -genkey -keyalg RSA

echo
echo "'$KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME' now contains a key pair and a"
echo "self-signed certificate. Again, this keystore can only be used for one broker or"
echo "one logical client. Other brokers or clients need to generate their own keystores."

echo
echo "Fetching the certificate from the trust store and storing in $CA_CERT_FILE."
echo
echo "You will be prompted for the trust store's password (labeled 'keystore')"

keytool -keystore $trust_store_file -export -alias CARoot -rfc -file $CA_CERT_FILE

echo
echo "Now a certificate signing request will be made to the keystore."
echo
echo "You will be prompted for the keystore's password."
keytool -keystore $KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME -alias localhost \
  -certreq -file $KEYSTORE_SIGN_REQUEST

echo
echo "Now the trust store's private key (CA) will sign the keystore's certificate."
echo
echo "You will be prompted for the trust store's private key password."
openssl x509 -req -CA $CA_CERT_FILE -CAkey $trust_store_private_key_file \
  -in $KEYSTORE_SIGN_REQUEST -out $KEYSTORE_SIGNED_CERT \
  -days $VALIDITY_IN_DAYS -CAcreateserial
# creates $KEYSTORE_SIGN_REQUEST_SRL which is never used or needed.

echo
echo "Now the CA will be imported into the keystore."
echo
echo "You will be prompted for the keystore's password and a confirmation that you want to"
echo "import the certificate."
keytool -keystore $KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME -alias CARoot \
  -import -file $CA_CERT_FILE
rm $CA_CERT_FILE # delete the trust store cert because it's stored in the trust store.

echo
echo "Now the keystore's signed certificate will be imported back into the keystore."
echo
echo "You will be prompted for the keystore's password."
keytool -keystore $KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME -alias localhost -import \
  -file $KEYSTORE_SIGNED_CERT

echo
echo "All done!"
echo
echo "Delete intermediate files? They are:"
echo " - '$KEYSTORE_SIGN_REQUEST_SRL': CA serial number"
echo " - '$KEYSTORE_SIGN_REQUEST': the keystore's certificate signing request"
echo "   (that was fulfilled)"
echo " - '$KEYSTORE_SIGNED_CERT': the keystore's certificate, signed by the CA, and stored back"
echo "    into the keystore"
echo -n "Delete? [yn] "
read delete_intermediate_files

if [ "$delete_intermediate_files" == "y" ]; then
  rm $KEYSTORE_SIGN_REQUEST_SRL
  rm $KEYSTORE_SIGN_REQUEST
  rm $KEYSTORE_SIGNED_CERT
fi

找个空文件夹执行后，得到
.
├── keystore
│ └── kafka.keystore.jks
└── truststore
├── ca-key
└── kafka.truststore.jks

最后，修改server.properties
只列出一部分内容，假设生成证书过程中所有密码都是123456

listeners=SSL://localhost:9090
#advertised.listeners=SSL://localhost:9090  
ssl.keystore.location=/opt/tools/kafka_2.12-0.10.2.1/ kafka.keystore.jks  
ssl.keystore.password=123456
ssl.key.password=123456
ssl.truststore.location=/opt/tools/kafka_2.12-0.10.2.1/kafka.truststore.jks  
ssl.truststore.password=123456 
ssl.client.auth=required  
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1  
ssl.keystore.type=JKS  
ssl.truststore.type=JKS  
security.inter.broker.protocol=SSL 

producer.properties

bootstrap.servers=localhost:9090

# specify the compression codec for all data generated: none, gzip, snappy, lz4
#compression.type=none
security.protocol=SSL
ssl.keystore.location=/opt/tools/kafka_2.12-0.10.2.1/ kafka.keystore.jks  
ssl.keystore.password=123456
ssl.key.password=123456
ssl.truststore.location=/opt/tools/kafka_2.12-0.10.2.1/kafka.truststore.jks  
ssl.truststore.password=123456 

producer consumer的启动方式

>bin/kafka-console-producer.sh --broker-st localhost:9090 --topic test6 --producer.config  ./config/producer.properties
>bin/kafka-console-consumer.sh --bootstrap-server localhost:9090 --topic test6 --new-consumer --consumer.config  config/producer.properties

到这步就可以看到consumer 和producer互通了。多台broker部署SSL证书和单台类似，需要在truststore加入目标主机的cert

其它参考资料
SSL公钥私钥证书常识 ：https://www.zhihu.com/question/29620953
无权限控制的kafka环境搭建：http://blog.csdn.net/catshitone/article/details/50462023
kafka SSL与ACL搭建，包含java版本kafka连接demo： http://blog.csdn.net/jsjsjs1789/article/details/53161985

golang的kafka库，
推荐使用这个 https://github.com/Shopify/sarama ，关于sarama连接tls请移步issue区
这个库功能够正常使用，由于未投入生产环境，无法判断性能。

oh!，问题来了，官方给出的指引生成的cert和key都是jks格式的，这个时候就需要把jks转换成pem格式了，http://blog.csdn.net/lsweetyy/article/details/51822496